Re: DistributeLoad across a NiFi cluster
If you're going to upgrade, I would recommend jumping to the latest version, 1.9.2 as of today. We ran into some issues in 1.8 with this feature that was fixed in 1.9.x. We're running 1.9.2 now with this feature now in production. Regards Dano On Tue, Jul 9, 2019, 6:58 AM wrote: > The feature requires NiFi > 1.8.x… Pierre describes it very well in his > blog : > https://pierrevillard.com/2018/10/29/nifi-1-8-revolutionizing-the-list-fetch-pattern-and-more/ > > > > > > *From: *James McMahon > *Reply-To: *"users@nifi.apache.org" > *Date: *Tuesday, 9 July 2019 at 14:46 > *To: *"users@nifi.apache.org" > *Subject: *Re: DistributeLoad across a NiFi cluster > > > > Andrew, when I right click on the connection between the two I do not see > a cluster distribution strategy in the queue connection. I am running > 1.7.1.g. Am I overlooking something? > > > > On Tue, Jul 2, 2019 at 12:29 PM Andrew Grande wrote: > > Jim, > > > > There's a better solution in NiFi. Right click on the connection between > ListFile and FetchFile and select a cluster distribution strategy in > options. That's it :) > > > > Andrew > > > > On Tue, Jul 2, 2019, 7:37 AM James McMahon wrote: > > We would like to employ a DistributeLoad processor, restricted to run on > the primary node of our cluster. Is there a recommended approach employed > to efficiently distribute across nodes in the cluster? > > > > As I understand it, and using a FetchFile running in "all nodes" as the > first processor following the DistributeLoad, I can have it distribute by > round robin, next available, or load distribution service. Can anyone > provide a link to an example that employs the load distribution service? Is > that the recommended distribution approach when running in clustered mode? > > > > I am interested in maintaining load balance across my cluster nodes when > running at high flowfile volumes. Flow files will vary greatly in contents, > so I'd like to design with an approach that helps me balance processing > distribution. > > > > Thanks very much in advance. -Jim > >
Re: DistributeLoad across a NiFi cluster
The feature requires NiFi > 1.8.x… Pierre describes it very well in his blog : https://pierrevillard.com/2018/10/29/nifi-1-8-revolutionizing-the-list-fetch-pattern-and-more/ From: James McMahon Reply-To: "users@nifi.apache.org" Date: Tuesday, 9 July 2019 at 14:46 To: "users@nifi.apache.org" Subject: Re: DistributeLoad across a NiFi cluster Andrew, when I right click on the connection between the two I do not see a cluster distribution strategy in the queue connection. I am running 1.7.1.g. Am I overlooking something? On Tue, Jul 2, 2019 at 12:29 PM Andrew Grande mailto:apere...@gmail.com>> wrote: Jim, There's a better solution in NiFi. Right click on the connection between ListFile and FetchFile and select a cluster distribution strategy in options. That's it :) Andrew On Tue, Jul 2, 2019, 7:37 AM James McMahon mailto:jsmcmah...@gmail.com>> wrote: We would like to employ a DistributeLoad processor, restricted to run on the primary node of our cluster. Is there a recommended approach employed to efficiently distribute across nodes in the cluster? As I understand it, and using a FetchFile running in "all nodes" as the first processor following the DistributeLoad, I can have it distribute by round robin, next available, or load distribution service. Can anyone provide a link to an example that employs the load distribution service? Is that the recommended distribution approach when running in clustered mode? I am interested in maintaining load balance across my cluster nodes when running at high flowfile volumes. Flow files will vary greatly in contents, so I'd like to design with an approach that helps me balance processing distribution. Thanks very much in advance. -Jim smime.p7s Description: S/MIME Cryptographic Signature
Re: DistributeLoad across a NiFi cluster
Andrew, when I right click on the connection between the two I do not see a cluster distribution strategy in the queue connection. I am running 1.7.1.g. Am I overlooking something? On Tue, Jul 2, 2019 at 12:29 PM Andrew Grande wrote: > Jim, > > There's a better solution in NiFi. Right click on the connection between > ListFile and FetchFile and select a cluster distribution strategy in > options. That's it :) > > Andrew > > On Tue, Jul 2, 2019, 7:37 AM James McMahon wrote: > >> We would like to employ a DistributeLoad processor, restricted to run on >> the primary node of our cluster. Is there a recommended approach employed >> to efficiently distribute across nodes in the cluster? >> >> As I understand it, and using a FetchFile running in "all nodes" as the >> first processor following the DistributeLoad, I can have it distribute by >> round robin, next available, or load distribution service. Can anyone >> provide a link to an example that employs the load distribution service? Is >> that the recommended distribution approach when running in clustered mode? >> >> I am interested in maintaining load balance across my cluster nodes when >> running at high flowfile volumes. Flow files will vary greatly in contents, >> so I'd like to design with an approach that helps me balance processing >> distribution. >> >> Thanks very much in advance. -Jim >> >
Re: DistributeLoad across a NiFi cluster
Edward, Documentation on this feature may be found here: https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#settings Warm regards, Jim Williams From: Edward Armes Sent: Thursday, July 4, 2019 5:54 AM To: users@nifi.apache.org Subject: Re: DistributeLoad across a NiFi cluster Hi Andrew, Is this functionality documented anywhere do you know? As I've had a quick look through the documentation and I haven't seen this. Edward On Tue, Jul 2, 2019 at 5:33 PM James McMahon mailto:jsmcmah...@gmail.com>> wrote: Excellent - thanks very much Andrew. This is my first crack at working with a clustered configuration, and I guess that shows by my question. Outstanding - thanks again. On Tue, Jul 2, 2019 at 12:29 PM Andrew Grande mailto:apere...@gmail.com>> wrote: Jim, There's a better solution in NiFi. Right click on the connection between ListFile and FetchFile and select a cluster distribution strategy in options. That's it :) Andrew On Tue, Jul 2, 2019, 7:37 AM James McMahon mailto:jsmcmah...@gmail.com>> wrote: We would like to employ a DistributeLoad processor, restricted to run on the primary node of our cluster. Is there a recommended approach employed to efficiently distribute across nodes in the cluster? As I understand it, and using a FetchFile running in "all nodes" as the first processor following the DistributeLoad, I can have it distribute by round robin, next available, or load distribution service. Can anyone provide a link to an example that employs the load distribution service? Is that the recommended distribution approach when running in clustered mode? I am interested in maintaining load balance across my cluster nodes when running at high flowfile volumes. Flow files will vary greatly in contents, so I'd like to design with an approach that helps me balance processing distribution. Thanks very much in advance. -Jim Confidentiality Notice | This email and any included attachments may be privileged, confidential and/or otherwise protected from disclosure. Access to this email by anyone other than the intended recipient is unauthorized. If you believe you have received this email in error, please contact the sender immediately and delete all copies. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.
Re: DistributeLoad across a NiFi cluster
Hi Andrew, Is this functionality documented anywhere do you know? As I've had a quick look through the documentation and I haven't seen this. Edward On Tue, Jul 2, 2019 at 5:33 PM James McMahon wrote: > Excellent - thanks very much Andrew. This is my first crack at working > with a clustered configuration, and I guess that shows by my question. > Outstanding - thanks again. > > On Tue, Jul 2, 2019 at 12:29 PM Andrew Grande wrote: > >> Jim, >> >> There's a better solution in NiFi. Right click on the connection between >> ListFile and FetchFile and select a cluster distribution strategy in >> options. That's it :) >> >> Andrew >> >> On Tue, Jul 2, 2019, 7:37 AM James McMahon wrote: >> >>> We would like to employ a DistributeLoad processor, restricted to run on >>> the primary node of our cluster. Is there a recommended approach employed >>> to efficiently distribute across nodes in the cluster? >>> >>> As I understand it, and using a FetchFile running in "all nodes" as the >>> first processor following the DistributeLoad, I can have it distribute by >>> round robin, next available, or load distribution service. Can anyone >>> provide a link to an example that employs the load distribution service? Is >>> that the recommended distribution approach when running in clustered mode? >>> >>> I am interested in maintaining load balance across my cluster nodes when >>> running at high flowfile volumes. Flow files will vary greatly in contents, >>> so I'd like to design with an approach that helps me balance processing >>> distribution. >>> >>> Thanks very much in advance. -Jim >>> >>
Re: DistributeLoad across a NiFi cluster
Excellent - thanks very much Andrew. This is my first crack at working with a clustered configuration, and I guess that shows by my question. Outstanding - thanks again. On Tue, Jul 2, 2019 at 12:29 PM Andrew Grande wrote: > Jim, > > There's a better solution in NiFi. Right click on the connection between > ListFile and FetchFile and select a cluster distribution strategy in > options. That's it :) > > Andrew > > On Tue, Jul 2, 2019, 7:37 AM James McMahon wrote: > >> We would like to employ a DistributeLoad processor, restricted to run on >> the primary node of our cluster. Is there a recommended approach employed >> to efficiently distribute across nodes in the cluster? >> >> As I understand it, and using a FetchFile running in "all nodes" as the >> first processor following the DistributeLoad, I can have it distribute by >> round robin, next available, or load distribution service. Can anyone >> provide a link to an example that employs the load distribution service? Is >> that the recommended distribution approach when running in clustered mode? >> >> I am interested in maintaining load balance across my cluster nodes when >> running at high flowfile volumes. Flow files will vary greatly in contents, >> so I'd like to design with an approach that helps me balance processing >> distribution. >> >> Thanks very much in advance. -Jim >> >
Re: DistributeLoad across a NiFi cluster
Jim, There's a better solution in NiFi. Right click on the connection between ListFile and FetchFile and select a cluster distribution strategy in options. That's it :) Andrew On Tue, Jul 2, 2019, 7:37 AM James McMahon wrote: > We would like to employ a DistributeLoad processor, restricted to run on > the primary node of our cluster. Is there a recommended approach employed > to efficiently distribute across nodes in the cluster? > > As I understand it, and using a FetchFile running in "all nodes" as the > first processor following the DistributeLoad, I can have it distribute by > round robin, next available, or load distribution service. Can anyone > provide a link to an example that employs the load distribution service? Is > that the recommended distribution approach when running in clustered mode? > > I am interested in maintaining load balance across my cluster nodes when > running at high flowfile volumes. Flow files will vary greatly in contents, > so I'd like to design with an approach that helps me balance processing > distribution. > > Thanks very much in advance. -Jim >
DistributeLoad across a NiFi cluster
We would like to employ a DistributeLoad processor, restricted to run on the primary node of our cluster. Is there a recommended approach employed to efficiently distribute across nodes in the cluster? As I understand it, and using a FetchFile running in "all nodes" as the first processor following the DistributeLoad, I can have it distribute by round robin, next available, or load distribution service. Can anyone provide a link to an example that employs the load distribution service? Is that the recommended distribution approach when running in clustered mode? I am interested in maintaining load balance across my cluster nodes when running at high flowfile volumes. Flow files will vary greatly in contents, so I'd like to design with an approach that helps me balance processing distribution. Thanks very much in advance. -Jim
