Thanks Koji, It was helpful. Mohit.
-----Original Message----- From: Koji Kawamura <[email protected]> Sent: 09 May 2018 05:53 To: [email protected] Subject: Re: Nifi Remote Process Group FlowFile Distribution among nodes Hi, You can see the description for default behavior by hovering your mouse on the question mark icon (?) next to the "Batch Settings". In short, for sending (to a remote input port), 500 milliseconds batch duration. For pulling (from a remote output port), 5 seconds batch duration. Thanks, Koji On Tue, May 8, 2018 at 4:56 PM, Mohit <[email protected]> wrote: > Hi, > > What are the configurations in default batch settings? > > Thanks, > Mohit > > -----Original Message----- > From: Koji Kawamura <[email protected]> > Sent: 08 May 2018 13:19 > To: [email protected] > Subject: Re: Nifi Remote Process Group FlowFile Distribution among > nodes > > Hi Mohit, > > NiFi RPG batches multiple FlowFiles into the same Site-to-Site transaction, > and the default batch settings are configured for higher throughput. > If you prefer more granular distribution, you can lower the batch > configurations from "Manage Remote Ports" context menu of a > RemoteProcessGroup. > The batch size configuration from UI is added since NiFi 1.2.0, and the JIRA > can be a reference. > https://issues.apache.org/jira/browse/NIFI-1202 > > Thanks, > Koji > > On Tue, May 8, 2018 at 2:24 PM, Mohit <[email protected]> wrote: >> Hi, >> >> >> >> I need some clarity on how flowfile is distributed among different >> nodes in a Nifi cluster. >> >> >> >> I have a flow where I’m using GenerateTableFetch to fetch the data >> from database. Source table has around 40 million records. I tried >> with different partition size which led to create different number of >> flowfiles. >> >> When there are less number of flowfiles(~40), RPG sends it to only >> one node(in a 4 node cluster) but when there are large number of >> flowfiles(~400), it distribute the flowfile among all the nodes. >> >> Are there some rules or best practices to fine tune the flow, so that >> the flowfiles are evenly distributed across the nodes even if there >> are less number of flowfiles. >> >> >> >> Regards, >> >> Mohit >
