Hi, What are the configurations in default batch settings?
Thanks, Mohit -----Original Message----- From: Koji Kawamura <[email protected]> Sent: 08 May 2018 13:19 To: [email protected] Subject: Re: Nifi Remote Process Group FlowFile Distribution among nodes Hi Mohit, NiFi RPG batches multiple FlowFiles into the same Site-to-Site transaction, and the default batch settings are configured for higher throughput. If you prefer more granular distribution, you can lower the batch configurations from "Manage Remote Ports" context menu of a RemoteProcessGroup. The batch size configuration from UI is added since NiFi 1.2.0, and the JIRA can be a reference. https://issues.apache.org/jira/browse/NIFI-1202 Thanks, Koji On Tue, May 8, 2018 at 2:24 PM, Mohit <[email protected]> wrote: > Hi, > > > > I need some clarity on how flowfile is distributed among different > nodes in a Nifi cluster. > > > > I have a flow where I’m using GenerateTableFetch to fetch the data > from database. Source table has around 40 million records. I tried > with different partition size which led to create different number of > flowfiles. > > When there are less number of flowfiles(~40), RPG sends it to only one > node(in a 4 node cluster) but when there are large number of > flowfiles(~400), it distribute the flowfile among all the nodes. > > Are there some rules or best practices to fine tune the flow, so that > the flowfiles are evenly distributed across the nodes even if there > are less number of flowfiles. > > > > Regards, > > Mohit
