Hello, I'm not sure I follow... wouldn't it be more efficient to merge multiple files in parallel across the cluster?
If you had to converge them all to one node, then this doesn't seem much different than just having a stand-alone NiFi, which would go against needing a cluster to achieve the desired through put. -Bryan -Bryan On Mon, Sep 17, 2018 at 4:02 PM Rob Verkuylen <[email protected]> wrote: > > I really went to replace Flume with Nifi, so for the simplest use case I > basically have Kafka->UpdateAttribute-> MergeContent(32->64->128MB)->PutHDFS. > > I need to run in cluster mode to get the thoughput I need, but run into the > problem of flowfiles assigned to nodes are only merged on those nodes. > Effectively splitting my merge efficiencly by the number of nifi nodes. > > Is there a workaround for this issue?
