That’s a cool workaround!! Thanks for sharing.
On Thu, 11 Jul 2019, at 16:57, James McMahon wrote:
> Update: we have another solution path that seems to give a very useful
> workaround for this challenge pre-1.8. If depends on having a messaging queue
> that can be read by ConsumeAMQP.
> Use Li
Update: we have another solution path that seems to give a very useful
workaround for this challenge pre-1.8. If depends on having a messaging
queue that can be read by ConsumeAMQP.
Use ListFile to establish your set of zero-byte flowfiles -- essentially,
metadata about the files you want to read
Great advice, thank you Joe. I had not realized Batch Settings Count was
even there, and so had left it unset. If any other folks are also still
using pre-1.8, you can set on your Remote Process Group, Manage Remote
Ports, parameter is Batch Settings Count. I still want to experiment to
gauge the e
James
Did you apply any specific batch settings on s2s? By default it sends
large chunks of messages at once. If you're testing on small scale you
might not see the distribution you would at typical/protracted scale.
Setting batch sizes smaller may be appropriate for your case or just
leaving th
Thank you Joe. We do one day intend to upgrade but are bound by enterprise
options available to us to 1.7 for the near-term. So, do I understand your
explanation correctly: behavior exhibited through the first 4000 flowfiles
as past performance may not represent future results. It will do what it
d
James
For distributing work across the cluster the load balanced connection
capability in NiFi 1.8 and beyond is the right answer - purpose built for
the job. I'd strongly recommend upgrading to avoid use of s2s for this
scenario and instead use load balanced connections. When using load
balance
We are on 1.7.1.g and have just recently established our first clustered
configuration. Using Pierre Villard's article from Feb 2017 (
https://pierrevillard.com/2017/02/23/listfetch-pattern-and-remote-process-group-in-apache-nifi/
) and a few other related technical articles to flesh out some detai
