It would buy time but either way it becomes a magic value people have
to know about. This is not unlike the SplitText scenario where we
recommend doing two-phase splits. The problem is that for the
ProcessSession we hold information about the flowfiles (not their
content) in memory and the
result. In
this scenario there was just a single FlowFile and everything went smoothly.
Thanks again!
Peter Wicks
From: Bryan Bende [mailto:bbe...@gmail.com]
Sent: Tuesday, September 20, 2016 3:38 PM
To: users@nifi.apache.org
Subject: Re: Requesting Obscene FlowFile Batch Sizes
Andy
Andy,
That was my thinking. An easy test might be to bump the threshold up to
100k (increase heap if needed) and see if it starts grabbing 100k every
time.
If it does then I would think it is swapping related, then need to figure
out if you really want to get all 1 million in a single batch, and
Bryan,
That’s a good point. Would running with a larger Java heap and higher swap
threshold allow Peter to get larger batches out?
Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Sep 20, 2016, at 1:41 PM,
Peter,
Does 10k happen to be your swap threshold in nifi.properties by any chance
(it defaults to 20k I believe)?
I suspect the behavior you are seeing could be due to the way swapping
works, but Mark or others could probably confirm.
I found this thread where Mark explained how swapping works
