Bryan,

That’s a good point. Would running with a larger Java heap and higher swap 
threshold allow Peter to get larger batches out?

Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Sep 20, 2016, at 1:41 PM, Bryan Bende <bbe...@gmail.com> wrote:
> 
> Peter,
> 
> Does 10k happen to be your swap threshold in nifi.properties by any chance 
> (it defaults to 20k I believe)?
> 
> I suspect the behavior you are seeing could be due to the way swapping works, 
> but Mark or others could probably confirm.
> 
> I found this thread where Mark explained how swapping works with a background 
> thread, and I believe it still works this way:
> http://apache-nifi.1125220.n5.nabble.com/Nifi-amp-Spark-receiver-performance-configuration-td524.html
>  
> <http://apache-nifi.1125220.n5.nabble.com/Nifi-amp-Spark-receiver-performance-configuration-td524.html>
> 
> -Bryan
> 
> On Tue, Sep 20, 2016 at 10:22 AM, Peter Wicks (pwicks) <pwi...@micron.com 
> <mailto:pwi...@micron.com>> wrote:
> I’m using JSONToSQL, followed by PutSQL.  I’m using Teradata, which supports 
> a special JDBC mode called FastLoad, designed for a minimum of 100,000 rows 
> of data per batch.
> 
> 
> 
> What I’m finding is that when PutSQL requests a new batch of FlowFiles from 
> the queue, which has over 1 million rows in it, with a batch size of 1000000, 
> it always returns a maximum of 10k.  How can I get my obscenely sized batch 
> request to return all the FlowFile’s I’m asking for?
> 
> 
> 
> Thanks,
> 
>   Peter
> 
> 

Attachment: signature.asc
Description: Message signed with OpenPGP using GPGMail

Reply via email to