Bryan, That’s a good point. Would running with a larger Java heap and higher swap threshold allow Peter to get larger batches out?
Andy LoPresto [email protected] [email protected] PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Sep 20, 2016, at 1:41 PM, Bryan Bende <[email protected]> wrote: > > Peter, > > Does 10k happen to be your swap threshold in nifi.properties by any chance > (it defaults to 20k I believe)? > > I suspect the behavior you are seeing could be due to the way swapping works, > but Mark or others could probably confirm. > > I found this thread where Mark explained how swapping works with a background > thread, and I believe it still works this way: > http://apache-nifi.1125220.n5.nabble.com/Nifi-amp-Spark-receiver-performance-configuration-td524.html > > <http://apache-nifi.1125220.n5.nabble.com/Nifi-amp-Spark-receiver-performance-configuration-td524.html> > > -Bryan > > On Tue, Sep 20, 2016 at 10:22 AM, Peter Wicks (pwicks) <[email protected] > <mailto:[email protected]>> wrote: > I’m using JSONToSQL, followed by PutSQL. I’m using Teradata, which supports > a special JDBC mode called FastLoad, designed for a minimum of 100,000 rows > of data per batch. > > > > What I’m finding is that when PutSQL requests a new batch of FlowFiles from > the queue, which has over 1 million rows in it, with a batch size of 1000000, > it always returns a maximum of 10k. How can I get my obscenely sized batch > request to return all the FlowFile’s I’m asking for? > > > > Thanks, > > Peter > >
signature.asc
Description: Message signed with OpenPGP using GPGMail
