Hi Vinay, Seems like this could be a nice addition. Do you have specific database systems in mind where you may have been experiencing issues? This would aid us in establishing a test scenario to evaluate against.
>From the NiFi side, we try to be quite good about handling data of all volumes and sizes, and scanning quickly through the code, it looks like we are treating that appropriately, streaming an item from the result set one at a time to the generated FlowFile. I think by spec the JDBC drivers call for not all data to be loaded into memory and keep a cursor to a given record, only retrieving that data piecewise when accessed. However, we have also found that certain JDBC implementations have varying idiosyncrasies in how their interpretation/implementation of the spec pans out. Doing some quick searching it seems that what certain drivers/databases may cache could lead to issues, and something like a setFetchSize controlled by a processor property could aid in this. Any additional guidance on which systems are being troublesome would be great so that we might find a solution that applies cleanly and uniformly. Thanks! On Wed, Dec 2, 2015 at 7:49 PM, Vinay <[email protected]> wrote: > Is there way to have streaming feature when large result set is fetched > from > database basically to reads data from the database in chunks of records > instead of loading the full result set into memory. > > As part of ExecuteSQL can a property be specified called "FetchSize" which > Indicates how many rows should be fetched from the resultSet. > > Since jam bit new in using NIFI , can any guide me on above. > > Thanks in advance > > > > > > > -- > View this message in context: > http://apache-nifi-developer-list.39713.n7.nabble.com/Streaming-large-result-set-ExecuteSQL-tp5573.html > Sent from the Apache NiFi Developer List mailing list archive at > Nabble.com. >
