[
https://issues.apache.org/jira/browse/NIFI-1251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16530273#comment-16530273
]
ASF GitHub Bot commented on NIFI-1251:
--------------------------------------
Github user mattyb149 commented on the issue:
https://github.com/apache/nifi/pull/2834
What do you think about naming conventions consistent between the
processors that do this "incremental commit"? I added this capability to
QueryDatabaseTable under
[NIFI-4836](https://issues.apache.org/jira/browse/NIFI-4836), and not to say we
have to use my namings, but as it is in product it might be better to have the
properties and such be named and documented consistently. For example, QDT has
a single property that takes a zero value to indicate that it should wait until
all results are processed before sending the files downstream. What are your
thoughts?
> Allow ExecuteSQL to send out large result sets in chunks
> --------------------------------------------------------
>
> Key: NIFI-1251
> URL: https://issues.apache.org/jira/browse/NIFI-1251
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
> Reporter: Mark Payne
> Assignee: Peter Wicks
> Priority: Major
>
> Currently, when using ExecuteSQL, if a result set is very large, it can take
> quite a long time to pull back all of the results. It would be nice to have
> the ability to specify the maximum number of records to put into a FlowFile,
> so that if we pull back say 1 million records we can configure it to create
> 1000 FlowFiles, each with 1000 records. This way, we can begin processing the
> first 1,000 records while the next 1000 are being pulled from the remote
> database.
> This suggestion comes from Vinay via the dev@ mailing list:
> Is there way to have streaming feature when large result set is fetched from
> database basically to reads data from the database in chunks of records
> instead of loading the full result set into memory.
> As part of ExecuteSQL can a property be specified called "FetchSize" which
> Indicates how many rows should be fetched from the resultSet.
> Since jam bit new in using NIFI , can any guide me on above.
> Thanks in advance
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
