Matt Burgess created NIFI-4836:
----------------------------------
Summary: Allow QueryDatabaseTables to send out batches of flow
files while result set is being processed
Key: NIFI-4836
URL: https://issues.apache.org/jira/browse/NIFI-4836
Project: Apache NiFi
Issue Type: Improvement
Components: Extensions
Reporter: Matt Burgess
Currently QueryDatabaseTable (QDT) will not transfer the outgoing flowfiles to
the downstream relationship(s) until the entire result set has been processed
(regardless of whether Max Rows Per Flow File is set). This is so the
maxvalue.* and fragment.count attributes can be set correctly for each flow
file.
However for very large result sets, the initial fetch can take a long time, and
depending on the setting of Max Rows Per FlowFile, there could be a great
number of FlowFiles transferred downstream as a large burst at the end of QDT
execution.
It would be nice for the user to be able to choose to have FlowFiles be
transferred downstream while the result set is still being processed. This
alleviates the "large burst at the end" by replacing it with smaller output
batches during processing. The tradeoff will be that if an Output Batch Size is
set, then the maxvalue.* and fragment.count attributes will not be set on the
outgoing flow files.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
