Matt Burgess created NIFI-5601:
----------------------------------
Summary: Add fragment.* attributes to GenerateTableFetch
Key: NIFI-5601
URL: https://issues.apache.org/jira/browse/NIFI-5601
Project: Apache NiFi
Issue Type: Improvement
Components: Extensions
Reporter: Matt Burgess
GenerateTableFetch will generate SQL statements in multiple flow files
corresponding to "pages" of data for a single execution of the processor.
QueryDatabaseTable does something similar, although it also does the fetching
of the rows and adds fragment.* attributes to the outgoing flow files, in case
a merge or other correlation needs to be done later.
GenerateTableFetch (although it doesn't produce source data per se) is usually
used in conjunction with ExecuteSQL to fetch the actual pages of rows, and
sometimes these would need to be merged/correlated. To that end, this Jira
proposes to add the same fragment.* attributes to the outgoing GTF flow files
for use downstream.
Note that if the SQL statements are distributed among a cluster (for parallel
execution by ExecuteSQL), they will not be able to be merged, as different
nodes may get different fragments and often no single node will get all the
fragments.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
