[
https://issues.apache.org/jira/browse/NIFI-5601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16672242#comment-16672242
]
ASF GitHub Bot commented on NIFI-5601:
--------------------------------------
Github user patricker commented on the issue:
https://github.com/apache/nifi/pull/3074
+1 LG - Reviewed, ran tests, built and ran the changes locally against a
larger table.
> Add fragment.* attributes to GenerateTableFetch
> -----------------------------------------------
>
> Key: NIFI-5601
> URL: https://issues.apache.org/jira/browse/NIFI-5601
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
> Reporter: Matt Burgess
> Assignee: Matt Burgess
> Priority: Major
>
> GenerateTableFetch will generate SQL statements in multiple flow files
> corresponding to "pages" of data for a single execution of the processor.
> QueryDatabaseTable does something similar, although it also does the fetching
> of the rows and adds fragment.* attributes to the outgoing flow files, in
> case a merge or other correlation needs to be done later.
> GenerateTableFetch (although it doesn't produce source data per se) is
> usually used in conjunction with ExecuteSQL to fetch the actual pages of
> rows, and sometimes these would need to be merged/correlated. To that end,
> this Jira proposes to add the same fragment.* attributes to the outgoing GTF
> flow files for use downstream.
> Note that if the SQL statements are distributed among a cluster (for parallel
> execution by ExecuteSQL), they will not be able to be merged, as different
> nodes may get different fragments and often no single node will get all the
> fragments.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
