[jira] [Commented] (NIFI-5604) Allow GenerateTableFetch to send empty flow files when no rows would be fetched
[ https://issues.apache.org/jira/browse/NIFI-5604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16688616#comment-16688616 ] ASF GitHub Bot commented on NIFI-5604: -- Github user patricker commented on the issue: https://github.com/apache/nifi/pull/3075 +1 LGTM. I tested this out with several different scenarios. After our chat on Slack, I squashed this down to a single commit under your name to make things cleaner. > Allow GenerateTableFetch to send empty flow files when no rows would be > fetched > --- > > Key: NIFI-5604 > URL: https://issues.apache.org/jira/browse/NIFI-5604 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > > Currently, GenerateTableFetch will not output a flow file if there are no > rows to be fetched. However, it may be desired (especially with incoming flow > files) that a flow file be sent out even if GTF does not generate any SQL. > This capability, along with the fragment attributes from NIFI-5601, would > allow the user to handle this downstream. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5604) Allow GenerateTableFetch to send empty flow files when no rows would be fetched
[ https://issues.apache.org/jira/browse/NIFI-5604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16688617#comment-16688617 ] ASF GitHub Bot commented on NIFI-5604: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/3075 > Allow GenerateTableFetch to send empty flow files when no rows would be > fetched > --- > > Key: NIFI-5604 > URL: https://issues.apache.org/jira/browse/NIFI-5604 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > > Currently, GenerateTableFetch will not output a flow file if there are no > rows to be fetched. However, it may be desired (especially with incoming flow > files) that a flow file be sent out even if GTF does not generate any SQL. > This capability, along with the fragment attributes from NIFI-5601, would > allow the user to handle this downstream. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5604) Allow GenerateTableFetch to send empty flow files when no rows would be fetched
[ https://issues.apache.org/jira/browse/NIFI-5604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16688615#comment-16688615 ] ASF subversion and git services commented on NIFI-5604: --- Commit 0207d0813ef164ee7227ded61fb10960a4842e2d in nifi's branch refs/heads/master from [~ca9mbu] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=0207d08 ] NIFI-5604: Added property to allow empty FlowFile when no SQL generated by GenerateTableFetch co-authored by: Peter Wicks Signed-off-by: Peter Wicks This closes #3075. > Allow GenerateTableFetch to send empty flow files when no rows would be > fetched > --- > > Key: NIFI-5604 > URL: https://issues.apache.org/jira/browse/NIFI-5604 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > > Currently, GenerateTableFetch will not output a flow file if there are no > rows to be fetched. However, it may be desired (especially with incoming flow > files) that a flow file be sent out even if GTF does not generate any SQL. > This capability, along with the fragment attributes from NIFI-5601, would > allow the user to handle this downstream. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5604) Allow GenerateTableFetch to send empty flow files when no rows would be fetched
[
https://issues.apache.org/jira/browse/NIFI-5604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16687049#comment-16687049
]
ASF GitHub Bot commented on NIFI-5604:
--
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3075#discussion_r233598457
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GenerateTableFetch.java
---
@@ -450,7 +450,33 @@ public void onTrigger(final ProcessContext context,
final ProcessSessionFactory
// If there are no SQL statements to be generated, still
output an empty flow file if specified by the user
if (numberOfFetches == 0 &&
outputEmptyFlowFileOnZeroResults) {
-session.transfer((fileToProcess == null) ?
session.create() : session.create(fileToProcess), REL_SUCCESS);
+FlowFile emptyFlowFile = (fileToProcess == null) ?
session.create() : session.create(fileToProcess);
+Map attributesToAdd = new HashMap<>();
+
+attributesToAdd.put("generatetablefetch.tableName",
tableName);
+if (columnNames != null) {
+
attributesToAdd.put("generatetablefetch.columnNames", columnNames);
+}
+whereClause = maxValueClauses.isEmpty() ? "1=1" :
StringUtils.join(maxValueClauses, " AND ");
+if (StringUtils.isNotBlank(whereClause)) {
--- End diff --
Yeah that's probably left over from before we added the "1=1", I'll take a
look and clean it up
> Allow GenerateTableFetch to send empty flow files when no rows would be
> fetched
> ---
>
> Key: NIFI-5604
> URL: https://issues.apache.org/jira/browse/NIFI-5604
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> Currently, GenerateTableFetch will not output a flow file if there are no
> rows to be fetched. However, it may be desired (especially with incoming flow
> files) that a flow file be sent out even if GTF does not generate any SQL.
> This capability, along with the fragment attributes from NIFI-5601, would
> allow the user to handle this downstream.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5604) Allow GenerateTableFetch to send empty flow files when no rows would be fetched
[
https://issues.apache.org/jira/browse/NIFI-5604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16686841#comment-16686841
]
ASF GitHub Bot commented on NIFI-5604:
--
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3075#discussion_r233524164
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GenerateTableFetch.java
---
@@ -450,7 +450,33 @@ public void onTrigger(final ProcessContext context,
final ProcessSessionFactory
// If there are no SQL statements to be generated, still
output an empty flow file if specified by the user
if (numberOfFetches == 0 &&
outputEmptyFlowFileOnZeroResults) {
-session.transfer((fileToProcess == null) ?
session.create() : session.create(fileToProcess), REL_SUCCESS);
+FlowFile emptyFlowFile = (fileToProcess == null) ?
session.create() : session.create(fileToProcess);
+Map attributesToAdd = new HashMap<>();
+
+attributesToAdd.put("generatetablefetch.tableName",
tableName);
+if (columnNames != null) {
+
attributesToAdd.put("generatetablefetch.columnNames", columnNames);
+}
+whereClause = maxValueClauses.isEmpty() ? "1=1" :
StringUtils.join(maxValueClauses, " AND ");
+if (StringUtils.isNotBlank(whereClause)) {
--- End diff --
I know it was already like this, but I'm not sure it's even possible for
`whereClause` to be blank based on the logic here. Same for existing code down
below for a normal run.
> Allow GenerateTableFetch to send empty flow files when no rows would be
> fetched
> ---
>
> Key: NIFI-5604
> URL: https://issues.apache.org/jira/browse/NIFI-5604
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> Currently, GenerateTableFetch will not output a flow file if there are no
> rows to be fetched. However, it may be desired (especially with incoming flow
> files) that a flow file be sent out even if GTF does not generate any SQL.
> This capability, along with the fragment attributes from NIFI-5601, would
> allow the user to handle this downstream.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5604) Allow GenerateTableFetch to send empty flow files when no rows would be fetched
[
https://issues.apache.org/jira/browse/NIFI-5604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16686842#comment-16686842
]
ASF GitHub Bot commented on NIFI-5604:
--
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3075#discussion_r233524977
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GenerateTableFetch.java
---
@@ -450,7 +450,33 @@ public void onTrigger(final ProcessContext context,
final ProcessSessionFactory
// If there are no SQL statements to be generated, still
output an empty flow file if specified by the user
if (numberOfFetches == 0 &&
outputEmptyFlowFileOnZeroResults) {
-session.transfer((fileToProcess == null) ?
session.create() : session.create(fileToProcess), REL_SUCCESS);
+FlowFile emptyFlowFile = (fileToProcess == null) ?
session.create() : session.create(fileToProcess);
+Map attributesToAdd = new HashMap<>();
+
+attributesToAdd.put("generatetablefetch.tableName",
tableName);
+if (columnNames != null) {
+
attributesToAdd.put("generatetablefetch.columnNames", columnNames);
+}
+whereClause = maxValueClauses.isEmpty() ? "1=1" :
StringUtils.join(maxValueClauses, " AND ");
+if (StringUtils.isNotBlank(whereClause)) {
+
attributesToAdd.put("generatetablefetch.whereClause", whereClause);
+}
+final String maxColumnNames =
StringUtils.join(maxValueColumnNameList, ", ");
+if (StringUtils.isNotBlank(maxColumnNames)) {
+
attributesToAdd.put("generatetablefetch.maxColumnNames", maxColumnNames);
+}
+attributesToAdd.put("generatetablefetch.limit", null);
--- End diff --
There is no existing test in `master`, but I think in normal circumstances
`limit` actually gets written as `"null"`.
```
if ((i == numberOfFetches - 1) && useColumnValsForPaging &&
(maxValueClauses.isEmpty() || customWhereClause != null)) {
maxValueClauses.add(columnForPartitioning + "
<= " + maxValueForPartitioning);
limit = null;
}
```
This value is then written using `String.valueOf(limit)`, which is going to
translate the `null` to `"null"`.
> Allow GenerateTableFetch to send empty flow files when no rows would be
> fetched
> ---
>
> Key: NIFI-5604
> URL: https://issues.apache.org/jira/browse/NIFI-5604
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> Currently, GenerateTableFetch will not output a flow file if there are no
> rows to be fetched. However, it may be desired (especially with incoming flow
> files) that a flow file be sent out even if GTF does not generate any SQL.
> This capability, along with the fragment attributes from NIFI-5601, would
> allow the user to handle this downstream.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5604) Allow GenerateTableFetch to send empty flow files when no rows would be fetched
[
https://issues.apache.org/jira/browse/NIFI-5604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685496#comment-16685496
]
ASF GitHub Bot commented on NIFI-5604:
--
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3075#discussion_r233142376
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GenerateTableFetch.java
---
@@ -434,48 +448,53 @@ public void onTrigger(final ProcessContext context,
final ProcessSessionFactory
numberOfFetches = (partitionSize == 0) ? 1 : (rowCount
/ partitionSize) + (rowCount % partitionSize == 0 ? 0 : 1);
}
-// Generate SQL statements to read "pages" of data
-Long limit = partitionSize == 0 ? null : (long)
partitionSize;
-final String fragmentIdentifier =
UUID.randomUUID().toString();
-for (long i = 0; i < numberOfFetches; i++) {
-// Add a right bounding for the partitioning column if
necessary (only on last partition, meaning we don't need the limit)
-if ((i == numberOfFetches - 1) &&
useColumnValsForPaging && (maxValueClauses.isEmpty() || customWhereClause !=
null)) {
-maxValueClauses.add(columnForPartitioning + " <= "
+ maxValueForPartitioning);
-limit = null;
-}
+// If there are no SQL statements to be generated, still
output an empty flow file if specified by the user
+if (numberOfFetches == 0 &&
outputEmptyFlowFileOnZeroResults) {
+session.transfer((fileToProcess == null) ?
session.create() : session.create(fileToProcess), REL_SUCCESS);
--- End diff --
Good call, will make the change.
> Allow GenerateTableFetch to send empty flow files when no rows would be
> fetched
> ---
>
> Key: NIFI-5604
> URL: https://issues.apache.org/jira/browse/NIFI-5604
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> Currently, GenerateTableFetch will not output a flow file if there are no
> rows to be fetched. However, it may be desired (especially with incoming flow
> files) that a flow file be sent out even if GTF does not generate any SQL.
> This capability, along with the fragment attributes from NIFI-5601, would
> allow the user to handle this downstream.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5604) Allow GenerateTableFetch to send empty flow files when no rows would be fetched
[
https://issues.apache.org/jira/browse/NIFI-5604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685476#comment-16685476
]
ASF GitHub Bot commented on NIFI-5604:
--
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3075#discussion_r233132498
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GenerateTableFetch.java
---
@@ -434,48 +448,53 @@ public void onTrigger(final ProcessContext context,
final ProcessSessionFactory
numberOfFetches = (partitionSize == 0) ? 1 : (rowCount
/ partitionSize) + (rowCount % partitionSize == 0 ? 0 : 1);
}
-// Generate SQL statements to read "pages" of data
-Long limit = partitionSize == 0 ? null : (long)
partitionSize;
-final String fragmentIdentifier =
UUID.randomUUID().toString();
-for (long i = 0; i < numberOfFetches; i++) {
-// Add a right bounding for the partitioning column if
necessary (only on last partition, meaning we don't need the limit)
-if ((i == numberOfFetches - 1) &&
useColumnValsForPaging && (maxValueClauses.isEmpty() || customWhereClause !=
null)) {
-maxValueClauses.add(columnForPartitioning + " <= "
+ maxValueForPartitioning);
-limit = null;
-}
+// If there are no SQL statements to be generated, still
output an empty flow file if specified by the user
+if (numberOfFetches == 0 &&
outputEmptyFlowFileOnZeroResults) {
+session.transfer((fileToProcess == null) ?
session.create() : session.create(fileToProcess), REL_SUCCESS);
--- End diff --
@mattyb149 It's not just the `fragment` attributes, I think you should add
all the standard attributes. Right now this Flowfile get's routed to success
with no attributes at all.
I would suggest refactoring this to include all the standard attributes
like `tablename`, `offset` etc... just like a real FlowFile would have. This
will make it much easier for downstream processors to use this file like
normal. I think the existing logic would cause `offset` to be `null`, which
would make sense for this scenario too.
> Allow GenerateTableFetch to send empty flow files when no rows would be
> fetched
> ---
>
> Key: NIFI-5604
> URL: https://issues.apache.org/jira/browse/NIFI-5604
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> Currently, GenerateTableFetch will not output a flow file if there are no
> rows to be fetched. However, it may be desired (especially with incoming flow
> files) that a flow file be sent out even if GTF does not generate any SQL.
> This capability, along with the fragment attributes from NIFI-5601, would
> allow the user to handle this downstream.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5604) Allow GenerateTableFetch to send empty flow files when no rows would be fetched
[ https://issues.apache.org/jira/browse/NIFI-5604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685470#comment-16685470 ] ASF GitHub Bot commented on NIFI-5604: -- Github user patricker commented on the issue: https://github.com/apache/nifi/pull/3075 Reviewing > Allow GenerateTableFetch to send empty flow files when no rows would be > fetched > --- > > Key: NIFI-5604 > URL: https://issues.apache.org/jira/browse/NIFI-5604 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > > Currently, GenerateTableFetch will not output a flow file if there are no > rows to be fetched. However, it may be desired (especially with incoming flow > files) that a flow file be sent out even if GTF does not generate any SQL. > This capability, along with the fragment attributes from NIFI-5601, would > allow the user to handle this downstream. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5604) Allow GenerateTableFetch to send empty flow files when no rows would be fetched
[ https://issues.apache.org/jira/browse/NIFI-5604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650745#comment-16650745 ] ASF GitHub Bot commented on NIFI-5604: -- GitHub user mattyb149 opened a pull request: https://github.com/apache/nifi/pull/3075 NIFI-5604: Added property to allow empty FlowFile when no SQL generated by GenerateTableFetch Note that this should work with NIFI-5601 (#3074) such that the fragment attributes would be written to the empty flow file as well. Depending on which gets merged first, the other probably needs to be updated. Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mattyb149/nifi NIFI-5604 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3075.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3075 commit 43bfe9531a0aa2a64179d27b4196a6062f60ebbe Author: Matthew Burgess Date: 2018-10-15T20:27:38Z NIFI-5604: Added property to allow empty FlowFile when no SQL generated by GenerateTableFetch > Allow GenerateTableFetch to send empty flow files when no rows would be > fetched > --- > > Key: NIFI-5604 > URL: https://issues.apache.org/jira/browse/NIFI-5604 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > > Currently, GenerateTableFetch will not output a flow file if there are no > rows to be fetched. However, it may be desired (especially with incoming flow > files) that a flow file be sent out even if GTF does not generate any SQL. > This capability, along with the fragment attributes from NIFI-5601, would > allow the user to handle this downstream. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5604) Allow GenerateTableFetch to send empty flow files when no rows would be fetched
[ https://issues.apache.org/jira/browse/NIFI-5604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16619233#comment-16619233 ] Matt Burgess commented on NIFI-5604: I propose a new property "Include Zero Record FlowFiles" that would behave in much the same way as it does for QueryRecord, where an empty flow file (not an incoming one even if present) would be generated if set to true. > Allow GenerateTableFetch to send empty flow files when no rows would be > fetched > --- > > Key: NIFI-5604 > URL: https://issues.apache.org/jira/browse/NIFI-5604 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Priority: Major > > Currently, GenerateTableFetch will not output a flow file if there are no > rows to be fetched. However, it may be desired (especially with incoming flow > files) that a flow file be sent out even if GTF does not generate any SQL. > This capability, along with the fragment attributes from NIFI-5601, would > allow the user to handle this downstream. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
