[jira] [Commented] (NIFI-1706) Extend QueryDatabaseTable to support arbitrary queries

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16352075#comment-16352075
 ] 

ASF GitHub Bot commented on NIFI-1706:
--

Github user patricker commented on the issue:

https://github.com/apache/nifi/pull/2162
  
@ijokarumawak Updated. 


> Extend QueryDatabaseTable to support arbitrary queries
> --
>
> Key: NIFI-1706
> URL: https://issues.apache.org/jira/browse/NIFI-1706
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 1.4.0
>Reporter: Paul Bormans
>Assignee: Peter Wicks
>Priority: Major
>  Labels: features
>
> The QueryDatabaseTable is able to observe a configured database table for new 
> rows and yield these into the flowfile. The model of an rdbms however is 
> often (if not always) normalized so you would need to join various tables in 
> order to "flatten" the data into useful events for a processing pipeline as 
> can be build with nifi or various tools within the hadoop ecosystem.
> The request is to extend the processor to specify an arbitrary sql query 
> instead of specifying the table name + columns.
> In addition (this may be another issue?) it is desired to limit the number of 
> rows returned per run. Not just because of bandwidth issue's from the nifi 
> pipeline onwards but mainly because huge databases may not be able to return 
> so many records within a reasonable time.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2162: NIFI-1706 Extend QueryDatabaseTable to support arbitrary q...

[patricker]

Github user patricker commented on the issue:

https://github.com/apache/nifi/pull/2162
  
@ijokarumawak Updated. 


---

[jira] [Commented] (NIFI-4164) Realistic Time Series Processor Simulator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351961#comment-16351961
 ] 

ASF GitHub Bot commented on NIFI-4164:
--

Github user cherrera2001 commented on the issue:

https://github.com/apache/nifi/pull/1997
  
Should be done by Tuesday night CST


> Realistic Time Series Processor Simulator
> -
>
> Key: NIFI-4164
> URL: https://issues.apache.org/jira/browse/NIFI-4164
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Chris Herrera
>Assignee: Chris Herrera
>Priority: Minor
>  Labels: features
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> In order to validate several flows that deal with sensor data, it would be 
> good to have a built in time series simulator processor that generates data 
> and can send it out via a flow file.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #1997: NIFI-4164 Adding a realistic time simulator processor to N...

[cherrera2001]

Github user cherrera2001 commented on the issue:

https://github.com/apache/nifi/pull/1997
  
Should be done by Tuesday night CST


---

[jira] [Commented] (NIFI-4840) Fix Formatting For Expression Language Guide (toRadix/fromRadix)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351943#comment-16351943
 ] 

ASF GitHub Bot commented on NIFI-4840:
--

GitHub user Leah-Anderson opened a pull request:

https://github.com/apache/nifi/pull/2449

NIFI-4840 Fix formatting for expression language guide for toRadix an…

…d fromRadix

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [X] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [X] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [X] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [X] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [X] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Leah-Anderson/nifi NIFI-4840

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2449.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2449


commit a445ee4d141ea1218d3cf5c47abb5171c5890f43
Author: Leah Anderson 
Date:   2018-02-05T00:32:02Z

NIFI-4840 Fix formatting for expression language guide for toRadix and 
fromRadix




> Fix Formatting For Expression Language Guide (toRadix/fromRadix)
> 
>
> Key: NIFI-4840
> URL: https://issues.apache.org/jira/browse/NIFI-4840
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation  Website
>Affects Versions: 1.5.0
>Reporter: Leah Anderson
>Priority: Trivial
>  Labels: documentation
> Fix For: 1.6.0
>
> Attachments: Screenshot from 2018-02-04 19-26-40.png
>
>
> The documentation in the expression language guide is not formatted correctly 
> for the toRadix/fromRadix sections due to a misplaced # sign. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2449: NIFI-4840 Fix formatting for expression language gu...

[Leah-Anderson]

GitHub user Leah-Anderson opened a pull request:

https://github.com/apache/nifi/pull/2449

NIFI-4840 Fix formatting for expression language guide for toRadix an…

…d fromRadix

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [X] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [X] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [X] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [X] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [X] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Leah-Anderson/nifi NIFI-4840

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2449.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2449


commit a445ee4d141ea1218d3cf5c47abb5171c5890f43
Author: Leah Anderson 
Date:   2018-02-05T00:32:02Z

NIFI-4840 Fix formatting for expression language guide for toRadix and 
fromRadix




---

[jira] [Created] (NIFI-4840) Fix Formatting For Expression Language Guide (toRadix/fromRadix)

[Leah Anderson (JIRA)]

Leah Anderson created NIFI-4840:
---

 Summary: Fix Formatting For Expression Language Guide 
(toRadix/fromRadix)
 Key: NIFI-4840
 URL: https://issues.apache.org/jira/browse/NIFI-4840
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Documentation  Website
Affects Versions: 1.5.0
Reporter: Leah Anderson
 Fix For: 1.6.0
 Attachments: Screenshot from 2018-02-04 19-26-40.png

The documentation in the expression language guide is not formatted correctly 
for the toRadix/fromRadix sections due to a misplaced # sign. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4164) Realistic Time Series Processor Simulator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351907#comment-16351907
 ] 

ASF GitHub Bot commented on NIFI-4164:
--

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/1997
  
@cherrera2001 Do you have any updates?


> Realistic Time Series Processor Simulator
> -
>
> Key: NIFI-4164
> URL: https://issues.apache.org/jira/browse/NIFI-4164
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Chris Herrera
>Assignee: Chris Herrera
>Priority: Minor
>  Labels: features
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> In order to validate several flows that deal with sensor data, it would be 
> good to have a built in time series simulator processor that generates data 
> and can send it out via a flow file.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4836) Allow QueryDatabaseTables to send out batches of flow files while result set is being processed

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351903#comment-16351903
 ] 

ASF GitHub Bot commented on NIFI-4836:
--

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2447#discussion_r165860112
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/QueryDatabaseTable.java
 ---
@@ -315,6 +334,12 @@ public void onTrigger(final ProcessContext context, 
final ProcessSessionFactory
 
 
session.getProvenanceReporter().receive(fileToProcess, jdbcURL, 
stopWatch.getElapsed(TimeUnit.MILLISECONDS));
 resultSetFlowFiles.add(fileToProcess);
+// If we've reached the batch size, send out the 
flow files
+if (outputBatchSize > 0 && 
resultSetFlowFiles.size() >= outputBatchSize) {
+session.transfer(resultSetFlowFiles, 
REL_SUCCESS);
+session.commit();
+resultSetFlowFiles.clear();
--- End diff --

I usually reinitialize the collection in cases like this. I think in the 
past I've had weird behavior from a collection when using clear() a lot on a 
lot of referenced objects. Not a needed change and maybe that's paranoia on my 
part based on weirdness from past projects.


> Allow QueryDatabaseTables to send out batches of flow files while result set 
> is being processed
> ---
>
> Key: NIFI-4836
> URL: https://issues.apache.org/jira/browse/NIFI-4836
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> Currently QueryDatabaseTable (QDT) will not transfer the outgoing flowfiles 
> to the downstream relationship(s) until the entire result set has been 
> processed (regardless of whether Max Rows Per Flow File is set). This is so 
> the maxvalue.* and fragment.count attributes can be set correctly for each 
> flow file.
> However for very large result sets, the initial fetch can take a long time, 
> and depending on the setting of Max Rows Per FlowFile, there could be a great 
> number of FlowFiles transferred downstream as a large burst at the end of QDT 
> execution.
> It would be nice for the user to be able to choose to have FlowFiles be 
> transferred downstream while the result set is still being processed. This 
> alleviates the "large burst at the end" by replacing it with smaller output 
> batches during processing. The tradeoff will be that if an Output Batch Size 
> is set, then the maxvalue.* and fragment.count attributes will not be set on 
> the outgoing flow files.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4836) Allow QueryDatabaseTables to send out batches of flow files while result set is being processed

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351904#comment-16351904
 ] 

ASF GitHub Bot commented on NIFI-4836:
--

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2447#discussion_r165436273
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/QueryDatabaseTable.java
 ---
@@ -123,8 +124,22 @@
 public static final PropertyDescriptor MAX_ROWS_PER_FLOW_FILE = new 
PropertyDescriptor.Builder()
 .name("qdbt-max-rows")
 .displayName("Max Rows Per Flow File")
-.description("The maximum number of result rows that will be 
included in a single FlowFile. " +
-"This will allow you to break up very large result 
sets into multiple FlowFiles. If the value specified is zero, then all rows are 
returned in a single FlowFile.")
+.description("The maximum number of result rows that will be 
included in a single FlowFile. This will allow you to break up very large "
++ "result sets into multiple FlowFiles. If the value 
specified is zero, then all rows are returned in a single FlowFile.")
+.defaultValue("0")
--- End diff --

When I did something similar on GetMongo, I think I chose to make it 
optional and have "blank" be the equivalent. Thoughts?


> Allow QueryDatabaseTables to send out batches of flow files while result set 
> is being processed
> ---
>
> Key: NIFI-4836
> URL: https://issues.apache.org/jira/browse/NIFI-4836
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> Currently QueryDatabaseTable (QDT) will not transfer the outgoing flowfiles 
> to the downstream relationship(s) until the entire result set has been 
> processed (regardless of whether Max Rows Per Flow File is set). This is so 
> the maxvalue.* and fragment.count attributes can be set correctly for each 
> flow file.
> However for very large result sets, the initial fetch can take a long time, 
> and depending on the setting of Max Rows Per FlowFile, there could be a great 
> number of FlowFiles transferred downstream as a large burst at the end of QDT 
> execution.
> It would be nice for the user to be able to choose to have FlowFiles be 
> transferred downstream while the result set is still being processed. This 
> alleviates the "large burst at the end" by replacing it with smaller output 
> batches during processing. The tradeoff will be that if an Output Batch Size 
> is set, then the maxvalue.* and fragment.count attributes will not be set on 
> the outgoing flow files.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2447: NIFI-4836: Allow output of FlowFiles during result ...

[MikeThomsen]

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2447#discussion_r165860112
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/QueryDatabaseTable.java
 ---
@@ -315,6 +334,12 @@ public void onTrigger(final ProcessContext context, 
final ProcessSessionFactory
 
 
session.getProvenanceReporter().receive(fileToProcess, jdbcURL, 
stopWatch.getElapsed(TimeUnit.MILLISECONDS));
 resultSetFlowFiles.add(fileToProcess);
+// If we've reached the batch size, send out the 
flow files
+if (outputBatchSize > 0 && 
resultSetFlowFiles.size() >= outputBatchSize) {
+session.transfer(resultSetFlowFiles, 
REL_SUCCESS);
+session.commit();
+resultSetFlowFiles.clear();
--- End diff --

I usually reinitialize the collection in cases like this. I think in the 
past I've had weird behavior from a collection when using clear() a lot on a 
lot of referenced objects. Not a needed change and maybe that's paranoia on my 
part based on weirdness from past projects.


---

[GitHub] nifi pull request #2447: NIFI-4836: Allow output of FlowFiles during result ...

[MikeThomsen]

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2447#discussion_r165436273
  
--- Diff: 
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/QueryDatabaseTable.java
 ---
@@ -123,8 +124,22 @@
 public static final PropertyDescriptor MAX_ROWS_PER_FLOW_FILE = new 
PropertyDescriptor.Builder()
 .name("qdbt-max-rows")
 .displayName("Max Rows Per Flow File")
-.description("The maximum number of result rows that will be 
included in a single FlowFile. " +
-"This will allow you to break up very large result 
sets into multiple FlowFiles. If the value specified is zero, then all rows are 
returned in a single FlowFile.")
+.description("The maximum number of result rows that will be 
included in a single FlowFile. This will allow you to break up very large "
++ "result sets into multiple FlowFiles. If the value 
specified is zero, then all rows are returned in a single FlowFile.")
+.defaultValue("0")
--- End diff --

When I did something similar on GetMongo, I think I chose to make it 
optional and have "blank" be the equivalent. Thoughts?


---

[GitHub] nifi pull request #2448: NIFI-4838 Added configurable progressive commits to...

[MikeThomsen]

GitHub user MikeThomsen opened a pull request:

https://github.com/apache/nifi/pull/2448

NIFI-4838 Added configurable progressive commits to GetMongo and also…

… added attributes that show the progress in the result set that each 
flowfile represents.

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/MikeThomsen/nifi NIFI-4838

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2448.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2448


commit 765720f7d087ec8b45ad6888f93910f8f6fdad45
Author: Mike Thomsen 
Date:   2018-02-04T21:51:47Z

NIFI-4838 Added configurable progressive commits to GetMongo and also added 
attributes that show the progress in the result set that each flowfile 
represents.




---