[GitHub] nifi pull request #3078: NIFI-4805 Allow Delayed Transfer
GitHub user patricker opened a pull request: https://github.com/apache/nifi/pull/3078 NIFI-4805 Allow Delayed Transfer Went looking for this processor, found a work around, and found that others had already discussed building one throughout 2018 (see ticket/email thread from today, October 15th). Happy to make changes, feedback welcome. ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/patricker/nifi NIFI-4805 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3078.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3078 commit badfe8651651e5cefe1d2447458ff43be7ba0cdc Author: patricker Date: 2018-10-16T02:55:20Z NIFI-4805 Allow Delayed Transfer ---
[jira] [Commented] (NIFI-4805) allow delayed transfer
[ https://issues.apache.org/jira/browse/NIFI-4805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16651070#comment-16651070 ] ASF GitHub Bot commented on NIFI-4805: -- GitHub user patricker opened a pull request: https://github.com/apache/nifi/pull/3078 NIFI-4805 Allow Delayed Transfer Went looking for this processor, found a work around, and found that others had already discussed building one throughout 2018 (see ticket/email thread from today, October 15th). Happy to make changes, feedback welcome. ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/patricker/nifi NIFI-4805 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3078.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3078 commit badfe8651651e5cefe1d2447458ff43be7ba0cdc Author: patricker Date: 2018-10-16T02:55:20Z NIFI-4805 Allow Delayed Transfer > allow delayed transfer > -- > > Key: NIFI-4805 > URL: https://issues.apache.org/jira/browse/NIFI-4805 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Martin Mucha >Assignee: Peter Wicks >Priority: Minor > Attachments: retry.xml > > > Nifi has concept of penalization, but this penalization has fixed delay, and > there isn't way how to change it dynamically. > If we want to implement retry flow, where FlowFile flows in loop, we can > either lower performance of Processor via yielding it, or we can do active > waiting. And this is actually recommended as a correct way how to do that. > It seems, that we can easily implement better RetryProcessor, all we missing > is `session.penalize` which accepts `penalizationPeriod`. Processor then can > gradually prolong waiting time after each failure. > > Would it be possible to make such method visible? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (NIFI-4805) allow delayed transfer
[ https://issues.apache.org/jira/browse/NIFI-4805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Wicks reassigned NIFI-4805: - Assignee: Peter Wicks > allow delayed transfer > -- > > Key: NIFI-4805 > URL: https://issues.apache.org/jira/browse/NIFI-4805 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Martin Mucha >Assignee: Peter Wicks >Priority: Minor > Attachments: retry.xml > > > Nifi has concept of penalization, but this penalization has fixed delay, and > there isn't way how to change it dynamically. > If we want to implement retry flow, where FlowFile flows in loop, we can > either lower performance of Processor via yielding it, or we can do active > waiting. And this is actually recommended as a correct way how to do that. > It seems, that we can easily implement better RetryProcessor, all we missing > is `session.penalize` which accepts `penalizationPeriod`. Processor then can > gradually prolong waiting time after each failure. > > Would it be possible to make such method visible? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5704) TestStandardProcessSession testBatchQueuedHaveSameQueuedTime is a brittle test
[ https://issues.apache.org/jira/browse/NIFI-5704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16651057#comment-16651057 ] ASF GitHub Bot commented on NIFI-5704: -- GitHub user patricker opened a pull request: https://github.com/apache/nifi/pull/3077 NIFI-5704 TestStandardProcessSession testBatchQueuedHaveSameQueuedTim… …e is a brittle test ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/patricker/nifi NIFI-5704 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3077.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3077 commit 6c7e38ebcedd5de518ca27c1a356ef5c06bc39b3 Author: Peter Wicks Date: 2018-10-16T02:34:10Z NIFI-5704 TestStandardProcessSession testBatchQueuedHaveSameQueuedTime is a brittle test > TestStandardProcessSession testBatchQueuedHaveSameQueuedTime is a brittle test > -- > > Key: NIFI-5704 > URL: https://issues.apache.org/jira/browse/NIFI-5704 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.8.0 >Reporter: Joseph Witt >Assignee: Peter Wicks >Priority: Major > > [ERROR] Failures: > [ERROR] TestStandardProcessSession.testBatchQueuedHaveSameQueuedTime:1691 > Queued times should not be equal.. Actual: 1539636497790 > [ERROR] Tests run: 59, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: > 0.102 s <<< FAILURE! - in > org.apache.nifi.controller.repository.TestStandardProcessSession > [ERROR] > testBatchQueuedHaveSameQueuedTime(org.apache.nifi.controller.repository.TestStandardProcessSession) > Time elapsed: 0.008 s <<< FAILURE! > java.lang.AssertionError: Queued times should not be equal.. Actual: > 1539636497790 > at > org.apache.nifi.controller.repository.TestStandardProcessSession.testBatchQueuedHaveSameQueuedTime(TestStandardProcessSession.java:1691) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3077: NIFI-5704 TestStandardProcessSession testBatchQueue...
GitHub user patricker opened a pull request: https://github.com/apache/nifi/pull/3077 NIFI-5704 TestStandardProcessSession testBatchQueuedHaveSameQueuedTim⦠â¦e is a brittle test ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/patricker/nifi NIFI-5704 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3077.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3077 commit 6c7e38ebcedd5de518ca27c1a356ef5c06bc39b3 Author: Peter Wicks Date: 2018-10-16T02:34:10Z NIFI-5704 TestStandardProcessSession testBatchQueuedHaveSameQueuedTime is a brittle test ---
[jira] [Assigned] (NIFI-5704) TestStandardProcessSession testBatchQueuedHaveSameQueuedTime is a brittle test
[ https://issues.apache.org/jira/browse/NIFI-5704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Wicks reassigned NIFI-5704: - Assignee: Peter Wicks > TestStandardProcessSession testBatchQueuedHaveSameQueuedTime is a brittle test > -- > > Key: NIFI-5704 > URL: https://issues.apache.org/jira/browse/NIFI-5704 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.8.0 >Reporter: Joseph Witt >Assignee: Peter Wicks >Priority: Major > > [ERROR] Failures: > [ERROR] TestStandardProcessSession.testBatchQueuedHaveSameQueuedTime:1691 > Queued times should not be equal.. Actual: 1539636497790 > [ERROR] Tests run: 59, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: > 0.102 s <<< FAILURE! - in > org.apache.nifi.controller.repository.TestStandardProcessSession > [ERROR] > testBatchQueuedHaveSameQueuedTime(org.apache.nifi.controller.repository.TestStandardProcessSession) > Time elapsed: 0.008 s <<< FAILURE! > java.lang.AssertionError: Queued times should not be equal.. Actual: > 1539636497790 > at > org.apache.nifi.controller.repository.TestStandardProcessSession.testBatchQueuedHaveSameQueuedTime(TestStandardProcessSession.java:1691) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5704) TestStandardProcessSession testBatchQueuedHaveSameQueuedTime is a brittle test
[ https://issues.apache.org/jira/browse/NIFI-5704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16651056#comment-16651056 ] Peter Wicks commented on NIFI-5704: --- The test adds files to a queue, and expects because they are added via a loop that the queue times will be slightly different between the first and the last files. However, apparently on your very fast system this fails sometimes, and all FlowFile's actually get the same queue time... I guess there is nothing wrong with this, it's a legitimate possibility. I'm adding in a 0.5 second delay when the last FlowFile is queued in the test to ensure the timestamps are different. Will PR. > TestStandardProcessSession testBatchQueuedHaveSameQueuedTime is a brittle test > -- > > Key: NIFI-5704 > URL: https://issues.apache.org/jira/browse/NIFI-5704 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.8.0 >Reporter: Joseph Witt >Priority: Major > > [ERROR] Failures: > [ERROR] TestStandardProcessSession.testBatchQueuedHaveSameQueuedTime:1691 > Queued times should not be equal.. Actual: 1539636497790 > [ERROR] Tests run: 59, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: > 0.102 s <<< FAILURE! - in > org.apache.nifi.controller.repository.TestStandardProcessSession > [ERROR] > testBatchQueuedHaveSameQueuedTime(org.apache.nifi.controller.repository.TestStandardProcessSession) > Time elapsed: 0.008 s <<< FAILURE! > java.lang.AssertionError: Queued times should not be equal.. Actual: > 1539636497790 > at > org.apache.nifi.controller.repository.TestStandardProcessSession.testBatchQueuedHaveSameQueuedTime(TestStandardProcessSession.java:1691) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5704) TestStandardProcessSession testBatchQueuedHaveSameQueuedTime is a brittle test
[ https://issues.apache.org/jira/browse/NIFI-5704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16651051#comment-16651051 ] Peter Wicks commented on NIFI-5704: --- I wrote it up, I'm checking it out. > TestStandardProcessSession testBatchQueuedHaveSameQueuedTime is a brittle test > -- > > Key: NIFI-5704 > URL: https://issues.apache.org/jira/browse/NIFI-5704 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.8.0 >Reporter: Joseph Witt >Priority: Major > > [ERROR] Failures: > [ERROR] TestStandardProcessSession.testBatchQueuedHaveSameQueuedTime:1691 > Queued times should not be equal.. Actual: 1539636497790 > [ERROR] Tests run: 59, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: > 0.102 s <<< FAILURE! - in > org.apache.nifi.controller.repository.TestStandardProcessSession > [ERROR] > testBatchQueuedHaveSameQueuedTime(org.apache.nifi.controller.repository.TestStandardProcessSession) > Time elapsed: 0.008 s <<< FAILURE! > java.lang.AssertionError: Queued times should not be equal.. Actual: > 1539636497790 > at > org.apache.nifi.controller.repository.TestStandardProcessSession.testBatchQueuedHaveSameQueuedTime(TestStandardProcessSession.java:1691) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5705) Add Hive 3 attribution to assembly and top-level NOTICEs
[ https://issues.apache.org/jira/browse/NIFI-5705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650966#comment-16650966 ] ASF GitHub Bot commented on NIFI-5705: -- GitHub user mattyb149 opened a pull request: https://github.com/apache/nifi/pull/3076 NIFI-5705: Added Hive 3 attribution to nifi-assembly and top-level NOTICEs Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [x] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mattyb149/nifi NIFI-5705 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3076.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3076 commit e4c1fc0d8e6bbe6ca4b07f7556f248567721a268 Author: Matthew Burgess Date: 2018-10-15T23:34:05Z NIFI-5705: Added Hive 3 attribution to nifi-assembly and top-level NOTICEs > Add Hive 3 attribution to assembly and top-level NOTICEs > > > Key: NIFI-5705 > URL: https://issues.apache.org/jira/browse/NIFI-5705 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation Website >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Fix For: 1.8.0 > > > NiFiRecordSerDe in the Hive 3 NAR derives from JsonSerDe in the Hive 3 > codebase. The attribution was added to the Hive 3 NAR's NOTICE, but at the > time it wasn't associated with a release (as the release hadn't happened), > and it wasn't added to the nifi-assembly and top-level NOTICEs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5705) Add Hive 3 attribution to assembly and top-level NOTICEs
[ https://issues.apache.org/jira/browse/NIFI-5705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-5705: --- Status: Patch Available (was: In Progress) > Add Hive 3 attribution to assembly and top-level NOTICEs > > > Key: NIFI-5705 > URL: https://issues.apache.org/jira/browse/NIFI-5705 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation Website >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Fix For: 1.8.0 > > > NiFiRecordSerDe in the Hive 3 NAR derives from JsonSerDe in the Hive 3 > codebase. The attribution was added to the Hive 3 NAR's NOTICE, but at the > time it wasn't associated with a release (as the release hadn't happened), > and it wasn't added to the nifi-assembly and top-level NOTICEs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (NIFI-5705) Add Hive 3 attribution to assembly and top-level NOTICEs
[ https://issues.apache.org/jira/browse/NIFI-5705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess reassigned NIFI-5705: -- Assignee: Matt Burgess > Add Hive 3 attribution to assembly and top-level NOTICEs > > > Key: NIFI-5705 > URL: https://issues.apache.org/jira/browse/NIFI-5705 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation Website >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Fix For: 1.8.0 > > > NiFiRecordSerDe in the Hive 3 NAR derives from JsonSerDe in the Hive 3 > codebase. The attribution was added to the Hive 3 NAR's NOTICE, but at the > time it wasn't associated with a release (as the release hadn't happened), > and it wasn't added to the nifi-assembly and top-level NOTICEs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3076: NIFI-5705: Added Hive 3 attribution to nifi-assembl...
GitHub user mattyb149 opened a pull request: https://github.com/apache/nifi/pull/3076 NIFI-5705: Added Hive 3 attribution to nifi-assembly and top-level NOTICEs Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [x] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mattyb149/nifi NIFI-5705 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3076.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3076 commit e4c1fc0d8e6bbe6ca4b07f7556f248567721a268 Author: Matthew Burgess Date: 2018-10-15T23:34:05Z NIFI-5705: Added Hive 3 attribution to nifi-assembly and top-level NOTICEs ---
[jira] [Created] (NIFI-5705) Add Hive 3 attribution to assembly and top-level NOTICEs
Matt Burgess created NIFI-5705: -- Summary: Add Hive 3 attribution to assembly and top-level NOTICEs Key: NIFI-5705 URL: https://issues.apache.org/jira/browse/NIFI-5705 Project: Apache NiFi Issue Type: Improvement Components: Documentation Website Reporter: Matt Burgess Fix For: 1.8.0 NiFiRecordSerDe in the Hive 3 NAR derives from JsonSerDe in the Hive 3 codebase. The attribution was added to the Hive 3 NAR's NOTICE, but at the time it wasn't associated with a release (as the release hadn't happened), and it wasn't added to the nifi-assembly and top-level NOTICEs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (NIFI-5659) Add documentation for Offloading Nodes functionality and new Node related CLI commands
[ https://issues.apache.org/jira/browse/NIFI-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Storck resolved NIFI-5659. --- Resolution: Done > Add documentation for Offloading Nodes functionality and new Node related CLI > commands > -- > > Key: NIFI-5659 > URL: https://issues.apache.org/jira/browse/NIFI-5659 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation Website >Reporter: Andrew Lim >Assignee: Andrew Lim >Priority: Minor > Fix For: 1.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5659) Add documentation for Offloading Nodes functionality and new Node related CLI commands
[ https://issues.apache.org/jira/browse/NIFI-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650872#comment-16650872 ] ASF GitHub Bot commented on NIFI-5659: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/3056 > Add documentation for Offloading Nodes functionality and new Node related CLI > commands > -- > > Key: NIFI-5659 > URL: https://issues.apache.org/jira/browse/NIFI-5659 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation Website >Reporter: Andrew Lim >Assignee: Andrew Lim >Priority: Minor > Fix For: 1.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5659) Add documentation for Offloading Nodes functionality and new Node related CLI commands
[ https://issues.apache.org/jira/browse/NIFI-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650871#comment-16650871 ] ASF subversion and git services commented on NIFI-5659: --- Commit fd5eccc593b97ec3c836d870adde079cd78b374c in nifi's branch refs/heads/master from [~andrewmlim] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=fd5eccc ] NIFI-5659 Add documentation for Offloading Nodes functionality and new Node related CLI commands This closes #3056 > Add documentation for Offloading Nodes functionality and new Node related CLI > commands > -- > > Key: NIFI-5659 > URL: https://issues.apache.org/jira/browse/NIFI-5659 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation Website >Reporter: Andrew Lim >Assignee: Andrew Lim >Priority: Minor > Fix For: 1.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3056: NIFI-5659 Add documentation for Offloading Nodes fu...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/3056 ---
[jira] [Updated] (NIFI-5659) Add documentation for Offloading Nodes functionality and new Node related CLI commands
[ https://issues.apache.org/jira/browse/NIFI-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Storck updated NIFI-5659: -- Fix Version/s: 1.8.0 > Add documentation for Offloading Nodes functionality and new Node related CLI > commands > -- > > Key: NIFI-5659 > URL: https://issues.apache.org/jira/browse/NIFI-5659 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation Website >Reporter: Andrew Lim >Assignee: Andrew Lim >Priority: Minor > Fix For: 1.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5704) TestStandardProcessSession testBatchQueuedHaveSameQueuedTime is a brittle test
[ https://issues.apache.org/jira/browse/NIFI-5704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650800#comment-16650800 ] Joseph Witt commented on NIFI-5704: --- [~patricker] [~mikerthomsen] can you look into the above test? It appears to not be written in such a way to work reliably on different systems. I've not looked in detail but it fails sometimes on one of my higher power build systems. > TestStandardProcessSession testBatchQueuedHaveSameQueuedTime is a brittle test > -- > > Key: NIFI-5704 > URL: https://issues.apache.org/jira/browse/NIFI-5704 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.8.0 >Reporter: Joseph Witt >Priority: Major > > [ERROR] Failures: > [ERROR] TestStandardProcessSession.testBatchQueuedHaveSameQueuedTime:1691 > Queued times should not be equal.. Actual: 1539636497790 > [ERROR] Tests run: 59, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: > 0.102 s <<< FAILURE! - in > org.apache.nifi.controller.repository.TestStandardProcessSession > [ERROR] > testBatchQueuedHaveSameQueuedTime(org.apache.nifi.controller.repository.TestStandardProcessSession) > Time elapsed: 0.008 s <<< FAILURE! > java.lang.AssertionError: Queued times should not be equal.. Actual: > 1539636497790 > at > org.apache.nifi.controller.repository.TestStandardProcessSession.testBatchQueuedHaveSameQueuedTime(TestStandardProcessSession.java:1691) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5704) TestStandardProcessSession testBatchQueuedHaveSameQueuedTime is a brittle test
Joseph Witt created NIFI-5704: - Summary: TestStandardProcessSession testBatchQueuedHaveSameQueuedTime is a brittle test Key: NIFI-5704 URL: https://issues.apache.org/jira/browse/NIFI-5704 Project: Apache NiFi Issue Type: Bug Components: Core Framework Affects Versions: 1.8.0 Reporter: Joseph Witt [ERROR] Failures: [ERROR] TestStandardProcessSession.testBatchQueuedHaveSameQueuedTime:1691 Queued times should not be equal.. Actual: 1539636497790 [ERROR] Tests run: 59, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 0.102 s <<< FAILURE! - in org.apache.nifi.controller.repository.TestStandardProcessSession [ERROR] testBatchQueuedHaveSameQueuedTime(org.apache.nifi.controller.repository.TestStandardProcessSession) Time elapsed: 0.008 s <<< FAILURE! java.lang.AssertionError: Queued times should not be equal.. Actual: 1539636497790 at org.apache.nifi.controller.repository.TestStandardProcessSession.testBatchQueuedHaveSameQueuedTime(TestStandardProcessSession.java:1691) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (NIFI-4811) Use a newer version of spring-data-redis
[ https://issues.apache.org/jira/browse/NIFI-4811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Witt resolved NIFI-4811. --- Resolution: Fixed > Use a newer version of spring-data-redis > > > Key: NIFI-4811 > URL: https://issues.apache.org/jira/browse/NIFI-4811 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Johan Oceanstorm >Assignee: Mike Thomsen >Priority: Major > Fix For: 1.8.0 > > Attachments: NIFI-4811-fixuplandn.patch > > > NiFi uses version 1.8.3 of spring-data-redis (as you can see > [here|[https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-redis-bundle/pom.xml#L31].] > This version is an old version. The newer stable version is 2.0.2. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4811) Use a newer version of spring-data-redis
[ https://issues.apache.org/jira/browse/NIFI-4811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650761#comment-16650761 ] ASF GitHub Bot commented on NIFI-4811: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/2856 > Use a newer version of spring-data-redis > > > Key: NIFI-4811 > URL: https://issues.apache.org/jira/browse/NIFI-4811 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Johan Oceanstorm >Assignee: Mike Thomsen >Priority: Major > Fix For: 1.8.0 > > Attachments: NIFI-4811-fixuplandn.patch > > > NiFi uses version 1.8.3 of spring-data-redis (as you can see > [here|[https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-redis-bundle/pom.xml#L31].] > This version is an old version. The newer stable version is 2.0.2. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #2856: NIFI-4811 Added two missing entries to the nifi-red...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/2856 ---
[jira] [Commented] (NIFI-4811) Use a newer version of spring-data-redis
[ https://issues.apache.org/jira/browse/NIFI-4811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650759#comment-16650759 ] ASF subversion and git services commented on NIFI-4811: --- Commit 11afb0115e0dac3092a02e9c5fb3997e79704113 in nifi's branch refs/heads/master from [~mike.thomsen] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=11afb01 ] NIFI-4811 This closes #2856. Removed no longer necessary license and updated NOTICE Signed-off-by: joewitt > Use a newer version of spring-data-redis > > > Key: NIFI-4811 > URL: https://issues.apache.org/jira/browse/NIFI-4811 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Johan Oceanstorm >Assignee: Mike Thomsen >Priority: Major > Fix For: 1.8.0 > > Attachments: NIFI-4811-fixuplandn.patch > > > NiFi uses version 1.8.3 of spring-data-redis (as you can see > [here|[https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-redis-bundle/pom.xml#L31].] > This version is an old version. The newer stable version is 2.0.2. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-4811) Use a newer version of spring-data-redis
[ https://issues.apache.org/jira/browse/NIFI-4811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Witt updated NIFI-4811: -- Attachment: NIFI-4811-fixuplandn.patch > Use a newer version of spring-data-redis > > > Key: NIFI-4811 > URL: https://issues.apache.org/jira/browse/NIFI-4811 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Johan Oceanstorm >Assignee: Mike Thomsen >Priority: Major > Fix For: 1.8.0 > > Attachments: NIFI-4811-fixuplandn.patch > > > NiFi uses version 1.8.3 of spring-data-redis (as you can see > [here|[https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-redis-bundle/pom.xml#L31].] > This version is an old version. The newer stable version is 2.0.2. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5604) Allow GenerateTableFetch to send empty flow files when no rows would be fetched
[ https://issues.apache.org/jira/browse/NIFI-5604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650745#comment-16650745 ] ASF GitHub Bot commented on NIFI-5604: -- GitHub user mattyb149 opened a pull request: https://github.com/apache/nifi/pull/3075 NIFI-5604: Added property to allow empty FlowFile when no SQL generated by GenerateTableFetch Note that this should work with NIFI-5601 (#3074) such that the fragment attributes would be written to the empty flow file as well. Depending on which gets merged first, the other probably needs to be updated. Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mattyb149/nifi NIFI-5604 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3075.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3075 commit 43bfe9531a0aa2a64179d27b4196a6062f60ebbe Author: Matthew Burgess Date: 2018-10-15T20:27:38Z NIFI-5604: Added property to allow empty FlowFile when no SQL generated by GenerateTableFetch > Allow GenerateTableFetch to send empty flow files when no rows would be > fetched > --- > > Key: NIFI-5604 > URL: https://issues.apache.org/jira/browse/NIFI-5604 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > > Currently, GenerateTableFetch will not output a flow file if there are no > rows to be fetched. However, it may be desired (especially with incoming flow > files) that a flow file be sent out even if GTF does not generate any SQL. > This capability, along with the fragment attributes from NIFI-5601, would > allow the user to handle this downstream. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5604) Allow GenerateTableFetch to send empty flow files when no rows would be fetched
[ https://issues.apache.org/jira/browse/NIFI-5604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-5604: --- Status: Patch Available (was: In Progress) > Allow GenerateTableFetch to send empty flow files when no rows would be > fetched > --- > > Key: NIFI-5604 > URL: https://issues.apache.org/jira/browse/NIFI-5604 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > > Currently, GenerateTableFetch will not output a flow file if there are no > rows to be fetched. However, it may be desired (especially with incoming flow > files) that a flow file be sent out even if GTF does not generate any SQL. > This capability, along with the fragment attributes from NIFI-5601, would > allow the user to handle this downstream. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3075: NIFI-5604: Added property to allow empty FlowFile w...
GitHub user mattyb149 opened a pull request: https://github.com/apache/nifi/pull/3075 NIFI-5604: Added property to allow empty FlowFile when no SQL generated by GenerateTableFetch Note that this should work with NIFI-5601 (#3074) such that the fragment attributes would be written to the empty flow file as well. Depending on which gets merged first, the other probably needs to be updated. Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mattyb149/nifi NIFI-5604 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3075.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3075 commit 43bfe9531a0aa2a64179d27b4196a6062f60ebbe Author: Matthew Burgess Date: 2018-10-15T20:27:38Z NIFI-5604: Added property to allow empty FlowFile when no SQL generated by GenerateTableFetch ---
[jira] [Assigned] (NIFI-5604) Allow GenerateTableFetch to send empty flow files when no rows would be fetched
[ https://issues.apache.org/jira/browse/NIFI-5604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess reassigned NIFI-5604: -- Assignee: Matt Burgess > Allow GenerateTableFetch to send empty flow files when no rows would be > fetched > --- > > Key: NIFI-5604 > URL: https://issues.apache.org/jira/browse/NIFI-5604 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > > Currently, GenerateTableFetch will not output a flow file if there are no > rows to be fetched. However, it may be desired (especially with incoming flow > files) that a flow file be sent out even if GTF does not generate any SQL. > This capability, along with the fragment attributes from NIFI-5601, would > allow the user to handle this downstream. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5601) Add fragment.* attributes to GenerateTableFetch
[ https://issues.apache.org/jira/browse/NIFI-5601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-5601: --- Status: Patch Available (was: In Progress) > Add fragment.* attributes to GenerateTableFetch > --- > > Key: NIFI-5601 > URL: https://issues.apache.org/jira/browse/NIFI-5601 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > > GenerateTableFetch will generate SQL statements in multiple flow files > corresponding to "pages" of data for a single execution of the processor. > QueryDatabaseTable does something similar, although it also does the fetching > of the rows and adds fragment.* attributes to the outgoing flow files, in > case a merge or other correlation needs to be done later. > GenerateTableFetch (although it doesn't produce source data per se) is > usually used in conjunction with ExecuteSQL to fetch the actual pages of > rows, and sometimes these would need to be merged/correlated. To that end, > this Jira proposes to add the same fragment.* attributes to the outgoing GTF > flow files for use downstream. > Note that if the SQL statements are distributed among a cluster (for parallel > execution by ExecuteSQL), they will not be able to be merged, as different > nodes may get different fragments and often no single node will get all the > fragments. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5601) Add fragment.* attributes to GenerateTableFetch
[ https://issues.apache.org/jira/browse/NIFI-5601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650713#comment-16650713 ] ASF GitHub Bot commented on NIFI-5601: -- GitHub user mattyb149 opened a pull request: https://github.com/apache/nifi/pull/3074 NIFI-5601: Add fragment.* attributes to GenerateTableFetch Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mattyb149/nifi NIFI-5601 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3074.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3074 commit bd6ce6d8c74b65a222fb2a842465e5aae2d40272 Author: Matthew Burgess Date: 2018-10-15T20:07:13Z NIFI-5601: Add fragment.* attributes to GenerateTableFetch > Add fragment.* attributes to GenerateTableFetch > --- > > Key: NIFI-5601 > URL: https://issues.apache.org/jira/browse/NIFI-5601 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > > GenerateTableFetch will generate SQL statements in multiple flow files > corresponding to "pages" of data for a single execution of the processor. > QueryDatabaseTable does something similar, although it also does the fetching > of the rows and adds fragment.* attributes to the outgoing flow files, in > case a merge or other correlation needs to be done later. > GenerateTableFetch (although it doesn't produce source data per se) is > usually used in conjunction with ExecuteSQL to fetch the actual pages of > rows, and sometimes these would need to be merged/correlated. To that end, > this Jira proposes to add the same fragment.* attributes to the outgoing GTF > flow files for use downstream. > Note that if the SQL statements are distributed among a cluster (for parallel > execution by ExecuteSQL), they will not be able to be merged, as different > nodes may get different fragments and often no single node will get all the > fragments. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3074: NIFI-5601: Add fragment.* attributes to GenerateTab...
GitHub user mattyb149 opened a pull request: https://github.com/apache/nifi/pull/3074 NIFI-5601: Add fragment.* attributes to GenerateTableFetch Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mattyb149/nifi NIFI-5601 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3074.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3074 commit bd6ce6d8c74b65a222fb2a842465e5aae2d40272 Author: Matthew Burgess Date: 2018-10-15T20:07:13Z NIFI-5601: Add fragment.* attributes to GenerateTableFetch ---
[jira] [Assigned] (NIFI-5601) Add fragment.* attributes to GenerateTableFetch
[ https://issues.apache.org/jira/browse/NIFI-5601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess reassigned NIFI-5601: -- Assignee: Matt Burgess > Add fragment.* attributes to GenerateTableFetch > --- > > Key: NIFI-5601 > URL: https://issues.apache.org/jira/browse/NIFI-5601 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > > GenerateTableFetch will generate SQL statements in multiple flow files > corresponding to "pages" of data for a single execution of the processor. > QueryDatabaseTable does something similar, although it also does the fetching > of the rows and adds fragment.* attributes to the outgoing flow files, in > case a merge or other correlation needs to be done later. > GenerateTableFetch (although it doesn't produce source data per se) is > usually used in conjunction with ExecuteSQL to fetch the actual pages of > rows, and sometimes these would need to be merged/correlated. To that end, > this Jira proposes to add the same fragment.* attributes to the outgoing GTF > flow files for use downstream. > Note that if the SQL statements are distributed among a cluster (for parallel > execution by ExecuteSQL), they will not be able to be merged, as different > nodes may get different fragments and often no single node will get all the > fragments. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5686) Test failure in TestStandardProcessScheduler
[
https://issues.apache.org/jira/browse/NIFI-5686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650655#comment-16650655
]
ASF subversion and git services commented on NIFI-5686:
---
Commit 32db43b3069b3c982b9c96576c11b447c229b360 in nifi's branch
refs/heads/master from [~markap14]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=32db43b ]
NIFI-5686: Updated StandardProcessScheduler so that if it fails to schedule a
Reporting Task, it re-schedules the @OnScheduled task instead of looping and
calling Thread.sleep. As it was, the single-threaded Process Scheduler was,
when calling ProcessScheduler.unschedule(), the unschedule task was not
executing because the schedule task was using the only thread. But switching
the logic to schedule the task for later and return, instead of calling
Thread.sleep and looping, we are able to avoid blocking the one thread in the
thread pool. Also, performed some trivial code cleanup and updated erroneous
links in Java-docs.
NIFI-5686: Fixed unit test in TestSocketLoadBalancedFlowFileQueue; renamed
TestProcessorLifecycle to ProcessorLifecycleIT as it is testing integration
between many components and largely focuses on high numbers of concurrent tasks
to see if it can trigger any threading bugs that may get introduced
NIFI-5686: Extended unit test timeouts
Signed-off-by: Matthew Burgess
This closes #3062
> Test failure in TestStandardProcessScheduler
>
>
> Key: NIFI-5686
> URL: https://issues.apache.org/jira/browse/NIFI-5686
> Project: Apache NiFi
> Issue Type: Bug
> Components: Tools and Build
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.8.0
>
>
> The {{testReportingTaskDoesntKeepRunningAfterStop}} test in
> {{TestStandardProcessScheduler}} now fails. This was \{{@Ignore}} d in
> NIFI-5666 but should be addressed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #3062: NIFI-5686: Updated StandardProcessScheduler so that...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/3062 ---
[jira] [Commented] (NIFI-5686) Test failure in TestStandardProcessScheduler
[
https://issues.apache.org/jira/browse/NIFI-5686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650657#comment-16650657
]
ASF subversion and git services commented on NIFI-5686:
---
Commit 32db43b3069b3c982b9c96576c11b447c229b360 in nifi's branch
refs/heads/master from [~markap14]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=32db43b ]
NIFI-5686: Updated StandardProcessScheduler so that if it fails to schedule a
Reporting Task, it re-schedules the @OnScheduled task instead of looping and
calling Thread.sleep. As it was, the single-threaded Process Scheduler was,
when calling ProcessScheduler.unschedule(), the unschedule task was not
executing because the schedule task was using the only thread. But switching
the logic to schedule the task for later and return, instead of calling
Thread.sleep and looping, we are able to avoid blocking the one thread in the
thread pool. Also, performed some trivial code cleanup and updated erroneous
links in Java-docs.
NIFI-5686: Fixed unit test in TestSocketLoadBalancedFlowFileQueue; renamed
TestProcessorLifecycle to ProcessorLifecycleIT as it is testing integration
between many components and largely focuses on high numbers of concurrent tasks
to see if it can trigger any threading bugs that may get introduced
NIFI-5686: Extended unit test timeouts
Signed-off-by: Matthew Burgess
This closes #3062
> Test failure in TestStandardProcessScheduler
>
>
> Key: NIFI-5686
> URL: https://issues.apache.org/jira/browse/NIFI-5686
> Project: Apache NiFi
> Issue Type: Bug
> Components: Tools and Build
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.8.0
>
>
> The {{testReportingTaskDoesntKeepRunningAfterStop}} test in
> {{TestStandardProcessScheduler}} now fails. This was \{{@Ignore}} d in
> NIFI-5666 but should be addressed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5686) Test failure in TestStandardProcessScheduler
[
https://issues.apache.org/jira/browse/NIFI-5686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650654#comment-16650654
]
ASF GitHub Bot commented on NIFI-5686:
--
Github user mattyb149 commented on the issue:
https://github.com/apache/nifi/pull/3062
+1 LGTM, ran full build with unit tests, tried on a live instance with
reporting tasks and various other flows. Thanks for the improvement! Merging to
master
> Test failure in TestStandardProcessScheduler
>
>
> Key: NIFI-5686
> URL: https://issues.apache.org/jira/browse/NIFI-5686
> Project: Apache NiFi
> Issue Type: Bug
> Components: Tools and Build
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.8.0
>
>
> The {{testReportingTaskDoesntKeepRunningAfterStop}} test in
> {{TestStandardProcessScheduler}} now fails. This was \{{@Ignore}} d in
> NIFI-5666 but should be addressed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5686) Test failure in TestStandardProcessScheduler
[
https://issues.apache.org/jira/browse/NIFI-5686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650658#comment-16650658
]
ASF GitHub Bot commented on NIFI-5686:
--
Github user asfgit closed the pull request at:
https://github.com/apache/nifi/pull/3062
> Test failure in TestStandardProcessScheduler
>
>
> Key: NIFI-5686
> URL: https://issues.apache.org/jira/browse/NIFI-5686
> Project: Apache NiFi
> Issue Type: Bug
> Components: Tools and Build
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.8.0
>
>
> The {{testReportingTaskDoesntKeepRunningAfterStop}} test in
> {{TestStandardProcessScheduler}} now fails. This was \{{@Ignore}} d in
> NIFI-5666 but should be addressed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5686) Test failure in TestStandardProcessScheduler
[
https://issues.apache.org/jira/browse/NIFI-5686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650656#comment-16650656
]
ASF subversion and git services commented on NIFI-5686:
---
Commit 32db43b3069b3c982b9c96576c11b447c229b360 in nifi's branch
refs/heads/master from [~markap14]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=32db43b ]
NIFI-5686: Updated StandardProcessScheduler so that if it fails to schedule a
Reporting Task, it re-schedules the @OnScheduled task instead of looping and
calling Thread.sleep. As it was, the single-threaded Process Scheduler was,
when calling ProcessScheduler.unschedule(), the unschedule task was not
executing because the schedule task was using the only thread. But switching
the logic to schedule the task for later and return, instead of calling
Thread.sleep and looping, we are able to avoid blocking the one thread in the
thread pool. Also, performed some trivial code cleanup and updated erroneous
links in Java-docs.
NIFI-5686: Fixed unit test in TestSocketLoadBalancedFlowFileQueue; renamed
TestProcessorLifecycle to ProcessorLifecycleIT as it is testing integration
between many components and largely focuses on high numbers of concurrent tasks
to see if it can trigger any threading bugs that may get introduced
NIFI-5686: Extended unit test timeouts
Signed-off-by: Matthew Burgess
This closes #3062
> Test failure in TestStandardProcessScheduler
>
>
> Key: NIFI-5686
> URL: https://issues.apache.org/jira/browse/NIFI-5686
> Project: Apache NiFi
> Issue Type: Bug
> Components: Tools and Build
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.8.0
>
>
> The {{testReportingTaskDoesntKeepRunningAfterStop}} test in
> {{TestStandardProcessScheduler}} now fails. This was \{{@Ignore}} d in
> NIFI-5666 but should be addressed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (NIFI-5686) Test failure in TestStandardProcessScheduler
[
https://issues.apache.org/jira/browse/NIFI-5686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess updated NIFI-5686:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Test failure in TestStandardProcessScheduler
>
>
> Key: NIFI-5686
> URL: https://issues.apache.org/jira/browse/NIFI-5686
> Project: Apache NiFi
> Issue Type: Bug
> Components: Tools and Build
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.8.0
>
>
> The {{testReportingTaskDoesntKeepRunningAfterStop}} test in
> {{TestStandardProcessScheduler}} now fails. This was \{{@Ignore}} d in
> NIFI-5666 but should be addressed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi issue #3062: NIFI-5686: Updated StandardProcessScheduler so that if it ...
Github user mattyb149 commented on the issue: https://github.com/apache/nifi/pull/3062 +1 LGTM, ran full build with unit tests, tried on a live instance with reporting tasks and various other flows. Thanks for the improvement! Merging to master ---
[jira] [Updated] (NIFI-5678) ValidateRecord does not handle Map type correctly
[
https://issues.apache.org/jira/browse/NIFI-5678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne updated NIFI-5678:
-
Resolution: Fixed
Fix Version/s: 1.8.0
Status: Resolved (was: Patch Available)
> ValidateRecord does not handle Map type correctly
> -
>
> Key: NIFI-5678
> URL: https://issues.apache.org/jira/browse/NIFI-5678
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
> Fix For: 1.8.0
>
>
> Consider the following Avro Schema:
> {code}
> {
> "name" : "test",
> "type" : "record",
> "fields" : [ {
> "name" : "field1",
> "type" : {
> "type" : "map",
> "values" : "string"
> }
> } ]
> }
> {code}
> and corresponding JSON data adhering to the schema:
> {code}
> [{
> "field1": {
> "toto" : "v1",
> "titi" : "v2"
> }
> }]
> {code}
> ValidateRecord marks the record as invalid though it should be valid. The
> message in the provenance event is "Record #1 is invalid due to:
> MapRecord[{toto=v1, titi=v2}] is not a valid value for /field1: Value is of
> type org.apache.nifi.serialization.record.MapRecord but was expected to be of
> type MAP[STRING]".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5669) Color on current thread count
[ https://issues.apache.org/jira/browse/NIFI-5669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650594#comment-16650594 ] Pierre Villard commented on NIFI-5669: -- Good point [~patricker]. I'd say that coloring should happen as soon as a single node is maxed out on threads - usually the primary node would use more threads than the other nodes. We have the details when going in the 'Cluster' view but the idea here would be to indicate to users that there could be something to tune / look at. > Color on current thread count > - > > Key: NIFI-5669 > URL: https://issues.apache.org/jira/browse/NIFI-5669 > Project: Apache NiFi > Issue Type: Improvement > Components: Core UI >Reporter: Pierre Villard >Priority: Minor > > It'd be very useful to change the color (or have another kind of visual > indicator) of the current threads count in the upper menu bar when the number > is reaching the maximum number of available threads defined by the thread > pool size. > Note - It could a bit tricky with the two thread pools (timer driven / event > driven) but that would be a very useful indicator as this is, quite often, > something users do not know about. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5678) ValidateRecord does not handle Map type correctly
[
https://issues.apache.org/jira/browse/NIFI-5678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650593#comment-16650593
]
ASF GitHub Bot commented on NIFI-5678:
--
Github user markap14 commented on the issue:
https://github.com/apache/nifi/pull/3060
Thanks @mattyb149 all looks good to me, was able to verify that the unit
test performs as expected and manual testing (including the example from the
dev@ mailing list, where I think this JIRA originated from) passes my tests. +1
merged to master.
> ValidateRecord does not handle Map type correctly
> -
>
> Key: NIFI-5678
> URL: https://issues.apache.org/jira/browse/NIFI-5678
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> Consider the following Avro Schema:
> {code}
> {
> "name" : "test",
> "type" : "record",
> "fields" : [ {
> "name" : "field1",
> "type" : {
> "type" : "map",
> "values" : "string"
> }
> } ]
> }
> {code}
> and corresponding JSON data adhering to the schema:
> {code}
> [{
> "field1": {
> "toto" : "v1",
> "titi" : "v2"
> }
> }]
> {code}
> ValidateRecord marks the record as invalid though it should be valid. The
> message in the provenance event is "Record #1 is invalid due to:
> MapRecord[{toto=v1, titi=v2}] is not a valid value for /field1: Value is of
> type org.apache.nifi.serialization.record.MapRecord but was expected to be of
> type MAP[STRING]".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5678) ValidateRecord does not handle Map type correctly
[
https://issues.apache.org/jira/browse/NIFI-5678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650592#comment-16650592
]
ASF GitHub Bot commented on NIFI-5678:
--
Github user asfgit closed the pull request at:
https://github.com/apache/nifi/pull/3060
> ValidateRecord does not handle Map type correctly
> -
>
> Key: NIFI-5678
> URL: https://issues.apache.org/jira/browse/NIFI-5678
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> Consider the following Avro Schema:
> {code}
> {
> "name" : "test",
> "type" : "record",
> "fields" : [ {
> "name" : "field1",
> "type" : {
> "type" : "map",
> "values" : "string"
> }
> } ]
> }
> {code}
> and corresponding JSON data adhering to the schema:
> {code}
> [{
> "field1": {
> "toto" : "v1",
> "titi" : "v2"
> }
> }]
> {code}
> ValidateRecord marks the record as invalid though it should be valid. The
> message in the provenance event is "Record #1 is invalid due to:
> MapRecord[{toto=v1, titi=v2}] is not a valid value for /field1: Value is of
> type org.apache.nifi.serialization.record.MapRecord but was expected to be of
> type MAP[STRING]".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi issue #3060: NIFI-5678: Fixed MAP type support of MapRecord objects in ...
Github user markap14 commented on the issue: https://github.com/apache/nifi/pull/3060 Thanks @mattyb149 all looks good to me, was able to verify that the unit test performs as expected and manual testing (including the example from the dev@ mailing list, where I think this JIRA originated from) passes my tests. +1 merged to master. ---
[GitHub] nifi pull request #3060: NIFI-5678: Fixed MAP type support of MapRecord obje...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/3060 ---
[jira] [Commented] (NIFI-5678) ValidateRecord does not handle Map type correctly
[
https://issues.apache.org/jira/browse/NIFI-5678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650590#comment-16650590
]
ASF subversion and git services commented on NIFI-5678:
---
Commit 218063a0b57c609cbc30276907c0c254d88c699c in nifi's branch
refs/heads/master from [~ca9mbu]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=218063a ]
NIFI-5678: Fixed MAP type support of MapRecord objects in
StandardSchemaValidator
This closes #3060.
Signed-off-by: Mark Payne
> ValidateRecord does not handle Map type correctly
> -
>
> Key: NIFI-5678
> URL: https://issues.apache.org/jira/browse/NIFI-5678
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> Consider the following Avro Schema:
> {code}
> {
> "name" : "test",
> "type" : "record",
> "fields" : [ {
> "name" : "field1",
> "type" : {
> "type" : "map",
> "values" : "string"
> }
> } ]
> }
> {code}
> and corresponding JSON data adhering to the schema:
> {code}
> [{
> "field1": {
> "toto" : "v1",
> "titi" : "v2"
> }
> }]
> {code}
> ValidateRecord marks the record as invalid though it should be valid. The
> message in the provenance event is "Record #1 is invalid due to:
> MapRecord[{toto=v1, titi=v2}] is not a valid value for /field1: Value is of
> type org.apache.nifi.serialization.record.MapRecord but was expected to be of
> type MAP[STRING]".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-4806) Upgrade to Tika/Tika-Parsers 1.19 to resolve TIKA-2535 and CVEs reported against all releases prior to 1.19
[ https://issues.apache.org/jira/browse/NIFI-4806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650543#comment-16650543 ] ASF GitHub Bot commented on NIFI-4806: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/3069 > Upgrade to Tika/Tika-Parsers 1.19 to resolve TIKA-2535 and CVEs reported > against all releases prior to 1.19 > --- > > Key: NIFI-4806 > URL: https://issues.apache.org/jira/browse/NIFI-4806 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.5.0 >Reporter: Joseph Witt >Assignee: Matt Gilman >Priority: Blocker > Fix For: 1.8.0 > > > The nifi-media-processors depend on Tika-Parsers 1.16 as of now. They need > to be upgraded to at least Tika-Parsers 1.18 to resolve the licensing problem > identified to the Tika team and which they resolved in > https://issues.apache.org/jira/browse/TIKA-2535 > > We aren't susceptible to the licensing problem at this time because when we > did the last update for Tika-Parsers this was flagged and excluded though we > could be exposing bugs for certain datayptes we'd do mime detection on > (maybe). I have a comment about this in our pom. > > This Jira is to upgrade, ensure no invalid libs are used, and clean up the > comments and move on. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-4806) Upgrade to Tika/Tika-Parsers 1.19 to resolve TIKA-2535 and CVEs reported against all releases prior to 1.19
[ https://issues.apache.org/jira/browse/NIFI-4806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Gilman updated NIFI-4806: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Upgrade to Tika/Tika-Parsers 1.19 to resolve TIKA-2535 and CVEs reported > against all releases prior to 1.19 > --- > > Key: NIFI-4806 > URL: https://issues.apache.org/jira/browse/NIFI-4806 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.5.0 >Reporter: Joseph Witt >Assignee: Matt Gilman >Priority: Blocker > Fix For: 1.8.0 > > > The nifi-media-processors depend on Tika-Parsers 1.16 as of now. They need > to be upgraded to at least Tika-Parsers 1.18 to resolve the licensing problem > identified to the Tika team and which they resolved in > https://issues.apache.org/jira/browse/TIKA-2535 > > We aren't susceptible to the licensing problem at this time because when we > did the last update for Tika-Parsers this was flagged and excluded though we > could be exposing bugs for certain datayptes we'd do mime detection on > (maybe). I have a comment about this in our pom. > > This Jira is to upgrade, ensure no invalid libs are used, and clean up the > comments and move on. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4806) Upgrade to Tika/Tika-Parsers 1.19 to resolve TIKA-2535 and CVEs reported against all releases prior to 1.19
[ https://issues.apache.org/jira/browse/NIFI-4806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650542#comment-16650542 ] ASF GitHub Bot commented on NIFI-4806: -- Github user mcgilman commented on the issue: https://github.com/apache/nifi/pull/3069 Thanks @joewitt > Upgrade to Tika/Tika-Parsers 1.19 to resolve TIKA-2535 and CVEs reported > against all releases prior to 1.19 > --- > > Key: NIFI-4806 > URL: https://issues.apache.org/jira/browse/NIFI-4806 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.5.0 >Reporter: Joseph Witt >Assignee: Matt Gilman >Priority: Blocker > Fix For: 1.8.0 > > > The nifi-media-processors depend on Tika-Parsers 1.16 as of now. They need > to be upgraded to at least Tika-Parsers 1.18 to resolve the licensing problem > identified to the Tika team and which they resolved in > https://issues.apache.org/jira/browse/TIKA-2535 > > We aren't susceptible to the licensing problem at this time because when we > did the last update for Tika-Parsers this was flagged and excluded though we > could be exposing bugs for certain datayptes we'd do mime detection on > (maybe). I have a comment about this in our pom. > > This Jira is to upgrade, ensure no invalid libs are used, and clean up the > comments and move on. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3069: NIFI-4806: Bumping to tika 1.19.1
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/3069 ---
[jira] [Commented] (NIFI-4806) Upgrade to Tika/Tika-Parsers 1.19 to resolve TIKA-2535 and CVEs reported against all releases prior to 1.19
[ https://issues.apache.org/jira/browse/NIFI-4806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650540#comment-16650540 ] ASF subversion and git services commented on NIFI-4806: --- Commit f55204cb69d065b8cedddaaa82c5e6e87e651c83 in nifi's branch refs/heads/master from [~mcgilman] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=f55204c ] NIFI-4806: - Bumping to tika 1.19.1. This closes #3069 > Upgrade to Tika/Tika-Parsers 1.19 to resolve TIKA-2535 and CVEs reported > against all releases prior to 1.19 > --- > > Key: NIFI-4806 > URL: https://issues.apache.org/jira/browse/NIFI-4806 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.5.0 >Reporter: Joseph Witt >Assignee: Matt Gilman >Priority: Blocker > Fix For: 1.8.0 > > > The nifi-media-processors depend on Tika-Parsers 1.16 as of now. They need > to be upgraded to at least Tika-Parsers 1.18 to resolve the licensing problem > identified to the Tika team and which they resolved in > https://issues.apache.org/jira/browse/TIKA-2535 > > We aren't susceptible to the licensing problem at this time because when we > did the last update for Tika-Parsers this was flagged and excluded though we > could be exposing bugs for certain datayptes we'd do mime detection on > (maybe). I have a comment about this in our pom. > > This Jira is to upgrade, ensure no invalid libs are used, and clean up the > comments and move on. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #3069: NIFI-4806: Bumping to tika 1.19.1
Github user mcgilman commented on the issue: https://github.com/apache/nifi/pull/3069 Thanks @joewitt ---
[jira] [Created] (NIFI-5703) SyslogParser regex fails to handle messages that end with Windows style new line
Joseph Percivall created NIFI-5703: -- Summary: SyslogParser regex fails to handle messages that end with Windows style new line Key: NIFI-5703 URL: https://issues.apache.org/jira/browse/NIFI-5703 Project: Apache NiFi Issue Type: Bug Reporter: Joseph Percivall The common SyslogParser class, which is used by ListenSyslog and ParseSyslog, has two different regexes to match the RFCs for syslog. Both of which end in a "(.*)$"[1]. There is also special handling if the last character is "\n"[2]. Note that "." does not match any of the newline characters[3]. The endline handling should be expanded to handle "\r\n". [1]https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-extension-utils/nifi-syslog-utils/src/main/java/org/apache/nifi/syslog/parsers/SyslogParser.java#L49 [2] https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-extension-utils/nifi-syslog-utils/src/main/java/org/apache/nifi/syslog/parsers/SyslogParser.java#L128 [3] https://stackoverflow.com/questions/3445326/regex-in-java-how-to-deal-with-newline -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (NIFI-3469) Add multipart request support to HandleHttpRequest Processor
[
https://issues.apache.org/jira/browse/NIFI-3469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne resolved NIFI-3469.
--
Resolution: Fixed
> Add multipart request support to HandleHttpRequest Processor
>
>
> Key: NIFI-3469
> URL: https://issues.apache.org/jira/browse/NIFI-3469
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: Koji Kawamura
>Assignee: Endre Kovacs
>Priority: Major
>
> Currently, HandleHttpRequest outputs a single FlowFile containing all
> multipart values as following:
> {code}
> --ef07e8bf36c274d3
> Content-Disposition: form-data; name="p1"
> v1
> --ef07e8bf36c274d3
> Content-Disposition: form-data; name="p2"
> v2
> --ef07e8bf36c274d3--
> {code}
> Many users requested adding upload files support to NiFi.
> In order for HandleHttpRequest to support multipart data we need to add
> followings (this is based on a brief researching and can be more complex or
> simple):
> We need to use HttpServletRequest#getParts() as written in this stackoverflow
> thread:
> http://stackoverflow.com/questions/3337056/convenient-way-to-parse-incoming-multipart-form-data-parameters-in-a-servlet
> Also, we probably need a custom MultiPartInputStreamParser implementation.
> Because Jetty's default implementation writes input data to temporary
> directory on file system, instead, we'd like NiFi to write those into output
> FlowFiles content in streaming fashion.
> And we need request size validation checks, threshold for those validation
> should be passed via javax.servlet.MultipartConfigElement.
> Finally, we have to do something with HandleHttpResponse processor.
> Once HandleHttpRequest processor start splitting incoming request into
> multiple output FlowFiles, we need to wait for every fragment to be
> processed, then execute HandleHttpRequest.
> I think Wait/Notify processors (available from next version) will be helpful
> here.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-3469) Add multipart request support to HandleHttpRequest Processor
[
https://issues.apache.org/jira/browse/NIFI-3469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650500#comment-16650500
]
ASF GitHub Bot commented on NIFI-3469:
--
Github user asfgit closed the pull request at:
https://github.com/apache/nifi/pull/2991
> Add multipart request support to HandleHttpRequest Processor
>
>
> Key: NIFI-3469
> URL: https://issues.apache.org/jira/browse/NIFI-3469
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: Koji Kawamura
>Assignee: Endre Kovacs
>Priority: Major
>
> Currently, HandleHttpRequest outputs a single FlowFile containing all
> multipart values as following:
> {code}
> --ef07e8bf36c274d3
> Content-Disposition: form-data; name="p1"
> v1
> --ef07e8bf36c274d3
> Content-Disposition: form-data; name="p2"
> v2
> --ef07e8bf36c274d3--
> {code}
> Many users requested adding upload files support to NiFi.
> In order for HandleHttpRequest to support multipart data we need to add
> followings (this is based on a brief researching and can be more complex or
> simple):
> We need to use HttpServletRequest#getParts() as written in this stackoverflow
> thread:
> http://stackoverflow.com/questions/3337056/convenient-way-to-parse-incoming-multipart-form-data-parameters-in-a-servlet
> Also, we probably need a custom MultiPartInputStreamParser implementation.
> Because Jetty's default implementation writes input data to temporary
> directory on file system, instead, we'd like NiFi to write those into output
> FlowFiles content in streaming fashion.
> And we need request size validation checks, threshold for those validation
> should be passed via javax.servlet.MultipartConfigElement.
> Finally, we have to do something with HandleHttpResponse processor.
> Once HandleHttpRequest processor start splitting incoming request into
> multiple output FlowFiles, we need to wait for every fragment to be
> processed, then execute HandleHttpRequest.
> I think Wait/Notify processors (available from next version) will be helpful
> here.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #2991: NIFI-3469: multipart request support added to Handl...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/2991 ---
[jira] [Commented] (NIFI-3469) Add multipart request support to HandleHttpRequest Processor
[
https://issues.apache.org/jira/browse/NIFI-3469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650497#comment-16650497
]
ASF subversion and git services commented on NIFI-3469:
---
Commit d5f071b2fe5240d4face9b7c8582ba45fd668294 in nifi's branch
refs/heads/master from Endre Zoltan Kovacs
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=d5f071b ]
NIFI-3469 multipart request support added to HandleHttpRequest
- introducing a in-memory-file-size-threashold, above which the incoming file
is written to local disk
- using java.io.tmpdir for such file writes
- enhancing documentation
- documenting how to avoid premature HTTP response
- fix and UT for unsuccessful request registration
This closes #2991.
Signed-off-by: Mark Payne
> Add multipart request support to HandleHttpRequest Processor
>
>
> Key: NIFI-3469
> URL: https://issues.apache.org/jira/browse/NIFI-3469
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: Koji Kawamura
>Assignee: Endre Kovacs
>Priority: Major
>
> Currently, HandleHttpRequest outputs a single FlowFile containing all
> multipart values as following:
> {code}
> --ef07e8bf36c274d3
> Content-Disposition: form-data; name="p1"
> v1
> --ef07e8bf36c274d3
> Content-Disposition: form-data; name="p2"
> v2
> --ef07e8bf36c274d3--
> {code}
> Many users requested adding upload files support to NiFi.
> In order for HandleHttpRequest to support multipart data we need to add
> followings (this is based on a brief researching and can be more complex or
> simple):
> We need to use HttpServletRequest#getParts() as written in this stackoverflow
> thread:
> http://stackoverflow.com/questions/3337056/convenient-way-to-parse-incoming-multipart-form-data-parameters-in-a-servlet
> Also, we probably need a custom MultiPartInputStreamParser implementation.
> Because Jetty's default implementation writes input data to temporary
> directory on file system, instead, we'd like NiFi to write those into output
> FlowFiles content in streaming fashion.
> And we need request size validation checks, threshold for those validation
> should be passed via javax.servlet.MultipartConfigElement.
> Finally, we have to do something with HandleHttpResponse processor.
> Once HandleHttpRequest processor start splitting incoming request into
> multiple output FlowFiles, we need to wait for every fragment to be
> processed, then execute HandleHttpRequest.
> I think Wait/Notify processors (available from next version) will be helpful
> here.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-3469) Add multipart request support to HandleHttpRequest Processor
[
https://issues.apache.org/jira/browse/NIFI-3469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650499#comment-16650499
]
ASF GitHub Bot commented on NIFI-3469:
--
Github user markap14 commented on the issue:
https://github.com/apache/nifi/pull/2991
@ekovacs thanks for the improvement! It appears that Koji is happy with
everything now, and a quick code review looks good to me as well. I was able to
test locally and all works as expected. +1 merged to master. Thanks!
> Add multipart request support to HandleHttpRequest Processor
>
>
> Key: NIFI-3469
> URL: https://issues.apache.org/jira/browse/NIFI-3469
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: Koji Kawamura
>Assignee: Endre Kovacs
>Priority: Major
>
> Currently, HandleHttpRequest outputs a single FlowFile containing all
> multipart values as following:
> {code}
> --ef07e8bf36c274d3
> Content-Disposition: form-data; name="p1"
> v1
> --ef07e8bf36c274d3
> Content-Disposition: form-data; name="p2"
> v2
> --ef07e8bf36c274d3--
> {code}
> Many users requested adding upload files support to NiFi.
> In order for HandleHttpRequest to support multipart data we need to add
> followings (this is based on a brief researching and can be more complex or
> simple):
> We need to use HttpServletRequest#getParts() as written in this stackoverflow
> thread:
> http://stackoverflow.com/questions/3337056/convenient-way-to-parse-incoming-multipart-form-data-parameters-in-a-servlet
> Also, we probably need a custom MultiPartInputStreamParser implementation.
> Because Jetty's default implementation writes input data to temporary
> directory on file system, instead, we'd like NiFi to write those into output
> FlowFiles content in streaming fashion.
> And we need request size validation checks, threshold for those validation
> should be passed via javax.servlet.MultipartConfigElement.
> Finally, we have to do something with HandleHttpResponse processor.
> Once HandleHttpRequest processor start splitting incoming request into
> multiple output FlowFiles, we need to wait for every fragment to be
> processed, then execute HandleHttpRequest.
> I think Wait/Notify processors (available from next version) will be helpful
> here.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi issue #2991: NIFI-3469: multipart request support added to HandleHttpRe...
Github user markap14 commented on the issue: https://github.com/apache/nifi/pull/2991 @ekovacs thanks for the improvement! It appears that Koji is happy with everything now, and a quick code review looks good to me as well. I was able to test locally and all works as expected. +1 merged to master. Thanks! ---
[jira] [Created] (NIFI-5702) FlowFileRepo should not discard data (at least not by default)
Brandon DeVries created NIFI-5702: - Summary: FlowFileRepo should not discard data (at least not by default) Key: NIFI-5702 URL: https://issues.apache.org/jira/browse/NIFI-5702 Project: Apache NiFi Issue Type: Improvement Reporter: Brandon DeVries The WriteAheadFlowFileRepository currently discards data it cannot find a queue for. Unfortunately, we have run in to issues where, when rejoining a node to a cluster, the flow.xml.gz can go "missing". This results in the instance creating a new, empty, flow.xml.gz and then continuing on... and not finding queues for any of its existing data, dropping it all. Regardless of the circumstances leading to an empty (or unexpectedly modified) flow.xml.gz, dropping data without user input seems less than ideal. Internally, my group has added a property "remove.orphaned.flowfiles.on.startup", defaulting to "false". On startup, rather than silently dropping data, the repo will throw an exception preventing startup. The operator can then choose to either "fix" any unexpected issues with the flow.xml.gz, or they can set the above property to "true" which restores the original behavior allowing the system to be restarted. When set to "true" this property also results in a warning message indicating that in this configuration the repo can drop data without (advance) warning. [1] https://github.com/apache/nifi/blob/support/nifi-1.7.x/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/WriteAheadFlowFileRepository.java#L596 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (NIFI-5701) Add documentation for Load Balancing connection UI
[ https://issues.apache.org/jira/browse/NIFI-5701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Lim reassigned NIFI-5701: Assignee: Andrew Lim > Add documentation for Load Balancing connection UI > -- > > Key: NIFI-5701 > URL: https://issues.apache.org/jira/browse/NIFI-5701 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Andrew Lim >Assignee: Andrew Lim >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5701) Add documentation for Load Balancing connection UI
Andrew Lim created NIFI-5701: Summary: Add documentation for Load Balancing connection UI Key: NIFI-5701 URL: https://issues.apache.org/jira/browse/NIFI-5701 Project: Apache NiFi Issue Type: Improvement Reporter: Andrew Lim -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5700) limit the size of nifi's flowfile repository
Brandon DeVries created NIFI-5700: - Summary: limit the size of nifi's flowfile repository Key: NIFI-5700 URL: https://issues.apache.org/jira/browse/NIFI-5700 Project: Apache NiFi Issue Type: Improvement Components: Core Framework Affects Versions: 1.7.1 Reporter: Brandon DeVries Creating this ticket so as not to lose the discussion from the dev list [1]. Internally, our team has been working with the idea of setting FlowFile count and heap usage limits, resulting in first stalls and then stops of writes to the FlowFile repository. Yes, stopping writes to the FlowFile repository isn't good... however, "overfilling" the FlowFile repository can potentially be bad to the point of being unrecoverable. Putting this safeguard in place may mean requiring manual intervention to get data moving again, but it should prevent the system from failing entirely. [1] [http://apache-nifi-developer-list.39713.n7.nabble.com/Is-there-a-configuration-to-limit-the-size-of-nifi-s-flowfile-repository-td18554.html] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5585) Prepare Nodes to be Offloaded
[
https://issues.apache.org/jira/browse/NIFI-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Storck updated NIFI-5585:
--
Resolution: Done
Status: Resolved (was: Patch Available)
> Prepare Nodes to be Offloaded
> -
>
> Key: NIFI-5585
> URL: https://issues.apache.org/jira/browse/NIFI-5585
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.7.1
>Reporter: Jeff Storck
>Assignee: Jeff Storck
>Priority: Major
>
> Allow a node in the cluster to be offloaded, rebalancing flowfiles on that
> node to the other active nodes.
> Only nodes that are DISCONNECTED can be transitioned to the OFFLOADING state.
> OFFLOADING nodes will transition to the OFFLOADED state once all flowfiles
> have been rebalanced to other connected nodes in the cluster.
> OFFLOADING nodes that remain in the OFFLOADING state (due to errors
> encountered while offloading) can be reconnected to the cluster by restarting
> NiFi on the node.
> OFFLOADED nodes can be reconnected to the cluster by issuing a connection
> request via the UI/CLI, or restarting NiFi on the node.
> OFFLOADED nodes can be deleted from the cluster.
> Once a node is OFFLOADED, it will behave like a DISCONNECTED node.
> OFFLOADING a node:
> * stops all processors
> * terminates all processors
> * stops transmitting on all remote process groups
> * rebalances flowfiles to other connected nodes in the cluster (via the work
> done in NIFI-5516)
> The steps to decommission a node and remove it from the cluster are:
> # Send request to disconnect the node
> # Once disconnect completes, send request to offload the node.
> # Once offload completes, send request to delete node.
> # Once the delete request has finished, the NiFi service on the host can be
> stopped/removed.
> When an error occurs and the node can not complete offloading, the user can:
> # Send request to delete the node from the cluster
> # Diagnose why the node had issues with the offload (out of memory, no
> network connection, etc) and address the issue
> # Restart NiFi on the node to so that it will reconnect to the cluster
> # Go through the steps to decommission a node
> The OFFLOADING request is idempotent.
> Toolkit CLI commands for retrieving a single node, list of nodes, and
> connecting/disconnecting/offloading/deleting nodes have been added.
> The cluster table UI has an icon to initiate the OFFLOADING of a DISCONNECTED
> node.
> Similar to the client sending PUT request with a DISCONNECTING message to
> cluster/nodes/\{id}, an OFFLOADING message can be sent as a PUT request to
> the same URI to initiate an OFFLOAD for a DISCONNECTED node.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #2861: NIFI-5248 Added new Elasticsearch json and record p...
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2861#discussion_r225216435
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchJson.java
---
@@ -0,0 +1,330 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.jayway.jsonpath.JsonPath;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.ElasticSearchError;
+import org.apache.nifi.elasticsearch.IndexOperationRequest;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import
org.apache.nifi.processors.elasticsearch.put.FlowFileJsonDescription;
+import org.apache.nifi.processors.elasticsearch.put.JsonProcessingError;
+import org.apache.nifi.util.StringUtils;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@Tags({"json", "elasticsearch", "elasticsearch5", "elasticsearch6", "put",
"index"})
+@CapabilityDescription("This processor puts user-supplied JSON into
ElasticSearch. It does not require a schema.")
+public class PutElasticsearchJson extends AbstractProcessor implements
ElasticSearchRestProcessor {
+
+static final PropertyDescriptor ID_ATTRIBUTE = new
PropertyDescriptor.Builder()
+.name("put-es-json-id-attribute")
+.displayName("ID Attribute")
+.description("The attribute to use for setting the document ID in
ElasticSearch. If the payload is an array, " +
+"and this option is used for getting the ID, an exception
will be raised.")
+.required(false)
+
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+.addValidator(Validator.VALID)
+.build();
+static final PropertyDescriptor ID_JSON_PATH = new
PropertyDescriptor.Builder()
+.name("put-es-json-id-json-path")
+.displayName("ID JSONPath")
+.description("If set, the document ID will be pulled from each
json block using this JSONPath operation.")
--- End diff --
Nitpick, "json" should be "JSON" everywhere in the doc for consistency
---
[GitHub] nifi pull request #2861: NIFI-5248 Added new Elasticsearch json and record p...
Github user mattyb149 commented on a diff in the pull request: https://github.com/apache/nifi/pull/2861#discussion_r225215375 --- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/pom.xml --- @@ -107,6 +146,38 @@ language governing permissions and limitations under the License. --> + +org.apache.maven.plugins +maven-compiler-plugin --- End diff -- I think we talked about this before but can't find it. Does it need to be here? I thought we can inherit the one from the top-level parent POM? ---
[jira] [Commented] (NIFI-5248) Create new put processors that use the ElasticSearch client service
[
https://issues.apache.org/jira/browse/NIFI-5248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650384#comment-16650384
]
ASF GitHub Bot commented on NIFI-5248:
--
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2861#discussion_r225216435
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchJson.java
---
@@ -0,0 +1,330 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.jayway.jsonpath.JsonPath;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.ElasticSearchError;
+import org.apache.nifi.elasticsearch.IndexOperationRequest;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import
org.apache.nifi.processors.elasticsearch.put.FlowFileJsonDescription;
+import org.apache.nifi.processors.elasticsearch.put.JsonProcessingError;
+import org.apache.nifi.util.StringUtils;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@Tags({"json", "elasticsearch", "elasticsearch5", "elasticsearch6", "put",
"index"})
+@CapabilityDescription("This processor puts user-supplied JSON into
ElasticSearch. It does not require a schema.")
+public class PutElasticsearchJson extends AbstractProcessor implements
ElasticSearchRestProcessor {
+
+static final PropertyDescriptor ID_ATTRIBUTE = new
PropertyDescriptor.Builder()
+.name("put-es-json-id-attribute")
+.displayName("ID Attribute")
+.description("The attribute to use for setting the document ID in
ElasticSearch. If the payload is an array, " +
+"and this option is used for getting the ID, an exception
will be raised.")
+.required(false)
+
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+.addValidator(Validator.VALID)
+.build();
+static final PropertyDescriptor ID_JSON_PATH = new
PropertyDescriptor.Builder()
+.name("put-es-json-id-json-path")
+.displayName("ID JSONPath")
+.description("If set, the document ID will be pulled from each
json block using this JSONPath operation.")
--- End diff --
Nitpick, "json" should be "JSON" everywhere in the doc for consistency
> Create new put processors that use the ElasticSearch client service
> ---
>
> Key: NIFI-5248
> URL:
[jira] [Commented] (NIFI-5248) Create new put processors that use the ElasticSearch client service
[
https://issues.apache.org/jira/browse/NIFI-5248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650383#comment-16650383
]
ASF GitHub Bot commented on NIFI-5248:
--
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2861#discussion_r225208197
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchRecord.java
---
@@ -0,0 +1,264 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.ElasticSearchError;
+import org.apache.nifi.elasticsearch.IndexOperationRequest;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.record.path.validation.RecordPathValidator;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.nifi.serialization.record.util.DataTypeUtils;
+
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@Tags({"json", "elasticsearch", "elasticsearch5", "elasticsearch6", "put",
"index", "record"})
+@CapabilityDescription("A record-aware ElasticSearch put processor that
uses the official Elastic REST client libraries.")
+public class PutElasticsearchRecord extends AbstractProcessor implements
ElasticSearchRestProcessor {
+static final PropertyDescriptor RECORD_READER = new
PropertyDescriptor.Builder()
+.name("put-es-record-reader")
+.displayName("Record Reader")
+.description("The record reader to use for reading incoming
records from flowfiles.")
+.identifiesControllerService(RecordReaderFactory.class)
+.required(true)
+.build();
+
+static final PropertyDescriptor OPERATION_RECORD_PATH = new
PropertyDescriptor.Builder()
--- End diff --
If this is specified, does the field remain in the document? What if the
user didn't want to have that field included in the document, but needed it in
order to specify the operation as a field?
What if the index operation were in an attribute versus a record field?
Like if you partitioned records based on whether they were upsert vs delete
operations (for example). Perhaps consider something like UpdateRecord's
"Replacement Value Strategy" property, which allows you to choose whether the
Operation field would be evaluated as a Record Path or a Literal Value. Both
can support EL, but the latter would allow the setting of a single operation
for all the records in the flow file.
> Create new put processors that use the
[jira] [Commented] (NIFI-5248) Create new put processors that use the ElasticSearch client service
[
https://issues.apache.org/jira/browse/NIFI-5248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650381#comment-16650381
]
ASF GitHub Bot commented on NIFI-5248:
--
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2861#discussion_r225215710
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/ElasticSearchRestProcessor.java
---
@@ -75,6 +76,19 @@
.required(true)
.build();
+Relationship REL_FAILURE = new Relationship.Builder()
+.name("failure")
+.description("All flowfiles that fail for reasons unrelated to
server availability go to this relationship.")
+.build();
+Relationship REL_RETRY = new Relationship.Builder()
+.name("retry")
+.description("All flowfiles that fail due to server/cluster
availability go to this relationship.")
+.build();
+Relationship REL_SUCCESS = new Relationship.Builder()
+.name("success")
+.description("All flowfiles that succeed in being indexed into
ElasticSearch go here.")
--- End diff --
Maybe "transmitted to ElasticSearch"? Unless "indexed" is a generic enough
term that users would know covers upserts, deletes, and any other supported
operation.
> Create new put processors that use the ElasticSearch client service
> ---
>
> Key: NIFI-5248
> URL: https://issues.apache.org/jira/browse/NIFI-5248
> Project: Apache NiFi
> Issue Type: New Feature
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Major
>
> Two new processors:
> * PutElasticsearchJson - put raw JSON.
> * PutElasticsearchRecord - put records.
> Both of them should support the general bulk load API and be able to do
> things like insert into multiple indexes from one payload.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5248) Create new put processors that use the ElasticSearch client service
[
https://issues.apache.org/jira/browse/NIFI-5248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650382#comment-16650382
]
ASF GitHub Bot commented on NIFI-5248:
--
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2861#discussion_r225208562
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchRecord.java
---
@@ -0,0 +1,264 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.ElasticSearchError;
+import org.apache.nifi.elasticsearch.IndexOperationRequest;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.record.path.validation.RecordPathValidator;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.nifi.serialization.record.util.DataTypeUtils;
+
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@Tags({"json", "elasticsearch", "elasticsearch5", "elasticsearch6", "put",
"index", "record"})
+@CapabilityDescription("A record-aware ElasticSearch put processor that
uses the official Elastic REST client libraries.")
+public class PutElasticsearchRecord extends AbstractProcessor implements
ElasticSearchRestProcessor {
+static final PropertyDescriptor RECORD_READER = new
PropertyDescriptor.Builder()
+.name("put-es-record-reader")
+.displayName("Record Reader")
+.description("The record reader to use for reading incoming
records from flowfiles.")
+.identifiesControllerService(RecordReaderFactory.class)
+.required(true)
+.build();
+
+static final PropertyDescriptor OPERATION_RECORD_PATH = new
PropertyDescriptor.Builder()
+.name("put-es-record-operation-path")
+.displayName("Operation Record Path")
+.description("A record path expression to retrieve index operation
setting from each record. If left blank, " +
+"all operations will be assumed to be index operations.")
+.addValidator(new RecordPathValidator())
+.required(false)
+
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+.build();
+
+static final PropertyDescriptor ID_RECORD_PATH = new
PropertyDescriptor.Builder()
+.name("put-es-record-id-path")
+.displayName("ID Record Path")
+.description("A record path expression to retrieve the ID field
for use with
[GitHub] nifi pull request #2861: NIFI-5248 Added new Elasticsearch json and record p...
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2861#discussion_r225215710
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/ElasticSearchRestProcessor.java
---
@@ -75,6 +76,19 @@
.required(true)
.build();
+Relationship REL_FAILURE = new Relationship.Builder()
+.name("failure")
+.description("All flowfiles that fail for reasons unrelated to
server availability go to this relationship.")
+.build();
+Relationship REL_RETRY = new Relationship.Builder()
+.name("retry")
+.description("All flowfiles that fail due to server/cluster
availability go to this relationship.")
+.build();
+Relationship REL_SUCCESS = new Relationship.Builder()
+.name("success")
+.description("All flowfiles that succeed in being indexed into
ElasticSearch go here.")
--- End diff --
Maybe "transmitted to ElasticSearch"? Unless "indexed" is a generic enough
term that users would know covers upserts, deletes, and any other supported
operation.
---
[GitHub] nifi pull request #2861: NIFI-5248 Added new Elasticsearch json and record p...
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2861#discussion_r225208197
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchRecord.java
---
@@ -0,0 +1,264 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.ElasticSearchError;
+import org.apache.nifi.elasticsearch.IndexOperationRequest;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.record.path.validation.RecordPathValidator;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.nifi.serialization.record.util.DataTypeUtils;
+
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@Tags({"json", "elasticsearch", "elasticsearch5", "elasticsearch6", "put",
"index", "record"})
+@CapabilityDescription("A record-aware ElasticSearch put processor that
uses the official Elastic REST client libraries.")
+public class PutElasticsearchRecord extends AbstractProcessor implements
ElasticSearchRestProcessor {
+static final PropertyDescriptor RECORD_READER = new
PropertyDescriptor.Builder()
+.name("put-es-record-reader")
+.displayName("Record Reader")
+.description("The record reader to use for reading incoming
records from flowfiles.")
+.identifiesControllerService(RecordReaderFactory.class)
+.required(true)
+.build();
+
+static final PropertyDescriptor OPERATION_RECORD_PATH = new
PropertyDescriptor.Builder()
--- End diff --
If this is specified, does the field remain in the document? What if the
user didn't want to have that field included in the document, but needed it in
order to specify the operation as a field?
What if the index operation were in an attribute versus a record field?
Like if you partitioned records based on whether they were upsert vs delete
operations (for example). Perhaps consider something like UpdateRecord's
"Replacement Value Strategy" property, which allows you to choose whether the
Operation field would be evaluated as a Record Path or a Literal Value. Both
can support EL, but the latter would allow the setting of a single operation
for all the records in the flow file.
---
[jira] [Commented] (NIFI-5248) Create new put processors that use the ElasticSearch client service
[ https://issues.apache.org/jira/browse/NIFI-5248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650380#comment-16650380 ] ASF GitHub Bot commented on NIFI-5248: -- Github user mattyb149 commented on a diff in the pull request: https://github.com/apache/nifi/pull/2861#discussion_r225215375 --- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/pom.xml --- @@ -107,6 +146,38 @@ language governing permissions and limitations under the License. --> + +org.apache.maven.plugins +maven-compiler-plugin --- End diff -- I think we talked about this before but can't find it. Does it need to be here? I thought we can inherit the one from the top-level parent POM? > Create new put processors that use the ElasticSearch client service > --- > > Key: NIFI-5248 > URL: https://issues.apache.org/jira/browse/NIFI-5248 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Mike Thomsen >Assignee: Mike Thomsen >Priority: Major > > Two new processors: > * PutElasticsearchJson - put raw JSON. > * PutElasticsearchRecord - put records. > Both of them should support the general bulk load API and be able to do > things like insert into multiple indexes from one payload. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #2861: NIFI-5248 Added new Elasticsearch json and record p...
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2861#discussion_r225208562
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchRecord.java
---
@@ -0,0 +1,264 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.ElasticSearchError;
+import org.apache.nifi.elasticsearch.IndexOperationRequest;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.record.path.FieldValue;
+import org.apache.nifi.record.path.RecordPath;
+import org.apache.nifi.record.path.RecordPathResult;
+import org.apache.nifi.record.path.util.RecordPathCache;
+import org.apache.nifi.record.path.validation.RecordPathValidator;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.nifi.serialization.record.util.DataTypeUtils;
+
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@Tags({"json", "elasticsearch", "elasticsearch5", "elasticsearch6", "put",
"index", "record"})
+@CapabilityDescription("A record-aware ElasticSearch put processor that
uses the official Elastic REST client libraries.")
+public class PutElasticsearchRecord extends AbstractProcessor implements
ElasticSearchRestProcessor {
+static final PropertyDescriptor RECORD_READER = new
PropertyDescriptor.Builder()
+.name("put-es-record-reader")
+.displayName("Record Reader")
+.description("The record reader to use for reading incoming
records from flowfiles.")
+.identifiesControllerService(RecordReaderFactory.class)
+.required(true)
+.build();
+
+static final PropertyDescriptor OPERATION_RECORD_PATH = new
PropertyDescriptor.Builder()
+.name("put-es-record-operation-path")
+.displayName("Operation Record Path")
+.description("A record path expression to retrieve index operation
setting from each record. If left blank, " +
+"all operations will be assumed to be index operations.")
+.addValidator(new RecordPathValidator())
+.required(false)
+
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+.build();
+
+static final PropertyDescriptor ID_RECORD_PATH = new
PropertyDescriptor.Builder()
+.name("put-es-record-id-path")
+.displayName("ID Record Path")
+.description("A record path expression to retrieve the ID field
for use with ElasticSearch. If left blank " +
+"the ID will be automatically generated by ElasticSearch.")
+.addValidator(new RecordPathValidator())
+
[jira] [Updated] (NIFI-5248) Create new put processors that use the ElasticSearch client service
[ https://issues.apache.org/jira/browse/NIFI-5248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-5248: --- Status: Patch Available (was: Open) > Create new put processors that use the ElasticSearch client service > --- > > Key: NIFI-5248 > URL: https://issues.apache.org/jira/browse/NIFI-5248 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Mike Thomsen >Assignee: Mike Thomsen >Priority: Major > > Two new processors: > * PutElasticsearchJson - put raw JSON. > * PutElasticsearchRecord - put records. > Both of them should support the general bulk load API and be able to do > things like insert into multiple indexes from one payload. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3056: NIFI-5659 Add documentation for Offloading Nodes fu...
Github user andrewmlim commented on a diff in the pull request: https://github.com/apache/nifi/pull/3056#discussion_r225181107 --- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc --- @@ -3939,8 +3976,7 @@ to the cluster. It provides an additional layer of security. This value is blank |`nifi.cluster.flow.election.max.candidates`|Specifies the number of Nodes required in the cluster to cause early election of Flows. This allows the Nodes in the cluster to avoid having to wait a long time before starting processing if we reach at least this number of nodes in the cluster. |`nifi.cluster.load.balance.port`|Specifies the port to listen on for incoming connections for load balancing data across the cluster. The default value is `6342`. --- End diff -- Just noting that the default value is 6342 not 6432. ---
[jira] [Commented] (NIFI-5659) Add documentation for Offloading Nodes functionality and new Node related CLI commands
[ https://issues.apache.org/jira/browse/NIFI-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650270#comment-16650270 ] ASF GitHub Bot commented on NIFI-5659: -- Github user andrewmlim commented on a diff in the pull request: https://github.com/apache/nifi/pull/3056#discussion_r225181107 --- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc --- @@ -3939,8 +3976,7 @@ to the cluster. It provides an additional layer of security. This value is blank |`nifi.cluster.flow.election.max.candidates`|Specifies the number of Nodes required in the cluster to cause early election of Flows. This allows the Nodes in the cluster to avoid having to wait a long time before starting processing if we reach at least this number of nodes in the cluster. |`nifi.cluster.load.balance.port`|Specifies the port to listen on for incoming connections for load balancing data across the cluster. The default value is `6342`. --- End diff -- Just noting that the default value is 6342 not 6432. > Add documentation for Offloading Nodes functionality and new Node related CLI > commands > -- > > Key: NIFI-5659 > URL: https://issues.apache.org/jira/browse/NIFI-5659 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation Website >Reporter: Andrew Lim >Assignee: Andrew Lim >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5667) Hive3 PutOrc processors, error when using nestled Avro Record types
[
https://issues.apache.org/jira/browse/NIFI-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bryan Bende updated NIFI-5667:
--
Resolution: Fixed
Fix Version/s: 1.8.0
Status: Resolved (was: Patch Available)
> Hive3 PutOrc processors, error when using nestled Avro Record types
> ---
>
> Key: NIFI-5667
> URL: https://issues.apache.org/jira/browse/NIFI-5667
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Environment: Centos 7 and Docker Image from Hortonworks
>Reporter: Viking Karstorp
>Assignee: Matt Burgess
>Priority: Major
> Fix For: 1.8.0
>
>
> I have been testing out the new PutOrc processor that was introduced in 1.7
> to see if I can replace the ConvertAvroToOrc processer I currently use.
> When I sent in some of the complex Avro messages in my flow I encountered the
> following error (see full stack further down)
> java.lang.IllegalArgumentException: Error converting object of type
> org.apache.nifi.serialization.record.MapRecord to ORC type The older
> ConvertAvroToOrc processor processed the flowfile without issues. Also to
> note is that the PutOrc processor handles the flowfile fine if there is no
> Avro data with only the schema present. It seems to be related to nestled
> "Record" types.
> How to reproduce:
> Avro schema: bug.avsc
> {code}
> {
> "name": "nifi_hive3_test",
> "namespace": "analytics.models.test",
> "type": "record",
> "fields": [
>{
> "name": "Serial",
> "type":
> {
> "name": "Serial",
> "namespace": "analytics.models.common.serial",
> "type": "record",
> "fields": [
> {
> "name": "Serial",
> "type": "long"
> }
> ]
> }
> }
> ]
> }
> {code}
> Small python script to create an Avro file.
> {code}
> import avro.schema
> from avro.datafile import DataFileReader, DataFileWriter
> from avro.io import DatumReader, DatumWriter
> schema = avro.schema.parse(open("bug.avsc", "rb").read())
> writer = DataFileWriter(open("bug.avro", "wb"), DatumWriter(), schema)
> writer.append({'Serial': {'Serial': 110881615L}})
> writer.close()
> #Print whats entered into the avro file
> reader1 = DataFileReader(open("bug.avro", "rb"), DatumReader())
> for user in reader1:
> print user
> {code}
> Then just load the avro file using ListFIle -> FetchFile
> Full error message:
> {code}
> 2018-10-06 15:54:10,201 ERROR [Timer-Driven Process Thread-8]
> org.apache.nifi.processors.orc.PutORC
> PutORC[id=8be207cb-b16e-3578-1765-1c9e0c0aa383] Failed to write due to
> java.lang.IllegalArgumentException: Error converting object of type
> org.apache.nifi.serialization.record.MapRecord to ORC type
> struct: java.lang.IllegalArgumentException: Error converting
> object of type org.apache.nifi.serialization.record.MapRecord to ORC type
> struct
> java.lang.IllegalArgumentException: Error converting object of type
> org.apache.nifi.serialization.record.MapRecord to ORC type
> struct
> at
> org.apache.hadoop.hive.ql.io.orc.NiFiOrcUtils.convertToORCObject(NiFiOrcUtils.java:206)
> at
> org.apache.nifi.processors.orc.record.ORCHDFSRecordWriter.write(ORCHDFSRecordWriter.java:71)
> at
> org.apache.nifi.processors.orc.record.ORCHDFSRecordWriter.write(ORCHDFSRecordWriter.java:91)
> at
> org.apache.nifi.processors.hadoop.AbstractPutHDFSRecord.lambda$null$0(AbstractPutHDFSRecord.java:324)
> at
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2218)
> at
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2186)
> at
> org.apache.nifi.processors.hadoop.AbstractPutHDFSRecord.lambda$onTrigger$1(AbstractPutHDFSRecord.java:305)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:360)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1662)
> at
> org.apache.nifi.processors.hadoop.AbstractPutHDFSRecord.onTrigger(AbstractPutHDFSRecord.java:272)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
>
[jira] [Commented] (NIFI-5667) Hive3 PutOrc processors, error when using nestled Avro Record types
[
https://issues.apache.org/jira/browse/NIFI-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650268#comment-16650268
]
ASF GitHub Bot commented on NIFI-5667:
--
Github user asfgit closed the pull request at:
https://github.com/apache/nifi/pull/3057
> Hive3 PutOrc processors, error when using nestled Avro Record types
> ---
>
> Key: NIFI-5667
> URL: https://issues.apache.org/jira/browse/NIFI-5667
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Environment: Centos 7 and Docker Image from Hortonworks
>Reporter: Viking Karstorp
>Assignee: Matt Burgess
>Priority: Major
> Fix For: 1.8.0
>
>
> I have been testing out the new PutOrc processor that was introduced in 1.7
> to see if I can replace the ConvertAvroToOrc processer I currently use.
> When I sent in some of the complex Avro messages in my flow I encountered the
> following error (see full stack further down)
> java.lang.IllegalArgumentException: Error converting object of type
> org.apache.nifi.serialization.record.MapRecord to ORC type The older
> ConvertAvroToOrc processor processed the flowfile without issues. Also to
> note is that the PutOrc processor handles the flowfile fine if there is no
> Avro data with only the schema present. It seems to be related to nestled
> "Record" types.
> How to reproduce:
> Avro schema: bug.avsc
> {code}
> {
> "name": "nifi_hive3_test",
> "namespace": "analytics.models.test",
> "type": "record",
> "fields": [
>{
> "name": "Serial",
> "type":
> {
> "name": "Serial",
> "namespace": "analytics.models.common.serial",
> "type": "record",
> "fields": [
> {
> "name": "Serial",
> "type": "long"
> }
> ]
> }
> }
> ]
> }
> {code}
> Small python script to create an Avro file.
> {code}
> import avro.schema
> from avro.datafile import DataFileReader, DataFileWriter
> from avro.io import DatumReader, DatumWriter
> schema = avro.schema.parse(open("bug.avsc", "rb").read())
> writer = DataFileWriter(open("bug.avro", "wb"), DatumWriter(), schema)
> writer.append({'Serial': {'Serial': 110881615L}})
> writer.close()
> #Print whats entered into the avro file
> reader1 = DataFileReader(open("bug.avro", "rb"), DatumReader())
> for user in reader1:
> print user
> {code}
> Then just load the avro file using ListFIle -> FetchFile
> Full error message:
> {code}
> 2018-10-06 15:54:10,201 ERROR [Timer-Driven Process Thread-8]
> org.apache.nifi.processors.orc.PutORC
> PutORC[id=8be207cb-b16e-3578-1765-1c9e0c0aa383] Failed to write due to
> java.lang.IllegalArgumentException: Error converting object of type
> org.apache.nifi.serialization.record.MapRecord to ORC type
> struct: java.lang.IllegalArgumentException: Error converting
> object of type org.apache.nifi.serialization.record.MapRecord to ORC type
> struct
> java.lang.IllegalArgumentException: Error converting object of type
> org.apache.nifi.serialization.record.MapRecord to ORC type
> struct
> at
> org.apache.hadoop.hive.ql.io.orc.NiFiOrcUtils.convertToORCObject(NiFiOrcUtils.java:206)
> at
> org.apache.nifi.processors.orc.record.ORCHDFSRecordWriter.write(ORCHDFSRecordWriter.java:71)
> at
> org.apache.nifi.processors.orc.record.ORCHDFSRecordWriter.write(ORCHDFSRecordWriter.java:91)
> at
> org.apache.nifi.processors.hadoop.AbstractPutHDFSRecord.lambda$null$0(AbstractPutHDFSRecord.java:324)
> at
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2218)
> at
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2186)
> at
> org.apache.nifi.processors.hadoop.AbstractPutHDFSRecord.lambda$onTrigger$1(AbstractPutHDFSRecord.java:305)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:360)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1662)
> at
> org.apache.nifi.processors.hadoop.AbstractPutHDFSRecord.onTrigger(AbstractPutHDFSRecord.java:272)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
>
[jira] [Commented] (NIFI-5667) Hive3 PutOrc processors, error when using nestled Avro Record types
[
https://issues.apache.org/jira/browse/NIFI-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650266#comment-16650266
]
ASF subversion and git services commented on NIFI-5667:
---
Commit ce25ae54196a318cbfcdb4dfe178607f4ac135c6 in nifi's branch
refs/heads/master from [~ca9mbu]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=ce25ae5 ]
NIFI-5667: Add nested record support for PutORC
NIFI-5667: Fixed default table name
NIFI-5667: Fixed handling of binary types
NIFI-5667: Added backticks in Hive DDL generation
This closes #3057.
Signed-off-by: Bryan Bende
> Hive3 PutOrc processors, error when using nestled Avro Record types
> ---
>
> Key: NIFI-5667
> URL: https://issues.apache.org/jira/browse/NIFI-5667
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Environment: Centos 7 and Docker Image from Hortonworks
>Reporter: Viking Karstorp
>Assignee: Matt Burgess
>Priority: Major
>
> I have been testing out the new PutOrc processor that was introduced in 1.7
> to see if I can replace the ConvertAvroToOrc processer I currently use.
> When I sent in some of the complex Avro messages in my flow I encountered the
> following error (see full stack further down)
> java.lang.IllegalArgumentException: Error converting object of type
> org.apache.nifi.serialization.record.MapRecord to ORC type The older
> ConvertAvroToOrc processor processed the flowfile without issues. Also to
> note is that the PutOrc processor handles the flowfile fine if there is no
> Avro data with only the schema present. It seems to be related to nestled
> "Record" types.
> How to reproduce:
> Avro schema: bug.avsc
> {code}
> {
> "name": "nifi_hive3_test",
> "namespace": "analytics.models.test",
> "type": "record",
> "fields": [
>{
> "name": "Serial",
> "type":
> {
> "name": "Serial",
> "namespace": "analytics.models.common.serial",
> "type": "record",
> "fields": [
> {
> "name": "Serial",
> "type": "long"
> }
> ]
> }
> }
> ]
> }
> {code}
> Small python script to create an Avro file.
> {code}
> import avro.schema
> from avro.datafile import DataFileReader, DataFileWriter
> from avro.io import DatumReader, DatumWriter
> schema = avro.schema.parse(open("bug.avsc", "rb").read())
> writer = DataFileWriter(open("bug.avro", "wb"), DatumWriter(), schema)
> writer.append({'Serial': {'Serial': 110881615L}})
> writer.close()
> #Print whats entered into the avro file
> reader1 = DataFileReader(open("bug.avro", "rb"), DatumReader())
> for user in reader1:
> print user
> {code}
> Then just load the avro file using ListFIle -> FetchFile
> Full error message:
> {code}
> 2018-10-06 15:54:10,201 ERROR [Timer-Driven Process Thread-8]
> org.apache.nifi.processors.orc.PutORC
> PutORC[id=8be207cb-b16e-3578-1765-1c9e0c0aa383] Failed to write due to
> java.lang.IllegalArgumentException: Error converting object of type
> org.apache.nifi.serialization.record.MapRecord to ORC type
> struct: java.lang.IllegalArgumentException: Error converting
> object of type org.apache.nifi.serialization.record.MapRecord to ORC type
> struct
> java.lang.IllegalArgumentException: Error converting object of type
> org.apache.nifi.serialization.record.MapRecord to ORC type
> struct
> at
> org.apache.hadoop.hive.ql.io.orc.NiFiOrcUtils.convertToORCObject(NiFiOrcUtils.java:206)
> at
> org.apache.nifi.processors.orc.record.ORCHDFSRecordWriter.write(ORCHDFSRecordWriter.java:71)
> at
> org.apache.nifi.processors.orc.record.ORCHDFSRecordWriter.write(ORCHDFSRecordWriter.java:91)
> at
> org.apache.nifi.processors.hadoop.AbstractPutHDFSRecord.lambda$null$0(AbstractPutHDFSRecord.java:324)
> at
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2218)
> at
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2186)
> at
> org.apache.nifi.processors.hadoop.AbstractPutHDFSRecord.lambda$onTrigger$1(AbstractPutHDFSRecord.java:305)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:360)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1662)
> at
> org.apache.nifi.processors.hadoop.AbstractPutHDFSRecord.onTrigger(AbstractPutHDFSRecord.java:272)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
> at
>
[jira] [Commented] (NIFI-5667) Hive3 PutOrc processors, error when using nestled Avro Record types
[
https://issues.apache.org/jira/browse/NIFI-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650263#comment-16650263
]
ASF subversion and git services commented on NIFI-5667:
---
Commit ce25ae54196a318cbfcdb4dfe178607f4ac135c6 in nifi's branch
refs/heads/master from [~ca9mbu]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=ce25ae5 ]
NIFI-5667: Add nested record support for PutORC
NIFI-5667: Fixed default table name
NIFI-5667: Fixed handling of binary types
NIFI-5667: Added backticks in Hive DDL generation
This closes #3057.
Signed-off-by: Bryan Bende
> Hive3 PutOrc processors, error when using nestled Avro Record types
> ---
>
> Key: NIFI-5667
> URL: https://issues.apache.org/jira/browse/NIFI-5667
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Environment: Centos 7 and Docker Image from Hortonworks
>Reporter: Viking Karstorp
>Assignee: Matt Burgess
>Priority: Major
>
> I have been testing out the new PutOrc processor that was introduced in 1.7
> to see if I can replace the ConvertAvroToOrc processer I currently use.
> When I sent in some of the complex Avro messages in my flow I encountered the
> following error (see full stack further down)
> java.lang.IllegalArgumentException: Error converting object of type
> org.apache.nifi.serialization.record.MapRecord to ORC type The older
> ConvertAvroToOrc processor processed the flowfile without issues. Also to
> note is that the PutOrc processor handles the flowfile fine if there is no
> Avro data with only the schema present. It seems to be related to nestled
> "Record" types.
> How to reproduce:
> Avro schema: bug.avsc
> {code}
> {
> "name": "nifi_hive3_test",
> "namespace": "analytics.models.test",
> "type": "record",
> "fields": [
>{
> "name": "Serial",
> "type":
> {
> "name": "Serial",
> "namespace": "analytics.models.common.serial",
> "type": "record",
> "fields": [
> {
> "name": "Serial",
> "type": "long"
> }
> ]
> }
> }
> ]
> }
> {code}
> Small python script to create an Avro file.
> {code}
> import avro.schema
> from avro.datafile import DataFileReader, DataFileWriter
> from avro.io import DatumReader, DatumWriter
> schema = avro.schema.parse(open("bug.avsc", "rb").read())
> writer = DataFileWriter(open("bug.avro", "wb"), DatumWriter(), schema)
> writer.append({'Serial': {'Serial': 110881615L}})
> writer.close()
> #Print whats entered into the avro file
> reader1 = DataFileReader(open("bug.avro", "rb"), DatumReader())
> for user in reader1:
> print user
> {code}
> Then just load the avro file using ListFIle -> FetchFile
> Full error message:
> {code}
> 2018-10-06 15:54:10,201 ERROR [Timer-Driven Process Thread-8]
> org.apache.nifi.processors.orc.PutORC
> PutORC[id=8be207cb-b16e-3578-1765-1c9e0c0aa383] Failed to write due to
> java.lang.IllegalArgumentException: Error converting object of type
> org.apache.nifi.serialization.record.MapRecord to ORC type
> struct: java.lang.IllegalArgumentException: Error converting
> object of type org.apache.nifi.serialization.record.MapRecord to ORC type
> struct
> java.lang.IllegalArgumentException: Error converting object of type
> org.apache.nifi.serialization.record.MapRecord to ORC type
> struct
> at
> org.apache.hadoop.hive.ql.io.orc.NiFiOrcUtils.convertToORCObject(NiFiOrcUtils.java:206)
> at
> org.apache.nifi.processors.orc.record.ORCHDFSRecordWriter.write(ORCHDFSRecordWriter.java:71)
> at
> org.apache.nifi.processors.orc.record.ORCHDFSRecordWriter.write(ORCHDFSRecordWriter.java:91)
> at
> org.apache.nifi.processors.hadoop.AbstractPutHDFSRecord.lambda$null$0(AbstractPutHDFSRecord.java:324)
> at
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2218)
> at
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2186)
> at
> org.apache.nifi.processors.hadoop.AbstractPutHDFSRecord.lambda$onTrigger$1(AbstractPutHDFSRecord.java:305)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:360)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1662)
> at
> org.apache.nifi.processors.hadoop.AbstractPutHDFSRecord.onTrigger(AbstractPutHDFSRecord.java:272)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
> at
>
[jira] [Commented] (NIFI-5667) Hive3 PutOrc processors, error when using nestled Avro Record types
[
https://issues.apache.org/jira/browse/NIFI-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650264#comment-16650264
]
ASF subversion and git services commented on NIFI-5667:
---
Commit ce25ae54196a318cbfcdb4dfe178607f4ac135c6 in nifi's branch
refs/heads/master from [~ca9mbu]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=ce25ae5 ]
NIFI-5667: Add nested record support for PutORC
NIFI-5667: Fixed default table name
NIFI-5667: Fixed handling of binary types
NIFI-5667: Added backticks in Hive DDL generation
This closes #3057.
Signed-off-by: Bryan Bende
> Hive3 PutOrc processors, error when using nestled Avro Record types
> ---
>
> Key: NIFI-5667
> URL: https://issues.apache.org/jira/browse/NIFI-5667
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Environment: Centos 7 and Docker Image from Hortonworks
>Reporter: Viking Karstorp
>Assignee: Matt Burgess
>Priority: Major
>
> I have been testing out the new PutOrc processor that was introduced in 1.7
> to see if I can replace the ConvertAvroToOrc processer I currently use.
> When I sent in some of the complex Avro messages in my flow I encountered the
> following error (see full stack further down)
> java.lang.IllegalArgumentException: Error converting object of type
> org.apache.nifi.serialization.record.MapRecord to ORC type The older
> ConvertAvroToOrc processor processed the flowfile without issues. Also to
> note is that the PutOrc processor handles the flowfile fine if there is no
> Avro data with only the schema present. It seems to be related to nestled
> "Record" types.
> How to reproduce:
> Avro schema: bug.avsc
> {code}
> {
> "name": "nifi_hive3_test",
> "namespace": "analytics.models.test",
> "type": "record",
> "fields": [
>{
> "name": "Serial",
> "type":
> {
> "name": "Serial",
> "namespace": "analytics.models.common.serial",
> "type": "record",
> "fields": [
> {
> "name": "Serial",
> "type": "long"
> }
> ]
> }
> }
> ]
> }
> {code}
> Small python script to create an Avro file.
> {code}
> import avro.schema
> from avro.datafile import DataFileReader, DataFileWriter
> from avro.io import DatumReader, DatumWriter
> schema = avro.schema.parse(open("bug.avsc", "rb").read())
> writer = DataFileWriter(open("bug.avro", "wb"), DatumWriter(), schema)
> writer.append({'Serial': {'Serial': 110881615L}})
> writer.close()
> #Print whats entered into the avro file
> reader1 = DataFileReader(open("bug.avro", "rb"), DatumReader())
> for user in reader1:
> print user
> {code}
> Then just load the avro file using ListFIle -> FetchFile
> Full error message:
> {code}
> 2018-10-06 15:54:10,201 ERROR [Timer-Driven Process Thread-8]
> org.apache.nifi.processors.orc.PutORC
> PutORC[id=8be207cb-b16e-3578-1765-1c9e0c0aa383] Failed to write due to
> java.lang.IllegalArgumentException: Error converting object of type
> org.apache.nifi.serialization.record.MapRecord to ORC type
> struct: java.lang.IllegalArgumentException: Error converting
> object of type org.apache.nifi.serialization.record.MapRecord to ORC type
> struct
> java.lang.IllegalArgumentException: Error converting object of type
> org.apache.nifi.serialization.record.MapRecord to ORC type
> struct
> at
> org.apache.hadoop.hive.ql.io.orc.NiFiOrcUtils.convertToORCObject(NiFiOrcUtils.java:206)
> at
> org.apache.nifi.processors.orc.record.ORCHDFSRecordWriter.write(ORCHDFSRecordWriter.java:71)
> at
> org.apache.nifi.processors.orc.record.ORCHDFSRecordWriter.write(ORCHDFSRecordWriter.java:91)
> at
> org.apache.nifi.processors.hadoop.AbstractPutHDFSRecord.lambda$null$0(AbstractPutHDFSRecord.java:324)
> at
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2218)
> at
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2186)
> at
> org.apache.nifi.processors.hadoop.AbstractPutHDFSRecord.lambda$onTrigger$1(AbstractPutHDFSRecord.java:305)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:360)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1662)
> at
> org.apache.nifi.processors.hadoop.AbstractPutHDFSRecord.onTrigger(AbstractPutHDFSRecord.java:272)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
> at
>
[jira] [Commented] (NIFI-5667) Hive3 PutOrc processors, error when using nestled Avro Record types
[
https://issues.apache.org/jira/browse/NIFI-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650265#comment-16650265
]
ASF subversion and git services commented on NIFI-5667:
---
Commit ce25ae54196a318cbfcdb4dfe178607f4ac135c6 in nifi's branch
refs/heads/master from [~ca9mbu]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=ce25ae5 ]
NIFI-5667: Add nested record support for PutORC
NIFI-5667: Fixed default table name
NIFI-5667: Fixed handling of binary types
NIFI-5667: Added backticks in Hive DDL generation
This closes #3057.
Signed-off-by: Bryan Bende
> Hive3 PutOrc processors, error when using nestled Avro Record types
> ---
>
> Key: NIFI-5667
> URL: https://issues.apache.org/jira/browse/NIFI-5667
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Environment: Centos 7 and Docker Image from Hortonworks
>Reporter: Viking Karstorp
>Assignee: Matt Burgess
>Priority: Major
>
> I have been testing out the new PutOrc processor that was introduced in 1.7
> to see if I can replace the ConvertAvroToOrc processer I currently use.
> When I sent in some of the complex Avro messages in my flow I encountered the
> following error (see full stack further down)
> java.lang.IllegalArgumentException: Error converting object of type
> org.apache.nifi.serialization.record.MapRecord to ORC type The older
> ConvertAvroToOrc processor processed the flowfile without issues. Also to
> note is that the PutOrc processor handles the flowfile fine if there is no
> Avro data with only the schema present. It seems to be related to nestled
> "Record" types.
> How to reproduce:
> Avro schema: bug.avsc
> {code}
> {
> "name": "nifi_hive3_test",
> "namespace": "analytics.models.test",
> "type": "record",
> "fields": [
>{
> "name": "Serial",
> "type":
> {
> "name": "Serial",
> "namespace": "analytics.models.common.serial",
> "type": "record",
> "fields": [
> {
> "name": "Serial",
> "type": "long"
> }
> ]
> }
> }
> ]
> }
> {code}
> Small python script to create an Avro file.
> {code}
> import avro.schema
> from avro.datafile import DataFileReader, DataFileWriter
> from avro.io import DatumReader, DatumWriter
> schema = avro.schema.parse(open("bug.avsc", "rb").read())
> writer = DataFileWriter(open("bug.avro", "wb"), DatumWriter(), schema)
> writer.append({'Serial': {'Serial': 110881615L}})
> writer.close()
> #Print whats entered into the avro file
> reader1 = DataFileReader(open("bug.avro", "rb"), DatumReader())
> for user in reader1:
> print user
> {code}
> Then just load the avro file using ListFIle -> FetchFile
> Full error message:
> {code}
> 2018-10-06 15:54:10,201 ERROR [Timer-Driven Process Thread-8]
> org.apache.nifi.processors.orc.PutORC
> PutORC[id=8be207cb-b16e-3578-1765-1c9e0c0aa383] Failed to write due to
> java.lang.IllegalArgumentException: Error converting object of type
> org.apache.nifi.serialization.record.MapRecord to ORC type
> struct: java.lang.IllegalArgumentException: Error converting
> object of type org.apache.nifi.serialization.record.MapRecord to ORC type
> struct
> java.lang.IllegalArgumentException: Error converting object of type
> org.apache.nifi.serialization.record.MapRecord to ORC type
> struct
> at
> org.apache.hadoop.hive.ql.io.orc.NiFiOrcUtils.convertToORCObject(NiFiOrcUtils.java:206)
> at
> org.apache.nifi.processors.orc.record.ORCHDFSRecordWriter.write(ORCHDFSRecordWriter.java:71)
> at
> org.apache.nifi.processors.orc.record.ORCHDFSRecordWriter.write(ORCHDFSRecordWriter.java:91)
> at
> org.apache.nifi.processors.hadoop.AbstractPutHDFSRecord.lambda$null$0(AbstractPutHDFSRecord.java:324)
> at
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2218)
> at
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2186)
> at
> org.apache.nifi.processors.hadoop.AbstractPutHDFSRecord.lambda$onTrigger$1(AbstractPutHDFSRecord.java:305)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:360)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1662)
> at
> org.apache.nifi.processors.hadoop.AbstractPutHDFSRecord.onTrigger(AbstractPutHDFSRecord.java:272)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
> at
>
[GitHub] nifi pull request #3057: NIFI-5667: Add nested record support for PutORC
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/3057 ---
[jira] [Commented] (NIFI-5667) Hive3 PutOrc processors, error when using nestled Avro Record types
[
https://issues.apache.org/jira/browse/NIFI-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650259#comment-16650259
]
ASF GitHub Bot commented on NIFI-5667:
--
Github user bbende commented on the issue:
https://github.com/apache/nifi/pull/3057
Looks good to me, was able to verify the functionality, going to merge
> Hive3 PutOrc processors, error when using nestled Avro Record types
> ---
>
> Key: NIFI-5667
> URL: https://issues.apache.org/jira/browse/NIFI-5667
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Environment: Centos 7 and Docker Image from Hortonworks
>Reporter: Viking Karstorp
>Assignee: Matt Burgess
>Priority: Major
>
> I have been testing out the new PutOrc processor that was introduced in 1.7
> to see if I can replace the ConvertAvroToOrc processer I currently use.
> When I sent in some of the complex Avro messages in my flow I encountered the
> following error (see full stack further down)
> java.lang.IllegalArgumentException: Error converting object of type
> org.apache.nifi.serialization.record.MapRecord to ORC type The older
> ConvertAvroToOrc processor processed the flowfile without issues. Also to
> note is that the PutOrc processor handles the flowfile fine if there is no
> Avro data with only the schema present. It seems to be related to nestled
> "Record" types.
> How to reproduce:
> Avro schema: bug.avsc
> {code}
> {
> "name": "nifi_hive3_test",
> "namespace": "analytics.models.test",
> "type": "record",
> "fields": [
>{
> "name": "Serial",
> "type":
> {
> "name": "Serial",
> "namespace": "analytics.models.common.serial",
> "type": "record",
> "fields": [
> {
> "name": "Serial",
> "type": "long"
> }
> ]
> }
> }
> ]
> }
> {code}
> Small python script to create an Avro file.
> {code}
> import avro.schema
> from avro.datafile import DataFileReader, DataFileWriter
> from avro.io import DatumReader, DatumWriter
> schema = avro.schema.parse(open("bug.avsc", "rb").read())
> writer = DataFileWriter(open("bug.avro", "wb"), DatumWriter(), schema)
> writer.append({'Serial': {'Serial': 110881615L}})
> writer.close()
> #Print whats entered into the avro file
> reader1 = DataFileReader(open("bug.avro", "rb"), DatumReader())
> for user in reader1:
> print user
> {code}
> Then just load the avro file using ListFIle -> FetchFile
> Full error message:
> {code}
> 2018-10-06 15:54:10,201 ERROR [Timer-Driven Process Thread-8]
> org.apache.nifi.processors.orc.PutORC
> PutORC[id=8be207cb-b16e-3578-1765-1c9e0c0aa383] Failed to write due to
> java.lang.IllegalArgumentException: Error converting object of type
> org.apache.nifi.serialization.record.MapRecord to ORC type
> struct: java.lang.IllegalArgumentException: Error converting
> object of type org.apache.nifi.serialization.record.MapRecord to ORC type
> struct
> java.lang.IllegalArgumentException: Error converting object of type
> org.apache.nifi.serialization.record.MapRecord to ORC type
> struct
> at
> org.apache.hadoop.hive.ql.io.orc.NiFiOrcUtils.convertToORCObject(NiFiOrcUtils.java:206)
> at
> org.apache.nifi.processors.orc.record.ORCHDFSRecordWriter.write(ORCHDFSRecordWriter.java:71)
> at
> org.apache.nifi.processors.orc.record.ORCHDFSRecordWriter.write(ORCHDFSRecordWriter.java:91)
> at
> org.apache.nifi.processors.hadoop.AbstractPutHDFSRecord.lambda$null$0(AbstractPutHDFSRecord.java:324)
> at
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2218)
> at
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2186)
> at
> org.apache.nifi.processors.hadoop.AbstractPutHDFSRecord.lambda$onTrigger$1(AbstractPutHDFSRecord.java:305)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:360)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1662)
> at
> org.apache.nifi.processors.hadoop.AbstractPutHDFSRecord.onTrigger(AbstractPutHDFSRecord.java:272)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
>
[GitHub] nifi issue #3057: NIFI-5667: Add nested record support for PutORC
Github user bbende commented on the issue: https://github.com/apache/nifi/pull/3057 Looks good to me, was able to verify the functionality, going to merge ---
[jira] [Created] (MINIFICPP-641) C API: add support to register failure callback
Arpad Boda created MINIFICPP-641: Summary: C API: add support to register failure callback Key: MINIFICPP-641 URL: https://issues.apache.org/jira/browse/MINIFICPP-641 Project: NiFi MiNiFi C++ Issue Type: New Feature Reporter: Arpad Boda Assignee: Arpad Boda Fix For: 0.6.0 Current implementation of C API provide no option to handle failure relationships of processors. Simply adding relationships one-by-one would make it unreasonably complicated in user's perspective. User should be able to add a simple callback, which - in the background - creates a processor and connects that all processors in the given flow with failure relationship. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5248) Create new put processors that use the ElasticSearch client service
[ https://issues.apache.org/jira/browse/NIFI-5248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650201#comment-16650201 ] ASF GitHub Bot commented on NIFI-5248: -- Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2861 @mattyb149 anything? > Create new put processors that use the ElasticSearch client service > --- > > Key: NIFI-5248 > URL: https://issues.apache.org/jira/browse/NIFI-5248 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Mike Thomsen >Assignee: Mike Thomsen >Priority: Major > > Two new processors: > * PutElasticsearchJson - put raw JSON. > * PutElasticsearchRecord - put records. > Both of them should support the general bulk load API and be able to do > things like insert into multiple indexes from one payload. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2861: NIFI-5248 Added new Elasticsearch json and record processo...
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2861 @mattyb149 anything? ---
[jira] [Commented] (NIFI-5605) Create Cassandra update/delete processor
[ https://issues.apache.org/jira/browse/NIFI-5605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650192#comment-16650192 ] ASF GitHub Bot commented on NIFI-5605: -- Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/3025 Going to close this because I think the consensus is that it should be folded into an existing processor. > Create Cassandra update/delete processor > > > Key: NIFI-5605 > URL: https://issues.apache.org/jira/browse/NIFI-5605 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Mike Thomsen >Assignee: Mike Thomsen >Priority: Major > > There should be a processor that can update and delete from Cassandra tables. > Where QueryCassandra is geared toward reading data, this should be geared > toward data mutations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5699) Make the property descriptors accessible to the subclasses of AbstractRecordProcessor
Alex Savitsky created NIFI-5699: --- Summary: Make the property descriptors accessible to the subclasses of AbstractRecordProcessor Key: NIFI-5699 URL: https://issues.apache.org/jira/browse/NIFI-5699 Project: Apache NiFi Issue Type: Improvement Components: Core Framework Affects Versions: 1.7.1 Reporter: Alex Savitsky Currently, RECORD_READER and RECORD_WRITER are declared with default visibility, but the class is abstract. I don't think it's reasonable to expect all subclasses to reside in org.apache.nifi.processors.standard to be able to access the property descriptors (e.g. for testing). Please make them public, they're final anyway. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5605) Create Cassandra update/delete processor
[ https://issues.apache.org/jira/browse/NIFI-5605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650193#comment-16650193 ] ASF GitHub Bot commented on NIFI-5605: -- Github user MikeThomsen closed the pull request at: https://github.com/apache/nifi/pull/3025 > Create Cassandra update/delete processor > > > Key: NIFI-5605 > URL: https://issues.apache.org/jira/browse/NIFI-5605 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Mike Thomsen >Assignee: Mike Thomsen >Priority: Major > > There should be a processor that can update and delete from Cassandra tables. > Where QueryCassandra is geared toward reading data, this should be geared > toward data mutations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3025: NIFI-5605 Added UpdateCassandra.
Github user MikeThomsen closed the pull request at: https://github.com/apache/nifi/pull/3025 ---
[GitHub] nifi issue #3025: NIFI-5605 Added UpdateCassandra.
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/3025 Going to close this because I think the consensus is that it should be folded into an existing processor. ---
[jira] [Commented] (NIFI-5698) DeleteAzureBlob fails to delete
[ https://issues.apache.org/jira/browse/NIFI-5698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650093#comment-16650093 ] ASF GitHub Bot commented on NIFI-5698: -- Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/3073 > > > Confirmed that the reported issue happened before applying fix. And resolved with this fix. +1, merging. Thanks again @zenfenan! Sorry for introducing this terrible bug.. @ijokarumawak No issues. Thanks for reviewing and merging. Appreciate the quick turnaround. :) > DeleteAzureBlob fails to delete > --- > > Key: NIFI-5698 > URL: https://issues.apache.org/jira/browse/NIFI-5698 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.7.0, 1.7.1 >Reporter: Sivaprasanna Sethuraman >Assignee: Sivaprasanna Sethuraman >Priority: Blocker > Fix For: 1.8.0 > > Attachments: NIFI-5698-Fixed-DeleteAzureBlobStorage-bug.patch > > > With NIFI-4199, proxy support has been added to Azure Storage processors. > However, that made way for a critical bug to sneak into the > DeleteAzureBlobStorage processor which effectively renders the processor > unusable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #3073: NIFI-5698: Fixed DeleteAzureBlobStorage bug
Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/3073 > > > Confirmed that the reported issue happened before applying fix. And resolved with this fix. +1, merging. Thanks again @zenfenan! Sorry for introducing this terrible bug.. @ijokarumawak No issues. Thanks for reviewing and merging. Appreciate the quick turnaround. :) ---
[jira] [Updated] (NIFI-5492) UDF in Expression Language
[
https://issues.apache.org/jira/browse/NIFI-5492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ed Berezitsky updated NIFI-5492:
Issue Type: New Feature (was: Wish)
> UDF in Expression Language
> --
>
> Key: NIFI-5492
> URL: https://issues.apache.org/jira/browse/NIFI-5492
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Core Framework
>Affects Versions: 1.5.0, 1.6.0, 1.7.0, 1.7.1
>Reporter: Ed Berezitsky
>Assignee: Ed Berezitsky
>Priority: Major
> Labels: features, patch
> Attachments: 0001-NIFI-5492_EXEC-Adding-UDF-to-EL.patch
>
>
> Set of functions available to use in expression language is limited by
> predefined ones.
> This request is to provide an ability to plug in custom/user defined
> functions.
> For example:
> ${*exec*('com.example.MyUDF', 'param1', 'param2')}
> Should be able to support:
> # Multiple, not limited number of parameters (including zero params)
> # Param data types should support all EL data types (dates, whole numbers,
> decimals, strings, booleans)
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5629) GetFile becomes slow listing vast directories
[
https://issues.apache.org/jira/browse/NIFI-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650073#comment-16650073
]
ASF GitHub Bot commented on NIFI-5629:
--
Github user adyoun2 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3033#discussion_r225123174
--- Diff:
nifi-commons/nifi-properties/src/main/java/org/apache/nifi/util/NiFiProperties.java
---
@@ -1422,10 +1424,15 @@ public String
getDefaultBackPressureDataSizeThreshold() {
public static NiFiProperties createBasicNiFiProperties(final String
propertiesFilePath, final Map additionalProperties) {
final Map addProps = (additionalProperties ==
null) ? Collections.EMPTY_MAP : additionalProperties;
final Properties properties = new Properties();
-final String nfPropertiesFilePath = (propertiesFilePath == null)
+String nfPropertiesFilePath = (propertiesFilePath == null)
? System.getProperty(NiFiProperties.PROPERTIES_FILE_PATH)
: propertiesFilePath;
if (nfPropertiesFilePath != null) {
+try {
--- End diff --
it was relevant, to get it working in my local environment, where someone
had put spaces in the path, %20 in resource paths needed converting back to ' '
> GetFile becomes slow listing vast directories
> -
>
> Key: NIFI-5629
> URL: https://issues.apache.org/jira/browse/NIFI-5629
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.6.0
>Reporter: Adam
>Priority: Minor
>
> GetFile repeatedly lists entire directories before applying batching, meaning
> for vast directories it spends a long time listing directories.
>
> Pull request to follow.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
