[jira] [Commented] (NIFI-5329) Add ability to "empty queue" on an entire process group
[ https://issues.apache.org/jira/browse/NIFI-5329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544660#comment-16544660 ] ASF GitHub Bot commented on NIFI-5329: -- Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2893 [Mongo_client_test.xml.txt](https://github.com/apache/nifi/files/2196064/Mongo_client_test.xml.txt) That is a flow for testing the functionality against a Mongo server running on `localhost:27017`. > Add ability to "empty queue" on an entire process group > --- > > Key: NIFI-5329 > URL: https://issues.apache.org/jira/browse/NIFI-5329 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.6.0 >Reporter: Mark Bean >Priority: Major > > It is desirable to clear all queues in a section of the flow. Specifically, a > Process Group can be taken as the unit to which 'empty queue' would be > applied. All connections within the Process Group are emptied. > If the user does not have permission to empty a given queue, it should be > excluded. However, all connections which the user has permission should > complete successfully. A dialog at the end of the operation should indicate > "X FlowFiles (Y bytes) were removed from Z queues". If appropriate, the > dialog should also include "X queues were not emptied due to insufficient > permission" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2893: NIFI-5329 Made a client service an optional source of conn...
Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2893 [Mongo_client_test.xml.txt](https://github.com/apache/nifi/files/2196064/Mongo_client_test.xml.txt) That is a flow for testing the functionality against a Mongo server running on `localhost:27017`. ---
[jira] [Commented] (NIFI-5329) Add ability to "empty queue" on an entire process group
[ https://issues.apache.org/jira/browse/NIFI-5329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544653#comment-16544653 ] ASF GitHub Bot commented on NIFI-5329: -- GitHub user MikeThomsen opened a pull request: https://github.com/apache/nifi/pull/2893 NIFI-5329 Made a client service an optional source of connection pool… …ing in Mongo processors. Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/MikeThomsen/nifi NIFI-5239 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2893.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2893 commit 92e9ead1ee8f1d55edd9b3be6991245abd18aa5e Author: Mike Thomsen Date: 2018-07-15T12:07:21Z NIFI-5329 Made a client service an optional source of connection pooling in Mongo processors. > Add ability to "empty queue" on an entire process group > --- > > Key: NIFI-5329 > URL: https://issues.apache.org/jira/browse/NIFI-5329 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.6.0 >Reporter: Mark Bean >Priority: Major > > It is desirable to clear all queues in a section of the flow. Specifically, a > Process Group can be taken as the unit to which 'empty queue' would be > applied. All connections within the Process Group are emptied. > If the user does not have permission to empty a given queue, it should be > excluded. However, all connections which the user has permission should > complete successfully. A dialog at the end of the operation should indicate > "X FlowFiles (Y bytes) were removed from Z queues". If appropriate, the > dialog should also include "X queues were not emptied due to insufficient > permission" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #2893: NIFI-5329 Made a client service an optional source ...
GitHub user MikeThomsen opened a pull request: https://github.com/apache/nifi/pull/2893 NIFI-5329 Made a client service an optional source of connection pool⦠â¦ing in Mongo processors. Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/MikeThomsen/nifi NIFI-5239 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2893.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2893 commit 92e9ead1ee8f1d55edd9b3be6991245abd18aa5e Author: Mike Thomsen Date: 2018-07-15T12:07:21Z NIFI-5329 Made a client service an optional source of connection pooling in Mongo processors. ---
[jira] [Commented] (NIFI-5406) Add new listing strategy by tracking listed entities to ListXXXX processors
[ https://issues.apache.org/jira/browse/NIFI-5406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544648#comment-16544648 ] ASF GitHub Bot commented on NIFI-5406: -- Github user ijokarumawak commented on the issue: https://github.com/apache/nifi/pull/2876 Docs and unit tests are added. All todo items are done. This PR is now fully ready for review. Thanks! > Add new listing strategy by tracking listed entities to List processors > --- > > Key: NIFI-5406 > URL: https://issues.apache.org/jira/browse/NIFI-5406 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Koji Kawamura >Assignee: Koji Kawamura >Priority: Major > > Current List processors (ListFile, ListFTP, ListSFTP ... etc) implementation > relies on file last modified timestamp to pick new or updated files. This > approach is efficient and lightweight in terms of state management, because > it only tracks latest modified timestamp and last executed timestamp. > However, timestamps do not work as expected in some file systems, causing > List processors missing files periodically. See NIFI-3332 comments for > details. > In order to pick every entity that has not seen before or has been updated > since it had seen last time, we need another set of processors using > different approach, that is by tracking listed entities: > * Add new abstract processor AbstractWatchEntries similar to > AbstractListProcessor but uses different approach > * Target entities have: name (path), size and last-modified-timestamp > * Implementation Processors have following properties: > ** 'Watch Time Window' to limit the maximum time period to hold the already > listed entries. E.g. if set as '30min', the processor keeps entities listed > in the last 30 mins. > ** 'Minimum File Age' to defer listing entities potentially being written > * Any entity added but not listed ever having last-modified-timestamp older > than configured 'Watch Time Window' will not be listed. If user needs to pick > these items, they have to make 'Watch Time Window' longer. It also increases > the size of data the processor has to persist in the K/V store. Efficiency vs > reliability trade-off. > * The already-listed entities are persisted into one of supported K/V store > through DistributedMapCacheClient service. User can chose what KVS to use > from HBase, Redis, Couchbase and File (DistributedMapCacheServer with > persistence file). > * The reason to use KVS instead of ManagedState is, to avoid hammering > Zookeeper too much with frequently updating Zk node with large amount of > data. The number of already-listed entries can be huge depending on > use-cases. Also, we can compress entities with DistributedMapCacheClient as > it supports putting byte array, while ManagedState only supports Map String>. > * On each onTrigger: > ** Processor performs listing. Listed entries meeting any of the following > condition will be written to the 'success' output FlowFile: > *** Not exists in the already-listed entities > *** Having newer last-modified-timestamp > *** Having different size > ** Already listed entries those are old enough compared to 'Watch Time > Window' are discarded from the already-listed entries. > * Initial supporting target is Local file system, FTP and SFTP -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2876: NIFI-5406: Adding new listing strategy to List processors.
Github user ijokarumawak commented on the issue: https://github.com/apache/nifi/pull/2876 Docs and unit tests are added. All todo items are done. This PR is now fully ready for review. Thanks! ---
[jira] [Updated] (NIFI-1184) Mock File System for PutHDFS Tests
[ https://issues.apache.org/jira/browse/NIFI-1184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uday Kale updated NIFI-1184: Fix Version/s: 1.8.0 Status: Patch Available (was: Open) > Mock File System for PutHDFS Tests > -- > > Key: NIFI-1184 > URL: https://issues.apache.org/jira/browse/NIFI-1184 > Project: Apache NiFi > Issue Type: Test >Reporter: Joseph Percivall >Assignee: Uday Kale >Priority: Minor > Fix For: 1.8.0 > > > Currently 2 of the PutHDFSTest unit test fail when run on Windows. For the > 0.4.0 released we decided to ignore those tests when running on Windows on > the grounds that they were failing due to issues unrelated to what was > attempting to be unit tested. > What needs to be done is to mock the File System being used for validation in > the PutHDFSTest instead of relying on Hadoop (which fails on Windows without > the "winutil.exe" binary). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (NIFI-1184) Mock File System for PutHDFS Tests
[ https://issues.apache.org/jira/browse/NIFI-1184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri reassigned NIFI-1184: - Assignee: Uday Kale (was: Uday) > Mock File System for PutHDFS Tests > -- > > Key: NIFI-1184 > URL: https://issues.apache.org/jira/browse/NIFI-1184 > Project: Apache NiFi > Issue Type: Test >Reporter: Joseph Percivall >Assignee: Uday Kale >Priority: Minor > > Currently 2 of the PutHDFSTest unit test fail when run on Windows. For the > 0.4.0 released we decided to ignore those tests when running on Windows on > the grounds that they were failing due to issues unrelated to what was > attempting to be unit tested. > What needs to be done is to mock the File System being used for validation in > the PutHDFSTest instead of relying on Hadoop (which fails on Windows without > the "winutil.exe" binary). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (NIFI-1184) Mock File System for PutHDFS Tests
[ https://issues.apache.org/jira/browse/NIFI-1184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri reassigned NIFI-1184: - Assignee: Uday > Mock File System for PutHDFS Tests > -- > > Key: NIFI-1184 > URL: https://issues.apache.org/jira/browse/NIFI-1184 > Project: Apache NiFi > Issue Type: Test >Reporter: Joseph Percivall >Assignee: Uday >Priority: Minor > > Currently 2 of the PutHDFSTest unit test fail when run on Windows. For the > 0.4.0 released we decided to ignore those tests when running on Windows on > the grounds that they were failing due to issues unrelated to what was > attempting to be unit tested. > What needs to be done is to mock the File System being used for validation in > the PutHDFSTest instead of relying on Hadoop (which fails on Windows without > the "winutil.exe" binary). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5305) Parameterized collection name fails with dynamic params
[
https://issues.apache.org/jira/browse/NIFI-5305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544566#comment-16544566
]
ASF GitHub Bot commented on NIFI-5305:
--
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2891#discussion_r202545693
--- Diff:
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/test/java/org/apache/nifi/processors/mongodb/GetMongoIT.java
---
@@ -468,4 +469,37 @@ public void testKeepOriginalAttributes() {
/*
* End query read behavior tests
*/
+
+/*
+ * Verify that behavior described in NIFI-5305 actually works.
--- End diff --
Maybe an explanation of the 'case' would be appropriate, like " Test that
when you do foo, bar happens and when you do baz, foo happens, see NIFI-5303" ?
That would help maintainers.
> Parameterized collection name fails with dynamic params
> ---
>
> Key: NIFI-5305
> URL: https://issues.apache.org/jira/browse/NIFI-5305
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.6.0
>Reporter: Luis M Calderon
>Assignee: Mike Thomsen
>Priority: Major
> Attachments: getmongo_error.txt
>
>
> When using a dynamic property for collection name, the creation of the full
> query reference name is not working as expected. The reference name seems to
> be the concatenation of the namespace and collection name. i.e.
> _.)_
> The following parameters are those which are provided to the GetMongo
> Processor:
> Param Definitions:
> my_schema = 'my_schema'
> collection_name =
> {code:java}
> Mongo Database Name: ${my_schema}
> Mongo Collection Name: ${collection_name}{code}
>
> Attached is a text file containing the NiFi log for this error.
>
> _There is no namespace error when the same values are hard coded._
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #2891: NIFI-5305 Added a unit test to check for the behavi...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2891#discussion_r202545693
--- Diff:
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/test/java/org/apache/nifi/processors/mongodb/GetMongoIT.java
---
@@ -468,4 +469,37 @@ public void testKeepOriginalAttributes() {
/*
* End query read behavior tests
*/
+
+/*
+ * Verify that behavior described in NIFI-5305 actually works.
--- End diff --
Maybe an explanation of the 'case' would be appropriate, like " Test that
when you do foo, bar happens and when you do baz, foo happens, see NIFI-5303" ?
That would help maintainers.
---
[jira] [Commented] (MINIFICPP-556) Create snap build
[ https://issues.apache.org/jira/browse/MINIFICPP-556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544556#comment-16544556 ] ASF GitHub Bot commented on MINIFICPP-556: -- Github user phrocker commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/371 I had a failure after approving. Will retry tomorrow before merging. Might be my issue. > Create snap build > - > > Key: MINIFICPP-556 > URL: https://issues.apache.org/jira/browse/MINIFICPP-556 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Andrew Christianson >Assignee: Andrew Christianson >Priority: Major > > Package nifi-minifi-cpp into a snap package for easy distribution to users > who use snap. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp issue #371: MINIFICPP-556 Added initial snapcraft build
Github user phrocker commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/371 I had a failure after approving. Will retry tomorrow before merging. Might be my issue. ---
[jira] [Commented] (NIFI-1184) Mock File System for PutHDFS Tests
[ https://issues.apache.org/jira/browse/NIFI-1184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544546#comment-16544546 ] ASF GitHub Bot commented on NIFI-1184: -- GitHub user udaykale opened a pull request: https://github.com/apache/nifi/pull/2892 NIFI-1184 Mock FileSystem for PutHDFSTest Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/udaykale/nifi NIFI-1184 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2892.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2892 commit 0640f80f7067867a2091531e2136196a7b87a4f6 Author: uday Date: 2018-07-14T20:54:58Z NIFI-1184 Mock FileSystem for PutHDFSTest > Mock File System for PutHDFS Tests > -- > > Key: NIFI-1184 > URL: https://issues.apache.org/jira/browse/NIFI-1184 > Project: Apache NiFi > Issue Type: Test >Reporter: Joseph Percivall >Priority: Minor > > Currently 2 of the PutHDFSTest unit test fail when run on Windows. For the > 0.4.0 released we decided to ignore those tests when running on Windows on > the grounds that they were failing due to issues unrelated to what was > attempting to be unit tested. > What needs to be done is to mock the File System being used for validation in > the PutHDFSTest instead of relying on Hadoop (which fails on Windows without > the "winutil.exe" binary). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #2892: NIFI-1184 Mock FileSystem for PutHDFSTest
GitHub user udaykale opened a pull request: https://github.com/apache/nifi/pull/2892 NIFI-1184 Mock FileSystem for PutHDFSTest Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/udaykale/nifi NIFI-1184 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2892.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2892 commit 0640f80f7067867a2091531e2136196a7b87a4f6 Author: uday Date: 2018-07-14T20:54:58Z NIFI-1184 Mock FileSystem for PutHDFSTest ---
[jira] [Commented] (NIFI-5426) Use NIO.2 API for ListFile to avoid multiple disk reads
[ https://issues.apache.org/jira/browse/NIFI-5426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544451#comment-16544451 ] ASF GitHub Bot commented on NIFI-5426: -- Github user mgaido91 commented on the issue: https://github.com/apache/nifi/pull/2889 Hi @markap14. I saw the comment on the JIRA that you merged this to master. Shall I close this PR then? thanks! > Use NIO.2 API for ListFile to avoid multiple disk reads > --- > > Key: NIFI-5426 > URL: https://issues.apache.org/jira/browse/NIFI-5426 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Mark Payne >Priority: Major > Fix For: 1.8.0 > > > NIFI-4631 attempted to do something similar but had issues when running > against CIFS mounts, reportedly. However, I think the issue was not the use > necessarily of NIO.2 but the way in which it was used. It attempted to use > walkTree and did did not appear to follow symlinks. As a result, this caused > failures in some scenarios. We should look at using NIO.2 and probably obtain > a Stream. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2889: NIFI-5426: Use NIO.2 API for ListFile
Github user mgaido91 commented on the issue: https://github.com/apache/nifi/pull/2889 Hi @markap14. I saw the comment on the JIRA that you merged this to master. Shall I close this PR then? thanks! ---
