[jira] [Updated] (NIFI-6569) Site-to-Site not timing out when reading from remote NiFi
[ https://issues.apache.org/jira/browse/NIFI-6569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Kawamura updated NIFI-6569: Resolution: Fixed Status: Resolved (was: Patch Available) > Site-to-Site not timing out when reading from remote NiFi > - > > Key: NIFI-6569 > URL: https://issues.apache.org/jira/browse/NIFI-6569 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 1.10.0 > > Time Spent: 40m > Remaining Estimate: 0h > > I created two simple flows: > Remote Input Port -> LogAttribute > GenerateFlowFile -> RemoteProcessGroup (pointing to self) > So that I can simply feed generated FlowFiles to the RPG to the LogAttribute. > When I ran this using Java 11, the listener failed to start (known issue, > NIFI-5952). However, what I noticed is that the RPG would not stop when I > clicked Disable Transmission. Instead, the web request to disable would hang. > Thread dumps show the call to try to find out which nodes are in the cluster > are never timing out. This seems to be related to NIFI-4461, though it's not > entirely clear at this point. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6570) Site-to-Site listener not starting if standalone and no Root Group Ports exist
[ https://issues.apache.org/jira/browse/NIFI-6570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16920583#comment-16920583 ] ASF subversion and git services commented on NIFI-6570: --- Commit 9ba280680ff8b40b15d460e6f822c2b10d4373c3 in nifi's branch refs/heads/master from Mark Payne [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=9ba2806 ] NIFI-6569, NIFI-6570: Fixed bug that caused read timeouts not to occur with site-to-site. Fixed bug that caused site-to-site listener not to accept connections if there are no input/output ports on the root group - this used to be a valid check and was done to prevent spawning extra threads and doing extra work if no ports exist but now that we have site-to-site ports outside of the root group it's no longer a reasonable condition to check. This closes #3658. Signed-off-by: Koji Kawamura > Site-to-Site listener not starting if standalone and no Root Group Ports exist > -- > > Key: NIFI-6570 > URL: https://issues.apache.org/jira/browse/NIFI-6570 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Minor > Fix For: 1.10.0 > > > I created a flow that has a Site-to-Site input port within a Process Group > but no ports at the root group. When I tried to send to the instance via > site-to-site, I could not connect to the instance. Thread dumps show that the > Site-to-Site listener had not yet begun to accept connections because no > Input Ports or Output Ports exist at the root level and the instance is not > clustered. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (NIFI-6570) Site-to-Site listener not starting if standalone and no Root Group Ports exist
[ https://issues.apache.org/jira/browse/NIFI-6570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Kawamura updated NIFI-6570: Resolution: Fixed Status: Resolved (was: Patch Available) > Site-to-Site listener not starting if standalone and no Root Group Ports exist > -- > > Key: NIFI-6570 > URL: https://issues.apache.org/jira/browse/NIFI-6570 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Minor > Fix For: 1.10.0 > > > I created a flow that has a Site-to-Site input port within a Process Group > but no ports at the root group. When I tried to send to the instance via > site-to-site, I could not connect to the instance. Thread dumps show that the > Site-to-Site listener had not yet begun to accept connections because no > Input Ports or Output Ports exist at the root level and the instance is not > clustered. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6569) Site-to-Site not timing out when reading from remote NiFi
[ https://issues.apache.org/jira/browse/NIFI-6569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16920582#comment-16920582 ] ASF subversion and git services commented on NIFI-6569: --- Commit 9ba280680ff8b40b15d460e6f822c2b10d4373c3 in nifi's branch refs/heads/master from Mark Payne [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=9ba2806 ] NIFI-6569, NIFI-6570: Fixed bug that caused read timeouts not to occur with site-to-site. Fixed bug that caused site-to-site listener not to accept connections if there are no input/output ports on the root group - this used to be a valid check and was done to prevent spawning extra threads and doing extra work if no ports exist but now that we have site-to-site ports outside of the root group it's no longer a reasonable condition to check. This closes #3658. Signed-off-by: Koji Kawamura > Site-to-Site not timing out when reading from remote NiFi > - > > Key: NIFI-6569 > URL: https://issues.apache.org/jira/browse/NIFI-6569 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 1.10.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > I created two simple flows: > Remote Input Port -> LogAttribute > GenerateFlowFile -> RemoteProcessGroup (pointing to self) > So that I can simply feed generated FlowFiles to the RPG to the LogAttribute. > When I ran this using Java 11, the listener failed to start (known issue, > NIFI-5952). However, what I noticed is that the RPG would not stop when I > clicked Disable Transmission. Instead, the web request to disable would hang. > Thread dumps show the call to try to find out which nodes are in the cluster > are never timing out. This seems to be related to NIFI-4461, though it's not > entirely clear at this point. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [nifi] asfgit closed pull request #3658: NIFI-6569, NIFI-6570: Fixed bug that caused read timeouts not to occu…
asfgit closed pull request #3658: NIFI-6569, NIFI-6570: Fixed bug that caused read timeouts not to occu… URL: https://github.com/apache/nifi/pull/3658 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] ijokarumawak commented on issue #3658: NIFI-6569, NIFI-6570: Fixed bug that caused read timeouts not to occu…
ijokarumawak commented on issue #3658: NIFI-6569, NIFI-6570: Fixed bug that caused read timeouts not to occu… URL: https://github.com/apache/nifi/pull/3658#issuecomment-526983549 @markap14 Thanks for finding and fixing the issues! Change LGTM +1. Merging to master. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (NIFI-6480) PutORC/PutParquet can't overwrite file even if set 'Overwrite Files' to true
[ https://issues.apache.org/jira/browse/NIFI-6480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16920575#comment-16920575 ] archon gum commented on NIFI-6480: -- BTW, that PR is opened on Jul 24 and it's Sep now.. > PutORC/PutParquet can't overwrite file even if set 'Overwrite Files' to true > > > Key: NIFI-6480 > URL: https://issues.apache.org/jira/browse/NIFI-6480 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: archon gum >Assignee: archon gum >Priority: Major > Attachments: Snipaste_2019-07-16_11-07-05.jpg > > Time Spent: 1h > Remaining Estimate: 0h > > !Snipaste_2019-07-16_11-07-05.jpg! > > Solution: > Reference to > [PutHDFS|https://github.com/apache/nifi/blob/72244d09ff193131119c10492b9327af35a64f02/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/PutHDFS.java], > delete dest file before rename temp file to dest file. > Tested on my local machine. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6480) PutORC/PutParquet can't overwrite file even if set 'Overwrite Files' to true
[ https://issues.apache.org/jira/browse/NIFI-6480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16920569#comment-16920569 ] HondaWei commented on NIFI-6480: OK, I didn't notice it. It's good. > PutORC/PutParquet can't overwrite file even if set 'Overwrite Files' to true > > > Key: NIFI-6480 > URL: https://issues.apache.org/jira/browse/NIFI-6480 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: archon gum >Assignee: archon gum >Priority: Major > Attachments: Snipaste_2019-07-16_11-07-05.jpg > > Time Spent: 1h > Remaining Estimate: 0h > > !Snipaste_2019-07-16_11-07-05.jpg! > > Solution: > Reference to > [PutHDFS|https://github.com/apache/nifi/blob/72244d09ff193131119c10492b9327af35a64f02/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/PutHDFS.java], > delete dest file before rename temp file to dest file. > Tested on my local machine. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [nifi] ijokarumawak commented on issue #3658: NIFI-6569, NIFI-6570: Fixed bug that caused read timeouts not to occu…
ijokarumawak commented on issue #3658: NIFI-6569, NIFI-6570: Fixed bug that caused read timeouts not to occu… URL: https://github.com/apache/nifi/pull/3658#issuecomment-526975224 Reviewing... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] ijokarumawak commented on a change in pull request #3483: NIFI-6275 ListHDFS now ignores scheme and authority when uses "Full P…
ijokarumawak commented on a change in pull request #3483: NIFI-6275 ListHDFS
now ignores scheme and authority when uses "Full P…
URL: https://github.com/apache/nifi/pull/3483#discussion_r319786069
##
File path:
nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/ListHDFS.java
##
@@ -527,7 +527,7 @@ private PathFilter createPathFilter(final ProcessContext
context) {
return path -> {
final boolean accepted;
if (FILTER_FULL_PATH_VALUE.getValue().equals(filterMode)) {
-accepted = filePattern.matcher(path.toString()).matches();
+accepted =
filePattern.matcher(Path.getPathWithoutSchemeAndAuthority(path).toString()).matches();
Review comment:
If it's possible that this improvement may break existing user flows, then
I'd like to discuss about other approaches to opt-in this.
We can provide different UX via different approaches:
1. Current approach: If existing flows regex contains schema or authority,
their flow will not list files as before. Users may wonder what goes wrong. May
not notice the change if they don't read docs..
2. Adding new 'Filter without Schema and Authority' property:
- A. If we leave its default value blank and implement a custom
validation to require it when filter regex is not empty, then we can make
existing ListHDFS invalid. That will give user to chance to review their
configuration.
- B. If we use `false` as default value, existing flows work as is.
While this improvement can be opt-in. The most safe approach, but a con is
people may forget enabling this option.
3. Adding new 'Full Path (without schema and authority)' filter mode, or add
new one and rename the existing one's display name to 'Full Path (include
schema and authority)': this guarantees existing flows work as is, while
providing easy configuration UX for new setups.
I personally prefer the option 3 above. How do you think?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (NIFI-6480) PutORC/PutParquet can't overwrite file even if set 'Overwrite Files' to true
[ https://issues.apache.org/jira/browse/NIFI-6480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16920554#comment-16920554 ] archon gum commented on NIFI-6480: -- [~hondawei] There is already a PR to this: https://github.com/apache/nifi/pull/3599 Have a look and share your thought, thanks > PutORC/PutParquet can't overwrite file even if set 'Overwrite Files' to true > > > Key: NIFI-6480 > URL: https://issues.apache.org/jira/browse/NIFI-6480 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: archon gum >Assignee: archon gum >Priority: Major > Attachments: Snipaste_2019-07-16_11-07-05.jpg > > Time Spent: 1h > Remaining Estimate: 0h > > !Snipaste_2019-07-16_11-07-05.jpg! > > Solution: > Reference to > [PutHDFS|https://github.com/apache/nifi/blob/72244d09ff193131119c10492b9327af35a64f02/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/PutHDFS.java], > delete dest file before rename temp file to dest file. > Tested on my local machine. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [nifi] granthenke commented on issue #3610: NIFI-6552 Kudu Put Operations
granthenke commented on issue #3610: NIFI-6552 Kudu Put Operations URL: https://github.com/apache/nifi/pull/3610#issuecomment-526919477 Thanks for the contribution @SandishKumarHN! And thank you for the reviews/commit @pvillard31! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] pvillard31 commented on issue #3394: NIFI-6159 - Add BigQuery processor using the Streaming API
pvillard31 commented on issue #3394: NIFI-6159 - Add BigQuery processor using the Streaming API URL: https://github.com/apache/nifi/pull/3394#issuecomment-526913127 Thanks for the review @arpadboda ! I believe I addressed your - very valid - points ! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] pvillard31 commented on issue #3611: NIFI-6009 ScanKudu Processor
pvillard31 commented on issue #3611: NIFI-6009 ScanKudu Processor URL: https://github.com/apache/nifi/pull/3611#issuecomment-526911416 @SandishKumarHN, now #3610 is merged, you can rebase this one against master This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (NIFI-6552) Kudu Processor: Kudu Put Operations
[ https://issues.apache.org/jira/browse/NIFI-6552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard updated NIFI-6552: - Labels: kudu (was: ) > Kudu Processor: Kudu Put Operations > > > Key: NIFI-6552 > URL: https://issues.apache.org/jira/browse/NIFI-6552 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Sandish Kumar HN >Assignee: Sandish Kumar HN >Priority: Major > Labels: kudu > Fix For: 1.10.0 > > Time Spent: 4.5h > Remaining Estimate: 0h > > Adding Kudu Operation's Like Delete, Update, Upsert to Kudu Put Processor -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (NIFI-6552) Kudu Processor: Kudu Put Operations
[ https://issues.apache.org/jira/browse/NIFI-6552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard updated NIFI-6552: - Component/s: Extensions > Kudu Processor: Kudu Put Operations > > > Key: NIFI-6552 > URL: https://issues.apache.org/jira/browse/NIFI-6552 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Sandish Kumar HN >Assignee: Sandish Kumar HN >Priority: Major > Fix For: 1.10.0 > > Time Spent: 4.5h > Remaining Estimate: 0h > > Adding Kudu Operation's Like Delete, Update, Upsert to Kudu Put Processor -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (NIFI-6552) Kudu Processor: Kudu Put Operations
[ https://issues.apache.org/jira/browse/NIFI-6552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard updated NIFI-6552: - Fix Version/s: 1.10.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Kudu Processor: Kudu Put Operations > > > Key: NIFI-6552 > URL: https://issues.apache.org/jira/browse/NIFI-6552 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Sandish Kumar HN >Assignee: Sandish Kumar HN >Priority: Major > Fix For: 1.10.0 > > Time Spent: 4.5h > Remaining Estimate: 0h > > Adding Kudu Operation's Like Delete, Update, Upsert to Kudu Put Processor -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [nifi] asfgit closed pull request #3610: NIFI-6552 Kudu Put Operations
asfgit closed pull request #3610: NIFI-6552 Kudu Put Operations URL: https://github.com/apache/nifi/pull/3610 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (NIFI-6552) Kudu Processor: Kudu Put Operations
[ https://issues.apache.org/jira/browse/NIFI-6552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16920376#comment-16920376 ] ASF subversion and git services commented on NIFI-6552: --- Commit 26b203616e79032bdcd47ac927d488a2ce68c656 in nifi's branch refs/heads/master from SandishKumarHN [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=26b2036 ] NIFI-6552 - Kudu Put Operations Signed-off-by: Pierre Villard This closes #3610. > Kudu Processor: Kudu Put Operations > > > Key: NIFI-6552 > URL: https://issues.apache.org/jira/browse/NIFI-6552 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Sandish Kumar HN >Assignee: Sandish Kumar HN >Priority: Major > Time Spent: 4h 20m > Remaining Estimate: 0h > > Adding Kudu Operation's Like Delete, Update, Upsert to Kudu Put Processor -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [nifi] pvillard31 commented on issue #3610: NIFI-6552 Kudu Put Operations
pvillard31 commented on issue #3610: NIFI-6552 Kudu Put Operations URL: https://github.com/apache/nifi/pull/3610#issuecomment-526910901 Things look good to me - I did a final round of review/testing and it's working as expected on the simple flows I have. Thanks for all the changes along the review. Will merge to master. Thanks @SandishKumarHN ! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
