Re: [PR] NIFI-12673 - ExtractKeywords processor [nifi]
pvillard31 commented on code in PR #8302: URL: https://github.com/apache/nifi/pull/8302#discussion_r1467295394 ## nifi-python-extensions/nifi-text-embeddings-module/src/main/python/ExtractKeywords.py: ## @@ -0,0 +1,95 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import yake + +from nifiapi.flowfiletransform import FlowFileTransform, FlowFileTransformResult +from nifiapi.properties import PropertyDescriptor, StandardValidators, ExpressionLanguageScope + +class ExtractKeywords(FlowFileTransform): +class Java: +implements = ["org.apache.nifi.python.processor.FlowFileTransform"] + +class ProcessorDetails: +version = "2.0.0-SNAPSHOT" +description = """Parses the content of the incoming FlowFile to extract the keywords and add those as attributes +using the convention 'keyword.i' where i is a number depending on how many keywords are extracted. The attribute +'keywords.length' will be set with the number of extracted keywords. The attribute 'keyword.0' will contain the +list of all extracted keywords with the associated score.""" +tags = ["text", "keyword", "vector", "extract"] +dependencies = ['yake'] Review Comment: Thanks @exceptionfactory - I noticed this but since we're not including anything, I was under the impression that we would not have licensing issues to be concerned about with the Python stuff. Do we have a clear position on this? Also, I agree this library has not been maintained in a while but the capability used here is pretty limited. I'm ok to look at other options though. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-12646) Conduct Apache NiFi 2.0.0-M2 Release
[ https://issues.apache.org/jira/browse/NIFI-12646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17811092#comment-17811092 ] ASF subversion and git services commented on NIFI-12646: Commit 5ca7b0df81300ce1bfda901c918842eca5847eb2 in nifi's branch refs/heads/NIFI-12646-RC4 from David Handermann [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=5ca7b0df81 ] NIFI-12646-RC4 prepare for next development iteration > Conduct Apache NiFi 2.0.0-M2 Release > > > Key: NIFI-12646 > URL: https://issues.apache.org/jira/browse/NIFI-12646 > Project: Apache NiFi > Issue Type: Task >Reporter: David Handermann >Assignee: David Handermann >Priority: Major > Fix For: 2.0.0-M2 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12646) Conduct Apache NiFi 2.0.0-M2 Release
[ https://issues.apache.org/jira/browse/NIFI-12646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17811090#comment-17811090 ] ASF subversion and git services commented on NIFI-12646: Commit 38fc0a4e8d966b6a185144e4ea6b34c948cdad1b in nifi's branch refs/heads/NIFI-12646-RC4 from David Handermann [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=38fc0a4e8d ] NIFI-12646 Set Python Processor version to 2.0.0-M2 Signed-off-by: David Handermann > Conduct Apache NiFi 2.0.0-M2 Release > > > Key: NIFI-12646 > URL: https://issues.apache.org/jira/browse/NIFI-12646 > Project: Apache NiFi > Issue Type: Task >Reporter: David Handermann >Assignee: David Handermann >Priority: Major > Fix For: 2.0.0-M2 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12646) Conduct Apache NiFi 2.0.0-M2 Release
[ https://issues.apache.org/jira/browse/NIFI-12646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17811091#comment-17811091 ] ASF subversion and git services commented on NIFI-12646: Commit 640b7bdfbbb8842f057a9bf49dc2b9b5d092abda in nifi's branch refs/heads/NIFI-12646-RC4 from David Handermann [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=640b7bdfbb ] NIFI-12646-RC4 prepare release nifi-2.0.0-M2-RC4 > Conduct Apache NiFi 2.0.0-M2 Release > > > Key: NIFI-12646 > URL: https://issues.apache.org/jira/browse/NIFI-12646 > Project: Apache NiFi > Issue Type: Task >Reporter: David Handermann >Assignee: David Handermann >Priority: Major > Fix For: 2.0.0-M2 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (NIFI-12668) Test for Git-backed Registry fails with global Git config value
[
https://issues.apache.org/jira/browse/NIFI-12668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann resolved NIFI-12668.
-
Fix Version/s: 2.0.0-M2
Assignee: Brian Ghigiarelli
Resolution: Fixed
> Test for Git-backed Registry fails with global Git config value
> ---
>
> Key: NIFI-12668
> URL: https://issues.apache.org/jira/browse/NIFI-12668
> Project: Apache NiFi
> Issue Type: Improvement
> Components: NiFi Registry
>Affects Versions: 2.0.0-M1
>Reporter: Brian Ghigiarelli
>Assignee: Brian Ghigiarelli
>Priority: Minor
> Fix For: 2.0.0-M2
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The Git-backed Registry provider, GitFlowPersistenceProvider, does not
> digitally sign commits to Git, nor does it specify the GPG configuration that
> would be used if it were to sign commits. See
> [GitFlowMetaData.java:478|https://github.com/apache/nifi/blob/89836f32d017d77972a4de09c4e864b0e11899a8/nifi-registry/nifi-registry-core/nifi-registry-framework/src/main/java/org/apache/nifi/registry/provider/flow/git/GitFlowMetaData.java#L478].
> When the GPG configuration is not specified, the underlying jgit library uses
> the default value of openpgp for the GPG format when it performs the commit.
> This works fine except when the system's global Git configuration has
> `gpg.format` set to `ssh`. Available options are:
> * openpgp (default)
> * x509
> * ssh
> The underlying jgit library does not recognize `ssh` as a valid GPG format.
> jgit validates the GPG configuration when committing to git, whether it signs
> the commit or not.
> This issue manifests when a user attempts to run NiFi's tests with their
> global Git configuration has `gpg.format` set to `ssh`. It would also occur
> if the NiFi Registry ran on a server with Git installed and configured in the
> same manner.
> h3. Steps to reproduce
> From the root NiFi source code directory, run:
> {code:java}
> git config --global gpg.format ssh
> cd ./nifi-registry/nifi-registry-core/nifi-registry-framework
> mvn clean test{code}
> h3. Observed Failure
> {code:java}
> [ERROR] Tests run: 6, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 3.387
> s <<< FAILURE! -- in
> org.apache.nifi.registry.provider.flow.git.TestGitFlowPersistenceProvider
> [ERROR]
> org.apache.nifi.registry.provider.flow.git.TestGitFlowPersistenceProvider.testLoadLargeFlow
> -- Time elapsed: 3.273 s <<< ERROR!
> java.lang.IllegalArgumentException: Invalid value: gpg.format=ssh
> at
> org.eclipse.jgit.lib.DefaultTypedConfigGetter.getEnum(DefaultTypedConfigGetter.java:103)
> at org.eclipse.jgit.lib.Config.getEnum(Config.java:453)
> at org.eclipse.jgit.lib.GpgConfig.(GpgConfig.java:86)
> at
> org.eclipse.jgit.api.CommitCommand.processOptions(CommitCommand.java:662)
> at org.eclipse.jgit.api.CommitCommand.call(CommitCommand.java:189)
> at
> org.apache.nifi.registry.provider.flow.git.GitFlowMetaData.commit(GitFlowMetaData.java:481)
> at
> org.apache.nifi.registry.provider.flow.git.GitFlowPersistenceProvider.saveFlowContent(GitFlowPersistenceProvider.java:187)
> at
> org.apache.nifi.registry.provider.flow.git.TestGitFlowPersistenceProvider.lambda$testLoadLargeFlow$14(TestGitFlowPersistenceProvider.java:311)
> at
> org.apache.nifi.registry.provider.flow.git.TestGitFlowPersistenceProvider.assertProvider(TestGitFlowPersistenceProvider.java:112)
> at
> org.apache.nifi.registry.provider.flow.git.TestGitFlowPersistenceProvider.testLoadLargeFlow(TestGitFlowPersistenceProvider.java:300)
> at java.base/java.lang.reflect.Method.invoke(Method.java:580)
> at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
> at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
> [ERROR]
> org.apache.nifi.registry.provider.flow.git.TestGitFlowPersistenceProvider.testLoadCommitHistories
> -- Time elapsed: 0.008 s <<< ERROR!
> java.lang.IllegalArgumentException: Invalid value: gpg.format=ssh
> at
> org.eclipse.jgit.lib.DefaultTypedConfigGetter.getEnum(DefaultTypedConfigGetter.java:103)
> at org.eclipse.jgit.lib.Config.getEnum(Config.java:453)
> at org.eclipse.jgit.lib.GpgConfig.(GpgConfig.java:86)
> at
> org.eclipse.jgit.api.CommitCommand.processOptions(CommitCommand.java:662)
> at org.eclipse.jgit.api.CommitCommand.call(CommitCommand.java:189)
> at
> org.apache.nifi.registry.provider.flow.git.GitFlowMetaData.commit(GitFlowMetaData.java:481)
> at
> org.apache.nifi.registry.provider.flow.git.GitFlowPersistenceProvider.saveFlowContent(GitFlowPersistenceProvider.java:187)
> at
>
[jira] [Commented] (NIFI-12668) Test for Git-backed Registry fails with global Git config value
[
https://issues.apache.org/jira/browse/NIFI-12668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17811060#comment-17811060
]
ASF subversion and git services commented on NIFI-12668:
Commit 49702f1f510cc33c9cb52bb5b9580009a1d8afe8 in nifi's branch
refs/heads/main from Brian Ghigiarelli
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=49702f1f51 ]
NIFI-12668 Fix conflict in Registry Git provider with gpg.format=ssh
NIFI-12667 Upgraded JGit from 6.7.0 to 6.8.0
This closes #8300
Signed-off-by: David Handermann
> Test for Git-backed Registry fails with global Git config value
> ---
>
> Key: NIFI-12668
> URL: https://issues.apache.org/jira/browse/NIFI-12668
> Project: Apache NiFi
> Issue Type: Improvement
> Components: NiFi Registry
>Affects Versions: 2.0.0-M1
>Reporter: Brian Ghigiarelli
>Priority: Minor
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The Git-backed Registry provider, GitFlowPersistenceProvider, does not
> digitally sign commits to Git, nor does it specify the GPG configuration that
> would be used if it were to sign commits. See
> [GitFlowMetaData.java:478|https://github.com/apache/nifi/blob/89836f32d017d77972a4de09c4e864b0e11899a8/nifi-registry/nifi-registry-core/nifi-registry-framework/src/main/java/org/apache/nifi/registry/provider/flow/git/GitFlowMetaData.java#L478].
> When the GPG configuration is not specified, the underlying jgit library uses
> the default value of openpgp for the GPG format when it performs the commit.
> This works fine except when the system's global Git configuration has
> `gpg.format` set to `ssh`. Available options are:
> * openpgp (default)
> * x509
> * ssh
> The underlying jgit library does not recognize `ssh` as a valid GPG format.
> jgit validates the GPG configuration when committing to git, whether it signs
> the commit or not.
> This issue manifests when a user attempts to run NiFi's tests with their
> global Git configuration has `gpg.format` set to `ssh`. It would also occur
> if the NiFi Registry ran on a server with Git installed and configured in the
> same manner.
> h3. Steps to reproduce
> From the root NiFi source code directory, run:
> {code:java}
> git config --global gpg.format ssh
> cd ./nifi-registry/nifi-registry-core/nifi-registry-framework
> mvn clean test{code}
> h3. Observed Failure
> {code:java}
> [ERROR] Tests run: 6, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 3.387
> s <<< FAILURE! -- in
> org.apache.nifi.registry.provider.flow.git.TestGitFlowPersistenceProvider
> [ERROR]
> org.apache.nifi.registry.provider.flow.git.TestGitFlowPersistenceProvider.testLoadLargeFlow
> -- Time elapsed: 3.273 s <<< ERROR!
> java.lang.IllegalArgumentException: Invalid value: gpg.format=ssh
> at
> org.eclipse.jgit.lib.DefaultTypedConfigGetter.getEnum(DefaultTypedConfigGetter.java:103)
> at org.eclipse.jgit.lib.Config.getEnum(Config.java:453)
> at org.eclipse.jgit.lib.GpgConfig.(GpgConfig.java:86)
> at
> org.eclipse.jgit.api.CommitCommand.processOptions(CommitCommand.java:662)
> at org.eclipse.jgit.api.CommitCommand.call(CommitCommand.java:189)
> at
> org.apache.nifi.registry.provider.flow.git.GitFlowMetaData.commit(GitFlowMetaData.java:481)
> at
> org.apache.nifi.registry.provider.flow.git.GitFlowPersistenceProvider.saveFlowContent(GitFlowPersistenceProvider.java:187)
> at
> org.apache.nifi.registry.provider.flow.git.TestGitFlowPersistenceProvider.lambda$testLoadLargeFlow$14(TestGitFlowPersistenceProvider.java:311)
> at
> org.apache.nifi.registry.provider.flow.git.TestGitFlowPersistenceProvider.assertProvider(TestGitFlowPersistenceProvider.java:112)
> at
> org.apache.nifi.registry.provider.flow.git.TestGitFlowPersistenceProvider.testLoadLargeFlow(TestGitFlowPersistenceProvider.java:300)
> at java.base/java.lang.reflect.Method.invoke(Method.java:580)
> at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
> at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
> [ERROR]
> org.apache.nifi.registry.provider.flow.git.TestGitFlowPersistenceProvider.testLoadCommitHistories
> -- Time elapsed: 0.008 s <<< ERROR!
> java.lang.IllegalArgumentException: Invalid value: gpg.format=ssh
> at
> org.eclipse.jgit.lib.DefaultTypedConfigGetter.getEnum(DefaultTypedConfigGetter.java:103)
> at org.eclipse.jgit.lib.Config.getEnum(Config.java:453)
> at org.eclipse.jgit.lib.GpgConfig.(GpgConfig.java:86)
> at
> org.eclipse.jgit.api.CommitCommand.processOptions(CommitCommand.java:662)
> at org.eclipse.jgit.api.CommitCommand.call(CommitCommand.java:189)
> at
> org.apache.nifi.registry.provider.flow.git.GitFlowMetaData.commit(GitFlowMetaData.java:481)
> at
>
[jira] [Resolved] (NIFI-12667) Upgrade JGit to 6.8.0
[ https://issues.apache.org/jira/browse/NIFI-12667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann resolved NIFI-12667. - Fix Version/s: 2.0.0-M2 Resolution: Fixed > Upgrade JGit to 6.8.0 > - > > Key: NIFI-12667 > URL: https://issues.apache.org/jira/browse/NIFI-12667 > Project: Apache NiFi > Issue Type: Improvement > Components: NiFi Registry >Reporter: David Handermann >Assignee: David Handermann >Priority: Minor > Fix For: 2.0.0-M2 > > > JGit should should be upgraded to > [6.8.0|https://projects.eclipse.org/projects/technology.jgit/releases/6.8.0] > incorporating several bug fixes and dependency upgrades. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12667) Upgrade JGit to 6.8.0
[ https://issues.apache.org/jira/browse/NIFI-12667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17811061#comment-17811061 ] ASF subversion and git services commented on NIFI-12667: Commit 49702f1f510cc33c9cb52bb5b9580009a1d8afe8 in nifi's branch refs/heads/main from Brian Ghigiarelli [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=49702f1f51 ] NIFI-12668 Fix conflict in Registry Git provider with gpg.format=ssh NIFI-12667 Upgraded JGit from 6.7.0 to 6.8.0 This closes #8300 Signed-off-by: David Handermann > Upgrade JGit to 6.8.0 > - > > Key: NIFI-12667 > URL: https://issues.apache.org/jira/browse/NIFI-12667 > Project: Apache NiFi > Issue Type: Improvement > Components: NiFi Registry >Reporter: David Handermann >Assignee: David Handermann >Priority: Minor > > JGit should should be upgraded to > [6.8.0|https://projects.eclipse.org/projects/technology.jgit/releases/6.8.0] > incorporating several bug fixes and dependency upgrades. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12668 Fix conflict in Registry Git provider with system config gpg.format=ssh [nifi]
exceptionfactory closed pull request #8300: NIFI-12668 Fix conflict in Registry Git provider with system config gpg.format=ssh URL: https://github.com/apache/nifi/pull/8300 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (NIFI-12664) Remove deprecated Distributed Map Cache Client in GetHBase
[ https://issues.apache.org/jira/browse/NIFI-12664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann resolved NIFI-12664. - Fix Version/s: 2.0.0-M2 (was: 2.0.0) Resolution: Fixed > Remove deprecated Distributed Map Cache Client in GetHBase > -- > > Key: NIFI-12664 > URL: https://issues.apache.org/jira/browse/NIFI-12664 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > Fix For: 2.0.0-M2 > > Time Spent: 20m > Remaining Estimate: 0h > > In 1.x, some code has been added in GetHBase processor to still support the > legacy Distributed Map Cache Client (for storing the state of the processor) > while migrating the state into the state manager of NiFi. However this DMC is > no longer used and the associated property/code should be removed for NiFi > 2.0. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12664) Remove deprecated Distributed Map Cache Client in GetHBase
[ https://issues.apache.org/jira/browse/NIFI-12664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17811059#comment-17811059 ] ASF subversion and git services commented on NIFI-12664: Commit 68228921a43c506e92a0476f550951e164dfb90e in nifi's branch refs/heads/main from Pierre Villard [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=68228921a4 ] NIFI-12664 Removed deprecated DMC in GetHBase This closes #8301 Signed-off-by: David Handermann > Remove deprecated Distributed Map Cache Client in GetHBase > -- > > Key: NIFI-12664 > URL: https://issues.apache.org/jira/browse/NIFI-12664 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > Fix For: 2.0.0 > > Time Spent: 20m > Remaining Estimate: 0h > > In 1.x, some code has been added in GetHBase processor to still support the > legacy Distributed Map Cache Client (for storing the state of the processor) > while migrating the state into the state manager of NiFi. However this DMC is > no longer used and the associated property/code should be removed for NiFi > 2.0. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12664 - remove deprecated DMC in GetHBase [nifi]
exceptionfactory closed pull request #8301: NIFI-12664 - remove deprecated DMC in GetHBase URL: https://github.com/apache/nifi/pull/8301 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-12676) HandleHttpRequest Servlet Not Registered
[ https://issues.apache.org/jira/browse/NIFI-12676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Witt updated NIFI-12676: Resolution: Fixed Status: Resolved (was: Patch Available) > HandleHttpRequest Servlet Not Registered > > > Key: NIFI-12676 > URL: https://issues.apache.org/jira/browse/NIFI-12676 > Project: Apache NiFi > Issue Type: Bug >Reporter: David Handermann >Assignee: David Handermann >Priority: Critical > Fix For: 2.0.0-M2 > > Time Spent: 20m > Remaining Estimate: 0h > > Following recent upgrades to Jetty 12, the HandleHttpRequest Processor is not > registered the standard Servlet that it uses to handle HTTP requests. > The Jetty ServletContextHandler should be used together with a root path > mapping to allow HandleHttpRequest to process requests according to > configured properties. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12676) HandleHttpRequest Servlet Not Registered
[ https://issues.apache.org/jira/browse/NIFI-12676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17811058#comment-17811058 ] ASF subversion and git services commented on NIFI-12676: Commit 103e350943233c6a780ab1086b2da9872df7c312 in nifi's branch refs/heads/main from David Handermann [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=103e350943 ] NIFI-12676 Fixed Servlet Registration in HandleHttpRequest This closes #8304. Signed-off-by: Joseph Witt > HandleHttpRequest Servlet Not Registered > > > Key: NIFI-12676 > URL: https://issues.apache.org/jira/browse/NIFI-12676 > Project: Apache NiFi > Issue Type: Bug >Reporter: David Handermann >Assignee: David Handermann >Priority: Critical > Fix For: 2.0.0-M2 > > Time Spent: 10m > Remaining Estimate: 0h > > Following recent upgrades to Jetty 12, the HandleHttpRequest Processor is not > registered the standard Servlet that it uses to handle HTTP requests. > The Jetty ServletContextHandler should be used together with a root path > mapping to allow HandleHttpRequest to process requests according to > configured properties. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12676 Fix Servlet Registration in HandleHttpRequest [nifi]
asfgit closed pull request #8304: NIFI-12676 Fix Servlet Registration in HandleHttpRequest URL: https://github.com/apache/nifi/pull/8304 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-12564) Processor 'ExecuteSQL' error, when uses unsigned mediumint in mysql
[
https://issues.apache.org/jira/browse/NIFI-12564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess updated NIFI-12564:
Summary: Processor 'ExecuteSQL' error, when uses unsigned mediumint in
mysql (was: Processor 'ExecuteSQL' error, when uses unsigned mdeiumint in
mysql)
> Processor 'ExecuteSQL' error, when uses unsigned mediumint in mysql
> ---
>
> Key: NIFI-12564
> URL: https://issues.apache.org/jira/browse/NIFI-12564
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: duhee
>Priority: Major
>
> * Nifi Version : NiFi 1.13.2
> * Processor : ExecuteSQL
> * Table schema
> {code:java}
> CREATE TABLE `nifi_test` (
> `clm1` varchar(15) NOT NULL DEFAULT '',
> `clm2` varchar(10) NOT NULL DEFAULT '',
> `clm3` varchar(12) NOT NULL DEFAULT '',
> `clm4` varchar(20) NOT NULL DEFAULT '',
> `clm5` mediumint(20) unsigned DEFAULT '1',
> PRIMARY KEY (`clm1`,`clm2`,`clm3`,`clm4`)
> );
> insert into nifi_test values ('1','2','3','4',1), ('11','12','13','14',100),
> ('21','22','23','24',0); {code}
> * Error Message
> {code:java}
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:95)
> 2024-01-04 10:55:09,064 ERROR [Timer-Driven Process Thread-91]
> o.a.nifi.processors.standard.ExecuteSQL
> ExecuteSQL[id=ce977968-018c-1000-e472-2fba6455ce07] Unable to execute SQL
> select query [SELECT * FROM nifi_test WHERE 1=1 LIMIT 10] for
> StandardFlowFileRecord[uuid=050b1e1c-85a4-440c-9dc4-27404cf1445d,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=170406508-74869,
> container=default, section=117], offset=0,
> length=46],offset=0,name=72fd1391-d014-4329-927b-fd037587ddbc,size=46]
> routing to failure
> org.apache.nifi.processor.exception.ProcessException:
> java.lang.RuntimeException: Unable to resolve union for value 1 with type
> java.lang.Integer while appending record {"clm1": "1", "clm2": "2", "clm3":
> "3", "clm4": "4","clm5": 1}
> at
> org.apache.nifi.processors.standard.AbstractExecuteSQL.lambda$onTrigger$4(AbstractExecuteSQL.java:336)
> at
> org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:3129)
> at
> org.apache.nifi.processors.standard.AbstractExecuteSQL.onTrigger(AbstractExecuteSQL.java:332)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1361)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:247)
> at
> org.apache.nifi.controller.scheduling.AbstractTimeBasedSchedulingAgent.lambda$doScheduleOnce$0(AbstractTimeBasedSchedulingAgent.java:59)
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750)
> Caused by: java.lang.RuntimeException: Unable to resolve union for value 1
> with type java.lang.Integer while appending record {"clm1": "1", "clm2": "2",
> "clm3": "3", "clm4": "4", "clm5": 1}
> at
> org.apache.nifi.util.db.JdbcCommon.convertToAvroStream(JdbcCommon.java:456)
> at
> org.apache.nifi.processors.standard.sql.DefaultAvroSqlWriter.writeResultSet(DefaultAvroSqlWriter.java:49)
> at
> org.apache.nifi.processors.standard.AbstractExecuteSQL.lambda$onTrigger$4(AbstractExecuteSQL.java:334)
> ... 14 common frames omitted
> Caused by: org.apache.avro.file.DataFileWriter$AppendWriteException:
> org.apache.avro.UnresolvedUnionException:Not in union ["null","long"]: 1
> (field=clm5)
> at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:317)
> at
> org.apache.nifi.util.db.JdbcCommon.convertToAvroStream(JdbcCommon.java:448)
> ... 16 common frames omitted
> Caused by: org.apache.avro.UnresolvedUnionException: Not in union
> ["null","long"]: 1 (field=clm5)
> at
> org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:247)
> at
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:234)
> at
>
Re: [PR] NIFI-12673 - ExtractKeywords processor [nifi]
exceptionfactory commented on code in PR #8302: URL: https://github.com/apache/nifi/pull/8302#discussion_r1467031775 ## nifi-python-extensions/nifi-text-embeddings-module/src/main/python/ExtractKeywords.py: ## @@ -0,0 +1,95 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import yake + +from nifiapi.flowfiletransform import FlowFileTransform, FlowFileTransformResult +from nifiapi.properties import PropertyDescriptor, StandardValidators, ExpressionLanguageScope + +class ExtractKeywords(FlowFileTransform): +class Java: +implements = ["org.apache.nifi.python.processor.FlowFileTransform"] + +class ProcessorDetails: +version = "2.0.0-SNAPSHOT" +description = """Parses the content of the incoming FlowFile to extract the keywords and add those as attributes +using the convention 'keyword.i' where i is a number depending on how many keywords are extracted. The attribute +'keywords.length' will be set with the number of extracted keywords. The attribute 'keyword.0' will contain the +list of all extracted keywords with the associated score.""" +tags = ["text", "keyword", "vector", "extract"] +dependencies = ['yake'] Review Comment: This dependency does not appear to be updated since 2021. https://pypi.org/project/yake/#history It is also licensed under GPLv3, which is not compatible with ASLv2. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-12500) Support dynamic host/port in Set/Get SNMP processors
[ https://issues.apache.org/jira/browse/NIFI-12500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-12500: Fix Version/s: 2.0.0-M2 (was: 2.0.0) > Support dynamic host/port in Set/Get SNMP processors > > > Key: NIFI-12500 > URL: https://issues.apache.org/jira/browse/NIFI-12500 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Lehel Boér >Assignee: Lehel Boér >Priority: Major > Fix For: 2.0.0-M2, 1.26.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > Currently GetSNMP initializes targets when the processor is first started. > Supporting dynamic hosts makes it possible to read hosts from flowfiles and > send SNMP GET requests to different targets dynamically. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12676) HandleHttpRequest Servlet Not Registered
[ https://issues.apache.org/jira/browse/NIFI-12676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-12676: Status: Patch Available (was: Open) > HandleHttpRequest Servlet Not Registered > > > Key: NIFI-12676 > URL: https://issues.apache.org/jira/browse/NIFI-12676 > Project: Apache NiFi > Issue Type: Bug >Reporter: David Handermann >Assignee: David Handermann >Priority: Critical > Fix For: 2.0.0-M2 > > Time Spent: 10m > Remaining Estimate: 0h > > Following recent upgrades to Jetty 12, the HandleHttpRequest Processor is not > registered the standard Servlet that it uses to handle HTTP requests. > The Jetty ServletContextHandler should be used together with a root path > mapping to allow HandleHttpRequest to process requests according to > configured properties. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12611) View State
[ https://issues.apache.org/jira/browse/NIFI-12611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-12611: Fix Version/s: 2.0.0-M2 > View State > -- > > Key: NIFI-12611 > URL: https://issues.apache.org/jira/browse/NIFI-12611 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Matt Gilman >Assignee: Matt Gilman >Priority: Major > Fix For: 2.0.0-M2 > > Time Spent: 1h > Remaining Estimate: 0h > > Introduce the ability to view and clear state for extension types that > support state. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] NIFI-12676 Fix Servlet Registration in HandleHttpRequest [nifi]
exceptionfactory opened a new pull request, #8304: URL: https://github.com/apache/nifi/pull/8304 # Summary [NIFI-12676](https://issues.apache.org/jira/browse/NIFI-12676) Corrects Servlet registration in the `HandleHttpRequest` Processor, which was not correctly configured following recent upgrades to Jetty 12. Changes include a new unit test method to ensure that the configured processing Servlet handles an HTTP request. # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [X] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [X] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [X] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [X] Pull Request based on current revision of the `main` branch - [X] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [X] Build completed using `mvn clean install -P contrib-check` - [X] JDK 21 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (NIFI-12676) HandleHttpRequest Servlet Not Registered
David Handermann created NIFI-12676: --- Summary: HandleHttpRequest Servlet Not Registered Key: NIFI-12676 URL: https://issues.apache.org/jira/browse/NIFI-12676 Project: Apache NiFi Issue Type: Bug Reporter: David Handermann Assignee: David Handermann Fix For: 2.0.0-M2 Following recent upgrades to Jetty 12, the HandleHttpRequest Processor is not registered the standard Servlet that it uses to handle HTTP requests. The Jetty ServletContextHandler should be used together with a root path mapping to allow HandleHttpRequest to process requests according to configured properties. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12663) Management Controller Services - General API error response handling
[ https://issues.apache.org/jira/browse/NIFI-12663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Gilman updated NIFI-12663: --- Description: Update the Management Controller Services page to show API error responses. (was: Update all pages to show API error response in a consistent way throughout the application.) > Management Controller Services - General API error response handling > > > Key: NIFI-12663 > URL: https://issues.apache.org/jira/browse/NIFI-12663 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Matt Gilman >Assignee: Matt Gilman >Priority: Major > > Update the Management Controller Services page to show API error responses. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12663) Management Controller Services - General API error response handling
[ https://issues.apache.org/jira/browse/NIFI-12663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Gilman updated NIFI-12663: --- Summary: Management Controller Services - General API error response handling (was: General API error response handling) > Management Controller Services - General API error response handling > > > Key: NIFI-12663 > URL: https://issues.apache.org/jira/browse/NIFI-12663 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Matt Gilman >Assignee: Matt Gilman >Priority: Major > > Update all pages to show API error response in a consistent way throughout > the application. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12611) View State
[ https://issues.apache.org/jira/browse/NIFI-12611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Gilman updated NIFI-12611: --- Resolution: Fixed Status: Resolved (was: Patch Available) > View State > -- > > Key: NIFI-12611 > URL: https://issues.apache.org/jira/browse/NIFI-12611 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Matt Gilman >Assignee: Matt Gilman >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > Introduce the ability to view and clear state for extension types that > support state. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12642 Added support for FileResourceService in PutS3Object [nifi]
turcsanyip commented on code in PR #8295:
URL: https://github.com/apache/nifi/pull/8295#discussion_r1466940824
##
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/PutS3Object.java:
##
@@ -501,6 +508,8 @@ public void onTrigger(final ProcessContext context, final
ProcessSession session
final FlowFile ff = flowFile;
final Map attributes = new HashMap<>();
final String ffFilename =
ff.getAttributes().get(CoreAttributes.FILENAME.key());
+final ResourceTransferSource resourceTransferSource =
ResourceTransferSource.valueOf(context.getProperty(RESOURCE_TRANSFER_SOURCE).getValue());
Review Comment:
Please use the recently added `PropertyValue.asAllowableValue()` method:
```suggestion
final ResourceTransferSource resourceTransferSource =
context.getProperty(RESOURCE_TRANSFER_SOURCE).asAllowableValue(ResourceTransferSource.class);
```
##
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/PutS3Object.java:
##
@@ -517,331 +526,328 @@ public void onTrigger(final ProcessContext context,
final ProcessSession session
/*
* Then
*/
-try {
-final FlowFile flowFileCopy = flowFile;
-session.read(flowFile, new InputStreamCallback() {
-@Override
-public void process(final InputStream in) throws IOException {
-final ObjectMetadata objectMetadata = new ObjectMetadata();
-objectMetadata.setContentLength(ff.getSize());
-
-final String contentType =
context.getProperty(CONTENT_TYPE)
-.evaluateAttributeExpressions(ff).getValue();
-if (contentType != null) {
-objectMetadata.setContentType(contentType);
-attributes.put(S3_CONTENT_TYPE, contentType);
-}
+final FlowFile flowFileCopy = session.clone(flowFile);
Review Comment:
I don't think we need to clone the FlowFile. What is the reason for that?
##
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/PutS3Object.java:
##
@@ -852,25 +858,27 @@ public void process(final InputStream in) throws
IOException {
final long millis =
TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - startNanos);
session.getProvenanceReporter().send(flowFile, url, millis);
-getLogger().info("Successfully put {} to Amazon S3 in {}
milliseconds", new Object[] {ff, millis});
+getLogger().info("Successfully put {} to Amazon S3 in {}
milliseconds", new Object[]{ff, millis});
try {
removeLocalState(cacheKey);
} catch (IOException e) {
getLogger().info("Error trying to delete key {} from cache:
{}",
new Object[]{cacheKey, e.getMessage()});
}
-} catch (final ProcessException | AmazonClientException pe) {
+
+} catch (final ProcessException | AmazonClientException | IOException
pe) {
Review Comment:
```suggestion
} catch (final ProcessException | AmazonClientException |
IOException e) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (NIFI-12611) View State
[ https://issues.apache.org/jira/browse/NIFI-12611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17811021#comment-17811021 ] ASF subversion and git services commented on NIFI-12611: Commit ecb87149fe6baea4a6707d98d491482461bcb8eb in nifi's branch refs/heads/main from Matt Gilman [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=ecb87149fe ] NIFI-12611: (#8296) - Component State. This closes #8296 > View State > -- > > Key: NIFI-12611 > URL: https://issues.apache.org/jira/browse/NIFI-12611 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Matt Gilman >Assignee: Matt Gilman >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > Introduce the ability to view and clear state for extension types that > support state. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12611: Component State [nifi]
rfellows merged PR #8296: URL: https://github.com/apache/nifi/pull/8296 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12611: Component State [nifi]
rfellows commented on code in PR #8296:
URL: https://github.com/apache/nifi/pull/8296#discussion_r1466894698
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-frontend/src/main/nifi/src/app/pages/flow-designer/service/canvas-utils.service.ts:
##
@@ -464,6 +464,28 @@ export class CanvasUtils {
return selection.size() === 1 && selection.classed('funnel');
}
+/**
+ * Determines whether the current selection is a stateful processor.
+ *
+ * @param {selection} selection
+ */
+public isStatefulProcessor(selection: any): boolean {
+// ensure the correct number of components are selected
+if (selection.size() !== 1) {
+return false;
+}
+if (this.canRead(selection) === false || this.canModify(selection) ===
false) {
+return false;
+}
+
+if (this.isProcessor(selection)) {
+const processorData: any = selection.datum();
+return processorData.component.persistsState === true;
+} else {
+return false;
+}
+}
Review Comment:
This name is fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-12611: Component State [nifi]
mcgilman commented on code in PR #8296:
URL: https://github.com/apache/nifi/pull/8296#discussion_r1466836265
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-frontend/src/main/nifi/src/app/pages/flow-designer/service/canvas-utils.service.ts:
##
@@ -464,6 +464,28 @@ export class CanvasUtils {
return selection.size() === 1 && selection.classed('funnel');
}
+/**
+ * Determines whether the current selection is a stateful processor.
+ *
+ * @param {selection} selection
+ */
+public isStatefulProcessor(selection: any): boolean {
+// ensure the correct number of components are selected
+if (selection.size() !== 1) {
+return false;
+}
+if (this.canRead(selection) === false || this.canModify(selection) ===
false) {
+return false;
+}
+
+if (this.isProcessor(selection)) {
+const processorData: any = selection.datum();
+return processorData.component.persistsState === true;
+} else {
+return false;
+}
+}
Review Comment:
This naming and logic was brought over from NiFi without much thought. I can
rename the method if desirable. The logic is correct. We want to verify the
selection contains a single Processor, the user has access, and it persists
state.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-12611: Component State [nifi]
mcgilman commented on code in PR #8296:
URL: https://github.com/apache/nifi/pull/8296#discussion_r1466833188
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-frontend/src/main/nifi/src/app/ui/common/component-state/component-state.component.html:
##
@@ -0,0 +1,86 @@
+
+
+
+Component State
+
+
+
+Name
+{{ componentName }}
+
+
+Description
+
+{{ stateDescription }}
+
+
+
+
+Displaying {{ filteredEntries }} of {{
totalEntries }}
+
+
+Filter
+
+
+
+
+Clear state
+
Review Comment:
This was a conscious decision.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [NIFI-12655] theme NiFi following Material Design spec and leveraging Angular Material themes [nifi]
rfellows commented on code in PR #8294:
URL: https://github.com/apache/nifi/pull/8294#discussion_r1466795410
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-frontend/src/main/nifi/src/app/pages/flow-designer/service/manager/process-group-manager.service.ts:
##
@@ -1075,22 +1075,22 @@ export class ProcessGroupManager {
const versionControl = processGroup
.select('text.version-control')
.style('visibility',
self.isUnderVersionControl(processGroupData) ? 'visible' : 'hidden')
-.style('fill', function () {
+.style('class', function () {
Review Comment:
i think you intended this to be
```suggestion
.attr('class', function () {
```
however, when doing that, it would wipe out the `version-control` class on
the same element (used when determining the selection on line 1076. Might need
to return both with this approach or find a way to concat with the existing
class(es)
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-frontend/src/main/nifi/src/app/pages/flow-designer/service/manager/processor-manager.service.ts:
##
@@ -669,20 +669,20 @@ export class ProcessorManager {
// update the run status
updated
.select('text.run-status-icon')
-.attr('fill', function (d: any) {
-let fill: string = '#728e9b';
+.attr('class', function (d: any) {
+let clazz: string = 'primary-500';
if (d.status.aggregateSnapshot.runStatus === 'Validating') {
-fill = '#a8a8a8';
+clazz = 'warn-contrast-300';
} else if (d.status.aggregateSnapshot.runStatus === 'Invalid')
{
-fill = '#cf9f5d';
+clazz = 'accent-A200';
} else if (d.status.aggregateSnapshot.runStatus === 'Running')
{
-fill = '#7dc7a0';
+clazz = 'accent-200';
} else if (d.status.aggregateSnapshot.runStatus === 'Stopped')
{
-fill = '#d18686';
+clazz = 'warn-200';
}
-return fill;
+return clazz;
Review Comment:
I think this is overwriting the `run-status-icon` class already on this
element.
It seems to be preventing the icon from updating when the processor is
started/stopped.
Ports have the same issue.
probably want something like:
```
.attr('class', function (d: any) {
let clazz: string = 'primary-500';
if (d.status.aggregateSnapshot.runStatus === 'Validating') {
clazz = 'warn-contrast-300';
} else if (d.status.aggregateSnapshot.runStatus ===
'Invalid') {
clazz = 'accent-A200';
} else if (d.status.aggregateSnapshot.runStatus ===
'Running') {
clazz = 'accent-200';
} else if (d.status.aggregateSnapshot.runStatus ===
'Stopped') {
clazz = 'warn-200';
}
return `run-status-icon ${clazz}`;
})
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (NIFI-12662) Conduct Apache NiFi 1.25.0 Release
[ https://issues.apache.org/jira/browse/NIFI-12662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17811008#comment-17811008 ] ASF subversion and git services commented on NIFI-12662: Commit 6ecc398d3f92425447e43242af4992757e25b3c5 in nifi's branch refs/heads/NIFI-12662-RC1 from Pierre Villard [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=6ecc398d3f ] NIFI-12662-RC1 prepare release nifi-1.25.0-RC1 > Conduct Apache NiFi 1.25.0 Release > -- > > Key: NIFI-12662 > URL: https://issues.apache.org/jira/browse/NIFI-12662 > Project: Apache NiFi > Issue Type: Task >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > Fix For: 1.25.0 > > > Conduct Apache NiFi 1.25.0 Release -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12662) Conduct Apache NiFi 1.25.0 Release
[ https://issues.apache.org/jira/browse/NIFI-12662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17811009#comment-17811009 ] ASF subversion and git services commented on NIFI-12662: Commit 65c33d08aac4a4563e030a515f13463dad8d4ee6 in nifi's branch refs/heads/NIFI-12662-RC1 from Pierre Villard [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=65c33d08aa ] NIFI-12662-RC1 prepare for next development iteration > Conduct Apache NiFi 1.25.0 Release > -- > > Key: NIFI-12662 > URL: https://issues.apache.org/jira/browse/NIFI-12662 > Project: Apache NiFi > Issue Type: Task >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > Fix For: 1.25.0 > > > Conduct Apache NiFi 1.25.0 Release -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [NIFI-12655] theme NiFi following Material Design spec and leveraging Angular Material themes [nifi]
scottyaslan commented on PR #8294: URL: https://github.com/apache/nifi/pull/8294#issuecomment-1910695264 > Looks pretty good to me. Left a couple of comments inline. The only thing that really stands out to me but I didn't understand how it all works to really comment inline... the text on primary buttons is too dark IMO. Makes the text hard to read. Can we customize the text color in those scenarios? Let me take a look and see what else we can do here... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] NIFI-12648 Refactor components in elasticsearch bundle using current API methods [nifi]
EndzeitBegins opened a new pull request, #8303: URL: https://github.com/apache/nifi/pull/8303 # Summary [NIFI-12648](https://issues.apache.org/jira/browse/NIFI-12648) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [x] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [x] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [x] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [x] Pull Request based on current revision of the `main` branch - [x] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [x] Build completed using `mvn clean install -P contrib-check` - [x] JDK 21 ### Licensing - [x] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [x] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [x] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (NIFI-12675) Python Processor erroring when creating custom relationships
Mark Payne created NIFI-12675:
-
Summary: Python Processor erroring when creating custom
relationships
Key: NIFI-12675
URL: https://issues.apache.org/jira/browse/NIFI-12675
Project: Apache NiFi
Issue Type: Bug
Components: Core Framework
Reporter: Mark Payne
Assignee: Mark Payne
>From apache Slack thread
>([https://apachenifi.slack.com/archives/C0L9VCD47/p1706176890922519):]
{quote}Hello, I am trying to test some custom python processors with nifi
2.0.0-M1
It works fine except when I try to add custom relationships to it (other than
the default success, failure and original).
Here's what I am trying:
{code:java}
self.matched = Relationship("matched", "flowfiles having a match with
the regex")
self.unmatched = Relationship("unmatched", "flowfiles not having any
match with regex")
self.failure = Relationship("failure", "flowfiles for which process
errored while matching")
self.relationships = {self.matched, self.unmatched, self.failure}
{code}
I get py4j complaining about AttributeError: 'set' object has no attribute
'_get_object_id'
which seems like the auto conversion of Python to java container is not
happening for "Relationship" class. Any idea what could be wrong here?
{quote}
The problem appears to be that Relationships created are of type
{{nifiapi.Relationship}} but that is being sent back to the Java side without
being converted into a {{org.apache.nifi.processor.Relationship}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] NIFI-11520: Add a menu to display Flow Analysis report results [nifi]
tpalfy commented on code in PR #8273:
URL: https://github.com/apache/nifi/pull/8273#discussion_r142453
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/js/nf/canvas/controllers/nf-ng-canvas-flow-status-controller.js:
##
@@ -400,6 +406,554 @@
}
}
+/**
+ * The flow analysis controller.
+ */
+
+this.flowAnalysis = {
+
+/**
+ * Create the list of rule violations
+ */
+buildRuleViolationsList: function(rules, violationsAndRecs) {
+var ruleViolationCountEl = $('#rule-violation-count');
+var ruleViolationListEl = $('#rule-violations-list');
+var ruleWarningCountEl = $('#rule-warning-count');
+var ruleWarningListEl = $('#rule-warnings-list');
+var violations = violationsAndRecs.filter(function
(violation) {
+return violation.enforcementPolicy === 'ENFORCE'
+});
+var warnings = violationsAndRecs.filter(function
(violation) {
+return violation.enforcementPolicy === 'WARN'
+});
+ruleViolationCountEl.empty().text('(' + violations.length
+ ')');
+ruleWarningCountEl.empty().text('(' + warnings.length +
')');
+ruleViolationListEl.empty();
+violations.forEach(function(violation) {
+var rule = rules.find(function(rule) {
+return rule.id === violation.ruleId;
+});
+// create DOM elements
+var violationListItemEl = $('');
+var violationEl = $('');
+var violationListItemWrapperEl = $('');
+var violationRuleEl = $('');
+var violationListItemNameEl = $('');
+var violationListItemIdEl = $('');
+var violationInfoButtonEl = $('');
+
+// add text content and button data
+$(violationRuleEl).text(rule.name);
+
$(violationListItemNameEl).text(violation.subjectDisplayName);
+$(violationListItemIdEl).text(violation.subjectId);
+
$(violationListItemEl).append(violationRuleEl).append(violationListItemWrapperEl);
+$(violationInfoButtonEl).data('violationInfo',
violation);
+
+// build list DOM structure
+
violationListItemWrapperEl.append(violationListItemNameEl).append(violationListItemIdEl);
+
violationEl.append(violationListItemWrapperEl).append(violationInfoButtonEl);
+
violationListItemEl.append(violationRuleEl).append(violationEl)
+ruleViolationListEl.append(violationListItemEl);
+});
+
+warnings.forEach(function(warning) {
+var rule = rules.find(function(rule) {
+return rule.id === warning.ruleId;
+});
+// create DOM elements
+var warningListItemEl = $('');
+var warningEl = $('');
+var warningListItemWrapperEl = $('');
+var warningRuleEl = $('');
+var warningListItemNameEl = $('');
+var warningListItemIdEl = $('');
+var warningInfoButtonEl = $('');
+
+// add text content and button data
+$(warningRuleEl).text(rule.name);
+
$(warningListItemNameEl).text(warning.subjectDisplayName);
+$(warningListItemIdEl).text(warning.subjectId);
+
$(warningListItemEl).append(warningRuleEl).append(warningListItemWrapperEl);
+$(warningInfoButtonEl).data('violationInfo', warning);
+
+// build list DOM structure
+
warningListItemWrapperEl.append(warningListItemNameEl).append(warningListItemIdEl);
+
warningEl.append(warningListItemWrapperEl).append(warningInfoButtonEl);
+
warningListItemEl.append(warningRuleEl).append(warningEl)
+ruleWarningListEl.append(warningListItemEl);
+});
+},
+
+/**
+ *
+ * Render a new list when it differs from the previous
violations response
+ */
+buildRuleViolations: function(rules, violations) {
+if
Re: [PR] NIFI-12611: Component State [nifi]
rfellows commented on code in PR #8296:
URL: https://github.com/apache/nifi/pull/8296#discussion_r1466653000
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-frontend/src/main/nifi/src/app/ui/common/component-state/component-state.component.html:
##
@@ -0,0 +1,86 @@
+
+
+
+Component State
+
+
+
+Name
+{{ componentName }}
+
+
+Description
+
+{{ stateDescription }}
+
+
+
+
+Displaying {{ filteredEntries }} of {{
totalEntries }}
+
+
+Filter
+
+
+
+
+Clear state
+
Review Comment:
NiFi actually shows the clear link but disabled rather than hiding it. I
think I prefer hiding it as well, just wanted to make sure this was a conscious
decision to deviate from NiFi.
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-frontend/src/main/nifi/src/app/pages/flow-designer/service/canvas-utils.service.ts:
##
@@ -464,6 +464,28 @@ export class CanvasUtils {
return selection.size() === 1 && selection.classed('funnel');
}
+/**
+ * Determines whether the current selection is a stateful processor.
+ *
+ * @param {selection} selection
+ */
+public isStatefulProcessor(selection: any): boolean {
+// ensure the correct number of components are selected
+if (selection.size() !== 1) {
+return false;
+}
+if (this.canRead(selection) === false || this.canModify(selection) ===
false) {
+return false;
+}
+
+if (this.isProcessor(selection)) {
+const processorData: any = selection.datum();
+return processorData.component.persistsState === true;
+} else {
+return false;
+}
+}
Review Comment:
This name is a bit misleading. It does more than just determines if
something is a stateful processor or not. Technically, a process can be
stateful even if you can't read/write it.
It probably doesn't matter though. We aren't really using this in any other
context.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Resolved] (NIFI-12590) Add an option to add a prefix on Kubernetes resources
[
https://issues.apache.org/jira/browse/NIFI-12590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julien G. resolved NIFI-12590.
--
Resolution: Resolved
> Add an option to add a prefix on Kubernetes resources
> -
>
> Key: NIFI-12590
> URL: https://issues.apache.org/jira/browse/NIFI-12590
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 2.0.0-M2
>Reporter: Julien G.
>Assignee: Julien G.
>Priority: Major
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> In case of using Kubernetes as a *Cluster Leader Election* and *State
> Mangement*, NiFi will create two types of resources: *ConfigMaps* and
> *Leases*.
> 2 *Leases* for the _Cluster Leader Election_ and multiple *ConfigMaps*
> depending the number of components which their state to be stored.
> Their names are normalized:
> - *Leases*: {{cluster-coordinator}} and {{primary-node}}.
> - *ConfigMap*: {{nifi-component-%s}} with {{%s}} the component identifier.
> But currently, if someone wants to deploy more than one cluster in one
> namespace, conflicts will arise. Because the resources will share their
> names. It has been done this way to prevent security concerns where a cluster
> will manipulate resources of another cluster.
> But as discussed on Slack, these concerns can't prevented with 100% certainty.
> So users should be able to configure a prefix to add to these resources to
> prevent conflict and assume the security concerns.
> It can be done by replicating the logic done with ZooKeeper by adding a
> property in {{nifi.properties}} and a new field in the
> {{state_management.xml}} file.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-12590) Add an option to add a prefix on Kubernetes resources
[
https://issues.apache.org/jira/browse/NIFI-12590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julien G. updated NIFI-12590:
-
Fix Version/s: 2.0.0-M2
> Add an option to add a prefix on Kubernetes resources
> -
>
> Key: NIFI-12590
> URL: https://issues.apache.org/jira/browse/NIFI-12590
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 2.0.0-M2
>Reporter: Julien G.
>Assignee: Julien G.
>Priority: Major
> Fix For: 2.0.0-M2
>
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> In case of using Kubernetes as a *Cluster Leader Election* and *State
> Mangement*, NiFi will create two types of resources: *ConfigMaps* and
> *Leases*.
> 2 *Leases* for the _Cluster Leader Election_ and multiple *ConfigMaps*
> depending the number of components which their state to be stored.
> Their names are normalized:
> - *Leases*: {{cluster-coordinator}} and {{primary-node}}.
> - *ConfigMap*: {{nifi-component-%s}} with {{%s}} the component identifier.
> But currently, if someone wants to deploy more than one cluster in one
> namespace, conflicts will arise. Because the resources will share their
> names. It has been done this way to prevent security concerns where a cluster
> will manipulate resources of another cluster.
> But as discussed on Slack, these concerns can't prevented with 100% certainty.
> So users should be able to configure a prefix to add to these resources to
> prevent conflict and assume the security concerns.
> It can be done by replicating the logic done with ZooKeeper by adding a
> property in {{nifi.properties}} and a new field in the
> {{state_management.xml}} file.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] NIFI-12643 Added support for FileResourceService in PutGCSObject [nifi]
turcsanyip commented on code in PR #8281:
URL: https://github.com/apache/nifi/pull/8281#discussion_r1466618437
##
nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/storage/PutGCSObject.java:
##
@@ -322,199 +328,196 @@ public void onTrigger(final ProcessContext context,
final ProcessSession session
return;
}
-final long startNanos = System.nanoTime();
+try {
+final long startNanos = System.nanoTime();
+
+final String bucket = context.getProperty(BUCKET)
+.evaluateAttributeExpressions(flowFile)
+.getValue();
+final String key = context.getProperty(KEY)
+.evaluateAttributeExpressions(flowFile)
+.getValue();
+final boolean overwrite =
context.getProperty(OVERWRITE).asBoolean();
+
+final FlowFile ff = flowFile;
+final String ffFilename =
ff.getAttributes().get(CoreAttributes.FILENAME.key());
+final Map attributes = new HashMap<>();
+final ResourceTransferSource resourceTransferSource =
ResourceTransferSource.valueOf(context.getProperty(RESOURCE_TRANSFER_SOURCE).getValue());
Review Comment:
Please use the recently added `PropertyValue.asAllowableValue()` method:
```suggestion
final ResourceTransferSource resourceTransferSource =
context.getProperty(RESOURCE_TRANSFER_SOURCE).asAllowableValue(ResourceTransferSource.class);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-11520: Add a menu to display Flow Analysis report results [nifi]
tpalfy commented on code in PR #8273:
URL: https://github.com/apache/nifi/pull/8273#discussion_r1463631993
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/js/nf/canvas/controllers/nf-ng-canvas-flow-status-controller.js:
##
@@ -400,6 +406,554 @@
}
}
+/**
+ * The flow analysis controller.
+ */
+
+this.flowAnalysis = {
+
+/**
+ * Create the list of rule violations
+ */
+buildRuleViolationsList: function(rules, violationsAndRecs) {
+var ruleViolationCountEl = $('#rule-violation-count');
+var ruleViolationListEl = $('#rule-violations-list');
+var ruleWarningCountEl = $('#rule-warning-count');
+var ruleWarningListEl = $('#rule-warnings-list');
+var violations = violationsAndRecs.filter(function
(violation) {
+return violation.enforcementPolicy === 'ENFORCE'
+});
+var warnings = violationsAndRecs.filter(function
(violation) {
+return violation.enforcementPolicy === 'WARN'
+});
+ruleViolationCountEl.empty().text('(' + violations.length
+ ')');
+ruleWarningCountEl.empty().text('(' + warnings.length +
')');
+ruleViolationListEl.empty();
+violations.forEach(function(violation) {
+var rule = rules.find(function(rule) {
+return rule.id === violation.ruleId;
+});
+// create DOM elements
+var violationListItemEl = $('');
+var violationEl = $('');
+var violationListItemWrapperEl = $('');
+var violationRuleEl = $('');
+var violationListItemNameEl = $('');
+var violationListItemIdEl = $('');
+var violationInfoButtonEl = $('');
+
+// add text content and button data
+$(violationRuleEl).text(rule.name);
+
$(violationListItemNameEl).text(violation.subjectDisplayName);
+$(violationListItemIdEl).text(violation.subjectId);
+
$(violationListItemEl).append(violationRuleEl).append(violationListItemWrapperEl);
+$(violationInfoButtonEl).data('violationInfo',
violation);
+
+// build list DOM structure
+
violationListItemWrapperEl.append(violationListItemNameEl).append(violationListItemIdEl);
+
violationEl.append(violationListItemWrapperEl).append(violationInfoButtonEl);
+
violationListItemEl.append(violationRuleEl).append(violationEl)
+ruleViolationListEl.append(violationListItemEl);
+});
+
+warnings.forEach(function(warning) {
+var rule = rules.find(function(rule) {
+return rule.id === warning.ruleId;
+});
+// create DOM elements
+var warningListItemEl = $('');
+var warningEl = $('');
+var warningListItemWrapperEl = $('');
+var warningRuleEl = $('');
+var warningListItemNameEl = $('');
+var warningListItemIdEl = $('');
+var warningInfoButtonEl = $('');
+
+// add text content and button data
+$(warningRuleEl).text(rule.name);
+
$(warningListItemNameEl).text(warning.subjectDisplayName);
+$(warningListItemIdEl).text(warning.subjectId);
+
$(warningListItemEl).append(warningRuleEl).append(warningListItemWrapperEl);
+$(warningInfoButtonEl).data('violationInfo', warning);
+
+// build list DOM structure
+
warningListItemWrapperEl.append(warningListItemNameEl).append(warningListItemIdEl);
+
warningEl.append(warningListItemWrapperEl).append(warningInfoButtonEl);
+
warningListItemEl.append(warningRuleEl).append(warningEl)
+ruleWarningListEl.append(warningListItemEl);
+});
+},
+
+/**
+ *
+ * Render a new list when it differs from the previous
violations response
+ */
+buildRuleViolations: function(rules, violations) {
+if
[jira] [Updated] (MINIFICPP-2291) Site-to-site client should treat TRANSACTION_FINISHED_BUT_DESTINATION_FULL as (qualified) success
[
https://issues.apache.org/jira/browse/MINIFICPP-2291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marton Szasz updated MINIFICPP-2291:
Fix Version/s: 0.16.0
> Site-to-site client should treat TRANSACTION_FINISHED_BUT_DESTINATION_FULL as
> (qualified) success
> -
>
> Key: MINIFICPP-2291
> URL: https://issues.apache.org/jira/browse/MINIFICPP-2291
> Project: Apache NiFi MiNiFi C++
> Issue Type: Bug
>Reporter: Ferenc Gerlits
>Assignee: Ferenc Gerlits
>Priority: Minor
> Fix For: 0.16.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Currently, the site-to-site client in MiNiFi only accepts
> {{TRANSACTION_FINISHED(13)}} as a success code, and it treats everything else
> as failure:
> https://github.com/apache/nifi-minifi-cpp/blob/9e4ef8d59dbbe345c889edb2f3ed13c98513c96d/libminifi/src/sitetosite/SiteToSiteClient.cpp#L385
> {code:c++}
> if (code == TRANSACTION_FINISHED) {
> logger_->log_info("Site2Site transaction {} peer finished transaction",
> transactionID.to_string());
> transaction->_state = TRANSACTION_COMPLETED;
> return true;
> } else {
> logger_->log_warn("Site2Site transaction {} peer unknown respond code {}",
> transactionID.to_string(), magic_enum::enum_underlying(code));
> return false;
> }{code}
> On the other hand, the NiFi site-to-site client treats both
> {{TRANSACTION_FINISHED(13)}} and
> {{TRANSACTION_FINISHED_BUT_DESTINATION_FULL(14)}} as success codes:
> https://github.com/apache/nifi/blob/3fcf5f25b4c705f6909d058d81a7b8b6d6a648f1/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-site-to-site/src/main/java/org/apache/nifi/remote/protocol/http/StandardHttpFlowFileServerProtocol.java#L120
> {code:java}
> switch (response) {
> case CONFIRM_TRANSACTION:
> logger.debug("{} Confirming transaction. checksum={}",
> this, explanation);
> commSession.setChecksum(explanation);
>
> commSession.setStatus(Transaction.TransactionState.DATA_EXCHANGED);
> break;
> case TRANSACTION_FINISHED:
> case TRANSACTION_FINISHED_BUT_DESTINATION_FULL:
> logger.debug("{} Transaction is completed.
> responseCode={}", this, response);
>
> commSession.setStatus(Transaction.TransactionState.TRANSACTION_COMPLETED);
> break;
> }{code}
> https://github.com/apache/nifi/blob/3fcf5f25b4c705f6909d058d81a7b8b6d6a648f1/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-site-to-site/src/main/java/org/apache/nifi/remote/protocol/AbstractFlowFileServerProtocol.java#L371
> {code:java}
> logger.debug("{} received {} from {}", new Object[]{this,
> transactionResponse, peer});
> if (transactionResponse.getCode() ==
> ResponseCode.TRANSACTION_FINISHED_BUT_DESTINATION_FULL) {
> peer.penalize(port.getIdentifier(),
> port.getYieldPeriod(TimeUnit.MILLISECONDS));
> } else if (transactionResponse.getCode() !=
> ResponseCode.TRANSACTION_FINISHED) {
> throw new ProtocolException("After sending data, expected
> TRANSACTION_FINISHED response but got " + transactionResponse);
> }{code}
> {{TRANSACTION_FINISHED_BUT_DESTINATION_FULL(14)}} can happen when the number
> or the total size of the files in the batch exceed the limits set in the Back
> Pressure settings of the outgoing connection of the input port on the server
> side.
> MiNiFi should behave in the same way as NiFi: accept
> {{TRANSACTION_FINISHED_BUT_DESTINATION_FULL(14)}} as success, but the remote
> processor group should yield, so the server has time to clear the connection.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (MINIFICPP-2291) Site-to-site client should treat TRANSACTION_FINISHED_BUT_DESTINATION_FULL as (qualified) success
[
https://issues.apache.org/jira/browse/MINIFICPP-2291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marton Szasz resolved MINIFICPP-2291.
-
Resolution: Fixed
> Site-to-site client should treat TRANSACTION_FINISHED_BUT_DESTINATION_FULL as
> (qualified) success
> -
>
> Key: MINIFICPP-2291
> URL: https://issues.apache.org/jira/browse/MINIFICPP-2291
> Project: Apache NiFi MiNiFi C++
> Issue Type: Bug
>Reporter: Ferenc Gerlits
>Assignee: Ferenc Gerlits
>Priority: Minor
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Currently, the site-to-site client in MiNiFi only accepts
> {{TRANSACTION_FINISHED(13)}} as a success code, and it treats everything else
> as failure:
> https://github.com/apache/nifi-minifi-cpp/blob/9e4ef8d59dbbe345c889edb2f3ed13c98513c96d/libminifi/src/sitetosite/SiteToSiteClient.cpp#L385
> {code:c++}
> if (code == TRANSACTION_FINISHED) {
> logger_->log_info("Site2Site transaction {} peer finished transaction",
> transactionID.to_string());
> transaction->_state = TRANSACTION_COMPLETED;
> return true;
> } else {
> logger_->log_warn("Site2Site transaction {} peer unknown respond code {}",
> transactionID.to_string(), magic_enum::enum_underlying(code));
> return false;
> }{code}
> On the other hand, the NiFi site-to-site client treats both
> {{TRANSACTION_FINISHED(13)}} and
> {{TRANSACTION_FINISHED_BUT_DESTINATION_FULL(14)}} as success codes:
> https://github.com/apache/nifi/blob/3fcf5f25b4c705f6909d058d81a7b8b6d6a648f1/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-site-to-site/src/main/java/org/apache/nifi/remote/protocol/http/StandardHttpFlowFileServerProtocol.java#L120
> {code:java}
> switch (response) {
> case CONFIRM_TRANSACTION:
> logger.debug("{} Confirming transaction. checksum={}",
> this, explanation);
> commSession.setChecksum(explanation);
>
> commSession.setStatus(Transaction.TransactionState.DATA_EXCHANGED);
> break;
> case TRANSACTION_FINISHED:
> case TRANSACTION_FINISHED_BUT_DESTINATION_FULL:
> logger.debug("{} Transaction is completed.
> responseCode={}", this, response);
>
> commSession.setStatus(Transaction.TransactionState.TRANSACTION_COMPLETED);
> break;
> }{code}
> https://github.com/apache/nifi/blob/3fcf5f25b4c705f6909d058d81a7b8b6d6a648f1/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-site-to-site/src/main/java/org/apache/nifi/remote/protocol/AbstractFlowFileServerProtocol.java#L371
> {code:java}
> logger.debug("{} received {} from {}", new Object[]{this,
> transactionResponse, peer});
> if (transactionResponse.getCode() ==
> ResponseCode.TRANSACTION_FINISHED_BUT_DESTINATION_FULL) {
> peer.penalize(port.getIdentifier(),
> port.getYieldPeriod(TimeUnit.MILLISECONDS));
> } else if (transactionResponse.getCode() !=
> ResponseCode.TRANSACTION_FINISHED) {
> throw new ProtocolException("After sending data, expected
> TRANSACTION_FINISHED response but got " + transactionResponse);
> }{code}
> {{TRANSACTION_FINISHED_BUT_DESTINATION_FULL(14)}} can happen when the number
> or the total size of the files in the batch exceed the limits set in the Back
> Pressure settings of the outgoing connection of the input port on the server
> side.
> MiNiFi should behave in the same way as NiFi: accept
> {{TRANSACTION_FINISHED_BUT_DESTINATION_FULL(14)}} as success, but the remote
> processor group should yield, so the server has time to clear the connection.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] MINIFICPP-2291 Fix site-to-site transfer of large files [nifi-minifi-cpp]
szaszm closed pull request #1720: MINIFICPP-2291 Fix site-to-site transfer of large files URL: https://github.com/apache/nifi-minifi-cpp/pull/1720 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12643 Added support for FileResourceService in PutGCSObject [nifi]
balazsgerner commented on code in PR #8281:
URL: https://github.com/apache/nifi/pull/8281#discussion_r1466548160
##
nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/storage/PutGCSObject.java:
##
@@ -291,6 +295,8 @@ public List
getSupportedPropertyDescriptors() {
descriptors.add(BUCKET);
descriptors.add(KEY);
descriptors.add(CONTENT_TYPE);
+descriptors.add(RESOURCE_TRANSFER_SOURCE);
+descriptors.add(FILE_RESOURCE_SERVICE);
descriptors.add(CRC32C);
descriptors.add(ACL);
Review Comment:
thanks, committed the suggestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (NIFI-12674) Modify ValidateCSV to make schema optional
Reynaldo Rea created NIFI-12674: --- Summary: Modify ValidateCSV to make schema optional Key: NIFI-12674 URL: https://issues.apache.org/jira/browse/NIFI-12674 Project: Apache NiFi Issue Type: Improvement Components: Extensions Reporter: Reynaldo Rea The schema for ValidateCSV should be optional. Whenever a schema is not supplied, the processor should only validate the structural consistency of the CSV. In addition, if a schema is not provided, the processor should attempt to derive it from the header row. The processor should also be able to validate against the flowfile content or a specified flowfile attribute. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12500) Support dynamic host/port in Set/Get SNMP processors
[ https://issues.apache.org/jira/browse/NIFI-12500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard updated NIFI-12500: -- Fix Version/s: 1.26.0 > Support dynamic host/port in Set/Get SNMP processors > > > Key: NIFI-12500 > URL: https://issues.apache.org/jira/browse/NIFI-12500 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Lehel Boér >Assignee: Lehel Boér >Priority: Major > Fix For: 2.0.0, 1.26.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > Currently GetSNMP initializes targets when the processor is first started. > Supporting dynamic hosts makes it possible to read hosts from flowfiles and > send SNMP GET requests to different targets dynamically. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12500) Support dynamic host/port in Set/Get SNMP processors
[ https://issues.apache.org/jira/browse/NIFI-12500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard updated NIFI-12500: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Support dynamic host/port in Set/Get SNMP processors > > > Key: NIFI-12500 > URL: https://issues.apache.org/jira/browse/NIFI-12500 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Lehel Boér >Assignee: Lehel Boér >Priority: Major > Fix For: 2.0.0, 1.26.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > Currently GetSNMP initializes targets when the processor is first started. > Supporting dynamic hosts makes it possible to read hosts from flowfiles and > send SNMP GET requests to different targets dynamically. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12500) Support dynamic host/port in Set/Get SNMP processors
[ https://issues.apache.org/jira/browse/NIFI-12500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-12500: Fix Version/s: 2.0.0 > Support dynamic host/port in Set/Get SNMP processors > > > Key: NIFI-12500 > URL: https://issues.apache.org/jira/browse/NIFI-12500 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Lehel Boér >Assignee: Lehel Boér >Priority: Major > Fix For: 2.0.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > Currently GetSNMP initializes targets when the processor is first started. > Supporting dynamic hosts makes it possible to read hosts from flowfiles and > send SNMP GET requests to different targets dynamically. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12643 Added support for FileResourceService in PutGCSObject [nifi]
turcsanyip commented on code in PR #8281:
URL: https://github.com/apache/nifi/pull/8281#discussion_r1466456472
##
nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/storage/PutGCSObject.java:
##
@@ -291,6 +295,8 @@ public List
getSupportedPropertyDescriptors() {
descriptors.add(BUCKET);
descriptors.add(KEY);
descriptors.add(CONTENT_TYPE);
+descriptors.add(RESOURCE_TRANSFER_SOURCE);
+descriptors.add(FILE_RESOURCE_SERVICE);
descriptors.add(CRC32C);
descriptors.add(ACL);
Review Comment:
I would move the new properties up, before `CONTENT_TYPE`. In that way, the
properties would be "grouped" like:
- target (Bucket, Key)
- source
- metadata (Content Type, CRC, etc)
```suggestion
descriptors.add(BUCKET);
descriptors.add(KEY);
descriptors.add(RESOURCE_TRANSFER_SOURCE);
descriptors.add(FILE_RESOURCE_SERVICE);
descriptors.add(CONTENT_TYPE);
descriptors.add(CRC32C);
descriptors.add(ACL);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-12500: Add dynamic target for Get/Set/SendTrapSNMP [nifi]
tpalfy commented on PR #8160: URL: https://github.com/apache/nifi/pull/8160#issuecomment-1910301209 Fixed some Java 8 compatibility issues and pushed to nifi-1.x -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-12500) Support dynamic host/port in Set/Get SNMP processors
[ https://issues.apache.org/jira/browse/NIFI-12500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17810914#comment-17810914 ] ASF subversion and git services commented on NIFI-12500: Commit c441f90cfcf332b367f32a57074dc975cf200e14 in nifi's branch refs/heads/support/nifi-1.x from lehelb [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=c441f90cfc ] NIFI-12500: Add dynamic target for Get/Set/SendTrapSNMP This closes #8160. Signed-off-by: Tamas Palfy (cherry picked from commit 7fc27651a45c21f1e47f33efbba5649b06113da0) > Support dynamic host/port in Set/Get SNMP processors > > > Key: NIFI-12500 > URL: https://issues.apache.org/jira/browse/NIFI-12500 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Lehel Boér >Assignee: Lehel Boér >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > > Currently GetSNMP initializes targets when the processor is first started. > Supporting dynamic hosts makes it possible to read hosts from flowfiles and > send SNMP GET requests to different targets dynamically. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12500: Add dynamic target for Get/Set/SendTrapSNMP [nifi]
tpalfy commented on PR #8160: URL: https://github.com/apache/nifi/pull/8160#issuecomment-1910260542 LGTM Thank you for your work @Lehel44 . Merged to main, merging to 1.x. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-12500) Support dynamic host/port in Set/Get SNMP processors
[ https://issues.apache.org/jira/browse/NIFI-12500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17810903#comment-17810903 ] ASF subversion and git services commented on NIFI-12500: Commit 7fc27651a45c21f1e47f33efbba5649b06113da0 in nifi's branch refs/heads/main from lehelb [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=7fc27651a4 ] NIFI-12500: Add dynamic target for Get/Set/SendTrapSNMP This closes #8160. Signed-off-by: Tamas Palfy > Support dynamic host/port in Set/Get SNMP processors > > > Key: NIFI-12500 > URL: https://issues.apache.org/jira/browse/NIFI-12500 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Lehel Boér >Assignee: Lehel Boér >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > Currently GetSNMP initializes targets when the processor is first started. > Supporting dynamic hosts makes it possible to read hosts from flowfiles and > send SNMP GET requests to different targets dynamically. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12500: Add dynamic target for Get/Set/SendTrapSNMP [nifi]
asfgit closed pull request #8160: NIFI-12500: Add dynamic target for Get/Set/SendTrapSNMP URL: https://github.com/apache/nifi/pull/8160 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-12673) Add ExtractKeywords processor
[ https://issues.apache.org/jira/browse/NIFI-12673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard updated NIFI-12673: -- Status: Patch Available (was: Open) > Add ExtractKeywords processor > - > > Key: NIFI-12673 > URL: https://issues.apache.org/jira/browse/NIFI-12673 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > > Add a python processor that is extracting the keywords of the content of the > incoming flow file and add the keywords as attributes of the flowfile. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] NIFI-12673 - ExtractKeywords processor [nifi]
pvillard31 opened a new pull request, #8302: URL: https://github.com/apache/nifi/pull/8302 # Summary [NIFI-12673](https://issues.apache.org/jira/browse/NIFI-12673) - ExtractKeywords processor # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [ ] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [ ] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [ ] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [ ] Pull Request based on current revision of the `main` branch - [ ] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [ ] JDK 21 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-12646) Conduct Apache NiFi 2.0.0-M2 Release
[ https://issues.apache.org/jira/browse/NIFI-12646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17810873#comment-17810873 ] ASF subversion and git services commented on NIFI-12646: Commit 28b254df9fd866b989b2b2cd15195c6ef168ce37 in nifi's branch refs/heads/NIFI-12646-RC3 from David Handermann [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=28b254df9f ] NIFI-12646 Set Python Processor version to 2.0.0-M2 Signed-off-by: David Handermann > Conduct Apache NiFi 2.0.0-M2 Release > > > Key: NIFI-12646 > URL: https://issues.apache.org/jira/browse/NIFI-12646 > Project: Apache NiFi > Issue Type: Task >Reporter: David Handermann >Assignee: David Handermann >Priority: Major > Fix For: 2.0.0-M2 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12646) Conduct Apache NiFi 2.0.0-M2 Release
[ https://issues.apache.org/jira/browse/NIFI-12646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17810875#comment-17810875 ] ASF subversion and git services commented on NIFI-12646: Commit c94ab4161dd1da0113ae551a9bc16adeb07ff817 in nifi's branch refs/heads/NIFI-12646-RC3 from David Handermann [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=c94ab4161d ] NIFI-12646-RC3 prepare for next development iteration > Conduct Apache NiFi 2.0.0-M2 Release > > > Key: NIFI-12646 > URL: https://issues.apache.org/jira/browse/NIFI-12646 > Project: Apache NiFi > Issue Type: Task >Reporter: David Handermann >Assignee: David Handermann >Priority: Major > Fix For: 2.0.0-M2 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12646) Conduct Apache NiFi 2.0.0-M2 Release
[ https://issues.apache.org/jira/browse/NIFI-12646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17810874#comment-17810874 ] ASF subversion and git services commented on NIFI-12646: Commit 439ac5f596fe90d2591376caec1499ed86abfd6b in nifi's branch refs/heads/NIFI-12646-RC3 from David Handermann [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=439ac5f596 ] NIFI-12646-RC3 prepare release nifi-2.0.0-M2-RC3 > Conduct Apache NiFi 2.0.0-M2 Release > > > Key: NIFI-12646 > URL: https://issues.apache.org/jira/browse/NIFI-12646 > Project: Apache NiFi > Issue Type: Task >Reporter: David Handermann >Assignee: David Handermann >Priority: Major > Fix For: 2.0.0-M2 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12672) Implement Azure option for FileResourceService
[ https://issues.apache.org/jira/browse/NIFI-12672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Balázs Gerner updated NIFI-12672: - Description: Add Azure based implementation for the FileResourceService interface to be able to fetch data directly from Azure storage. (was: Add S3 based implementation for the FileResourceService interface to be able to fetch data directly from AWS.) > Implement Azure option for FileResourceService > -- > > Key: NIFI-12672 > URL: https://issues.apache.org/jira/browse/NIFI-12672 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Balázs Gerner >Assignee: Balázs Gerner >Priority: Major > > Add Azure based implementation for the FileResourceService interface to be > able to fetch data directly from Azure storage. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-12672) Implement Azure option for FileResourceService
Balázs Gerner created NIFI-12672: Summary: Implement Azure option for FileResourceService Key: NIFI-12672 URL: https://issues.apache.org/jira/browse/NIFI-12672 Project: Apache NiFi Issue Type: New Feature Reporter: Balázs Gerner Assignee: Balázs Gerner Add S3 based implementation for the FileResourceService interface to be able to fetch data directly from AWS. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12671) Implement AWS option for FileResourceService
[ https://issues.apache.org/jira/browse/NIFI-12671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Balázs Gerner updated NIFI-12671: - Fix Version/s: (was: 2.0.0-M2) > Implement AWS option for FileResourceService > > > Key: NIFI-12671 > URL: https://issues.apache.org/jira/browse/NIFI-12671 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Balázs Gerner >Assignee: Balázs Gerner >Priority: Major > > Add GCS based implementation for the FileResourceService interface to be able > to fetch data directly from Google Cloud Storage. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-12671) Implement AWS option for FileResourceService
[ https://issues.apache.org/jira/browse/NIFI-12671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Balázs Gerner reassigned NIFI-12671: Assignee: Balázs Gerner (was: Mark Bathori) > Implement AWS option for FileResourceService > > > Key: NIFI-12671 > URL: https://issues.apache.org/jira/browse/NIFI-12671 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Balázs Gerner >Assignee: Balázs Gerner >Priority: Major > Fix For: 2.0.0-M2 > > > Add GCS based implementation for the FileResourceService interface to be able > to fetch data directly from Google Cloud Storage. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12671) Implement AWS option for FileResourceService
[ https://issues.apache.org/jira/browse/NIFI-12671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Balázs Gerner updated NIFI-12671: - Description: Add S3 based implementation for the FileResourceService interface to be able to fetch data directly from AWS. (was: Add GCS based implementation for the FileResourceService interface to be able to fetch data directly from Google Cloud Storage.) > Implement AWS option for FileResourceService > > > Key: NIFI-12671 > URL: https://issues.apache.org/jira/browse/NIFI-12671 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Balázs Gerner >Assignee: Balázs Gerner >Priority: Major > > Add S3 based implementation for the FileResourceService interface to be able > to fetch data directly from AWS. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-12671) Implement AWS option for FileResourceService
Balázs Gerner created NIFI-12671: Summary: Implement AWS option for FileResourceService Key: NIFI-12671 URL: https://issues.apache.org/jira/browse/NIFI-12671 Project: Apache NiFi Issue Type: New Feature Reporter: Balázs Gerner Assignee: Mark Bathori Fix For: 2.0.0-M2 Add GCS based implementation for the FileResourceService interface to be able to fetch data directly from Google Cloud Storage. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12669) EvaluateXQuery processor incorrectly encodes result attributes
[
https://issues.apache.org/jira/browse/NIFI-12669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17810842#comment-17810842
]
René Zeidler commented on NIFI-12669:
-
NIFI-12670 is a similar encoding bug.
> EvaluateXQuery processor incorrectly encodes result attributes
> --
>
> Key: NIFI-12669
> URL: https://issues.apache.org/jira/browse/NIFI-12669
> Project: Apache NiFi
> Issue Type: Bug
> Components: Configuration, Extensions
>Affects Versions: 2.0.0-M1, 1.24.0
> Environment: JVM with non-UTF-8 default encoding (e.g. default
> Windows installation)
>Reporter: René Zeidler
>Priority: Major
> Labels: encoding, utf8, windows, xml
> Attachments: EvaluateXQuery_Encoding_Bug.json,
> image-2024-01-25-10-24-17-005.png, image-2024-01-25-10-31-35-200.png
>
>
> h2. Environment
> This issue affects environments where the JVM default encoding is not
> {{{}UTF-8{}}}. Standard Java installations on Windows are affected, as they
> usually use the default encoding {{{}windows-1252{}}}. To reproduce the issue
> on Linux, change the default encoding to {{windows-1252}} by adding the
> following line to your {{{}bootstrap.conf{}}}:
> {quote}{{java.arg.21=-Dfile.encoding=windows-1252}}
> {quote}
> h2. Summary
> The EvaluateXQuery incorrectly encodes result values when storing them in
> attributes. This causes non-ASCII characters to be garbled.
> Example:
> !image-2024-01-25-10-24-17-005.png!
> h2. Steps to reproduce
> # Make sure NiFi runs with a non-UTF-8 default encoding, see "Environment"
> # Create a GenerateFlowFile processor with the following content:
> {quote}{{}}
> {{}}
> {{ This text contains non-ASCII characters: ÄÖÜäöüßéèóò}}
> {{}}
> {quote}
> # Connect the processor to an EvaluateXQuery processor.
> Set the {{Destination}} to {{{}flowfile-attribute{}}}.
> Create a custom property {{myData}} with value {{{}string(/myRoot/myData){}}}.
> # Connect the outputs of the EvaluateXQuery processor to funnels to be able
> to observe the result in the queue.
> # Start the EvaluateXQuery processor and run the GenerateFlowFile processor
> once.
> The flow should look similar to this:
> !image-2024-01-25-10-31-35-200.png!
> I also attached a JSON export of the example flow.
> # Observe the attributes of the resulting FlowFile in the queue.
> h3. Expected Result
> The FlowFile should contain an attribute {{myData}} with the value {{{}"This
> text contains non-ASCII characters: ÄÖÜäöüßéèóò"{}}}.
> h3. Actual Result
> The attribute has the value "This text contains non-ASCII characters:
> ÄÖÜäöüßéèóò".
> h2. Root Cause Analysis
> EvaluateXQuery uses the method
> [{{formatItem}}|https://github.com/apache/nifi/blob/2e3f83eb54cbc040b5a1da5bce9a74a558f08ea4/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/EvaluateXQuery.java#L368-L372]
> to write the query result to an attribute. This method calls
> {{{}ByteArrayOutputStream{}}}'s
> [toString|https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/io/ByteArrayOutputStream.html#toString()]
> method without an encoding argument, which then defaults to the default
> charset of the environment. Bytes are always written to this output stream
> using UTF-8
> ([.getBytes(StandardCharsets.UTF8)|https://github.com/apache/nifi/blob/2e3f83eb54cbc040b5a1da5bce9a74a558f08ea4/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/EvaluateXQuery.java#L397]).
> When the default charset is not UTF-8, this results in UTF-8 bytes to be
> interpreted in a different encoding when converting to a string, resulting in
> garbled text (see above).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (NIFI-12670) JoltTransform processors incorrectly encode/decode text in the Jolt Specification
[
https://issues.apache.org/jira/browse/NIFI-12670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17810840#comment-17810840
]
René Zeidler edited comment on NIFI-12670 at 1/25/24 11:14 AM:
---
NIFI-12669 is a similar encoding bug
was (Author: JIRAUSER297247):
Similar encoding bug
> JoltTransform processors incorrectly encode/decode text in the Jolt
> Specification
> -
>
> Key: NIFI-12670
> URL: https://issues.apache.org/jira/browse/NIFI-12670
> Project: Apache NiFi
> Issue Type: Bug
> Components: Configuration, Extensions
>Affects Versions: 2.0.0-M1, 1.24.0
> Environment: JVM with non-UTF-8 default encoding (e.g. default
> Windows installation)
>Reporter: René Zeidler
>Priority: Minor
> Labels: encoding, jolt, json, utf8, windows
> Attachments: Jolt_Transform_Encoding_Bug.json,
> image-2024-01-25-11-01-15-405.png, image-2024-01-25-11-59-56-662.png,
> image-2024-01-25-12-00-09-544.png
>
>
> h2. Environment
> This issue affects environments where the JVM default encoding is not
> {{{}UTF-8{}}}. Standard Java installations on Windows are affected, as they
> usually use the default encoding {{{}windows-1252{}}}. To reproduce the issue
> on Linux, change the default encoding to {{windows-1252}} by adding the
> following line to your {{{}bootstrap.conf{}}}:
> {quote}{{java.arg.21=-Dfile.encoding=windows-1252}}
> {quote}
> h2. Summary
> The Jolt Specification of both the JoltTransformJSON and JoltTransformRecord
> processors is read interally using the system default encoding, even though
> it is always stored in UTF-8. This causes non-ASCII characters to be garbled
> in the Jolt Specification, resulting in incorrect transformations (missing
> data or garbled keys).
> h2. Steps to reproduce
> # Make sure NiFi runs with a non-UTF-8 default encoding, see "Environment"
> # Create a GenerateFlowFile processor with the following content:
> {quote}{
> "regularString": "string with only ASCII characters",
> "umlautString": "string with non-ASCII characters: ÄÖÜäöüßéèóò",
> "keyWithÜmlaut": "any string"
> }
> {quote}
> # Connect the processor to a JoltTransformJSON and/or JoltTransformRecord
> processor.
> (If using the record based processor, use a default JsonTreeReader and
> JsonRecordSetWriter. The record reader/writer don't affect this bug.)
> Set the Jolt Specification to:
> {quote}[
> {
> "operation": "shift",
> "spec": {
> "regularString": "Remapped to Umlaut ÄÖÜ",
> "umlautString": "Umlaut String",
> "keyWithÜmlaut": "Key with Umlaut"
> }
> }
> ]
> {quote}
> # Connect the outputs of the Jolt processor(s) to funnels to be able to
> observe the result in the queue.
> # Start the Jolt processor(s) and run the GenerateFlowFile processor once.
> The flow should look similar to this:
> !image-2024-01-25-11-01-15-405.png!
> I also attached a JSON export of the example flow.
> # Observe the content of the resulting FlowFile(s) in the queue.
> h3. Expected Result
> !image-2024-01-25-12-00-09-544.png!
> h3. Actual Result
> !image-2024-01-25-11-59-56-662.png!
> * Remapped key containing non-ASCII characters is garbled, since the key
> value originated from the Jolt Specification.
> * The key "{{{}keyWithÜmlaut{}}}" could not be matched at all, since it
> contains non-ASCII characters, resulting in missing data in the output.
> h2. Root Cause Analysis
> Both processors use the
> {{[readTransform|https://github.com/apache/nifi/blob/2e3f83eb54cbc040b5a1da5bce9a74a558f08ea4/nifi-nar-bundles/nifi-jolt-bundle/nifi-jolt-processors/src/main/java/org/apache/nifi/processors/jolt/AbstractJoltTransform.java#L242-L249]}}
> method of {{AbstractJoltTransform}} to read the Jolt Specification property.
> This method uses an
> [{{InputStreamReader}}|https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/io/InputStreamReader.html]
> without specifying an encoding, which then defaults to the default charset
> of the environment. Text properties are [always encoded in
> UTF-8|https://github.com/apache/nifi/blob/89836f32d017d77972a4de09c4e864b0e11899a8/nifi-api/src/main/java/org/apache/nifi/components/resource/StandardResourceReferenceFactory.java#L111].
> When the default charset is not UTF-8, this results in UTF-8 bytes to be
> interpreted in a different encoding when converting to a string, resulting in
> a garbled Jolt Specification being used.
> h2. Workaround
> This issue is not present when any attribute expression language is present
> in the Jolt Specification. Simply adding {{${literal('')}}} anywhere in the
> Jolt Specification works around this issue.
> This happens because [a different code path is
>
[jira] [Commented] (NIFI-12670) JoltTransform processors incorrectly encode/decode text in the Jolt Specification
[
https://issues.apache.org/jira/browse/NIFI-12670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17810840#comment-17810840
]
René Zeidler commented on NIFI-12670:
-
Similar encoding bug
> JoltTransform processors incorrectly encode/decode text in the Jolt
> Specification
> -
>
> Key: NIFI-12670
> URL: https://issues.apache.org/jira/browse/NIFI-12670
> Project: Apache NiFi
> Issue Type: Bug
> Components: Configuration, Extensions
>Affects Versions: 2.0.0-M1, 1.24.0
> Environment: JVM with non-UTF-8 default encoding (e.g. default
> Windows installation)
>Reporter: René Zeidler
>Priority: Minor
> Labels: encoding, jolt, json, utf8, windows
> Attachments: Jolt_Transform_Encoding_Bug.json,
> image-2024-01-25-11-01-15-405.png, image-2024-01-25-11-59-56-662.png,
> image-2024-01-25-12-00-09-544.png
>
>
> h2. Environment
> This issue affects environments where the JVM default encoding is not
> {{{}UTF-8{}}}. Standard Java installations on Windows are affected, as they
> usually use the default encoding {{{}windows-1252{}}}. To reproduce the issue
> on Linux, change the default encoding to {{windows-1252}} by adding the
> following line to your {{{}bootstrap.conf{}}}:
> {quote}{{java.arg.21=-Dfile.encoding=windows-1252}}
> {quote}
> h2. Summary
> The Jolt Specification of both the JoltTransformJSON and JoltTransformRecord
> processors is read interally using the system default encoding, even though
> it is always stored in UTF-8. This causes non-ASCII characters to be garbled
> in the Jolt Specification, resulting in incorrect transformations (missing
> data or garbled keys).
> h2. Steps to reproduce
> # Make sure NiFi runs with a non-UTF-8 default encoding, see "Environment"
> # Create a GenerateFlowFile processor with the following content:
> {quote}{
> "regularString": "string with only ASCII characters",
> "umlautString": "string with non-ASCII characters: ÄÖÜäöüßéèóò",
> "keyWithÜmlaut": "any string"
> }
> {quote}
> # Connect the processor to a JoltTransformJSON and/or JoltTransformRecord
> processor.
> (If using the record based processor, use a default JsonTreeReader and
> JsonRecordSetWriter. The record reader/writer don't affect this bug.)
> Set the Jolt Specification to:
> {quote}[
> {
> "operation": "shift",
> "spec": {
> "regularString": "Remapped to Umlaut ÄÖÜ",
> "umlautString": "Umlaut String",
> "keyWithÜmlaut": "Key with Umlaut"
> }
> }
> ]
> {quote}
> # Connect the outputs of the Jolt processor(s) to funnels to be able to
> observe the result in the queue.
> # Start the Jolt processor(s) and run the GenerateFlowFile processor once.
> The flow should look similar to this:
> !image-2024-01-25-11-01-15-405.png!
> I also attached a JSON export of the example flow.
> # Observe the content of the resulting FlowFile(s) in the queue.
> h3. Expected Result
> !image-2024-01-25-12-00-09-544.png!
> h3. Actual Result
> !image-2024-01-25-11-59-56-662.png!
> * Remapped key containing non-ASCII characters is garbled, since the key
> value originated from the Jolt Specification.
> * The key "{{{}keyWithÜmlaut{}}}" could not be matched at all, since it
> contains non-ASCII characters, resulting in missing data in the output.
> h2. Root Cause Analysis
> Both processors use the
> {{[readTransform|https://github.com/apache/nifi/blob/2e3f83eb54cbc040b5a1da5bce9a74a558f08ea4/nifi-nar-bundles/nifi-jolt-bundle/nifi-jolt-processors/src/main/java/org/apache/nifi/processors/jolt/AbstractJoltTransform.java#L242-L249]}}
> method of {{AbstractJoltTransform}} to read the Jolt Specification property.
> This method uses an
> [{{InputStreamReader}}|https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/io/InputStreamReader.html]
> without specifying an encoding, which then defaults to the default charset
> of the environment. Text properties are [always encoded in
> UTF-8|https://github.com/apache/nifi/blob/89836f32d017d77972a4de09c4e864b0e11899a8/nifi-api/src/main/java/org/apache/nifi/components/resource/StandardResourceReferenceFactory.java#L111].
> When the default charset is not UTF-8, this results in UTF-8 bytes to be
> interpreted in a different encoding when converting to a string, resulting in
> a garbled Jolt Specification being used.
> h2. Workaround
> This issue is not present when any attribute expression language is present
> in the Jolt Specification. Simply adding {{${literal('')}}} anywhere in the
> Jolt Specification works around this issue.
> This happens because [a different code path is
>
[jira] [Created] (NIFI-12670) JoltTransform processors incorrectly encode/decode text in the Jolt Specification
René Zeidler created NIFI-12670:
---
Summary: JoltTransform processors incorrectly encode/decode text
in the Jolt Specification
Key: NIFI-12670
URL: https://issues.apache.org/jira/browse/NIFI-12670
Project: Apache NiFi
Issue Type: Bug
Components: Configuration, Extensions
Affects Versions: 1.24.0, 2.0.0-M1
Environment: JVM with non-UTF-8 default encoding (e.g. default Windows
installation)
Reporter: René Zeidler
Attachments: Jolt_Transform_Encoding_Bug.json,
image-2024-01-25-11-01-15-405.png, image-2024-01-25-11-59-56-662.png,
image-2024-01-25-12-00-09-544.png
h2. Environment
This issue affects environments where the JVM default encoding is not
{{{}UTF-8{}}}. Standard Java installations on Windows are affected, as they
usually use the default encoding {{{}windows-1252{}}}. To reproduce the issue
on Linux, change the default encoding to {{windows-1252}} by adding the
following line to your {{{}bootstrap.conf{}}}:
{quote}{{java.arg.21=-Dfile.encoding=windows-1252}}
{quote}
h2. Summary
The Jolt Specification of both the JoltTransformJSON and JoltTransformRecord
processors is read interally using the system default encoding, even though it
is always stored in UTF-8. This causes non-ASCII characters to be garbled in
the Jolt Specification, resulting in incorrect transformations (missing data or
garbled keys).
h2. Steps to reproduce
# Make sure NiFi runs with a non-UTF-8 default encoding, see "Environment"
# Create a GenerateFlowFile processor with the following content:
{quote}{
"regularString": "string with only ASCII characters",
"umlautString": "string with non-ASCII characters: ÄÖÜäöüßéèóò",
"keyWithÜmlaut": "any string"
}
{quote}
# Connect the processor to a JoltTransformJSON and/or JoltTransformRecord
processor.
(If using the record based processor, use a default JsonTreeReader and
JsonRecordSetWriter. The record reader/writer don't affect this bug.)
Set the Jolt Specification to:
{quote}[
{
"operation": "shift",
"spec": {
"regularString": "Remapped to Umlaut ÄÖÜ",
"umlautString": "Umlaut String",
"keyWithÜmlaut": "Key with Umlaut"
}
}
]
{quote}
# Connect the outputs of the Jolt processor(s) to funnels to be able to
observe the result in the queue.
# Start the Jolt processor(s) and run the GenerateFlowFile processor once.
The flow should look similar to this:
!image-2024-01-25-11-01-15-405.png!
I also attached a JSON export of the example flow.
# Observe the content of the resulting FlowFile(s) in the queue.
h3. Expected Result
!image-2024-01-25-12-00-09-544.png!
h3. Actual Result
!image-2024-01-25-11-59-56-662.png!
* Remapped key containing non-ASCII characters is garbled, since the key value
originated from the Jolt Specification.
* The key "{{{}keyWithÜmlaut{}}}" could not be matched at all, since it
contains non-ASCII characters, resulting in missing data in the output.
h2. Root Cause Analysis
Both processors use the
{{[readTransform|https://github.com/apache/nifi/blob/2e3f83eb54cbc040b5a1da5bce9a74a558f08ea4/nifi-nar-bundles/nifi-jolt-bundle/nifi-jolt-processors/src/main/java/org/apache/nifi/processors/jolt/AbstractJoltTransform.java#L242-L249]}}
method of {{AbstractJoltTransform}} to read the Jolt Specification property.
This method uses an
[{{InputStreamReader}}|https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/io/InputStreamReader.html]
without specifying an encoding, which then defaults to the default charset of
the environment. Text properties are [always encoded in
UTF-8|https://github.com/apache/nifi/blob/89836f32d017d77972a4de09c4e864b0e11899a8/nifi-api/src/main/java/org/apache/nifi/components/resource/StandardResourceReferenceFactory.java#L111].
When the default charset is not UTF-8, this results in UTF-8 bytes to be
interpreted in a different encoding when converting to a string, resulting in a
garbled Jolt Specification being used.
h2. Workaround
This issue is not present when any attribute expression language is present in
the Jolt Specification. Simply adding {{${literal('')}}} anywhere in the Jolt
Specification works around this issue.
This happens because [a different code path is
used|https://github.com/apache/nifi/blob/2e3f83eb54cbc040b5a1da5bce9a74a558f08ea4/nifi-nar-bundles/nifi-jolt-bundle/nifi-jolt-processors/src/main/java/org/apache/nifi/processors/jolt/AbstractJoltTransform.java#L233-L237]
when expression language is present.
I don't know why the property is even read line-by-line using a stream reader
when no expression language is present. It seems like just using {{getValue()}}
would work fine even without expression language, and that method doesn't have
the encoding bug.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-12669) EvaluateXQuery processor incorrectly encodes result attributes
[
https://issues.apache.org/jira/browse/NIFI-12669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
René Zeidler updated NIFI-12669:
Description:
h2. Environment
This issue affects environments where the JVM default encoding is not
{{{}UTF-8{}}}. Standard Java installations on Windows are affected, as they
usually use the default encoding {{{}windows-1252{}}}. To reproduce the issue
on Linux, change the default encoding to {{windows-1252}} by adding the
following line to your {{{}bootstrap.conf{}}}:
{quote}{{java.arg.21=-Dfile.encoding=windows-1252}}
{quote}
h2. Summary
The EvaluateXQuery incorrectly encodes result values when storing them in
attributes. This causes non-ASCII characters to be garbled.
Example:
!image-2024-01-25-10-24-17-005.png!
h2. Steps to reproduce
# Make sure NiFi runs with a non-UTF-8 default encoding, see "Environment"
# Create a GenerateFlowFile processor with the following content:
{quote}{{}}
{{}}
{{ This text contains non-ASCII characters: ÄÖÜäöüßéèóò}}
{{}}
{quote}
# Connect the processor to an EvaluateXQuery processor.
Set the {{Destination}} to {{{}flowfile-attribute{}}}.
Create a custom property {{myData}} with value {{{}string(/myRoot/myData){}}}.
# Connect the outputs of the EvaluateXQuery processor to funnels to be able to
observe the result in the queue.
# Start the EvaluateXQuery processor and run the GenerateFlowFile processor
once.
The flow should look similar to this:
!image-2024-01-25-10-31-35-200.png!
I also attached a JSON export of the example flow.
# Observe the attributes of the resulting FlowFile in the queue.
h3. Expected Result
The FlowFile should contain an attribute {{myData}} with the value {{{}"This
text contains non-ASCII characters: ÄÖÜäöüßéèóò"{}}}.
h3. Actual Result
The attribute has the value "This text contains non-ASCII characters:
ÄÖÜäöüßéèóò".
h2. Root Cause Analysis
EvaluateXQuery uses the method
[{{formatItem}}|https://github.com/apache/nifi/blob/2e3f83eb54cbc040b5a1da5bce9a74a558f08ea4/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/EvaluateXQuery.java#L368-L372]
to write the query result to an attribute. This method calls
{{{}ByteArrayOutputStream{}}}'s
[toString|https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/io/ByteArrayOutputStream.html#toString()]
method without an encoding argument, which then defaults to the default
charset of the environment. Bytes are always written to this output stream
using UTF-8
([.getBytes(StandardCharsets.UTF8)|https://github.com/apache/nifi/blob/2e3f83eb54cbc040b5a1da5bce9a74a558f08ea4/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/EvaluateXQuery.java#L397]).
When the default charset is not UTF-8, this results in UTF-8 bytes to be
interpreted in a different encoding when converting to a string, resulting in
garbled text (see above).
was:
h2. Environment
This issue affects environments where the JVM default encoding is not
{{{}UTF-8{}}}. Standard Java installations on Windows are affected, as they
usually use the default encoding {{{}windows-1252{}}}. To reproduce the issue
on Linux, change the default encoding to {{windows-1252}} by adding the
following line to your {{{}bootstrap.conf{}}}:
{quote}{{java.arg.21=-Dfile.encoding=windows-1252}}
{quote}
h2. Summary
The EvaluateXQuery incorrectly encodes result values when storing them in
attributes. This causes non-ASCII characters to be garbled.
Example:
!image-2024-01-25-10-24-17-005.png!
h2. Steps to reproduce
# Make sure NiFi runs with a non-UTF-8 default encoding, see "Environment"
# Create a GenerateFlowFile processor with the following content:
{quote}{{}}
{{}}
{{ This text contains non-ASCII characters: ÄÖÜäöüßéèóò}}
{{}}
{quote}
# Connect the processor to an EvaluateXQuery processor.
Set the {{Destination}} to {{{}flowfile-attribute{}}}.
Create a custom property {{myData}} with value {{{}string(/myRoot/myData){}}}.
# Connect the outputs of the EvaluateXQuery processor to funnels to be able to
observe the result in the queue.
# Start the EvaluateXQuery processor and run the GenerateFlowFile processor
once.
The flow should look similar to this:
!image-2024-01-25-10-31-35-200.png!
I also attached a JSON export of the example flow.
# Observe the attributes of the resulting FlowFile in the queue.
h3. Expected Result
The FlowFile should contain an attribute {{myData}} with the value {{{}"This
text contains non-ASCII characters: ÄÖÜäöüßéèóò"{}}}.
h3. Actual Result
The attribute has the value {{{}"This text contains non-ASCII characters:
ÄÖÜäöüßéèóò"{}}}.
h2. Root Cause Analysis
EvaluateXQuery uses the method
[jira] [Created] (NIFI-12669) EvaluateXQuery processor incorrectly encodes result attributes
René Zeidler created NIFI-12669:
---
Summary: EvaluateXQuery processor incorrectly encodes result
attributes
Key: NIFI-12669
URL: https://issues.apache.org/jira/browse/NIFI-12669
Project: Apache NiFi
Issue Type: Bug
Components: Configuration, Extensions
Affects Versions: 1.24.0, 2.0.0-M1
Environment: JVM with non-UTF-8 default encoding (e.g. default Windows
installation)
Reporter: René Zeidler
Attachments: EvaluateXQuery_Encoding_Bug.json,
image-2024-01-25-10-24-17-005.png, image-2024-01-25-10-31-35-200.png
h2. Environment
This issue affects environments where the JVM default encoding is not
{{{}UTF-8{}}}. Standard Java installations on Windows are affected, as they
usually use the default encoding {{{}windows-1252{}}}. To reproduce the issue
on Linux, change the default encoding to {{windows-1252}} by adding the
following line to your {{{}bootstrap.conf{}}}:
{quote}{{java.arg.21=-Dfile.encoding=windows-1252}}
{quote}
h2. Summary
The EvaluateXQuery incorrectly encodes result values when storing them in
attributes. This causes non-ASCII characters to be garbled.
Example:
!image-2024-01-25-10-24-17-005.png!
h2. Steps to reproduce
# Make sure NiFi runs with a non-UTF-8 default encoding, see "Environment"
# Create a GenerateFlowFile processor with the following content:
{quote}{{}}
{{}}
{{ This text contains non-ASCII characters: ÄÖÜäöüßéèóò}}
{{}}
{quote}
# Connect the processor to an EvaluateXQuery processor.
Set the {{Destination}} to {{{}flowfile-attribute{}}}.
Create a custom property {{myData}} with value {{{}string(/myRoot/myData){}}}.
# Connect the outputs of the EvaluateXQuery processor to funnels to be able to
observe the result in the queue.
# Start the EvaluateXQuery processor and run the GenerateFlowFile processor
once.
The flow should look similar to this:
!image-2024-01-25-10-31-35-200.png!
I also attached a JSON export of the example flow.
# Observe the attributes of the resulting FlowFile in the queue.
h3. Expected Result
The FlowFile should contain an attribute {{myData}} with the value {{{}"This
text contains non-ASCII characters: ÄÖÜäöüßéèóò"{}}}.
h3. Actual Result
The attribute has the value {{{}"This text contains non-ASCII characters:
ÄÖÜäöüßéèóò"{}}}.
h2. Root Cause Analysis
EvaluateXQuery uses the method
[{{formatItem}}|https://github.com/apache/nifi/blob/2e3f83eb54cbc040b5a1da5bce9a74a558f08ea4/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/EvaluateXQuery.java#L368-L372]
to write the query result to an attribute. This method calls
{{{}ByteArrayOutputStream{}}}'s
[toString|https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/io/ByteArrayOutputStream.html#toString()]
method without an encoding argument, which then defaults to the default
charset of the environment. Bytes are always written to this output stream
using UTF-8
([.getBytes(StandardCharsets.UTF8)|https://github.com/apache/nifi/blob/2e3f83eb54cbc040b5a1da5bce9a74a558f08ea4/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/EvaluateXQuery.java#L397]).
When the default charset is not UTF-8, this results in UTF-8 bytes to be
interpreted in a different encoding when converting to a string, resulting in
garbled text (see above).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[PR] MINIFICPP-2291 Fix site-to-site transfer of large files [nifi-minifi-cpp]
fgerlits opened a new pull request, #1720: URL: https://github.com/apache/nifi-minifi-cpp/pull/1720 Previously, the site-to-site client treated the `TRANSACTION_FINISHED_BUT_DESTINATION_FULL` message from the server as a failure, and kept rolling back and retrying the flow file. Now, we treat it as success but add a yield so the server has time to clear the incoming connection, the same way as NiFi does. https://issues.apache.org/jira/browse/MINIFICPP-2287 https://issues.apache.org/jira/browse/MINIFICPP-2291 I have also included the fix of two unrelated bugs I noticed during testing; these are too small to merit their own pull requests. --- Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with MINIFICPP- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically main)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
