Re: EncrpytContent Question
Great answer Jim! The source of the credentials and the intended use of the credentials are important to know to ensure a secure flow design. NiFi supports repository encryption, which encrypts FlowFile content and metadata on the node filesystem, and that applies regardless of the Processors configured. As Jim said, Parameter Providers would be a better fit if the goal is to supply credentials to a particular Processor or Controller Service. If the goal is to send information to another system, then I concur with the recommendation to use either EncryptContentAge or EncryptContenPGP, since the EncryptContent Processor is deprecated and removed from NiFi 2.0.0. Regards, David Handermann On Thu, Apr 11, 2024 at 10:26 AM Jim Steinebrey wrote: > > Hi Cecil, > In response to your questions, > if you store credentials in the contents of an encrypted flow file, then the > credentials contents will be encrypted at rest. > Flow file contents are not accessible from metadata. > > How are you going to use the stored credentials? > Are the credentials going to be used inside NiFi? > > Depending on your use case, you might be able to use NiFi parameters and > ParameterProviders for getting your credentials into NiFi. > I suggest you google NiFi Parameter Providers and read about them. > > I would suggest you develop your flow on NiFi 2.0.0-M2 because it is the > latest and greatest. > but I would not recommend going to production with a milestone release for an > enterprise critical flow, but that is up to you. > I do not know an estimate for the 2.0 final release date. > Be aware EncryptContent is removed in NiFI 2, so I suggest you use > EncryptContentAge or EncryptContentGPG > > Best, > Jim > > > On Apr 11, 2024, at 8:24 AM, Cecil McKenna wrote: > > > > Hello, > > > > I am a developer new to Apache Nifi, and I am trying to encrypt and store > > credentials in Nifi. I was looking for information regarding the different > > EncryptContent processors, but there isn't much provided in the > > documentation. My question is, if I encrypt a flowfile containing user > > credentials and store them in Nifi, will the credentials themselves remain > > encrypted at rest? From my understanding, the data itself will remain > > encrypted, but the metadata will not. However, the user credentials should > > not be accessible in the flowfile metadata, correct? Could you please > > verify or point me in the right direction? > > > > Thank you! > > > > Best, > > Cecil McKenna >
Re: Build of 2.x failing on Centos 7
Dan, Reviewing the previous thread, the last reply from Matt Gilman still applies [1] meaning that CentOS 7 is not recent enough to support the version of Node.js required. CentOS 7 is also scheduled for End of Life [2] on June 30, 2024. Based on the compatibility issues with the core OS libraries, I would not expect CentOS 7 to be supported for building NiFi 2.0. Regards, David Handermann [1] https://lists.apache.org/thread/jdmw7dsjryyf8vbjgq84tl8kpv68pplr [2] https://www.redhat.com/en/topics/linux/centos-linux-eol On Fri, Apr 19, 2024 at 10:51 AM Dan S wrote: > > I am trying to build the 2.x branch for a PR I am trying to submit for > NIFI-13069 and it is failing. Here are the messages I see from the build > > [INFO] > git/NIFI-FORK/nifi-2.x/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node/node: > /lib64/libm.so.6: version `GLIBC_2.27' not found (required by > git/NIFI-FORK/nifi-2.x/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node/node) > [INFO] > git/NIFI-FORK/nifi-2.x/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node/node: > /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by > git/NIFI-FORK/nifi-2.x/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node/node) > [INFO] > git/NIFI-FORK/nifi-2.x/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node/node: > /lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by > git/NIFI-FORK/nifi-2.x/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node/node) > [INFO] > git/NIFI-FORK/nifi-2.x/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node/node: > /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by > git/NIFI-FORK/nifi-2.x/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node/node) > [INFO] > git/NIFI-FORK/nifi-2.x/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node/node: > /lib64/libc.so.6: version `GLIBC_2.27' not found (required by > git/NIFI-FORK/nifi-2.x/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node/node) > [INFO] > git/NIFI-FORK/nifi-2.x/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node/node: > /lib64/libc.so.6: version `GLIBC_2.28' not found (required by > git/NIFI-FORK/nifi-2.x/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node/node) > [INFO] > git/NIFI-FORK/nifi-2.x/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node/node: > /lib64/libc.so.6: version `GLIBC_2.25' not found (required by > git/NIFI-FORK/nifi-2.x/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node/node) > > [ERROR] Failed to execute goal > com.github.eirslett:frontend-maven-plugin:1.14.2:npm (npm install) on > project nifi-web-ui: Failed to run task: 'npm run ci' failed. > org.apache.commons.exec.ExecuteException: Process exited with an error: 1 > (Exit value: 1) -> [Help 1] > > This seems the same problem I had back In November 2023 on this thread > <https://lists.apache.org/thread/pm5yqognkt62gw6z6g2rlhcl872wgw3z> > Please advise.
Re: Build of 2.x failing on Centos 7
Dan, Reviewing recent changes, NIFI-13035 [1] updated the frontend build process to use Node.js 21 for both the current nifi-web-ui and the new nifi-web-frontend modules. For that reason, a more recent OS and glibc version is now required. Regards, David Handermann [1] https://issues.apache.org/jira/browse/NIFI-13035 On Fri, Apr 19, 2024 at 11:06 AM Dan S wrote: > > Thanks David! > I did not have this problem the other day when I submitted a PR for > NIFI-12960. > > On Fri, Apr 19, 2024 at 12:02 PM David Handermann < > exceptionfact...@apache.org> wrote: > > > Dan, > > > > Reviewing the previous thread, the last reply from Matt Gilman still > > applies [1] meaning that CentOS 7 is not recent enough to support the > > version of Node.js required. > > > > CentOS 7 is also scheduled for End of Life [2] on June 30, 2024. Based > > on the compatibility issues with the core OS libraries, I would not > > expect CentOS 7 to be supported for building NiFi 2.0. > > > > Regards, > > David Handermann > > > > [1] https://lists.apache.org/thread/jdmw7dsjryyf8vbjgq84tl8kpv68pplr > > [2] https://www.redhat.com/en/topics/linux/centos-linux-eol > > > > On Fri, Apr 19, 2024 at 10:51 AM Dan S wrote: > > > > > > I am trying to build the 2.x branch for a PR I am trying to submit for > > > NIFI-13069 and it is failing. Here are the messages I see from the build > > > > > > [INFO] > > > > > git/NIFI-FORK/nifi-2.x/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node/node: > > > /lib64/libm.so.6: version `GLIBC_2.27' not found (required by > > > > > git/NIFI-FORK/nifi-2.x/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node/node) > > > [INFO] > > > > > git/NIFI-FORK/nifi-2.x/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node/node: > > > /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by > > > > > git/NIFI-FORK/nifi-2.x/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node/node) > > > [INFO] > > > > > git/NIFI-FORK/nifi-2.x/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node/node: > > > /lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by > > > > > git/NIFI-FORK/nifi-2.x/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node/node) > > > [INFO] > > > > > git/NIFI-FORK/nifi-2.x/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node/node: > > > /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by > > > > > git/NIFI-FORK/nifi-2.x/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node/node) > > > [INFO] > > > > > git/NIFI-FORK/nifi-2.x/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node/node: > > > /lib64/libc.so.6: version `GLIBC_2.27' not found (required by > > > > > git/NIFI-FORK/nifi-2.x/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node/node) > > > [INFO] > > > > > git/NIFI-FORK/nifi-2.x/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node/node: > > > /lib64/libc.so.6: version `GLIBC_2.28' not found (required by > > > > > git/NIFI-FORK/nifi-2.x/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node/node) > > > [INFO] > > > > > git/NIFI-FORK/nifi-2.x/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node/node: > > > /lib64/libc.so.6: version `GLIBC_2.25' not found (required by > > > > > git/NIFI-FORK/nifi-2.x/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node/node) > > > > > > [ERROR] Failed to execute goal > > > com.github.eirslett:frontend-maven-plugin:1.14.2:npm (npm install) on > > > project nifi-web-ui: Failed to run task: 'npm run ci' failed. > > > org.apache.commons.exec.ExecuteException: Process exited with an error: 1 > > > (Exit value: 1) -> [Help 1] > > > > > > This seems the same problem I had back In November 2023 on this thread > > > <https://lists.apache.org/thread/pm5yqognkt62gw6z6g2rlhcl872wgw3z> > > > Please advise. > >
Re: Build of 2.x failing on Centos 7
Mike, The change was intentional from my understanding as Node.js 16 reached end of life last year [1]. Registry should also be updated to build with a more recent version of Node.js, but I believe there are some details to sort out before that can be changed. Regards, David Handermann [1] https://nodejs.org/en/blog/announcements/nodejs16-eol On Fri, Apr 19, 2024 at 11:12 AM David Handermann wrote: > > Dan, > > Reviewing recent changes, NIFI-13035 [1] updated the frontend build > process to use Node.js 21 for both the current nifi-web-ui and the new > nifi-web-frontend modules. For that reason, a more recent OS and glibc > version is now required. > > Regards, > David Handermann > > [1] https://issues.apache.org/jira/browse/NIFI-13035 > > On Fri, Apr 19, 2024 at 11:06 AM Dan S wrote: > > > > Thanks David! > > I did not have this problem the other day when I submitted a PR for > > NIFI-12960. > > > > On Fri, Apr 19, 2024 at 12:02 PM David Handermann < > > exceptionfact...@apache.org> wrote: > > > > > Dan, > > > > > > Reviewing the previous thread, the last reply from Matt Gilman still > > > applies [1] meaning that CentOS 7 is not recent enough to support the > > > version of Node.js required. > > > > > > CentOS 7 is also scheduled for End of Life [2] on June 30, 2024. Based > > > on the compatibility issues with the core OS libraries, I would not > > > expect CentOS 7 to be supported for building NiFi 2.0. > > > > > > Regards, > > > David Handermann > > > > > > [1] https://lists.apache.org/thread/jdmw7dsjryyf8vbjgq84tl8kpv68pplr > > > [2] https://www.redhat.com/en/topics/linux/centos-linux-eol > > > > > > On Fri, Apr 19, 2024 at 10:51 AM Dan S wrote: > > > > > > > > I am trying to build the 2.x branch for a PR I am trying to submit for > > > > NIFI-13069 and it is failing. Here are the messages I see from the build > > > > > > > > [INFO] > > > > > > > git/NIFI-FORK/nifi-2.x/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node/node: > > > > /lib64/libm.so.6: version `GLIBC_2.27' not found (required by > > > > > > > git/NIFI-FORK/nifi-2.x/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node/node) > > > > [INFO] > > > > > > > git/NIFI-FORK/nifi-2.x/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node/node: > > > > /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by > > > > > > > git/NIFI-FORK/nifi-2.x/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node/node) > > > > [INFO] > > > > > > > git/NIFI-FORK/nifi-2.x/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node/node: > > > > /lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by > > > > > > > git/NIFI-FORK/nifi-2.x/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node/node) > > > > [INFO] > > > > > > > git/NIFI-FORK/nifi-2.x/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node/node: > > > > /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by > > > > > > > git/NIFI-FORK/nifi-2.x/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node/node) > > > > [INFO] > > > > > > > git/NIFI-FORK/nifi-2.x/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node/node: > > > > /lib64/libc.so.6: version `GLIBC_2.27' not found (required by > > > > > > > git/NIFI-FORK/nifi-2.x/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node/node) > > > > [INFO] > > > > > > > git/NIFI-FORK/nifi-2.x/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/target/frontend-working-directory/node/node: > > > > /lib64/libc.so.6: version `GLIBC_2.28' not found (required by > > > > > > > git/NIFI-FORK/nifi-2.x/nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-frame
[DISCUSS] Release Timeline for NiFi 2.0.0-M3
Team, We are approaching 300 Jira issues resolved for version 2.0.0-M3 [1], including a number of important dependency upgrades and Python framework improvements. There are several open pull requests that are getting close to completion, which should be possible to include in an upcoming release version. There is also a significant pull request [2] for NIFI-12998 [3] that includes some important changes to module directory hierarchy, providing a clearer distinction between framework and extensions, and implementing significant cleanup of dependency declarations across the repository. With these changes in view, we should be ready for another milestone release soon after merging the above pull request. Another milestone release seems to be the best course of action for now, providing the opportunity for additional review and testing before a full 2.0.0 release version. I would be glad to handle Release Manager responsibilities for 2.0.0-M3 when things are ready. Regards, David Handermann [1] https://issues.apache.org/jira/issues/?jql=project%20%3D%20NIFI%20AND%20fixVersion%20%3D%202.0.0-M3 [2] https://github.com/apache/nifi/pull/8677 [3] https://issues.apache.org/jira/browse/NIFI-12998
Re: [DISCUSS] Release Timeline for NiFi 2.0.0-M3
Thanks for the replies thus far! With the goal of including the new UI, it seems like a good time to move the module from an optional profile to part of the standard build. That would make the release candidate build and review process simpler, not requiring the optional build profile. Regards, David Handermann On Tue, Apr 23, 2024 at 2:42 PM Matt Gilman wrote: > > I agree. Including the updated UI and getting some feedback would be great. > > On Mon, Apr 22, 2024 at 8:50 PM Joe Witt wrote: > > > Id be a big fan of including the new UI. > > > > On Mon, Apr 22, 2024 at 5:45 PM Pierre Villard < > > pierre.villard...@gmail.com> > > wrote: > > > > > Thanks David, I'm happy to take care of a 1.26 release at the same time. > > > > > > Regarding the M3 release, should the new UI be included and instructions > > > given on how to access it to start collecting feedback? I'm under the > > > impression that almost all of the work has been done, no? > > > > > > Thanks, > > > Pierre > > > > > > Le mar. 23 avr. 2024, 02:03, David Handermann < > > exceptionfact...@apache.org > > > > > > > a écrit : > > > > > > > Team, > > > > > > > > We are approaching 300 Jira issues resolved for version 2.0.0-M3 [1], > > > > including a number of important dependency upgrades and Python > > > > framework improvements. > > > > > > > > There are several open pull requests that are getting close to > > > > completion, which should be possible to include in an upcoming release > > > > version. There is also a significant pull request [2] for NIFI-12998 > > > > [3] that includes some important changes to module directory > > > > hierarchy, providing a clearer distinction between framework and > > > > extensions, and implementing significant cleanup of dependency > > > > declarations across the repository. > > > > > > > > With these changes in view, we should be ready for another milestone > > > > release soon after merging the above pull request. > > > > > > > > Another milestone release seems to be the best course of action for > > > > now, providing the opportunity for additional review and testing > > > > before a full 2.0.0 release version. > > > > > > > > I would be glad to handle Release Manager responsibilities for > > > > 2.0.0-M3 when things are ready. > > > > > > > > Regards, > > > > David Handermann > > > > > > > > [1] > > > > > > > > > https://issues.apache.org/jira/issues/?jql=project%20%3D%20NIFI%20AND%20fixVersion%20%3D%202.0.0-M3 > > > > [2] https://github.com/apache/nifi/pull/8677 > > > > [3] https://issues.apache.org/jira/browse/NIFI-12998 > > > > > > > > >
Re: [DISCUSS] Release Timeline for NiFi 2.0.0-M3
Team, We are getting closer to ready for a release candidate build. Based on some conversations with those working on the user interface, there are still a couple more items to complete this week. Getting a solid build of the new UI into this next version will be very helpful, so getting a few more of those issues resolved should put things in a good position. Regards, David Handermann On Wed, Apr 24, 2024 at 8:32 AM Joe Witt wrote: > > Also agreed there. Profile should be there to exclude perhaps but include > should be default. > > On Wed, Apr 24, 2024 at 6:30 AM David Handermann < > exceptionfact...@apache.org> wrote: > > > Thanks for the replies thus far! > > > > With the goal of including the new UI, it seems like a good time to > > move the module from an optional profile to part of the standard > > build. That would make the release candidate build and review process > > simpler, not requiring the optional build profile. > > > > Regards, > > David Handermann > > > > On Tue, Apr 23, 2024 at 2:42 PM Matt Gilman > > wrote: > > > > > > I agree. Including the updated UI and getting some feedback would be > > great. > > > > > > On Mon, Apr 22, 2024 at 8:50 PM Joe Witt wrote: > > > > > > > Id be a big fan of including the new UI. > > > > > > > > On Mon, Apr 22, 2024 at 5:45 PM Pierre Villard < > > > > pierre.villard...@gmail.com> > > > > wrote: > > > > > > > > > Thanks David, I'm happy to take care of a 1.26 release at the same > > time. > > > > > > > > > > Regarding the M3 release, should the new UI be included and > > instructions > > > > > given on how to access it to start collecting feedback? I'm under the > > > > > impression that almost all of the work has been done, no? > > > > > > > > > > Thanks, > > > > > Pierre > > > > > > > > > > Le mar. 23 avr. 2024, 02:03, David Handermann < > > > > exceptionfact...@apache.org > > > > > > > > > > > a écrit : > > > > > > > > > > > Team, > > > > > > > > > > > > We are approaching 300 Jira issues resolved for version 2.0.0-M3 > > [1], > > > > > > including a number of important dependency upgrades and Python > > > > > > framework improvements. > > > > > > > > > > > > There are several open pull requests that are getting close to > > > > > > completion, which should be possible to include in an upcoming > > release > > > > > > version. There is also a significant pull request [2] for > > NIFI-12998 > > > > > > [3] that includes some important changes to module directory > > > > > > hierarchy, providing a clearer distinction between framework and > > > > > > extensions, and implementing significant cleanup of dependency > > > > > > declarations across the repository. > > > > > > > > > > > > With these changes in view, we should be ready for another > > milestone > > > > > > release soon after merging the above pull request. > > > > > > > > > > > > Another milestone release seems to be the best course of action for > > > > > > now, providing the opportunity for additional review and testing > > > > > > before a full 2.0.0 release version. > > > > > > > > > > > > I would be glad to handle Release Manager responsibilities for > > > > > > 2.0.0-M3 when things are ready. > > > > > > > > > > > > Regards, > > > > > > David Handermann > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > https://issues.apache.org/jira/issues/?jql=project%20%3D%20NIFI%20AND%20fixVersion%20%3D%202.0.0-M3 > > > > > > [2] https://github.com/apache/nifi/pull/8677 > > > > > > [3] https://issues.apache.org/jira/browse/NIFI-12998 > > > > > > > > > > > > > > > > >
Re: [VOTE] Release Apache NiFi 1.26.0 (RC1)
+1 binding - Verified signatures and hashes - Ran build using Maven Wrapper 3.9.6 - Ran build on Ubuntu 22.04 with Azul Zulu JDK 1.8.0-382 x86_64 Thanks Pierre! Regards, David Handermann On Fri, May 3, 2024 at 7:47 AM Pierre Villard wrote: > > Team, > > I am pleased to be calling this vote for the source release of Apache NiFi > 1.26.0. > > Please review the following guide for how to verify a release candidate > build: > > https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification > > The source being voted on the and the convenience binaries are available on > the Apache Distribution Repository: > > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.26.0 > > The build artifacts are available on the Apache Nexus Repository: > > https://repository.apache.org/content/repositories/orgapachenifi-1265 > > Git Tag: nifi-1.26.0-RC1 > Git Commit ID: 338083f6850b61cd2651e41bbd73b3cb5330d98c > GitHub Commit Link: > https://github.com/apache/nifi/commit/338083f6850b61cd2651e41bbd73b3cb5330d98c > > Checksums of nifi-1.26.0-source-release.zip > > SHA256: 75ea201c12bb99672b1f075c9520b5f7df8b24e033ec47121848b351a0d47790 > SHA512: > 5b754d899ce8cff900d5871d44c2fda9224e69fe8a0fe9a7121f3359ed8881300e4d4d0b2fe5befea276e0495c7ebbed04cc307c18527aa7a1cea25a923a > > Release artifacts are signed with the following key: > > https://people.apache.org/keys/committer/pvillard.asc > > KEYS file is available on the Apache Distribution Repository: > > https://dist.apache.org/repos/dist/release/nifi/KEYS > > Issues resolved for this version: 128 > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12354185 > > Release note highlights can be found on the project wiki: > > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.26.0 > > The vote will be open for 72 hours. > > Please download the release candidate and evaluate the necessary items > including checking hashes, signatures, build from source, and test. Then > please vote: > > [] +1 Release this package as nifi-1.26.0 > [] +0 no opinion > [] -1 Do not release this package because...
Re: [discuss] removal of nifi-external module in 2.x/deprecate in 1.x
Joe, Thanks for highlighting the Kafka Connect external module. I concur with your evaluation, I have not observed community usage or questions, so it seems best to remove from the main branch. Regards, David Handermann On Tue, May 7, 2024 at 2:18 PM Otto Fowler wrote: > > I would say nuke it, as you are saying. I would also say you should cross > post this to users@ as they will have relevant input > > > On May 7, 2024 at 1:32:11 PM, Joe Witt (joew...@apache.org) wrote: > > Team, > > The only component remaining is kafka-connect in the nifi-external module. > As noted in [1] it does not see much active maintenance and I never see in > the apache community any questions/usage related either. Kafka is > currently on 3.7.0 and this component is Kafka 2.6 for example. > > We should not carry this forward in the 2.x line. To be clear Kafka > remains a super important component to interact with and our Kafka > components within NiFi do see maintenance including an active PR under > review right now. But this is our last 'nifi-external' remaining and is > not maintained. > > I'll file a PR under this JIRA [1] and its child JIRA for removal on 2x and > deprecation on 1x but wanted to flag this here in case there is some > substantial usage and/or maintenance/update coming soon. Even if desire > remains to keep it alive that should be considered to occur in another repo > or by another group that leverages it - but please let me know if there is > apache community usage/mx I'm not aware of. > > Thoughts? > > [1] https://issues.apache.org/jira/browse/NIFI-13171 > > Thanks > Joe
Re: [discuss] identifying key components needed updates which require deeper review/consideration
Joe, Thanks for highlighting these areas to be updated. I was able to address the QuestDB update with a couple minor adjustments, and submitted a PR [1]. Regards, David Handermann [1] https://github.com/apache/nifi/pull/8773 On Tue, May 7, 2024 at 2:58 PM Chris Sampson wrote: > > I’ll pick up the Elasticsearch dependency upgrade. > > I’ll try to test it again OpenSearch as well, if for no reason other than to > prove either way whether the current NiFi components will work with the AWS > service, although I may not get to do that and instead testing will focus on > Elasticsearch 8.x instances (and the existing Integration Tests within NiFi). > > > Cheers, > > --- > Chris Sampson > IT Consultant > chris.samp...@naimuri.com > > > > On 7 May 2024, at 20:54, Joe Witt wrote: > > > > Yeah sorry - what I wrote for the Hive one was a bit nonsensical. I tried > > to clarify it better in the JIRA. > > > > Thanks > > > > On Tue, May 7, 2024 at 12:38 PM Matt Burgess wrote: > > > >> For [1] Hive 3 was removed in > >> https://issues.apache.org/jira/browse/NIFI-12981, if you mean the Hive > >> dependency from Apache Iceberg we probably need a release from them that > >> brings in Hive 4 dependencies? Or do you suggest we bring the Hive > >> components back as long as we update the version? > >> > >> I'll take the Flyway one in the meantime, database integrations are my > >> blessing and my curse :) If that goes more smoothly than I expect I can > >> grab the QuestDB one too, will update the Jiras as appropriate. > >> > >> Regards, > >> Matt > >> > >> On Tue, May 7, 2024 at 1:56 PM Joe Witt wrote: > >> > >>> Team, > >>> > >>> We have made massive strides going from thousands of out of date > >> libraries > >>> to dozens in the 2.x line vs any prior releases. And we can/should stay > >> on > >>> top of these better going forward. A few key bits that are outstanding > >>> with no obvious immediate work underway are as follows. If anyone is > >>> working on these, willing to work on these, please volunteer and let us > >>> know. For components that we cannot find active engagement on we should > >>> consider removal though these aren't great examples as they're indeed > >>> common usage components. > >>> > >>> [1] https://issues.apache.org/jira/browse/NIFI-13173 > >>> Hive 4 recently came out but Hive also gives us some of the most > >>> complicated POM/library mangling and oldest dependencies to > >> override/etc.. > >>> Putting serious detailed focus here is very valuable from a maintenance > >>> point of view. While existing usage of Hive 3 likely remains for some > >>> time, those users are like NiFi 1.x users anyway. In any case if there > >> are > >>> motivated developers interested to see our Hive components be healthy > >>> please do dive in on this. > >>> > >>> [2] https://issues.apache.org/jira/browse/NIFI-13169 > >>> The NiFI registry is a major release behind Flyway DB. We need to stay > >>> current given the importance of this component. > >>> > >>> [3] https://issues.apache.org/jira/browse/NIFI-13168 > >>> We are a few incrementals behind and a minor release behind QuestDB. I > >>> tried bumping the incremental but it broke tests. > >>> > >>> [4] https://issues.apache.org/jira/browse/NIFI-11686 > >>> We are a full major release behind and several incremental/minors as > >> well. > >>> Apparently there is some concern related to the community changes in > >>> Elastic vs OpenSearch/etc.. but regardless we need to stay active in > >>> maintaining this component. We really should get to the latest main. > >>> > >>> Thanks to anyone who can help tackle these and make progress both in > >> terms > >>> of doing the upgrade/update work, validating tests, fixing licenses, but > >>> also reviews. > >>> > >>> Thanks > >>> Joe > >>> > >> >
Re: [DISCUSS] Release Timeline for NiFi 2.0.0-M3
Team, There has been a significant amount of progress on the new UI, and it is now part of the default build in the main branch. There are a couple more dependency updates to be incorporated, and I plan initiating a release candidate build for 2.0.0-M3 in the next day or two at the latest. Regards, David Handermann On Fri, May 3, 2024 at 4:02 AM Pierre Villard wrote: > > I'm starting the process for the 1.26 RC as I don't believe we're waiting > for any additional PRs there. > > Le jeu. 2 mai 2024 à 07:30, Joe Witt a écrit : > > > Matt > > > > You are referring to NIFI-12998 and it causing rebasing to be necessary for > > outstanding PRs? > > > > The rebase should be quite straight forward for most cases. > > > > If you have any questions Im happy to help. > > > > I rebased a tremendous number of times to keep up with main changing while > > the PR took several weeks to create and test and validate against previous > > builds. Also rebased repeatedly during the review and feedback stage. As > > a result I got to practice my rebase skills about 20 times across 50 or > > more commits to main during that time. > > > > On your pr branch > > > > git fetch —all > > git rebase upstream/main > > git status > > fix any conflicts > > git add . > > git rebase —continue > > > > If any conflicts it would likely be for new modules created in the pr which > > would just be moving from a path in nifi-nar-bundles to > > nifi-extension-bundles. > > > > If you have a pr in mind you have a question on for the rebase let me know. > > > > Thanks > > Joe > > > > On Wed, May 1, 2024 at 9:51 PM Matt Burgess wrote: > > > > > One thing that was mentioned was an included Jira for "providing a > > clearer > > > distinction between framework and > > > extensions," This involved moving extensions to a new module and for > > > community folks this is causing problems with any > > > new improvements in-flight before that. Seems like it needed more > > > announcement than an upcoming RC due to the impact? > > > > > > On Mon, Apr 29, 2024 at 10:22 AM David Handermann < > > > exceptionfact...@apache.org> wrote: > > > > > > > Team, > > > > > > > > We are getting closer to ready for a release candidate build. Based on > > > > some conversations with those working on the user interface, there are > > > > still a couple more items to complete this week. Getting a solid build > > > > of the new UI into this next version will be very helpful, so getting > > > > a few more of those issues resolved should put things in a good > > > > position. > > > > > > > > Regards, > > > > David Handermann > > > > > > > > On Wed, Apr 24, 2024 at 8:32 AM Joe Witt wrote: > > > > > > > > > > Also agreed there. Profile should be there to exclude perhaps but > > > > include > > > > > should be default. > > > > > > > > > > On Wed, Apr 24, 2024 at 6:30 AM David Handermann < > > > > > exceptionfact...@apache.org> wrote: > > > > > > > > > > > Thanks for the replies thus far! > > > > > > > > > > > > With the goal of including the new UI, it seems like a good time to > > > > > > move the module from an optional profile to part of the standard > > > > > > build. That would make the release candidate build and review > > process > > > > > > simpler, not requiring the optional build profile. > > > > > > > > > > > > Regards, > > > > > > David Handermann > > > > > > > > > > > > On Tue, Apr 23, 2024 at 2:42 PM Matt Gilman < > > matt.c.gil...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > > > > I agree. Including the updated UI and getting some feedback would > > > be > > > > > > great. > > > > > > > > > > > > > > On Mon, Apr 22, 2024 at 8:50 PM Joe Witt > > > wrote: > > > > > > > > > > > > > > > Id be a big fan of including the new UI. > > > > > > > > > > > > > > > > On Mon, Apr 22, 2024 at 5:45 PM Pierre Villard < > > > > > > > > pierre.villard.
Re: [DISCUSS] Release Timeline for NiFi 2.0.0-M3
Team, We have had great progress this week on a number of important details. At this point, I am planning on starting the release candidate build process first thing next week. Regards, David Handermann On Tue, May 7, 2024 at 4:09 PM David Handermann wrote: > > Team, > > There has been a significant amount of progress on the new UI, and it > is now part of the default build in the main branch. There are a > couple more dependency updates to be incorporated, and I plan > initiating a release candidate build for 2.0.0-M3 in the next day or > two at the latest. > > Regards, > David Handermann > > On Fri, May 3, 2024 at 4:02 AM Pierre Villard > wrote: > > > > I'm starting the process for the 1.26 RC as I don't believe we're waiting > > for any additional PRs there. > > > > Le jeu. 2 mai 2024 à 07:30, Joe Witt a écrit : > > > > > Matt > > > > > > You are referring to NIFI-12998 and it causing rebasing to be necessary > > > for > > > outstanding PRs? > > > > > > The rebase should be quite straight forward for most cases. > > > > > > If you have any questions Im happy to help. > > > > > > I rebased a tremendous number of times to keep up with main changing > > > while > > > the PR took several weeks to create and test and validate against previous > > > builds. Also rebased repeatedly during the review and feedback stage. As > > > a result I got to practice my rebase skills about 20 times across 50 or > > > more commits to main during that time. > > > > > > On your pr branch > > > > > > git fetch —all > > > git rebase upstream/main > > > git status > > > fix any conflicts > > > git add . > > > git rebase —continue > > > > > > If any conflicts it would likely be for new modules created in the pr > > > which > > > would just be moving from a path in nifi-nar-bundles to > > > nifi-extension-bundles. > > > > > > If you have a pr in mind you have a question on for the rebase let me > > > know. > > > > > > Thanks > > > Joe > > > > > > On Wed, May 1, 2024 at 9:51 PM Matt Burgess wrote: > > > > > > > One thing that was mentioned was an included Jira for "providing a > > > clearer > > > > distinction between framework and > > > > extensions," This involved moving extensions to a new module and for > > > > community folks this is causing problems with any > > > > new improvements in-flight before that. Seems like it needed more > > > > announcement than an upcoming RC due to the impact? > > > > > > > > On Mon, Apr 29, 2024 at 10:22 AM David Handermann < > > > > exceptionfact...@apache.org> wrote: > > > > > > > > > Team, > > > > > > > > > > We are getting closer to ready for a release candidate build. Based on > > > > > some conversations with those working on the user interface, there are > > > > > still a couple more items to complete this week. Getting a solid build > > > > > of the new UI into this next version will be very helpful, so getting > > > > > a few more of those issues resolved should put things in a good > > > > > position. > > > > > > > > > > Regards, > > > > > David Handermann > > > > > > > > > > On Wed, Apr 24, 2024 at 8:32 AM Joe Witt wrote: > > > > > > > > > > > > Also agreed there. Profile should be there to exclude perhaps but > > > > > include > > > > > > should be default. > > > > > > > > > > > > On Wed, Apr 24, 2024 at 6:30 AM David Handermann < > > > > > > exceptionfact...@apache.org> wrote: > > > > > > > > > > > > > Thanks for the replies thus far! > > > > > > > > > > > > > > With the goal of including the new UI, it seems like a good time > > > > > > > to > > > > > > > move the module from an optional profile to part of the standard > > > > > > > build. That would make the release candidate build and review > > > process > > > > > > > simpler, not requiring the optional build profile. > > > > > > > > > > > > > > Regards, > > > > > > > David
[VOTE] Release Apache NiFi 2.0.0-M3 (RC1)
Team, I am pleased to be calling this vote for the source release of Apache NiFi 2.0.0-M3. Please review the following guide for how to verify a release candidate build: https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification The source being voted on and the convenience binaries are available on the Apache Distribution Repository: https://dist.apache.org/repos/dist/dev/nifi/nifi-2.0.0-M3 The build artifacts are available on the Apache Nexus Repository: https://repository.apache.org/content/repositories/orgapachenifi-1269 Git Tag: nifi-2.0.0-M3-RC1 Git Commit ID: f2215c6522a5571189290760c55f0317f8562cbd GitHub Commit Link: https://github.com/apache/nifi/commit/f2215c6522a5571189290760c55f0317f8562cbd Checksums of nifi-2.0.0-M3-source-release.zip SHA256: d471a0a9e4e04faf13bbe13c7d83be6f8002fba598bf0968a3c1b339802a185a SHA512: cd0b8bbd3fe4ea7ebe8cdac6ac8a7afa97fd7b6a521c2cbcb2c0cdc94899b652bf3726c8fe432e16f44a9dc81907414bbb42e03113f0cd9fb51aa1de9cd727a7 Release artifacts are signed with the following key: https://people.apache.org/keys/committer/exceptionfactory.asc KEYS file is available on the Apache Distribution Repository: https://dist.apache.org/repos/dist/release/nifi/KEYS Issues resolved for this version: 411 https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12354155 Release note highlights can be found on the project wiki: https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.0.0-M3 The vote will be open for 72 hours. Please download the release candidate and evaluate the necessary items including checking hashes, signatures, build from source, and test. Then please vote: [] +1 Release this package as nifi-2.0.0-M3 [] +0 no opinion [] -1 Do not release this package because...
Re: [VOTE] Release Apache NiFi 2.0.0-M3 (RC1)
+1 binding On Mon, May 13, 2024 at 10:48 PM David Handermann wrote: > > Team, > > I am pleased to be calling this vote for the source release of Apache > NiFi 2.0.0-M3. > > Please review the following guide for how to verify a release candidate build: > > https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification > > The source being voted on and the convenience binaries are available > on the Apache Distribution Repository: > > https://dist.apache.org/repos/dist/dev/nifi/nifi-2.0.0-M3 > > The build artifacts are available on the Apache Nexus Repository: > > https://repository.apache.org/content/repositories/orgapachenifi-1269 > > Git Tag: nifi-2.0.0-M3-RC1 > Git Commit ID: f2215c6522a5571189290760c55f0317f8562cbd > GitHub Commit Link: > https://github.com/apache/nifi/commit/f2215c6522a5571189290760c55f0317f8562cbd > > Checksums of nifi-2.0.0-M3-source-release.zip > > SHA256: d471a0a9e4e04faf13bbe13c7d83be6f8002fba598bf0968a3c1b339802a185a > SHA512: > cd0b8bbd3fe4ea7ebe8cdac6ac8a7afa97fd7b6a521c2cbcb2c0cdc94899b652bf3726c8fe432e16f44a9dc81907414bbb42e03113f0cd9fb51aa1de9cd727a7 > > Release artifacts are signed with the following key: > > https://people.apache.org/keys/committer/exceptionfactory.asc > > KEYS file is available on the Apache Distribution Repository: > > https://dist.apache.org/repos/dist/release/nifi/KEYS > > Issues resolved for this version: 411 > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12354155 > > Release note highlights can be found on the project wiki: > > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.0.0-M3 > > The vote will be open for 72 hours. > > Please download the release candidate and evaluate the necessary items > including checking hashes, signatures, build from source, and test. > Then please vote: > > [] +1 Release this package as nifi-2.0.0-M3 > [] +0 no opinion > [] -1 Do not release this package because...
[RESULT][VOTE] Release Apache NiFi 2.0.0-M3 (RC1)
Apache NiFi Community, I am pleased to announce that the 2.0.0-M3 release of Apache NiFi passes: 9 +1 (binding) votes 10 +1 (non-binding) votes 0 0 votes 0 -1 votes Thanks to all who helped make this release possible! The release artifacts will be published in the next 24 hours. Here is the vote thread: https://lists.apache.org/thread/ocxc43gklt2qb36n6wxjp0x868bq71gq
[ANNOUNCE] Apache NiFi 2.0.0-M3 Released
The Apache NiFi Team is pleased to announce the release of Apache NiFi 2.0.0-M3. Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data. https://nifi.apache.org The release artifacts can be downloaded from the project website. https://nifi.apache.org/download/ Maven artifacts have been released and mirrored according to ASF artifact distribution processes. Issues resolved in Apache NiFi 2.0.0-M3 are listed in Jira Release Notes. https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12354155 Highlights of the release are available on the project wiki page: https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.0.0-M3 Thank you, The Apache NiFi Team
Re: [DISCUSS] Release NiFi NAR Maven Plugin
Thanks Pierre, sounds good! Regards, David Handermann On Tue, May 21, 2024 at 8:36 AM Pierre Villard wrote: > > Thanks will go ahead with 1.6.0 release process > > Le lun. 20 mai 2024 à 19:30, Matt Burgess a écrit : > > > +1 for a 1.6.0 release > > > > On Mon, May 20, 2024 at 1:25 PM Bryan Bende wrote: > > > > > Thanks Pierre. I agree it would be good to kick out a release. I would > > > lean towards 1.6.0 since the commits seem to be improvements rather > > > than bug fixes for a patch. > > > > > > On Mon, May 20, 2024 at 1:08 PM Pierre Villard > > > wrote: > > > > > > > > Hi all, > > > > > > > > We just merged a couple of improvements for the NiFi NAR Maven Plugin > > and > > > > another improvement has been merged some time ago for which the > > community > > > > expressed interest [1]. I think it could be a good time to release a > > new > > > > version, either as 1.5.2 or as 1.6.0. Once we have a new release, there > > > > will be a need for updating the NiFi code base and I have a PR ready > > for > > > > that [2]. > > > > > > > > I'm happy to help with the release process. > > > > > > > > Do you agree it is time for a new release or do you see additional > > > changes > > > > we should make? > > > > > > > > Thanks, > > > > Pierre > > > > > > > > [1] https://github.com/apache/nifi-maven/pull/35 > > > > [2] https://issues.apache.org/jira/browse/NIFI-13267 > > > > >
Re: [DISCUSS] Release NiFi NAR Maven Plugin
Pierre, Thanks for highlighting the differences for Flow Analysis Rules. As far as making this a 2.0.0 release version, do you anticipate including any breaking changes? The current version of the plugin is built with Java 8, so this could be updated to Java 21, which would be a breaking change, however, that should be handled before the release process. If there are no breaking changes, I recommend a version 1.6.0 release before a 2.0.0 release of the plugin. Regards, David Handermann On Tue, May 21, 2024 at 10:19 AM Pierre Villard wrote: > > Given that the flow analysis rules are something specific to NiFi 2, and > given that this is an opportunity to do some house cleaning, I'll go > directly to a 2.0.0 release directly and have it used only on the main > branch. The things that have been added as improvements are not really > required for 1.x anyway. > > Le mar. 21 mai 2024 à 15:40, David Handermann > a écrit : > > > Thanks Pierre, sounds good! > > > > Regards, > > David Handermann > > > > On Tue, May 21, 2024 at 8:36 AM Pierre Villard > > wrote: > > > > > > Thanks will go ahead with 1.6.0 release process > > > > > > Le lun. 20 mai 2024 à 19:30, Matt Burgess a écrit > > : > > > > > > > +1 for a 1.6.0 release > > > > > > > > On Mon, May 20, 2024 at 1:25 PM Bryan Bende wrote: > > > > > > > > > Thanks Pierre. I agree it would be good to kick out a release. I > > would > > > > > lean towards 1.6.0 since the commits seem to be improvements rather > > > > > than bug fixes for a patch. > > > > > > > > > > On Mon, May 20, 2024 at 1:08 PM Pierre Villard > > > > > wrote: > > > > > > > > > > > > Hi all, > > > > > > > > > > > > We just merged a couple of improvements for the NiFi NAR Maven > > Plugin > > > > and > > > > > > another improvement has been merged some time ago for which the > > > > community > > > > > > expressed interest [1]. I think it could be a good time to release > > a > > > > new > > > > > > version, either as 1.5.2 or as 1.6.0. Once we have a new release, > > there > > > > > > will be a need for updating the NiFi code base and I have a PR > > ready > > > > for > > > > > > that [2]. > > > > > > > > > > > > I'm happy to help with the release process. > > > > > > > > > > > > Do you agree it is time for a new release or do you see additional > > > > > changes > > > > > > we should make? > > > > > > > > > > > > Thanks, > > > > > > Pierre > > > > > > > > > > > > [1] https://github.com/apache/nifi-maven/pull/35 > > > > > > [2] https://issues.apache.org/jira/browse/NIFI-13267 > > > > > > > > > > >
Re: [DISCUSS] Release NiFi NAR Maven Plugin
Pierre, Thanks for clarifying. I agree that going to Java 21 makes sense, and aligns with the NiFi main branch. That would be a breaking change, warranting the 2.0.0 version, so that sounds like a good plan. Regards, David Handermann On Tue, May 21, 2024 at 10:32 AM Pierre Villard wrote: > > My plan was to go with the latest for everything (including Java 21). > This way, NiFi NAR Maven plugin 2.0.0 would be used only for NiFi 2.x. > I actually think that the addition of the extension type "flow analysis > rules" would break things if used in the 1.x line (unless I also update the > enum in NiFi 1.x to include flow analysis rules as an extension type but > not sure that would be a great approach). > > Le mar. 21 mai 2024 à 17:24, David Handermann > a écrit : > > > Pierre, > > > > Thanks for highlighting the differences for Flow Analysis Rules. As > > far as making this a 2.0.0 release version, do you anticipate > > including any breaking changes? > > > > The current version of the plugin is built with Java 8, so this could > > be updated to Java 21, which would be a breaking change, however, that > > should be handled before the release process. > > > > If there are no breaking changes, I recommend a version 1.6.0 release > > before a 2.0.0 release of the plugin. > > > > Regards, > > David Handermann > > > > On Tue, May 21, 2024 at 10:19 AM Pierre Villard > > wrote: > > > > > > Given that the flow analysis rules are something specific to NiFi 2, and > > > given that this is an opportunity to do some house cleaning, I'll go > > > directly to a 2.0.0 release directly and have it used only on the main > > > branch. The things that have been added as improvements are not really > > > required for 1.x anyway. > > > > > > Le mar. 21 mai 2024 à 15:40, David Handermann < > > exceptionfact...@apache.org> > > > a écrit : > > > > > > > Thanks Pierre, sounds good! > > > > > > > > Regards, > > > > David Handermann > > > > > > > > On Tue, May 21, 2024 at 8:36 AM Pierre Villard > > > > wrote: > > > > > > > > > > Thanks will go ahead with 1.6.0 release process > > > > > > > > > > Le lun. 20 mai 2024 à 19:30, Matt Burgess a > > écrit > > > > : > > > > > > > > > > > +1 for a 1.6.0 release > > > > > > > > > > > > On Mon, May 20, 2024 at 1:25 PM Bryan Bende > > wrote: > > > > > > > > > > > > > Thanks Pierre. I agree it would be good to kick out a release. I > > > > would > > > > > > > lean towards 1.6.0 since the commits seem to be improvements > > rather > > > > > > > than bug fixes for a patch. > > > > > > > > > > > > > > On Mon, May 20, 2024 at 1:08 PM Pierre Villard > > > > > > > wrote: > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > We just merged a couple of improvements for the NiFi NAR Maven > > > > Plugin > > > > > > and > > > > > > > > another improvement has been merged some time ago for which the > > > > > > community > > > > > > > > expressed interest [1]. I think it could be a good time to > > release > > > > a > > > > > > new > > > > > > > > version, either as 1.5.2 or as 1.6.0. Once we have a new > > release, > > > > there > > > > > > > > will be a need for updating the NiFi code base and I have a PR > > > > ready > > > > > > for > > > > > > > > that [2]. > > > > > > > > > > > > > > > > I'm happy to help with the release process. > > > > > > > > > > > > > > > > Do you agree it is time for a new release or do you see > > additional > > > > > > > changes > > > > > > > > we should make? > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Pierre > > > > > > > > > > > > > > > > [1] https://github.com/apache/nifi-maven/pull/35 > > > > > > > > [2] https://issues.apache.org/jira/browse/NIFI-13267 > > > > > > > > > > > > > > > > > > >
Re: Handle of Datagram Socket used in ListenUdp
Hi Atol, The use case you described for two-way communication over UDP is understandable, but I recommend taking a different approach for implementation. The ListenUDP Processor is designed to receive packets without sending a response, so the DatagramChannelDispatcher is not designed for two-way communication. Rather than attempting to build on these internal NiFi classes, I recommend researching and using the Netty framework [1] to build a datagram service and corresponding Processor. Netty is not trivial, but if you are able to follow the architecture, it provides a good foundation for building network services. Going through the Netty User Guide [1] provides a helpful introduction. The NettyEventServerFactory [2] in the nifi-event-transport module provides some low-level bootstrap support, but still requires wiring in custom Netty Channel Handlers. The Netty examples directory has a number of protocol implementations that are useful for understanding the framework. In particular, the quote-of-the-moment [3] example includes a UDP-based server implementation that could be helpful. Although this is not a simple exercise, investing the time in Netty should provide a much better foundation for integration than the purpose-built components that support the current ListenUDP Processor. Regards, David Handermann [1] https://netty.io/wiki/user-guide-for-4.x.html [2] https://github.com/apache/nifi/blob/main/nifi-extension-bundles/nifi-extension-utils/nifi-event-transport/src/main/java/org/apache/nifi/event/transport/netty/NettyEventServerFactory.java [3] https://github.com/netty/netty/tree/4.1/example/src/main/java/io/netty/example/qotm On Thu, Jun 6, 2024 at 9:26 AM Atul Saroha wrote: > > Hi, > > We are overriding ListenUDP to support response on the Datagram channel to > support response event. OnTrigger method calls "postProcess(context, > session, allEvents)" method which we can override. > However, DatagramChannelDispatcher is setting StandardEvent with > ChannelResponder as null. Also we are not able to find any example on how > to create ChannelResponder to send udp responses. > > Kindly help us with code references which we can use to create Datagram > ChannelResponder to send UDP response . > > > Thanks and Regards, > Atul Saroha > > > > On Wed, Jun 5, 2024 at 10:59 AM Atul Saroha wrote: > > > Hi, > > > > We are using Nifi to listen the UDP packets. However, we also have to send > > UDP packets as a response back to the calling sender IP and port using the > > same receiver port used in NiFi ListenUDP processor > > We are looking to write a custom processor to send UDP packets back. To do > > this, we need the handle of the Datagram socket which is used in ListenUDP. > > > > Please help us with some code snapshots to use the same datagram socket > > object in our custom processor. > > > > Thanks and Regards, > > Atul Saroha > >
[DISCUSS] Release Planning for NiFi 2.0.0-M4 and 1.26.1
Team, With recent changes on the main branch to make the nifi-web-frontend module the default version of the user interface, we are in a good position for another milestone release of NiFi 2.0.0. As always, we also have a number of bug fixes and dependency upgrades that are ready for release. It seems helpful to have a Milestone 4 release version, with a focus on the new user interface, as it will provide an opportunity for additional feedback before a full general availability version of NiFi 2.0.0. Thanks again to Matt Gilman, Rob Fellows, and Scott Aslan for their incredible effort to rebuild the user interface on modern frameworks and libraries! Although NiFi 2.0.0 remains the primary focus, there have been several important bug fixes applied to the support branch [1]. I renamed the 1.27.0 version to 1.26.1 since the number of changes and the types of changes are narrowly focused. Pierre Villard mentioned that he is ready to handle the Release Manager responsibilities for the 1.26.1 version, and I would be glad to handle the 2.0.0-M4 version. I am in the process of reviewing open pull requests, and I believe we are close to ready for release candidates this week. Regards, David Handermann [1] https://issues.apache.org/jira/projects/NIFI/versions/12354651
Re: Python extensions in their own repo...
Joe, Thanks for raising the discussion, and thanks to everyone for the feedback thus far. This tracks our previous discussion on the topic [1]. I am also strongly in favor of separating out extensions into their own repositories for many of the reasons already mentioned. Starting with a single dedicated repository named nifi-python-extensions should be a good opportunity to prove out the concept. I agree considering the Java extensions is more involved, and I think should consider that separately. I would be glad to handle the initial work of creating the new repository and setting up the initial build structure. I am familiar with the work necessary to publish to the PyPI repository, and that could provide an optional distribution channel. We would still make the source distribution available through standard Apache channels, following standard project policies. Based on the current structure, it should be straightforward to download an archive of the Python extensions and expand that for those who want to have them as part of their NiFi installation. Once the initial repository is in place, that last initial step would be a pull request to remove the Python extensions from the main repository. I can proceed along these lines, unless any substantive objections come up, and the pull request process will also provide opportunity for additional consideration and review. Regards, David Handermann [1] https://lists.apache.org/thread/nok561sg1dzw3zrott06gkl34hdjxbb3 On Fri, Jun 21, 2024, 9:14 PM Marton Szasz wrote: > Hi Joe, Arpad and all, > > I'm strongly in favor of moving all Python components to a separate > repository. It could be called apache/nifi-python-extensions or > -components, and contain all Python components that this community > maintains. I would prefer that over a separate repo for each extension, > because it seems easier to keep track of all components maintained by > the community if they are in the same repo, than if they were separate. > > Since MiNiFi C++ implemented a large subset of the same Python API, I > think it makes our lives easier if we share the code, and keep the > Python components in their own dedicated location. As NiFi and MiNiFi > C++ both approach their next major version, and we commit to a stable > Python API, I expect it to become easier to maintain the Python > components separately, targeting this stable API, and we could align the > release frequency with the maintenance needs of the Python components. > > I'm neutral of whether to package them with convenience binaries or > leave that up to the user. Hopefully we can come up with a user-friendly > way to install them if they're not included. I wouldn't include them in > the source tarballs. > > I would keep the Java components separate from Python components. > Whether that's in the NiFi repo or somewhere else, both are fine with me. > > Regarding introducing breaking changes: on the NiFi side, unit tests > should cover the API well enough, and after 2.0 GA, I expect it to > remain backwards-compatible until the next major version. So I think the > API will not be a moving target (things only added, not changed), and it > will be easy to keep things working. But I think we should set up > automated testing that runs tests with the extensions, checking their > functionality with NiFi 2.0, NiFi latest, and at least one MiNiFi C++ > version, to catch breakages early if they rear up their head anyway. > > Thanks and have a great weekend, > Marton > > > On 6/21/24 23:18, Joe Witt wrote: > > "I would suggest starting with > > moving the Python ones to a dedicated repo, let's have a workflow > > established and polished there, might follow with some Java ones in case > it > > works well." > > > > Yeah kinda where my head is too > > > > On Fri, Jun 21, 2024 at 2:07 PM Arpad Boda wrote: > > > >> Joe, > >> > >> Interesting thoughts, I see a lot of pros and cons. Let me list the most > >> important ones of both: > >> +cves in extensions doesn't make nifi "vulnerable" automatically as they > >> live in a different repo. > >> +the responsibility of being up-to-date is being moved to the > maintainers > >> of the given extension, same applies for the stability of the tests > >> covering that extension > >> > >> -easier to introduce breaking changes accidentally: a breaking change > might > >> go through and get committed. Especially in case of Java extensions, > they > >> python api is pretty thin (yet!). Only an extension developer will find > it, > >> most probably not immediately, when things already depend on the > breaking > >> change and
Re: [DISCUSS] Release Planning for NiFi 2.0.0-M4 and 1.26.1
Team, Thanks for the replies thus far. In a conversation with Shane, and reviewing the current status of the PR [1], it looks like there is a bit more work to do, plus review, to get that completed. Other than that, I am planning to review the Python API PR for NIFI-13427 [2] and then I think we are in a good position for an initial release candidate build. Regards, David Handermann [1] https://github.com/apache/nifi/pull/8974 [2] https://github.com/apache/nifi/pull/9000 On Mon, Jun 17, 2024 at 11:12 AM Shane Ardell wrote: > > Hi team, > > Pierre is correct that I'm aiming to have a PR open today to expose the > flow analysis rules engine in the new UI. I'll be sure to address any > feedback left on my PR asap. > > Best, > Shane > > On Mon, Jun 17, 2024 at 12:03 PM Pierre Villard > wrote: > > > Hi team, > > > > Happy to take care of the 1.26.1. > > > > I believe Shane will open a PR today to have the UI bits of the rules > > engine available in the new UI. We may want to wait for this, assuming this > > would be merged quickly, to get as much feedback as possible on the new UI. > > > > Very exciting to see all of this coming together! Thanks David for starting > > this discussion. > > > > Pierre > > > > Le lun. 17 juin 2024 à 17:39, David Handermann < > > exceptionfact...@apache.org> > > a écrit : > > > > > Team, > > > > > > With recent changes on the main branch to make the nifi-web-frontend > > > module the default version of the user interface, we are in a good > > > position for another milestone release of NiFi 2.0.0. > > > > > > As always, we also have a number of bug fixes and dependency upgrades > > > that are ready for release. It seems helpful to have a Milestone 4 > > > release version, with a focus on the new user interface, as it will > > > provide an opportunity for additional feedback before a full general > > > availability version of NiFi 2.0.0. Thanks again to Matt Gilman, Rob > > > Fellows, and Scott Aslan for their incredible effort to rebuild the > > > user interface on modern frameworks and libraries! > > > > > > Although NiFi 2.0.0 remains the primary focus, there have been several > > > important bug fixes applied to the support branch [1]. I renamed the > > > 1.27.0 version to 1.26.1 since the number of changes and the types of > > > changes are narrowly focused. Pierre Villard mentioned that he is > > > ready to handle the Release Manager responsibilities for the 1.26.1 > > > version, and I would be glad to handle the 2.0.0-M4 version. > > > > > > I am in the process of reviewing open pull requests, and I believe we > > > are close to ready for release candidates this week. > > > > > > Regards, > > > David Handermann > > > > > > [1] https://issues.apache.org/jira/projects/NIFI/versions/12354651 > > > > >
Re: Python extensions in their own repo...
Team, Thanks to collaboration with Joe Witt, the new nifi-python-extensions repository [1] is now populated with the initial set of Python Processors. The repository includes a standard GitHub workflow for pull request validation that checks license headers and Python code formatting. The project uses Hatch [2] to run code formatting as well as build source and binary distribution packages. The source distribution and binary wheel packages both contain the Python Processors, which can be placed into an Apache NiFi 2 installation. The source distribution archive will provide a suitable release candidate file when we are ready to release a version of nifi-python-extensions. In Jira, there is a new python-extensions-2.0.0 version to target for features and fixes. There is certainly more room for documentation and improvement, but this should provide a reasonable foundation for decoupled Python Processor development efforts. Regards, David Handermann [1] https://github.com/apache/nifi-python-extensions [2] https://hatch.pypa.io On Sat, Jun 22, 2024 at 9:06 AM David Handermann wrote: > > Joe, > > Thanks for raising the discussion, and thanks to everyone for the feedback > thus far. This tracks our previous discussion on the topic [1]. > > I am also strongly in favor of separating out extensions into their own > repositories for many of the reasons already mentioned. Starting with a > single dedicated repository named nifi-python-extensions should be a good > opportunity to prove out the concept. I agree considering the Java extensions > is more involved, and I think should consider that separately. > > I would be glad to handle the initial work of creating the new repository and > setting up the initial build structure. I am familiar with the work necessary > to publish to the PyPI repository, and that could provide an optional > distribution channel. We would still make the source distribution available > through standard Apache channels, following standard project policies. Based > on the current structure, it should be straightforward to download an archive > of the Python extensions and expand that for those who want to have them as > part of their NiFi installation. > > Once the initial repository is in place, that last initial step would be a > pull request to remove the Python extensions from the main repository. > > I can proceed along these lines, unless any substantive objections come up, > and the pull request process will also provide opportunity for additional > consideration and review. > > Regards, > David Handermann > > [1] https://lists.apache.org/thread/nok561sg1dzw3zrott06gkl34hdjxbb3 > > On Fri, Jun 21, 2024, 9:14 PM Marton Szasz wrote: >> >> Hi Joe, Arpad and all, >> >> I'm strongly in favor of moving all Python components to a separate >> repository. It could be called apache/nifi-python-extensions or >> -components, and contain all Python components that this community >> maintains. I would prefer that over a separate repo for each extension, >> because it seems easier to keep track of all components maintained by >> the community if they are in the same repo, than if they were separate. >> >> Since MiNiFi C++ implemented a large subset of the same Python API, I >> think it makes our lives easier if we share the code, and keep the >> Python components in their own dedicated location. As NiFi and MiNiFi >> C++ both approach their next major version, and we commit to a stable >> Python API, I expect it to become easier to maintain the Python >> components separately, targeting this stable API, and we could align the >> release frequency with the maintenance needs of the Python components. >> >> I'm neutral of whether to package them with convenience binaries or >> leave that up to the user. Hopefully we can come up with a user-friendly >> way to install them if they're not included. I wouldn't include them in >> the source tarballs. >> >> I would keep the Java components separate from Python components. >> Whether that's in the NiFi repo or somewhere else, both are fine with me. >> >> Regarding introducing breaking changes: on the NiFi side, unit tests >> should cover the API well enough, and after 2.0 GA, I expect it to >> remain backwards-compatible until the next major version. So I think the >> API will not be a moving target (things only added, not changed), and it >> will be easy to keep things working. But I think we should set up >> automated testing that runs tests with the extensions, checking their >> functionality with NiFi 2.0, NiFi latest, and at least one MiNiFi C++ >> version, to catch breakages early if they rear up
[VOTE] Release Apache NiFi 2.0.0-M4 (RC1)
Team, I am pleased to be calling this vote for the source release of Apache NiFi 2.0.0-M4. Please review the following guide for how to verify a release candidate build: https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification The source being voted on the and the convenience binaries are available on the Apache Distribution Repository: https://dist.apache.org/repos/dist/dev/nifi/nifi-2.0.0-M4 The build artifacts are available on the Apache Nexus Repository: https://repository.apache.org/content/repositories/orgapachenifi-1275 Git Tag: nifi-2.0.0-M4-RC1 Git Commit ID: 19c5be01d463bc38ec0f5008549a2a42e589436d GitHub Commit Link: https://github.com/apache/nifi/commit/19c5be01d463bc38ec0f5008549a2a42e589436d Checksums of nifi-2.0.0-M4-source-release.zip SHA256: d882f05cec09ee1bfafaa3d4cde8f8660512d09765b5c400471f3a6e014029a6 SHA512: d429cd67fb0b7d9737c59cb834106d7b6e25cbdb91e3ecc5290be865a1313cbebbc314c2e0e54228226f021c44f0a86c745a18c148247c632a739c871c5fa013 Release artifacts are signed with the following key: https://people.apache.org/keys/committer/exceptionfactory.asc KEYS file is available on the Apache Distribution Repository: https://dist.apache.org/repos/dist/release/nifi/KEYS Issues resolved for this version: 153 https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12354678 Release note highlights can be found on the project wiki: https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.0.0-M4 The vote will be open for 72 hours. Please download the release candidate and evaluate the necessary items including checking hashes, signatures, build from source, and test. Then please vote: [] +1 Release this package as nifi-2.0.0-M4 [] +0 no opinion [] -1 Do not release this package because...
Re: [VOTE] Release Apache NiFi 2.0.0-M4 (RC1)
+1 (binding) On Fri, Jun 28, 2024 at 8:40 AM David Handermann wrote: > > Team, > > I am pleased to be calling this vote for the source release of Apache > NiFi 2.0.0-M4. > > Please review the following guide for how to verify a release candidate build: > > https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification > > The source being voted on the and the convenience binaries are > available on the Apache Distribution Repository: > > https://dist.apache.org/repos/dist/dev/nifi/nifi-2.0.0-M4 > > The build artifacts are available on the Apache Nexus Repository: > > https://repository.apache.org/content/repositories/orgapachenifi-1275 > > Git Tag: nifi-2.0.0-M4-RC1 > Git Commit ID: 19c5be01d463bc38ec0f5008549a2a42e589436d > GitHub Commit Link: > https://github.com/apache/nifi/commit/19c5be01d463bc38ec0f5008549a2a42e589436d > > Checksums of nifi-2.0.0-M4-source-release.zip > > SHA256: d882f05cec09ee1bfafaa3d4cde8f8660512d09765b5c400471f3a6e014029a6 > SHA512: > d429cd67fb0b7d9737c59cb834106d7b6e25cbdb91e3ecc5290be865a1313cbebbc314c2e0e54228226f021c44f0a86c745a18c148247c632a739c871c5fa013 > > Release artifacts are signed with the following key: > > https://people.apache.org/keys/committer/exceptionfactory.asc > > KEYS file is available on the Apache Distribution Repository: > > https://dist.apache.org/repos/dist/release/nifi/KEYS > > Issues resolved for this version: 153 > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12354678 > > Release note highlights can be found on the project wiki: > > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.0.0-M4 > > The vote will be open for 72 hours. > > Please download the release candidate and evaluate the necessary items > including checking hashes, signatures, build from source, and test. > Then please vote: > > [] +1 Release this package as nifi-2.0.0-M4 > [] +0 no opinion > [] -1 Do not release this package because...
[RESULT][VOTE] Release Apache NiFi 2.0.0-M4 (RC1)
Apache NiFi Community, I am pleased to announce that the 2.0.0-M4 release of Apache NiFi passes: 6 +1 (binding) votes 4 +1 (non-binding) votes 0 0 votes 0 -1 votes Thanks to all who helped make this release possible! The release artifacts will be published in the next 24 hours. Here is the vote thread: https://lists.apache.org/thread/jdb2j398ow6221rx4og7p958nnbj7bs9 Regards, David Handermann
[ANNOUNCE] Apache NiFi 2.0.0-M4 Released
The Apache NiFi Team is pleased to announce the release of Apache NiFi 2.0.0-M4. Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data. https://nifi.apache.org The release artifacts can be downloaded from the project website. https://nifi.apache.org/download/ Maven artifacts have been released and mirrored according to ASF artifact distribution processes. Issues resolved in Apache NiFi 2.0.0-M4 are listed in Jira Release Notes. https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12354678 Highlights of the release are available on the project wiki page: https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.0.0-M4 Regards, The Apache NiFi Team
Re: [VOTE] Release Apache NiFi 1.27.0 (RC2)
+1 binding - Verified signatures and hashes - Ran build using Maven Wrapper 3.9.6 - Ran build on Ubuntu 22.04 with Azul Zulu JDK 1.8.0-412 x86_64 - Ran build on Ubuntu 22.04 with Azul Zulu JDK 17.0.11 x86_64 - Ran NiFi on Ubuntu 22.04 with Azul Zulu JDK 17.0.11 x86_64 - Ran NiFi on macOS 14.5 with Azul Zulu JDK 21.0.1 Aarch64 - Ran NiFi Registry on Ubuntu 22.04 with Azul Zulu JDK 17.0.11 x86_64 - Verified NIFI-13429 with EncryptContentPGP and DecryptContentPGP using sample JPEG - Verified NIFI-13464 NiFi Registry starts without issues following deprecation-log dependency adjustments Thanks Pierre! Regards, David Handermann On Wed, Jul 3, 2024 at 7:42 AM Pierre Villard wrote: > > Team, > > I am pleased to be calling this vote for the source release of Apache NiFi > 1.27.0. > > Please review the following guide for how to verify a release candidate > build: > > https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification > > The source being voted on the and the convenience binaries are available on > the Apache Distribution Repository: > > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.27.0 > > The build artifacts are available on the Apache Nexus Repository: > > https://repository.apache.org/content/repositories/orgapachenifi-1276 > > Git Tag: nifi-1.27.0-RC2 > Git Commit ID: e0c4461d90bd4f6e5f2b81765bcff5cd97ed3e18 > GitHub Commit Link: > https://github.com/apache/nifi/commit/e0c4461d90bd4f6e5f2b81765bcff5cd97ed3e18 > > Checksums of nifi-1.27.0-source-release.zip > > SHA256: 0ac01d54eeceb4a4f4863880bf67dfa71e6a585036c5caf0546c592bf55ced48 > SHA512: > 675c7750752bf3092061c9eaac39a975955e9bf881e6bee3124c5842738d8c8626b6a551f33ef7a678018bd83e0323c1f0f0d79101255494d8ca91be7fc750f5 > > Release artifacts are signed with the following key: > > https://people.apache.org/keys/committer/pvillard.asc > > KEYS file is available on the Apache Distribution Repository: > > https://dist.apache.org/repos/dist/release/nifi/KEYS > > Issues resolved for this version: 37 > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12354832 > > Release note highlights can be found on the project wiki: > > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.27.0 > > The vote will be open for 72 hours. > > Please download the release candidate and evaluate the necessary items > including checking hashes, signatures, build from source, and test. Then > please vote: > > [] +1 Release this package as nifi-1.27.0 > [] +0 no opinion > [] -1 Do not release this package because...
Re: [DISCUSS] 1.x Maintenance
All, The discussion thus far has highlighted some important aspects of project maintenance at both general and specific levels. Clearly everyone shares a common goal to provide a supported and maintainable product. The concerns mentioned reflect the natural tension in any kind of technical debt evaluation, so the challenge is to determine how to navigate those concerns. As Joe said, nothing is lost, the source code and previous releases remain available for both internal and external use. Regarding the timeline for handling the removal of particular features, it is important to note that nothing has occurred outside of standard procedures. Every change has had a Jira issue, a pull request, feedback, and signed-off commits. Regarding updates to the deprecated features page, that has always occurred after merging, not before, because it reflects committed changes, rather than proposed changes. I understand the concern Pierre raised regarding the pace of some changes, so if contributing members of the PMC see a need to have a standard waiting period for Jira issues or pull requests, we should consider that on a separate thread. Not having that right now, it remains important that both removals and additions provide some sort of rationale as background. Sometimes that only requires a sentence or two, sometimes more, but that also characterizes the majority of changes. We have had past discussions about the need for feature proposals, and if it would improve maintainability to consider review timelines, active contributors should discuss the options. For now, however, raising issues even after merging changes continues to be a valid way to handle things that may have been missed. On repository encryption itself, the conversation thus far rightly acknowledged the problems noted in the Jira issue [1] NIFI-13494. Removal of repository encryption is one several security-focused removals, along with support for historical encryption processors and application property protection. Although these removals will undoubtedly impact some users, this is an area where we cannot maintain extended support at the community level. Encryption features include an inherent promise of security. If certain encryption features have fundamental flaws, it is better to remove those features than to hold on to compatibility. Major version changes are the standard way to introduce breaking changes, and that has been one of the primary themes for NiFi 2.0. With that understanding in mind, there is value in considering future alternatives. If there are any unanswered questions about security-focused feature removals, I would be glad to provide more background as one who has both introduced features and removed them during this process. The project has enjoyed significant adoption and contribution over the years. The process of preparing for NiFi 2.0 has highlighted a number of things, particularly that it is far easier to introduce capabilities than to remove them. I believe we are on a solid path forward. When we focus on specific concerns, such as those Pierre raised regarding Kafka or Kudu, we can incorporate solutions as we go forward. Regards, David Handermann [1] https://issues.apache.org/jira/browse/NIFI-13494 On Sun, Jul 7, 2024 at 11:44 PM Joe Witt wrote: > > Adam, > > We are all sympathetic to Pierre's view. As one of the people who > participates in contributing code, code reviews, discussions, conducting > releases, and more, his concerns and actions carry considerable weight for > those of us that do so as well and the broader community. > > For repository encryption we should have raised a discussion thread. > Fortunately every code change is reversible. Nothing is lost. If there is > anything needing undone let's discuss them. There are no ships sailing. > Just contributors contributing. > > If you keep the rest of the sentence in the quote you snipped it reads > "let's surface them and address them" which does the opposite of close off > conversation. > > Thanks > Joe > > > On Mon, Jul 8, 2024 at 12:07 AM Adam Taft wrote: > > > Feeling sympathetic to Pierre's point of view here. > > > > Picking specifically on the repository encryption issue itself, we have > > deprecation of a fundamental and well-known capability of NiFi. Repository > > encryption has been a key highlight and feature for those using NiFi in > > highly secure enclaves that maintain the strictest of auditing oversight. > > This change will no doubt cause reassessment efforts and scrutiny over what > > will be perceived as a loss of information security. > > > > The justification for the change is probably warranted. Maybe repository > > encryption should never have been a problem NiFi attempted to solve. But > > regardless, the change was made without reasonable discussion, at
[ANNOUNCE] Apache NiFi 1.27.0 Released
Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data. https://nifi.apache.org The release artifacts can be downloaded from the project website. https://nifi.apache.org/download/ Maven artifacts have been released and mirrored according to ASF artifact distribution processes. Issues resolved in Apache NiFi 1.27.0 are listed in Jira Release Notes. https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12354832 Highlights of the release are available on the project wiki page: https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.27.0 Regards, The Apache NiFi Team
Re: [DISCUSS] 1.x Maintenance
Paul, Thanks for providing some additional perspective on the Kafka components, and for your extensive work on the new Processors. It sounds like some next steps involve writing up some Jira issues for additional features. NIFI-12952 outlines the authentication features as you noted, so it sounds like another for a new KafkaRecordSink would be worth considering. The RecordSink interface has more narrow usage, so that might be worth additional consideration before proceeding directly to implementation. As far as exactly once semantics, it would also be helpful to outline the scope of implementation for that work. With NiFi 2.0 supporting stateless execution for any process group, and with the new Kafka Connection Service architecture, there may be some opportunity for taking a different approach than previously implemented. With that background, I recognize the challenge of migration from existing Processors, and the gaps for some use cases. Although one option could be to continue maintaining the Kafka 2.6 Processors, this presents a larger maintenance effort together with the new components. Recognizing the concerns Pierre highlighted about migration, focusing on implementing features following the new design seems to be the better way forward from a project maintenance perspective. One important detail here we can note in Migration Guidance is that the nifi-kafka-nar version 2.0.0-M4 is available in Maven Central. Loading that NAR should provide an option for existing users to run with subsequent versions of NiFI 2.0, supporting a transitional path to the new components. Although it is not an automated migration, it does provide a path forward. I recognize that Kafka support is a key capability, and this migration is bound to be one of the more challenging aspects. However, NiFi 2.0 presents an important step forward on a number of levels. Tracking and addressing issues with the new Kafka components should provide the best path for maintainability. Regards, David Handermann On Mon, Jul 8, 2024 at 9:08 AM Paul Grey wrote: > > > > > Kafka 2.6 parity concern - Can you share more on what is missing? > > Obviously supporting Kafka users within NiFi is quite important but I think > > we're making the right steps here. Whatever gap we have now let's identify > > and see who can knock it out. > > > I cited a few in the PR description [1]. > > > >- The PR as is should support the following authentication strategies: > >PLAINTEXT, SSL, SASL_PLAINTEXT, and SASL_SSL. Support for additional > >authentication strategies could be handled via follow on JIRAs. > >- Support for the KafkaRecordSink form factor could be handled via > >follow on JIRAs. > >- Support for Kafka "exactly once" semantics could be handled via > >follow on JIRAs. > >- Migration documentation could be handled via follow on JIRAs. > > > > This should not be considered a complete list. I consider it likely that > there are others that I did not identify. I did try to capture that > thinking in the PR [2]. > > Of note, there is a follow on JIRA [3] to flag the authentication features > that were backed in *Kafka_2_6 by processor properties. There likely > should be other JIRAs. > > In my opinion, this upgrade is more involved than the upgrade from > *Kafka*_2_0 to *Kafka*_2_6, from a design perspective, an implementation > perspective, and from the perspective of representing usages in a flow > definition. This has the potential to cause a lot of problems for the > community. > > It is hard for me to weigh all of the factors that should go into the > retain / remove decision for Kafka 2.6, and I understand that I am a guest > at this party. It does complicate things that the community is trying to > close the book on NiFi 2.0 with a minimum of technical debt. But I'm not > comfortable saying that the known (and unknown) functionality gaps can be > addressed in the timeline I've seen discussed elsewhere for 2.0 GA. And I > don't think that 2.0 should be held up for this issue. There is so much > good in the main line, when compared against 1.x. > > > [1] https://github.com/apache/nifi/pull/8463 > [2] https://github.com/apache/nifi/pull/8463#issuecomment-2090939791 > [3] https://issues.apache.org/jira/browse/NIFI-12952 > > > On Mon, Jul 8, 2024 at 8:24 AM David Handermann > wrote: > > > All, > > > > The discussion thus far has highlighted some important aspects of > > project maintenance at both general and specific levels. Clearly > > everyone shares a common goal to provide a supported and maintainable > > product. The concerns mentioned reflect the natural tension in any > > kind of technical debt evaluation, so the challe
Re: [DISCUSS] 1.x Maintenance
Matt, Although it is technically possible to have encrypted repository implementations in a separate NAR, the original design did not take that approach. Improvements to the property configuration process also abstracted away the need to specify individual repository implementation classes. The removal pull request [1] provides a good reference point for the classes and supporting modules that were necessary for repository encryption. This presents an opportunity for anyone to create a separate project based on the historical capabilities. With that being said, the original reasons for deprecating and removing repository encryption still stand. Refactoring the design to isolated modules would require a notable amount of work, as many of the implementation classes extend from the standard framework classes. Moving support to an optional profile does not solve the fundamental problems related to continued maintenance of a problematic implementation. Regards, David Handermann [1] https://github.com/apache/nifi/pull/9039 On Thu, Jul 18, 2024 at 1:49 PM Matt Burgess wrote: > > Couldn't we restore the encrypted repository implementation as a NAR and > possibly make adding it a non-active profile by default? It doesn't solve > the security issue but it does remove it from the convenience binary yet > leaves it to be added by whoever wants to build NiFi their own way. > Arguably all things with public API interfaces (extensions and framework) > should be extensible so it probably should've been in its own NAR anyway, > like the Provenance Repository implementations. > > Regards, > Matt > > On Mon, Jul 8, 2024 at 12:43 AM Joe Witt wrote: > > > Adam, > > > > We are all sympathetic to Pierre's view. As one of the people who > > participates in contributing code, code reviews, discussions, conducting > > releases, and more, his concerns and actions carry considerable weight for > > those of us that do so as well and the broader community. > > > > For repository encryption we should have raised a discussion thread. > > Fortunately every code change is reversible. Nothing is lost. If there is > > anything needing undone let's discuss them. There are no ships sailing. > > Just contributors contributing. > > > > If you keep the rest of the sentence in the quote you snipped it reads > > "let's surface them and address them" which does the opposite of close off > > conversation. > > > > Thanks > > Joe > > > > > > On Mon, Jul 8, 2024 at 12:07 AM Adam Taft wrote: > > > > > Feeling sympathetic to Pierre's point of view here. > > > > > > Picking specifically on the repository encryption issue itself, we have > > > deprecation of a fundamental and well-known capability of NiFi. > > Repository > > > encryption has been a key highlight and feature for those using NiFi in > > > highly secure enclaves that maintain the strictest of auditing oversight. > > > This change will no doubt cause reassessment efforts and scrutiny over > > what > > > will be perceived as a loss of information security. > > > > > > The justification for the change is probably warranted. Maybe repository > > > encryption should never have been a problem NiFi attempted to solve. But > > > regardless, the change was made without reasonable discussion, at > > minimum, > > > here on the mailing lists where an Apache project is supposed to have > > such > > > discussions. > > > > > > - The PR to remove this capability was created 5 days ago. [1] > > > - The github PR was accepted and merged 3 days ago, on July 4th, a US > > > national holiday no less. [2] > > > - The deprecated components page was updated to reflect this change > > > *retroactively* 1 day ago. [3] > > > > > > Granted, I might not have seen additional mailing list discussions on > > this > > > topic. I grepped my inbox, but maybe I missed it. But still, for a > > security > > > feature to be removed in the timeline outlined above, even if it was > > > brought to the mailing list, would have been aggressive. > > > > > > So yes, there's a smell here. Clearly there needs to be a balance of > > doing > > > small/normal changes without a lot (or any) needed discussion on the > > > mailing lists, using Jira as the main project driver. But likewise, there > > > has to be a sense of when to promote a discussion (or even awareness) to > > > the mailing list level. A removed feature that has any sort of touchpoint > > > to information security s
Re: NiFi 2.0.0-M4 RHEL 9 FIPS mode
Hi Will, Thanks for providing the stack trace associated with the Bouncy Castle FIPS configuration. Enabling FIPS is a complex topic that impacts both the NiFi framework and individual extension components, so it usually requires vendor support for correct implementation. The particular stack trace points to a problem loading the default trust store, and the MAC calculation failure usually indicates a bad password. Any troubleshooting would require understanding the exact changes made to java.security and java.policy files. This configuration is outside of standard Apache NiFi deployments, and any changes to those files will have direct implications for the security of the installation. If you have any additional background on the intended use case for NiFi with FIPS support, that may be helpful to know. Regards, David Handermann On Sat, Jul 20, 2024 at 3:21 PM William Mallett < wmall...@provisus-solutions.com> wrote: > Hello Dev, > > > > Not sure if you would be able to help me, but I wanted to provide an > update on the effort to try to run NiFi in FIPS Mode using bouncycastle: > > > >1. I installed the bouncycastle jar files in ./lib, and I can see them >load in bootstrap.conf >2. I modified java.security and java .policy for bouncycastle related >configuration changes (like adding the bouncycastle providers, and more) >3. My keystore and truststore for NiFi are converted to BCFKS, and I >can open them using keytool with specifying the provider info, password, > etc >4. I have converted the default cacerts truststore from JKS to BCFKS >(which resolved the nifi error “o.b.jsse.provider.DefaultSSLContextSpi >Failed to load default trust managers java.io.IOException: DER length more >than 4 bytes: 109), and I have verified I can open with keytool with >specifying the needed info (provider, pwd, etc) >5. But now starting nifi, I get the error: >“o.b.jsse.provider.DefaultSSLContextSpi Failed to load default trust >managers java.io.IOException: BCFKS KeyStore corrupted: MAC calculation >failed (full error at the bottom of the email). > 1. One note: when using keytool, this is the exact same error I get > for BCFKS stores if I get the password for the store wrong, or I don’t > include it and it is required for the store. > 2. I have verified that cacerts is BCFKS, and that its password is > “changeit”. > 3. Just to see if it would work, I have changed cacerts to the same > password as my NiFi keystore/truststore (same error). > > > > Please help if you would like, any help is much appreciated! > > > > > > 2024-07-20 12:04:23,897 WARN [main] o.b.jsse.provider.DefaultSSLContextSpi > Failed to load default trust managers > > java.io.IOException: BCFKS KeyStore corrupted: MAC calculation failed. > > at > org.bouncycastle.jcajce.provider.ProvBCFKS$BCFIPSKeyStoreSpi.verifyMac(Unknown > Source) > > at > org.bouncycastle.jcajce.provider.ProvBCFKS$BCFIPSKeyStoreSpi.engineLoad(Unknown > Source) > > at java.base/java.security.KeyStore.load(KeyStore.java:1500) > > at > org.bouncycastle.jsse.provider.ProvTrustManagerFactorySpi.getDefaultTrustStore(ProvTrustManagerFactorySpi.java:112) > > at > org.bouncycastle.jsse.provider.ProvSSLContextSpi.getDefaultTrustManagers(ProvSSLContextSpi.java:545) > > at > org.bouncycastle.jsse.provider.DefaultSSLContextSpi$LazyManagers.(DefaultSSLContextSpi.java:65) > > at > org.bouncycastle.jsse.provider.DefaultSSLContextSpi.(DefaultSSLContextSpi.java:113) > > at > org.bouncycastle.jsse.provider.BouncyCastleJsseProvider$8.createInstance(BouncyCastleJsseProvider.java:223) > > at > org.bouncycastle.jsse.provider.BouncyCastleJsseProvider$BcJsseService.newInstance(BouncyCastleJsseProvider.java:407) > > at > java.base/sun.security.jca.GetInstance.getInstance(GetInstance.java:236) > > at > java.base/sun.security.jca.GetInstance.getInstance(GetInstance.java:164) > > at > java.base/javax.net.ssl.SSLContext.getInstance(SSLContext.java:185) > > at > java.base/javax.net.ssl.SSLContext.getDefault(SSLContext.java:110) > > at > org.apache.nifi.security.util.TlsPlatform.getDefaultSslContext(TlsPlatform.java:102) > > at > org.apache.nifi.security.util.TlsPlatform.getDefaultSslContextProtocols(TlsPlatform.java:83) > > at > org.apache.nifi.security.util.TlsPlatform.(TlsPlatform.java:45) > > at > org.apache.nifi.web.server.connector.FrameworkServerConnectorFactory.(FrameworkServerConnectorFactory.java:81) > > at > org.apache.nifi.web.server.JettyServer.configu
[DISCUSS] Python pip in Docker Images
Team, Apache NiFi 2.0.0-M1 introduced support for native Python Processors, bringing new capabilities and some new questions. Part of that initial release included the installation of Python pip for dynamic dependency installation in the unofficial Docker images. Although this feature is useful in some development scenarios, pip has several known vulnerabilities, including CVE-2018-20225, related to the potential for passing an option to reference additional repositories. Although NiFi does not use this option, the presence of pip is a notable security concern at the level of runtime package retrieval. With that background, it seems that we should remove pip from the unofficial Docker image builds. This would bring closer alignment with the convenience binary downloads, which do not have Python support enabled in the default configuration. Although this would require users to build their own images to add pip, it follows the general principle of providing secure defaults. There is a developer usability tradeoff, but we have had similar considerations in the past, and have leaned in the direction of security. Regards, David Handermann
Re: [DISCUSS] Python pip in Docker Images
Hi Matthew, Thanks for highlighting the Red Hat response to CVE-2018-20225. Although the CVE was not withdrawn, the Red Hat response is worth noting. The point of noting that CVE was not the issue itself, but the level of support for additional package installation in runtime deployments. Focusing on that particular point, Maven is not included in the Apache NiFi Docker image. As a composable processing system, NiFi supports custom solutions that could be considered potentially dangerous, so the purpose is not to simply check a box for security. The purpose of raising this particular question is to consider the default distribution approach, allowing users to make further decisions. Regards, David Handermann On Wed, Jul 24, 2024 at 6:51 PM Matthew Hawkins wrote: > > Hi David, > > See the RHEL bug [1] for the shellacking this now rescinded CVE received. > > Removing pip from the python side should also be accompanied by removing > maven from the Java side, if you are serious about addressing the actual > security concern raised in this CVE. > (That malicious content may exist somewhere on the internet, and people can > download it via http). Also, remove every node from NiFi that allows the > ingestion of data from any source, by the same token. That'll also help > reduce developer maintenance workload on these nasty CVE filled nodes ;) ;) > ;) > > > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1835736 > > > On Thu, 25 July 2024, 06:16 David Handermann, > wrote: > > > Team, > > > > Apache NiFi 2.0.0-M1 introduced support for native Python Processors, > > bringing new capabilities and some new questions. Part of that initial > > release included the installation of Python pip for dynamic dependency > > installation in the unofficial Docker images. Although this feature is > > useful in some development scenarios, pip has several known > > vulnerabilities, including CVE-2018-20225, related to the potential > > for passing an option to reference additional repositories. Although > > NiFi does not use this option, the presence of pip is a notable > > security concern at the level of runtime package retrieval. > > > > With that background, it seems that we should remove pip from the > > unofficial Docker image builds. This would bring closer alignment with > > the convenience binary downloads, which do not have Python support > > enabled in the default configuration. Although this would require > > users to build their own images to add pip, it follows the general > > principle of providing secure defaults. There is a developer usability > > tradeoff, but we have had similar considerations in the past, and have > > leaned in the direction of security. > > > > Regards, > > David Handermann > >
Re: [Discuss] NiFi 2.0 milestones, where we are?
Arpad, Thanks for initiating the discussion! I think we are getting very close to ready for a GA release of NiFi 2.0. The last major element I am aware of right now is some reworking of content viewer integration. Jira issue NIFI-13632 [1] highlights some general issues to be addressed, bringing the content viewer integration in line with the rest of the redesigned web user interface. I will defer to Matt Gilman and others for details. I'm not aware of anything else in particular, but this particular area is important as it relates to the contract between the main application and content viewers. Beyond that, there are other framework issues I would like to address in the future, such as the configuration structure that requires multiple XML files, and the Site-to-Site client library. However, in the interest of keeping changes scoped, I believe those can be addressed down the road. With that background, as soon as we are in a good position with the content viewer integration, we should proceed to preparing for a release. Regards, David Handermann [1] https://issues.apache.org/jira/browse/NIFI-13632 On Thu, Aug 8, 2024 at 5:57 PM Arpad Boda wrote: > > Team, > > Tremendous amount of effort has been put into NiFi 2.0 so far and I haven't > seen breaking changes introduced during the last few weeks, so I wonder > where you think we are in the process of making it final(GA)? > Do you see features missing, things that need to be either updated or > deprecated, breaking api changes introduced? > > Don't get me wrong, I have no intent to push the community to release 2.0 > as soon as we can, the goal of this thread is to have a common > understanding of where we are in the process and what we need to achieve to > get there. > > Thanks in advance for sharing your thoughts, opinions! > > Cheers, > Arpad
[DISCUSS] Introducing NiFi Improvement Proposals
Team, As we wrap up remaining items for NiFi 2, we should consider how to continue improving the quality and maintainability of the NiFi ecosystem. One primary focus of NiFi 2 has been the reduction of technical debt, which involved the removal of numerous modules and thousands of lines of code. In that process, it is worth highlighting that the core NiFi API, and the NiFi Framework API, and the NiFi REST API have had comparatively few breaking changes. This a testament to the quality of the API design itself. The NiFi Version Schema and API Compatibility [1] has provided a strong direction thus far. With that background, we should consider adopting a more formal process around changes that impact the fundamental API contracts that NiFi provides. NiFi Feature Proposals [2] have provided elements of this in the past, but did not include approval requirements. Kafka [3] and Airflow [4] have more structured improvement proposal processes, and that is what we should adopt going forward. Part of moving in this direction requires identifying the areas that would require going through the Improvement Proposal process itself. At minimum, this should include the nifi-api [5] module. The nifi-framework-api [6] is also worth consideration for inclusion in this category. The NiFi REST API does not have quite the same level of concern, but may warrant inclusion. As part of this discussion, we should consider separating the nifi-api module into its own repository, with its own versioning scheme. This will provide a helpful distinction in terms of the scope of changes, and allow the API to be released independently of the application, providing strong version compatibility guarantees. Based on feedback for the general idea, I would be glad to draft a NiFi Improvement Proposal page, outlining the recommended steps in more detail so we can come to consensus on how this should work. Regards, David Handermann [1] https://cwiki.apache.org/confluence/display/NIFI/Version+Scheme+and+API+Compatibility [2] https://cwiki.apache.org/confluence/display/NIFI/NiFi+Feature+Proposals [3] https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals [4] https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals [5] https://github.com/apache/nifi/tree/main/nifi-api [6] https://github.com/apache/nifi/tree/main/nifi-framework-api
Re: [DISCUSS] Introducing NiFi Improvement Proposals
Thanks for the replies! I will move forward with drafting a NiFi Improvements Proposal page to keep the discussion going. Regards, David Handermann On Tue, Aug 20, 2024 at 3:22 PM Mark Payne wrote: > > David, > > +1 to all of this. Especially for guaranteeing compatibility and > maintainability, I think instrumenting a more > formal approach for updating the API is a step in the right direction. The > huge amount of purging, cleanup, > and refactoring that has gone into 2.0 helps to highlight the importance of > ensuring maintainability. > > Thanks > -Mark > > > > On Aug 20, 2024, at 2:00 AM, Joe Witt wrote: > > > > David > > > > Yeah well written and good points. > > > > I dont think this says we would relax our existing commitment we have shown > > for things such as the http based api but rather is calling out specific > > things which we will make changes to even more formalized. > > > > To that end +1 on the concept of proposals and +1 to breaking out the > > nifi-api (fundamental designed extension points for flow development) in > > particular. > > > > Thanks > > Joe > > > > On Fri, Aug 16, 2024 at 8:20 PM Adam Taft wrote: > > > >> These are all really appreciated concepts, David. Thank you for putting the > >> thoughts and time in on this! > >> > >> Regarding this: > >>> The NiFi REST API does not have quite the same level of concern, but may > >> warrant inclusion. > >> > >> I hear what you're saying. However, the REST API (from my > >> observation/experience) has gathered quite a number of useful tools and > >> "hacks" for NiFi. Quite often, many different monitoring and alerting tools > >> have been developed against the REST API by third parties and/or > >> integrators of NiFi against their internal workflows. Having stable API > >> versioning in the REST API possibly makes just as much sense as having the > >> same for the nifi-api itself. This is a prime entry point for extensions > >> and other features developed alongside NiFi, maybe even the weird stuff > >> that you can't do with the nifi-api directly. > >> > >> Food for thought of course, but I would hope that we can treat the REST API > >> as a proper first-class citizen in terms of documented versioning. It turns > >> out it's quite a useful means for interacting with a running NiFi instance. > >> > >> /Adam > >> > >> On Fri, Aug 16, 2024 at 9:59 AM David Handermann < > >> exceptionfact...@apache.org> wrote: > >> > >>> Team, > >>> > >>> As we wrap up remaining items for NiFi 2, we should consider how to > >>> continue improving the quality and maintainability of the NiFi > >>> ecosystem. > >>> > >>> One primary focus of NiFi 2 has been the reduction of technical debt, > >>> which involved the removal of numerous modules and thousands of lines > >>> of code. In that process, it is worth highlighting that the core NiFi > >>> API, and the NiFi Framework API, and the NiFi REST API have had > >>> comparatively few breaking changes. This a testament to the quality of > >>> the API design itself. The NiFi Version Schema and API Compatibility > >>> [1] has provided a strong direction thus far. > >>> > >>> With that background, we should consider adopting a more formal > >>> process around changes that impact the fundamental API contracts that > >>> NiFi provides. NiFi Feature Proposals [2] have provided elements of > >>> this in the past, but did not include approval requirements. Kafka [3] > >>> and Airflow [4] have more structured improvement proposal processes, > >>> and that is what we should adopt going forward. > >>> > >>> Part of moving in this direction requires identifying the areas that > >>> would require going through the Improvement Proposal process itself. > >>> At minimum, this should include the nifi-api [5] module. The > >>> nifi-framework-api [6] is also worth consideration for inclusion in > >>> this category. The NiFi REST API does not have quite the same level of > >>> concern, but may warrant inclusion. > >>> > >>> As part of this discussion, we should consider separating the nifi-api > >>> module into its own repository, with its own versioning scheme. This > >>> will provide a helpful distinction in terms of the scope of ch
Re: [DISCUSS] Introducing NiFi Improvement Proposals
Team, I have drafted an initial version of the NiFi Improvement Proposal Process: https://cwiki.apache.org/confluence/display/NIFI/NiFi+Improvement+Proposal+Process The document outlines the scope of project components requiring a proposal, as well as the requirements for a proposal, and the review process. The general approach for review and approval is designed to be similar to the release process. The requirements for a proposal are similar to other project improvement proposals. I recommend creating a new Jira project for handling Improvement Proposals, as opposed to using Confluence pages. Please review the draft and I will look to call for a vote within the next week. As mentioned earlier, with the nifi-api module being the core contract for internal and external extensions, I would like to include moving the nifi-api to a separate Git repository as part of the vote for instituting the Improvement Proposal Process. If this proves to be too much of a concern, we can divide the question and vote separately on the Improvement Proposal Process and moving the nifi-api. Thanks for the feedback! Regards, David Handermann On Wed, Aug 21, 2024 at 11:09 AM David Handermann wrote: > > Thanks for the replies! > > I will move forward with drafting a NiFi Improvements Proposal page to > keep the discussion going. > > Regards, > David Handermann > > On Tue, Aug 20, 2024 at 3:22 PM Mark Payne wrote: > > > > David, > > > > +1 to all of this. Especially for guaranteeing compatibility and > > maintainability, I think instrumenting a more > > formal approach for updating the API is a step in the right direction. The > > huge amount of purging, cleanup, > > and refactoring that has gone into 2.0 helps to highlight the importance of > > ensuring maintainability. > > > > Thanks > > -Mark > > > > > > > On Aug 20, 2024, at 2:00 AM, Joe Witt wrote: > > > > > > David > > > > > > Yeah well written and good points. > > > > > > I dont think this says we would relax our existing commitment we have > > > shown > > > for things such as the http based api but rather is calling out specific > > > things which we will make changes to even more formalized. > > > > > > To that end +1 on the concept of proposals and +1 to breaking out the > > > nifi-api (fundamental designed extension points for flow development) in > > > particular. > > > > > > Thanks > > > Joe > > > > > > On Fri, Aug 16, 2024 at 8:20 PM Adam Taft wrote: > > > > > >> These are all really appreciated concepts, David. Thank you for putting > > >> the > > >> thoughts and time in on this! > > >> > > >> Regarding this: > > >>> The NiFi REST API does not have quite the same level of concern, but may > > >> warrant inclusion. > > >> > > >> I hear what you're saying. However, the REST API (from my > > >> observation/experience) has gathered quite a number of useful tools and > > >> "hacks" for NiFi. Quite often, many different monitoring and alerting > > >> tools > > >> have been developed against the REST API by third parties and/or > > >> integrators of NiFi against their internal workflows. Having stable API > > >> versioning in the REST API possibly makes just as much sense as having > > >> the > > >> same for the nifi-api itself. This is a prime entry point for extensions > > >> and other features developed alongside NiFi, maybe even the weird stuff > > >> that you can't do with the nifi-api directly. > > >> > > >> Food for thought of course, but I would hope that we can treat the REST > > >> API > > >> as a proper first-class citizen in terms of documented versioning. It > > >> turns > > >> out it's quite a useful means for interacting with a running NiFi > > >> instance. > > >> > > >> /Adam > > >> > > >> On Fri, Aug 16, 2024 at 9:59 AM David Handermann < > > >> exceptionfact...@apache.org> wrote: > > >> > > >>> Team, > > >>> > > >>> As we wrap up remaining items for NiFi 2, we should consider how to > > >>> continue improving the quality and maintainability of the NiFi > > >>> ecosystem. > > >>> > > >>> One primary focus of NiFi 2 has been the reduction of technical debt, > > >>> which involved the removal of numerous mod
[RESULT][VOTE] Adopt NiFi Improvement Proposal Process and Move nifi-api
Apache NiFi Community, I am pleased to announce that the vote for adopting NiFi Improvement Proposal process and moving the nifi-api module passes: 7 +1 (binding) votes 2 +1 (non-binding) votes 0 0 votes 0 -1 votes Thanks to everyone who participated in the discussion and voting! I will proceed to create the new Jira project for NiFi Improvement Proposals and the Git repository for nifi-api. Here is the vote thread: https://lists.apache.org/thread/59o0d9cv4mo4g5hpr8fprtkh6dt7c953 Regards, David Handermann
Re: Jira contributor access
Hi Cory, Thanks for submitting the issue and requesting access to Jira, I'm sure one of the project maintainers will be able to add you soon. I added a comment to the issue you submitted and I would be interested to hear more details on your proposed implementation. As mentioned in the comment, I have another open issue that is closely related to S/MIME handling, and includes some Controller Services that could be useful in implementing signed S/MIME emails. Feel free to provide your comments on the related issue and PR in GitHub. Regards, David Handermann On Tue, Dec 1, 2020 at 10:00 PM cWix Software wrote: > Hi there, > > I would like to work on Jira ticket NIFI-8059. > > I was reading you had to give me permissions so I could. My login is > cwixom. > > Please let me know if there's anything else I need to provide. > > Thanks! > > Cory Wixom >
Re: [VOTE] Release Apache NiFi 1.13.0
-1 non-binding. Verified release with successful build on Azul Zulu JDK 11.0.10 on Ubuntu 20.0.10. Verified sample flow with InvokeHTTP and ListenHTTP processors using multiple keystore types and TLS configuration options. Unfortunately found bcprov-ext-jdk15on-1.60.jar together with bcprov-jdk15on-1.68.jar in nifi-framework-nar. The bcprov-ext-jdk15on-1.60.jar library is apparently a transitive dependency of spring-security-saml2-core through a library named com.narupley:not-going-to-be-commons-ssl. The Bouncy Castle libraries should be version 1.68 throughout the NiFi framework. The bcprov-ext-jdk15on library contains the same classes as bcprov-jdk15on plus a handful of additional classes for infrequently used algorithms. The presence of both versions did not appear to cause problems during initial tests, but it could cause unexpected behavior at runtime depending on which version gets loaded. If the Spring Security SAML2 library requires the algorithms present in bcprov-ext-jdk15on, it will probably be necessary to change dependencies in NiFi to replace references to bcprov-jdk15on with bcprov-ext-jdk15on to ensure a consistent version and avoid duplication. Regards, David Handermann On Fri, Jan 29, 2021 at 5:33 PM M Tien wrote: > +1 non-binding. > > Went through the release guide > Verified a full build on JDK 1.8.0_275 and JDK 11.0.5 > Verified a secure instance of NiFi > Verified I was able to authenticate with OIDC using Google, Okta, and > Azure and I can successfully log out and invalidate the JWT. > > - Margot
Re: [VOTE] Release Apache NiFi 1.13.0 (rc2)
+1 non-binding Verified release signatures and expected files. Verified build on Ubuntu 20.10 using Apache Maven 3.6.3 with Azul Zulu JDK 11.0.10 and AdoptOpenJDK 1.8.0_282. Tested sample flows with ListenHTTP and InvokeHTTP using different TLS versions and keystore types to verify BCFKS support as well as TLS protocol support differences on Java 8 and 11. Confirmed absence of bcprov-ext-jdk15on from nifi-framework-nar as documented in NIFI-8186. Regards, David Handermann On Tue, Feb 2, 2021 at 12:18 PM Mark Payne wrote: > +1 (binding) > > Build details: > Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f) > Maven home: /usr/local/Cellar/maven/3.6.3_1/libexec > Java version: 11.0.8, vendor: AdoptOpenJDK, runtime: > /Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home > Default locale: en_US, platform encoding: UTF-8 > OS name: "mac os x", version: "10.15.6", arch: "x86_64", family: “mac" > > Verified hashes, README, NOTICE, and LICENSE files look as expected. > > Performed some smoke tests. Verified new dependent properties feature. > Exported flow and ran it with bin/nifi.sh stateless. > > Thanks for putting together the release, Joe! > > > > > On Feb 2, 2021, at 12:32 PM, Peter Turcsanyi > wrote: > > > > +1 non-binding > > > > Went through the release helper guide. > > Verified full build with empty local maven repo with Java 8 (AdoptOpenJDK > > 1.8.0_282-b08) and 11 (AdoptOpenJDK 11.0.10+9) on Ubuntu 20.04. > > Ran full builds with different time zone settings to verify NIFI-8023, > also > > tested it with flows. > > Ran some SSL related test flows with processors changed recently > > (ListenHTTP, GRPC, MQTT). > > > > Thanks for RMing Joe! > > > > On Tue, Feb 2, 2021 at 6:04 PM Otto Fowler > wrote: > > > >> +1 > >> Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f) > >> Maven home: /usr/local/Cellar/maven/3.6.3_1/libexec > >> Java version: 1.8.0_282, vendor: AdoptOpenJDK, runtime: > >> /Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/jre > >> Default locale: en_US, platform encoding: UTF-8 > >> OS name: "mac os x", version: "10.16", arch: "x86_64", family: “mac" > >> > >> Thanks Joe! > >> > >> > >>> On Feb 1, 2021, at 20:10, Joe Witt wrote: > >>> > >>> Hello, > >>> > >>> I am pleased to be calling this vote for the source release of Apache > >> NiFi > >>> 1.13.0. > >>> > >>> The source zip, including signatures, digests, etc. can be found at: > >>> https://repository.apache.org/content/repositories/orgapachenifi-1176 > >>> > >>> The source being voted upon and the convenience binaries can be found > at: > >>> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.13.0/ > >>> > >>> A helpful reminder on how the release candidate verification process > >> works: > >>> > >> > https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate > >>> > >>> The Git tag is nifi-1.13.0-RC2 > >>> The Git commit ID is c27e59fc679a2e982102a75b8b8df2b0f062af23 > >>> > >> > https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=c27e59fc679a2e982102a75b8b8df2b0f062af23 > >>> > >>> Checksums of nifi-1.13.0-source-release.zip: > >>> SHA256: > 4913dd3d943afac710d1a277bf31beebf7a6207a20e1148849d69511f44fc97b > >>> SHA512: > >>> > >> > dc9935f0eb8692cd8493f5863bc8ae2ef0b52653fa69ff8b9a7e8db7dbd9ec6561f6ffdca4a1b55e43b289d04f5671f5ab4de30999838c5fca5c282c00a7c7f8 > >>> > >>> Release artifacts are signed with the following key: > >>> https://people.apache.org/keys/committer/joewitt.asc > >>> > >>> KEYS file available here: > >>> https://dist.apache.org/repos/dist/release/nifi/KEYS > >>> > >>> 238 issues were closed/resolved for this release: > >>> > >> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12348700 > >>> > >>> Release note highlights can be found here: > >>> > >> > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.13.0 > >>> > >>> The vote will be open for 72 hours. > >>> Please download the release candidate and evaluate the necessary items > >>> including checking hashes, signatures, build from source, and test. > Then > >>> please vote: > >>> > >>> [ ] +1 Release this package as nifi-1.13.0 > >>> [ ] +0 no opinion > >>> [ ] -1 Do not release this package because... > >> > >> > >
Re: [discuss] we need to enable secure by default...
Having NiFi enforce authentication by default seems like the right way to go, especially given the capabilities of the system. Bryan raises a good point about storage of account information, so weighing the positives and negatives of various identity providers should be considered. Following on Joe's point about disabling plain HTTP, one option could be generating both client and server certificates and using Mutual TLS for authentication. This would obviously require installing the client certificate in a browser, which is not trivial, but could be part of an installation guide. This approach definitely provides a high barrier of entry of new users, but provides a strong level of security while avoiding the need for some other identity provider service to be configured. As others have mentioned, this seems necessary to address the situation of someone installing NiFi without a full understanding of the configuration required, so it is important to keep the audience in mind when evaluating a solution. Regards, David Handermann On Wed, Feb 10, 2021 at 7:39 AM Bryan Bende wrote: > Just to clarify, I was not suggesting that we make a default secure > setup that requires LDAP. > > I was just saying that in order to provide a default secure setup, > we'd have to provide a login identity provider implementation that is > backed by a file or database table, which in the past we decided > against. > > On Wed, Feb 10, 2021 at 8:28 AM Russell Bateman > wrote: > > > > I second the concerns expressed, but second especially Bryan's pointing > > out that requiring LDAP/AD to be set up in order even to begin to use > > our framework would be a bit onerous for developers just interested in > > getting work done and a barrier to considering the framework should it > > be erected a little too high. Should we at least glance at how this is > > solved by the likes of other projects, Kafka and Cassandra come to mind, > > even if it means resorting to a store of a name or two? I didn't find > > getting into developing with them a pain, but making me jump through the > > hoop of setting up LDAP may very well have changed that. > > > > These unsecure instances of NiFi out there are not our community's > > fault. I suppose we're worried about getting splattered by bad press? > > > > On 2/10/21 5:47 AM, Bryan Bende wrote: > > > I agree with the overall idea, although I would think it requires a > > > major release to make this kind of change to the default behavior. > > > > > > Also, we have always avoided NiFi being a store of usernames and > > > passwords, so we don't have a login provider that uses a local file or > > > a database, we've always said you connect to LDAP/AD for that. > > > > > > Obviously it can be implemented, but just pointing out that we'd have > > > to change our stance here if we want to provide a default username and > > > password to authenticate with. > > > > > > On Tue, Feb 9, 2021 at 11:25 PM Andrew Grande > wrote: > > >> Mysql has been generating an admin password on default installs for, > like, > > >> forever. This workflow should be familiar for many users. > > >> > > >> I'd suggest taking the automation tooling into account and how a > production > > >> rollout (user-provided password) would fit into the workflow. > > >> > > >> Andrew > > >> > > >> On Tue, Feb 9, 2021, 8:15 PM Tony Kurc wrote: > > >> > > >>> Joe, > > >>> In addition to your suggestions, were you thinking of making this > processor > > >>> disabled by default as well? > > >>> > > >>> Tony > > >>> > > >>> > > >>> On Tue, Feb 9, 2021, 11:04 PM Joe Witt wrote: > > >>> > > >>>> Team > > >>>> > > >>>> While secure by default may not be practical perhaps ‘not blatantly > wide > > >>>> open’ by default should be adopted. > > >>>> > > >>>> I think we should consider killing support for http entirely and > support > > >>>> only https. We should consider auto generating a user and password > and > > >>>> possibly server cert if nothing is configured and log the generated > user > > >>>> and password. Sure it could still be configured to be non secure > but > > >>> that > > >>>> would truly be an admins fault. Now its just ‘on’ > > >>>> > > >>>> This tweet is a great example of why > > >>>> > > >>>> https://twitter.com/_escctrl_/status/1359280656174510081?s=21 > > >>>> > > >>>> > > >>>> Who agrees? Who disagrees? Please share ideas. > > >>>> > > >>>> Thanks > > >>>> >
Re: [discuss] we need to enable secure by default...
Mark, Thanks for your perspective on certificate generation, I agree that troubleshooting certificates can be confusing due to unclear error messaging. Generating self-signed certificates for one-way TLS still requires the user to accept the certificate presented, but at least that is more common in various products. Are you concerned about that situation, or are you more concerned about the additional challenges of mutual TLS authentication? Generating a random password in absence of any other configuration would certainly be easier from a new user perspective. Some of the security concerns with password authentication can be mitigated with one-way TLS, so a blending of these approaches, as Joe describes in NIFI-8220, seems like a good way to go. Regards, David Handermann On Wed, Feb 10, 2021 at 8:23 AM Mark Payne wrote: > I would be in favor of this as well. I agree that there is no need to wait > for a 2.0 version - there is no inherit backward compatibility challenge > here. > Requiring that new application configs be set, etc. I think is completely > fair game between minor versions. > > Personally, though, I would very much stray away from auto-generating > certificates. For those of us who have dealt with certificates significantly > In the past, minor configuration issues are easy to address. But for > someone who hasn’t spent much time dealing with certificates, the error > messages > that are often intentionally vague can quickly result in users being > overwhelmed. This is especially true if browser configurations are already > setup to > automatically offer certificates that area already installed - this can > result in weird error messages about untrusted certificates when the user > thinks > they haven’t even provided a certificate, etc. It gets really hairy. > > I am more in favor of a default username/password personally. It would > require implementing a new auth provider. And it’s one that historically we > have > avoided implementing because a basic auth type of approach is less secure > than mutual auth, ldap, etc. But it’s more secure than nothing. > > Thanks > -Mark > > > > On Feb 10, 2021, at 9:13 AM, Joe Witt wrote: > > > > There are a range of things to consider. And yes whatever we do will > > involve writing code. We also have to find the right place for the bar > > here. > > > > 1. Disable HTTP by default. And if they want to enable HTTP they should > > also have to make a config change indicating they are fully willing to > > accept that it is an entirely non secure configuration as far as the > > application is concerned. > > 2. Provide a default username/password provider out of the box configured > > by default and an auto generate self-signed server cert. This means the > > NiFi server will provide the client browser a cert. It won't be > > known/trusted so the browser will advise of this. And on startup NiFi > can > > auto generate and log this username and password. It would truly be a > > single username/password and not some store for various users. We > > disallow access to all restricted components by default. > > > > This default configuration at least means someone cannot start NiFi and > it > > is totally exposed by default while also preserving a pretty simple 'get > > started' model for the user. They'd have to take action to make it less > > secure. > > > > The option DavidH mentions of mutual auth could work well also and as he > > mentioned avoids the need for anything special in terms of auth provider > > which is compelling for us but I do worry about the user experience on > that. > > > > I'll add to this that I think we cannot afford to wait for a NiFi 2.0 > > release to take action here. Or that we should expedite a NiFi 2.0 > release > > to take action and that we should just not make the bar for what 2.0 is > so > > high that we never get this done. But frankly I think we could make this > > change in NiFi 1.15 and document what is happening. For existing > > installs/configs we'd not be changing anything except maybe when they're > > using HTTP and don't set the 'no seriously i want this thing wide open > > option'. > > > > Thanks > > > > On Wed, Feb 10, 2021 at 6:58 AM David Handermann < > exceptionfact...@gmail.com> > > wrote: > > > >> Having NiFi enforce authentication by default seems like the right way > to > >> go, especially given the capabilities of the system. > >> > >> Bryan raises a good point about storage of account information, so > weighing > >> the positives
Re: [discuss] we need to enable secure by default...
Mark, Thanks for clarifying, that makes sense. Regards, David Handermann On Wed, Feb 10, 2021 at 8:41 AM Mark Payne wrote: > David, > > My concern was purely around generating client certs and using mutual TLS. > I definitely think we should have a server cert if using username & > password. > > Thanks > -Mark > > > > On Feb 10, 2021, at 9:37 AM, David Handermann < > exceptionfact...@gmail.com> wrote: > > > > Mark, > > > > Thanks for your perspective on certificate generation, I agree that > > troubleshooting certificates can be confusing due to unclear error > > messaging. Generating self-signed certificates for one-way TLS still > > requires the user to accept the certificate presented, but at least that > is > > more common in various products. Are you concerned about that situation, > > or are you more concerned about the additional challenges of mutual TLS > > authentication? > > > > Generating a random password in absence of any other configuration would > > certainly be easier from a new user perspective. Some of the security > > concerns with password authentication can be mitigated with one-way TLS, > so > > a blending of these approaches, as Joe describes in NIFI-8220, seems > like a > > good way to go. > > > > Regards, > > David Handermann > > > > On Wed, Feb 10, 2021 at 8:23 AM Mark Payne wrote: > > > >> I would be in favor of this as well. I agree that there is no need to > wait > >> for a 2.0 version - there is no inherit backward compatibility challenge > >> here. > >> Requiring that new application configs be set, etc. I think is > completely > >> fair game between minor versions. > >> > >> Personally, though, I would very much stray away from auto-generating > >> certificates. For those of us who have dealt with certificates > significantly > >> In the past, minor configuration issues are easy to address. But for > >> someone who hasn’t spent much time dealing with certificates, the error > >> messages > >> that are often intentionally vague can quickly result in users being > >> overwhelmed. This is especially true if browser configurations are > already > >> setup to > >> automatically offer certificates that area already installed - this can > >> result in weird error messages about untrusted certificates when the > user > >> thinks > >> they haven’t even provided a certificate, etc. It gets really hairy. > >> > >> I am more in favor of a default username/password personally. It would > >> require implementing a new auth provider. And it’s one that > historically we > >> have > >> avoided implementing because a basic auth type of approach is less > secure > >> than mutual auth, ldap, etc. But it’s more secure than nothing. > >> > >> Thanks > >> -Mark > >> > >> > >>> On Feb 10, 2021, at 9:13 AM, Joe Witt wrote: > >>> > >>> There are a range of things to consider. And yes whatever we do will > >>> involve writing code. We also have to find the right place for the bar > >>> here. > >>> > >>> 1. Disable HTTP by default. And if they want to enable HTTP they > should > >>> also have to make a config change indicating they are fully willing to > >>> accept that it is an entirely non secure configuration as far as the > >>> application is concerned. > >>> 2. Provide a default username/password provider out of the box > configured > >>> by default and an auto generate self-signed server cert. This means > the > >>> NiFi server will provide the client browser a cert. It won't be > >>> known/trusted so the browser will advise of this. And on startup NiFi > >> can > >>> auto generate and log this username and password. It would truly be a > >>> single username/password and not some store for various users. We > >>> disallow access to all restricted components by default. > >>> > >>> This default configuration at least means someone cannot start NiFi and > >> it > >>> is totally exposed by default while also preserving a pretty simple > 'get > >>> started' model for the user. They'd have to take action to make it > less > >>> secure. > >>> > >>> The option DavidH mentions of mutual auth could work well also and as > he > >>> mentioned avoids the need for anything spec
Re: [discuss] we need to enable secure by default...
I agree that a generated password is the way to go, and the challenge is making it available to the user. Depending on how it is stored for the identity provider, having a command line tool to read and print it could be a reasonable option. Although this particular thread referenced a specific Twitter post, this general discussion is more of a reflection of the need to make things more secure by default as other products have followed similar approaches. Regards, David Handermann On Wed, Feb 10, 2021 at 8:53 AM Kevin Doran wrote: > I am in favor of requiring some authentication by default. > > I’m in favor of an admin username and generated password. (It sounds li9ke > most of us are on the same page, but I don’t think a static, default > password buys us much against the types of abuse shown on the twitter > thread Joe shared.) > > We need some way of making the generated password discoverable… Print to > the logs on first boot? Not great but are there other mechanisms? A setup > CLI utility? > > To help not impede automated deployments, maybe we should change the > startup flow such that there is a way to provide this password. That would > be better than people just disabling whatever default authentication we set. > > > > On Feb 10, 2021, at 09:45, David Handermann > wrote: > > > > Mark, > > > > Thanks for clarifying, that makes sense. > > > > Regards, > > David Handermann > > > > On Wed, Feb 10, 2021 at 8:41 AM Mark Payne wrote: > > > >> David, > >> > >> My concern was purely around generating client certs and using mutual > TLS. > >> I definitely think we should have a server cert if using username & > >> password. > >> > >> Thanks > >> -Mark > >> > >> > >>> On Feb 10, 2021, at 9:37 AM, David Handermann < > >> exceptionfact...@gmail.com> wrote: > >>> > >>> Mark, > >>> > >>> Thanks for your perspective on certificate generation, I agree that > >>> troubleshooting certificates can be confusing due to unclear error > >>> messaging. Generating self-signed certificates for one-way TLS still > >>> requires the user to accept the certificate presented, but at least > that > >> is > >>> more common in various products. Are you concerned about that > situation, > >>> or are you more concerned about the additional challenges of mutual TLS > >>> authentication? > >>> > >>> Generating a random password in absence of any other configuration > would > >>> certainly be easier from a new user perspective. Some of the security > >>> concerns with password authentication can be mitigated with one-way > TLS, > >> so > >>> a blending of these approaches, as Joe describes in NIFI-8220, seems > >> like a > >>> good way to go. > >>> > >>> Regards, > >>> David Handermann > >>> > >>> On Wed, Feb 10, 2021 at 8:23 AM Mark Payne > wrote: > >>> > >>>> I would be in favor of this as well. I agree that there is no need to > >> wait > >>>> for a 2.0 version - there is no inherit backward compatibility > challenge > >>>> here. > >>>> Requiring that new application configs be set, etc. I think is > >> completely > >>>> fair game between minor versions. > >>>> > >>>> Personally, though, I would very much stray away from auto-generating > >>>> certificates. For those of us who have dealt with certificates > >> significantly > >>>> In the past, minor configuration issues are easy to address. But for > >>>> someone who hasn’t spent much time dealing with certificates, the > error > >>>> messages > >>>> that are often intentionally vague can quickly result in users being > >>>> overwhelmed. This is especially true if browser configurations are > >> already > >>>> setup to > >>>> automatically offer certificates that area already installed - this > can > >>>> result in weird error messages about untrusted certificates when the > >> user > >>>> thinks > >>>> they haven’t even provided a certificate, etc. It gets really hairy. > >>>> > >>>> I am more in favor of a default username/password personally. It would > >>>> require implementing a new auth provider. And it’s one that > >> historically we > >>>> h
Re: [discuss] we need to enable secure by default...
Integrating an option to use Let's Encrypt would be a great improvement over self-signed certificates, but it is difficult to support that for instances of NiFi running on servers without Internet access. Even when running NiFi for local development purposes, the development system is not likely to have a publicly addressable DNS name, so Let's Encrypt is not an option in those cases. Regards, David Handermann On Wed, Feb 10, 2021 at 11:09 AM Joe Witt wrote: > Otto > > Installers like you mention are inherently brutal for portability so very > difficult for us in the community. From the vendor world we of course do > things like that. I think in apache nifi land we need a default 'even for > devs' which is not wide open. > > James > > I do think adding such a warning is good. But it doesn't help avoid these > wildly abusive cases. We need a secure by default path. We can probably > do better than the self signed path if we add a 'before running' step in > the CLI for the user. Not sure. > > Thanks > > On Wed, Feb 10, 2021 at 10:05 AM James Srinivasan < > james.sriniva...@gmail.com> wrote: > > > Would a suitably large warning on the http ui be a good starting point? > > > > Browsers are getting increasingly wary of self signed certs and we > probably > > don't want to be encouraging people to ignore them. > > > > What about easier acme+certbot support? (notwithstanding all the non > public > > deployments) > > > > On Wed, 10 Feb 2021, 15:25 Otto Fowler, wrote: > > > > > Aren’t most of these applications installed by an installer? > > > Maybe we can have one or more installers that setup a secure instance, > > and > > > those installers > > > could be the *production* nifi, and keep the zip distribution open for > > > developers? > > > > > > > > > > On Feb 10, 2021, at 10:04, David Handermann < > > exceptionfact...@gmail.com> > > > wrote: > > > > > > > > I agree that a generated password is the way to go, and the challenge > > is > > > > making it available to the user. Depending on how it is stored for > the > > > > identity provider, having a command line tool to read and print it > > could > > > be > > > > a reasonable option. > > > > > > > > Although this particular thread referenced a specific Twitter post, > > this > > > > general discussion is more of a reflection of the need to make things > > > more > > > > secure by default as other products have followed similar approaches. > > > > > > > > Regards, > > > > David Handermann > > > > > > > > On Wed, Feb 10, 2021 at 8:53 AM Kevin Doran > wrote: > > > > > > > >> I am in favor of requiring some authentication by default. > > > >> > > > >> I’m in favor of an admin username and generated password. (It sounds > > > li9ke > > > >> most of us are on the same page, but I don’t think a static, default > > > >> password buys us much against the types of abuse shown on the > twitter > > > >> thread Joe shared.) > > > >> > > > >> We need some way of making the generated password discoverable… > Print > > to > > > >> the logs on first boot? Not great but are there other mechanisms? A > > > setup > > > >> CLI utility? > > > >> > > > >> To help not impede automated deployments, maybe we should change the > > > >> startup flow such that there is a way to provide this password. That > > > would > > > >> be better than people just disabling whatever default authentication > > we > > > set. > > > >> > > > >> > > > >>> On Feb 10, 2021, at 09:45, David Handermann < > > > exceptionfact...@gmail.com> > > > >> wrote: > > > >>> > > > >>> Mark, > > > >>> > > > >>> Thanks for clarifying, that makes sense. > > > >>> > > > >>> Regards, > > > >>> David Handermann > > > >>> > > > >>> On Wed, Feb 10, 2021 at 8:41 AM Mark Payne > > > wrote: > > > >>> > > > >>>> David, > > > >>>> > > > >>>> My concern was purely around generating client certs and using > > mutual > > > >> TLS. &
Re: [VOTE] Release Apache NiFi 1.13.0 (rc4)
+1 non-binding Verified release signatures and expected files. Verified build on Ubuntu 20.10 using Apache Maven 3.6.3 with Azul Zulu JDK 11.0.10 and AdoptOpenJDK 1.8.0_282. Verified absence of startup script shell warnings on Ubuntu 20.10. Verified service listening on localhost when using default nifi.properties. Configured and tested mutual TLS authentication on a standalone server with BCFKS key store and trust store. Regards, David Handermann On Fri, Feb 12, 2021 at 5:45 PM Muazma Zahid wrote: > +1 (non-binding) > > - Ran build with OpenJDK 1.8.0_275 on Linux > - Deployed cluster on Azure and tested flows with Blob, ADLS, and CosmosDB > processors. > > Looks good to me. > > Thanks > Muazma > > On Fri, Feb 12, 2021 at 3:38 PM Sushil Kumar wrote: > > > +1 (non-binding) Release this package as nifi-1.13.0 > > > > Deployed this via helm chart(https://github.com/sushilkm/nifi-chart) on > > kubernetes. > > Thank you to all the contributors and reviewers. > > > > On Fri, Feb 12, 2021 at 2:13 PM Marc Parisi wrote: > > > > > +1 > > > - Verified sigs and hashes > > > - Built on PopOS w/ java 8 && 11 > > > - Tested with basic flow sending data to various devices w/ Secured > > > transport > > > - Tested secured w/ secured nifi reg. > > > > > > Thanks, > > > Marc > > > > > > On Fri, Feb 12, 2021 at 2:56 PM Andrew Lim > > > > wrote: > > > > > > > +1 (binding) > > > > > > > > -Tested secure NiFi with secure NiFi Registry 0.8.0 > > > > -Ran basic flows successfully > > > > -Tested basic versioned flow functionality > > > > -Reviewed and tested Core UI fixes and Documentation updates > > > > > > > > Drew > > > > > > > > > On Feb 10, 2021, at 11:37 PM, Joe Witt wrote: > > > > > > > > > > Hello, > > > > > > > > > > I am pleased to be calling this vote for the source release of > Apache > > > > NiFi > > > > > 1.13.0. > > > > > > > > > > The source zip, including signatures, digests, etc. can be found > at: > > > > > > > https://repository.apache.org/content/repositories/orgapachenifi-1178 > > > > > > > > > > The source being voted upon and the convenience binaries can be > found > > > at: > > > > > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.13.0/ > > > > > > > > > > A helpful reminder on how the release candidate verification > process > > > > works: > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate > > > > > > > > > > The Git tag is nifi-1.13.0-RC4 > > > > > The Git commit ID is 3bc6a122091214b33eee17a270163d7ca26e2a0c > > > > > > > > > > > > > > > https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=3bc6a122091214b33eee17a270163d7ca26e2a0c > > > > > > > > > > Checksums of nifi-1.13.0-source-release.zip: > > > > > SHA256: > > > 95efe5db38e973c9f4062a7b2c95fdc5dad31d7c5e1d36ce1776b9b227c89b9c > > > > > SHA512: > > > > > > > > > > > > > > > d7dd9e851341ebd605784142a7861935f6f814bc612499013456a15713bc9119e426df8f26445c260bdb25cbfc21822cf0d44985bf372a065c9d8597953a3c4a > > > > > > > > > > Release artifacts are signed with the following key: > > > > > https://people.apache.org/keys/committer/joewitt.asc > > > > > > > > > > KEYS file available here: > > > > > https://dist.apache.org/repos/dist/release/nifi/KEYS > > > > > > > > > > 260 issues were closed/resolved for this release: > > > > > > > > > > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12348700 > > > > > > > > > > Release note highlights can be found here: > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.13.0 > > > > > > > > > > The vote will be open for 72 hours. > > > > > Please download the release candidate and evaluate the necessary > > items > > > > > including checking hashes, signatures, build from source, and test. > > > Then > > > > > please vote: > > > > > > > > > > [ ] +1 Release this package as nifi-1.13.0 > > > > > [ ] +0 no opinion > > > > > [ ] -1 Do not release this package because... > > > > > > > > > > > > > >
Re: [VOTE] Release Apache NiFi 1.13.1
+1 non-binding Verified release signatures and expected files. Verified build on Ubuntu 20.10 using Apache Maven 3.6.3 with Azul Zulu JDK 11.0.10. Configured and tested InvokeHTTP with multiple configurations including disabling HTTP/2. Regards, David Handermann On Fri, Mar 12, 2021 at 2:19 PM Joey Frazee wrote: > +1 (non-binding) > > - Verified checksums and signatures > - Ran full build on Java 1.8 and 11 on Linux > - Verified NIFI-3383, NIFI-8231, and NIFI-8200 > > -joey > > > On Mar 12, 2021, at 10:28 AM, Bryan Bende wrote: > > > > +1 (binding) > > > > - Verified everything in the standard release helper > > - Setup secure standalone instance with SAML authentication > > > >> On Fri, Mar 12, 2021 at 10:17 AM Marton Szasz > wrote: > >> > >> +1 (non-binding) > >> > >> Followed the release helper guide, tested the binary with a simple flow. > >> > >> Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f) > >> Maven home: /usr/share/maven-bin-3.6 > >> Java version: 1.8.0_252, vendor: IcedTea, runtime: > /opt/icedtea-bin-3.16.0/jre > >> Default locale: en_US, platform encoding: UTF-8 > >> OS name: "linux", version: "5.10.12-gentoo-x86_64", arch: "amd64", > >> family: "unix" > >> > >> > >>> On Fri, 12 Mar 2021 at 13:35, Otto Fowler > wrote: > >>> > >>> +1 > >>> > >>> Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f) > >>> Maven home: /usr/local/Cellar/maven/3.6.3_1/libexec > >>> Java version: 1.8.0_282, vendor: AdoptOpenJDK, runtime: > /Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/jre > >>> Default locale: en_US, platform encoding: UTF-8 > >>> OS name: "mac os x", version: "10.16", arch: "x86_64", family: “mac" > >>> > >>> > >>>> On Mar 11, 2021, at 11:29, Joe Witt wrote: > >>>> > >>>> Hello, > >>>> > >>>> I am pleased to be calling this vote for the source release of Apache > NiFi > >>>> 1.13.1. > >>>> > >>>> The source zip, including signatures, digests, etc. can be found at: > >>>> https://repository.apache.org/content/repositories/orgapachenifi-1179 > >>>> > >>>> The source being voted upon and the convenience binaries can be found > at: > >>>> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.13.1/ > >>>> > >>>> A helpful reminder on how the release candidate verification process > works: > >>>> > https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate > >>>> > >>>> The Git tag is nifi-1.13.1-RC1 > >>>> The Git commit ID is acbc217cb7002d7489421f4d346b995a43b6ea01 > >>>> > https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=acbc217cb7002d7489421f4d346b995a43b6ea01 > >>>> > >>>> Checksums of nifi-1.13.1-source-release.zip: > >>>> SHA256: > 0a397df640e579720e26699e38a3738c5be05af01aad8aaeefc04eb58591faac > >>>> SHA512: > >>>> > 7f8df759d4345ccd6e75c169bd0aab1b7f4f64bf5a8b11b45bc1d7c8163acd0035922dcdbef232392279f4ea0710d4a97c55d480281bfe1d50b6401295633d48 > >>>> > >>>> Release artifacts are signed with the following key: > >>>> https://people.apache.org/keys/committer/joewitt.asc > >>>> > >>>> KEYS file available here: > >>>> https://dist.apache.org/repos/dist/release/nifi/KEYS > >>>> > >>>> 48 issues were closed/resolved for this release: > >>>> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12348700 > >>>> > >>>> Release note highlights can be found here: > >>>> > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.13.1 > >>>> > >>>> The vote will be open for 72 hours. > >>>> Please download the release candidate and evaluate the necessary items > >>>> including checking hashes, signatures, build from source, and test. > Then > >>>> please vote: > >>>> > >>>> [ ] +1 Release this package as nifi-1.13.1 > >>>> [ ] +0 no opinion > >>>> [ ] -1 Do not release this package because... > >>> >
Re: [VOTE] Release Apache NiFi 1.13.2
+1 non-binding - Verified release signatures and expected files - Verified build on Ubuntu 20.10 using Apache Maven 3.6.3 with Azul Zulu JDK 11.0.10 - Verified Admin Guide update for NIFI-8324 - Verified valid JSON output from SplitJson for NIFI-8342 Regards, David On Thu, Mar 18, 2021 at 1:09 PM Pierre Villard wrote: > +1 binding > > Confirmed the main fix with a flow provided in the JIRA. > > Thanks, > Pierre > > Le jeu. 18 mars 2021 à 21:37, Mark Payne a écrit : > > > +1 (binding) > > > > Was able to build, and successfully run all system tests, including those > > added for NIFI-8337. Also started up and manually verified that NIFI-8337 > > has been addressed. > > > > Thanks > > -Mark > > > > > On Mar 18, 2021, at 1:09 PM, Matt Burgess > wrote: > > > > > > +1 Release this package as nifi-1.13.2 > > > > > > Ran through release helper, verified fixes for NIFI-8297 and > > > NIFI-8334/8342 on an unsecured standalone instance. Thanks for RM'ing > > > and for the quick turnaround Joe! > > > > > > Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f) > > > Maven home: /Users/mburgess/.sdkman/candidates/maven/current > > > Java version: 1.8.0_282, vendor: AdoptOpenJDK, runtime: > > > /Users/mburgess/.sdkman/candidates/java/8.0.282.hs-adpt/jre > > > Default locale: en_US, platform encoding: UTF-8 > > > OS name: "mac os x", version: "10.15.7", arch: "x86_64", family: "mac" > > > > > > On Thu, Mar 18, 2021 at 1:29 AM Joe Witt wrote: > > >> > > >> Hello, > > >> > > >> Please note this has a shorter than normal vote period as this > > >> corrects an important regression introduced in 1.13.1 and is purely a > > >> focused bug fix release. > > >> > > >> I am pleased to be calling this vote for the source release of Apache > > >> NiFi 1.13.2. > > >> > > >> The source zip, including signatures, digests, etc. can be found at: > > >> https://repository.apache.org/content/repositories/orgapachenifi-1180 > > >> > > >> The source being voted upon and the convenience binaries can be found > > at: > > >> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.13.2/ > > >> > > >> A helpful reminder on how the release candidate verification process > > works: > > >> > > > https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate > > >> > > >> The Git tag is nifi-1.13.2-RC1 > > >> The Git commit ID is 174938e5e3bfe36951d5607d0d53e78604b0e07b > > >> > > > https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=174938e5e3bfe36951d5607d0d53e78604b0e07b > > >> > > >> Checksums of nifi-1.13.2-source-release.zip: > > >> SHA256: > db18f57b4629367f21f70ec9a332294ec6b6c090a661c974b0fa7a80b09a79be > > >> SHA512: > > > 279c5701fd7b6d76206f5e79bdf1340d432a8e4834b8e786559095781c77c96a4594e1da040c6b404d1bab615acf5aa9dc60d289e460e6d7261f454f6210c66b > > >> > > >> Release artifacts are signed with the following key: > > >> https://people.apache.org/keys/committer/joewitt.asc > > >> > > >> KEYS file available here: > > >> https://dist.apache.org/repos/dist/release/nifi/KEYS > > >> > > >> 5 issues were closed/resolved for this release: > > >> > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12350008 > > >> > > >> Release note highlights can be found here: > > >> > > > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.13.2 > > >> > > >> The vote will be open for 36 hours. > > >> Please download the release candidate and evaluate the necessary items > > >> including checking hashes, signatures, build from source, and test. > > >> Then please vote: > > >> > > >> [ ] +1 Release this package as nifi-1.13.2 > > >> [ ] +0 no opinion > > >> [ ] -1 Do not release this package because... > > > > >
Re: Apache Project issues with available GitHub Action resources across projects
This background is very helpful to keep in mind when evaluating new and updated unit tests. There are definitely some expensive tests that could be streamlined, but introducing a separate version lifecycle for framework and extensions seems like it is becoming more necessary. Moving to a Java 11 baseline would also reduce the need to build on multiple versions, but based on other discussions, it sounds like that is not currently scheduled. I have noticed that Windows builds can run for a longer period of time, is there a reason that the Windows workflow definition does not have the same timeout setting as the Linux and macOS builds? Introducing one would at least kill off problematic Windows builds. Regards, David Handermann On Mon, Apr 19, 2021 at 10:08 AM Joe Witt wrote: > Thanks for bringing this up. The most clear next step I can envision > at this point is that we break up our core framework from our > extensions. Not obvious how best to break this up but we need to. > The build times are insane. > > Joe > > On Mon, Apr 19, 2021 at 7:57 AM Otto Fowler > wrote: > > > > As you can probably imagine, it takes a lot of resources in order to CI > for all the Apache projects. Periodically this becomes an issue, as the > donated resources from cloud CI providers ( Travis and now GitHub Actions ) > end up queuing and delaying builds across Apache projects because of larger > projects and their requirements. > > > > The discussions center around a few common themes: > > > > - the CI requirements of large complex projects dominate the Apache Queue > > - how those projects can supplement their builds in a way acceptable to > ASF INFRA > > - how those projects can have per project metering such that a project > can pay for hours over the ’norm’ > > - how to optimize projects > > > > This issue is currently being discussed again on the @builds apache > list. I’m sending this over to the Nifi Dev community because Nifi itself > has been mentioned as one of the top users of GitHub action resources by > some measure. Many of the other projects are actively taking measures to > decrease or optimize their usage, and we should probably think about how we > can do so as well. > > > > Here is the *current* thread: > > > https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3cCAMFhwAbyoDPM8V7bt6=40m++GTdbXK64Q5LjwD=cvrghs7d...@mail.gmail.com%3e > < > https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3CCAMFhwAbyoDPM8V7bt6=40m++GTdbXK64Q5LjwD=cvrghs7d...@mail.gmail.com%3E > > > > > > Here is the message where 13 projects are listed > > > https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3c86d8e65a-4943-cbf1-ec46-3980128b8...@apache.org%3e > < > https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3c86d8e65a-4943-cbf1-ec46-3980128b8...@apache.org%3E > > > > > > There are many other threads regarding GitHub Action limits and > resources if you start scrolling back through the archives. > > > > I’m posting this to hopefully kick off some discussions. >
Re: Apache Project issues with available GitHub Action resources across projects
Chris,, Thanks for the description of the modular approach. Here is one GitHub action that supports conditional execution based on Git changes: https://github.com/dorny/paths-filter It would take some effort to implement an approach that covers all the bases, but using the Maven also-make and project-list command line options should support building both a module and its dependencies. The list of changes files can be passed to another action that could determine one or more modules to build. Regards, David Handermann On Mon, Apr 19, 2021 at 12:56 PM Chris Sampson wrote: > Our approach is the use a `git diff` between source and destination > branches when a branch is merged (and the same for on dev branch builds). > > Each component within the repo then checks for whether any files within its > directories were changed (bearing in mind a dev branch may consist of > multiple commits over time, so don't just look at changes in individual > commits). > > I'm not sure if the mechanics for the existing nifi repo though with its > use of: > > * GutHub Actions (can a git diff be performed at the start and then > following actions use the output?) > > * Maven with sub-modules (can they be optionally built based on such > conditions? The GitHub Action could create files depending upon what's been > changed in order to activate profiles maybe, would that work?) > > This is broadly how I've implemented a multi-component build previously, > but not using Maven sub-modules (each component is typically its own docker > image and built separately from others). > > > Cheers, > > Chris Sampson > > On Mon, 19 Apr 2021, 18:35 Joe Witt, wrote: > > > Chris, > > > > Yeah that would be very helpful. But do you have any idea how that > > might be achieved in this environment? > > > > Thanks > > > > On Mon, Apr 19, 2021 at 10:33 AM Chris Sampson > > wrote: > > > > > > Could an approach of building only the updated parts of the repo help > to > > > reduce build times? > > > > > > For example, changes to the classes under the AWS bundle (and only that > > > bundle) would only need those classes to be built and tested. > > > > > > Where such an approach gets a bit more complex is interdependence > between > > > parts of the repo. For example, if nifi-api is updated, should all > > classes > > > be built and tested? > > > > > > As part of a release, the entire repo could then be built and tested. > > > > > > This approach might help if the majority of repo changes are to > > individual > > > NARs rather than wider ranging. I'm also assuming this would be > possible > > > via GitHub Actions (I have no experience of them, but have implemented > > this > > > kind of approach in a Drone.io mono-repo previously). > > > > > > > > > Cheers, > > > > > > Chris Sampson > > > > > > On Mon, 19 Apr 2021, 17:16 David Handermann, < > exceptionfact...@gmail.com > > > > > > wrote: > > > > > > > This background is very helpful to keep in mind when evaluating new > and > > > > updated unit tests. There are definitely some expensive tests that > > could > > > > be streamlined, but introducing a separate version lifecycle for > > framework > > > > and extensions seems like it is becoming more necessary. Moving to a > > Java > > > > 11 baseline would also reduce the need to build on multiple versions, > > but > > > > based on other discussions, it sounds like that is not currently > > scheduled. > > > > > > > > I have noticed that Windows builds can run for a longer period of > > time, is > > > > there a reason that the Windows workflow definition does not have the > > same > > > > timeout setting as the Linux and macOS builds? Introducing one would > > at > > > > least kill off problematic Windows builds. > > > > > > > > Regards, > > > > David Handermann > > > > > > > > On Mon, Apr 19, 2021 at 10:08 AM Joe Witt > wrote: > > > > > > > > > Thanks for bringing this up. The most clear next step I can > envision > > > > > at this point is that we break up our core framework from our > > > > > extensions. Not obvious how best to break this up but we need to. > > > > > The build times are insane. > > > > > > > > > > Joe > > > > > > >
Re: [ANNOUNCE] New Apache NiFi Committer Chris Sampson
Congratulations Chris! Looking forward to your continued contributions! Regards, David Handermann On Thu, May 13, 2021 at 3:38 PM Matt Burgess wrote: > Congratulations and welcome aboard Chris! > > On Thu, May 13, 2021 at 4:25 PM Joe Witt wrote: > > > > On behalf of the Apache NiFI PMC, I am very pleased to announce that > > Chris has accepted the PMC's invitation to become a committer on the > > Apache NiFi project. We greatly appreciate the contributions from Chris > > on the mailing list, PR reviews, code contributions, and especially rich > > communications in the Apache NiFi slack channels. > > > > Welcome and congratulations! >
Re: [discuss] nifi 1.14.0
Thanks for kicking off the discussion Joe! Of the many items that could be included in the next release, securing the default configuration as described in NIFI-8220 <https://issues.apache.org/jira/browse/NIFI-8220> would be great to have completed. Most of the elements are in place, and the current Pull Request for NIFI-8516 <https://github.com/apache/nifi/pull/5068> is under review. If there are any other achievable items that should be included as part of a secure default installation for NiFi, it would be helpful to add sub-tasks to NIFI-8220. The current scope is limited to a standalone installation, so issues regarding clustered deployments can be handled separately. If others are interested in evaluating the proposed new default configuration that requires HTTPS and leverages a generated username and password, feel free to provide feedback on NIFI-8516. Regards, David Handermann On Thu, May 27, 2021 at 6:51 PM Otto Fowler wrote: > I think NIFI-8625 and NIFI-8461 need to be understood and addressed. > > > > On May 27, 2021, at 13:29, Joe Witt wrote: > > > > Team, > > > > There has been a tremendous amount of work already on the 1.14 line as > shown: > > > > https://issues.apache.org/jira/projects/NIFI/versions/12349644 > > > > These include merging the nifi registry and minifi java into the nifi > > line itself. So when we release these things stay in sync and > > maintained. The release will now produce things like Apache NiFi, the > > Apache NiFi toolkit, Apache NiFi Registry, and Apache NiFi MiNiFi Java > > and the Apache NiFi stateless runtime as well. There have been many > > improvements to core nifi and stateless nifi now meaning we have the > > traditional execution form factor and this new stateless mode. We can > > now hot load nars from HDFS storage locations which could mean HDFS, > > blob storage in the cloud, etc.. There is a lot more. > > > > Anyway, I wanted to start circling the wagons for a 1.14 release. I'm > > happy to take on RM duties especially since there will be new elements > > to the release process. > > > > Thanks > >
Re: SSLPeerUnverifiedException following upgrade from 1.6.0 to 1.13.2
Hi Phil, Thanks for providing the stack trace. Recent versions of NiFi include updates to the OkHttp library, which modified the hostname verification process. OkHttp starting with version 3.10.0 made changes to TLS hostname verification, requiring that a certificate contain DNS Subject Alternative Names matching the connection hostname. Based on the error message, it appears that the certificates configured do not have any Subject Alternative Names, resulting in the SSLPeerUnverifiedException. Generating or obtaining new certificates that include the required DNS Subject Alternative Names should resolve the problem. Here's the release notes for OkHttp 3.10.0, referencing RFC 2818, which deprecated falling back to certificate common names for hostname verification: https://square.github.io/okhttp/changelog_3x/#version-3100 Regards, David Handermann On Thu, Jun 10, 2021 at 11:16 PM Phil H wrote: > Hi there, > > I upgraded an older dev setup today from 1.6.0 to 1.13.2. After a > couple of config tweaks, it’s “working”, but if I try and access the > interface at https://nifi2.domain.blah/ I get a message on screen > stating that nifi1.domain.blah is not verified. The logs contain this > same message, along with the stack trace. (This also happens if I > access nifi1 – it complains nifi2 is not verified). > > My keystore and truststore on both servers both contain the certs for > both servers, and the truststore additionally contains the CA that > signed both servers’ certificates. > > What am I missing? > > Thanks, > Phil > > > > > > 2021-06-11 23:51:20,970 WARN [Replicate Request Thread-1] > o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request > GET /nifi-api/flow/current-user to nifi1.domain.blah:443 due to > javax.net.ssl.SSLPeerUnverifiedException: Hostname nifi1.domain.blah > not verified: > > certificate: sha256/Wv+eIBMlpsSS95xKF+Fry9C/jQhFbNS35yfJGK92/5U= > > DN: CN=nifi1.domain.blah, OU=domain, O=blah > > subjectAltNames: [] > > 2021-06-11 23:51:20,970 WARN [Replicate Request Thread-1] > o.a.n.c.c.h.r.ThreadPoolRequestReplicator > > javax.net.ssl.SSLPeerUnverifiedException: Hostname nifi1. > nifi1.domain.blah not verified: > > certificate: sha256/Wv+eIBMlpsSS95xKF+Fry9C/jQhFbNS35yfJGK92/5U= > > DN: CN=nifi1.domain.blah OU=domain, O=blah > > subjectAltNames: [] > > at > > okhttp3.internal.connection.RealConnection.connectTls(RealConnection.kt:389) > > at > > okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.kt:337) > > at > okhttp3.internal.connection.RealConnection.connect(RealConnection.kt:209) > > at > > okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.kt:226) > > at > > okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.kt:106) > > at > okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.kt:74) > > at > okhttp3.internal.connection.RealCall.initExchange$okhttp(RealCall.kt:255) > > at > > okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:32) > > at > > okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) > > at > okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95) > > at > > okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) > > at > okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83) > > at > > okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) > > at > > okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76) > > at > > okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) > > at > > okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201) > > at > okhttp3.internal.connection.RealCall.execute(RealCall.kt:154) > > at > > org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:132) > > at > > org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:126) > > at > > org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestR
Re: [discuss] nifi 1.14.0
Joe, Thanks for following up. The PR for NIFI-8516 has gone through several rounds of feedback, I believe it is about ready to go, pending confirmation that the ability to set custom credentials addresses the ease of use concern. Regards, David Handermann On Fri, Jun 11, 2021 at 11:41 AM Joe Witt wrote: > David, > > Ok thanks - do you have a sense of when what you see as good 1.14 > specific work will be merged? Do you have the reviewers/engagement > you need? > > This 1.14 is already pretty packed but definitely agree we need to > make real progress on secure by default and this release is a great > time to take the first big step. > > Thanks > > On Mon, May 31, 2021 at 5:52 AM David Handermann > wrote: > > > > Thanks for kicking off the discussion Joe! > > > > Of the many items that could be included in the next release, securing > the > > default configuration as described in NIFI-8220 > > <https://issues.apache.org/jira/browse/NIFI-8220> would be great to have > > completed. Most of the elements are in place, and the current Pull > Request > > for NIFI-8516 <https://github.com/apache/nifi/pull/5068> is under > review. > > If there are any other achievable items that should be included as part > of > > a secure default installation for NiFi, it would be helpful to add > > sub-tasks to NIFI-8220. The current scope is limited to a standalone > > installation, so issues regarding clustered deployments can be handled > > separately. If others are interested in evaluating the proposed new > > default configuration that requires HTTPS and leverages a generated > > username and password, feel free to provide feedback on NIFI-8516. > > > > Regards, > > David Handermann > > > > On Thu, May 27, 2021 at 6:51 PM Otto Fowler > wrote: > > > > > I think NIFI-8625 and NIFI-8461 need to be understood and addressed. > > > > > > > > > > On May 27, 2021, at 13:29, Joe Witt wrote: > > > > > > > > Team, > > > > > > > > There has been a tremendous amount of work already on the 1.14 line > as > > > shown: > > > > > > > > https://issues.apache.org/jira/projects/NIFI/versions/12349644 > > > > > > > > These include merging the nifi registry and minifi java into the nifi > > > > line itself. So when we release these things stay in sync and > > > > maintained. The release will now produce things like Apache NiFi, > the > > > > Apache NiFi toolkit, Apache NiFi Registry, and Apache NiFi MiNiFi > Java > > > > and the Apache NiFi stateless runtime as well. There have been many > > > > improvements to core nifi and stateless nifi now meaning we have the > > > > traditional execution form factor and this new stateless mode. We > can > > > > now hot load nars from HDFS storage locations which could mean HDFS, > > > > blob storage in the cloud, etc.. There is a lot more. > > > > > > > > Anyway, I wanted to start circling the wagons for a 1.14 release. > I'm > > > > happy to take on RM duties especially since there will be new > elements > > > > to the release process. > > > > > > > > Thanks > > > > > > >
Re: [discuss] nifi 1.14.0
Thanks to Mark Payne, NIFI-8516 is now merged, so that covers current open issues around securing the default configuration. Regards, David Handermann On Fri, Jun 11, 2021 at 11:55 AM David Handermann < exceptionfact...@apache.org> wrote: > Joe, > > Thanks for following up. The PR for NIFI-8516 has gone through several > rounds of feedback, I believe it is about ready to go, pending confirmation > that the ability to set custom credentials addresses the ease of use > concern. > > Regards, > David Handermann > > On Fri, Jun 11, 2021 at 11:41 AM Joe Witt wrote: > >> David, >> >> Ok thanks - do you have a sense of when what you see as good 1.14 >> specific work will be merged? Do you have the reviewers/engagement >> you need? >> >> This 1.14 is already pretty packed but definitely agree we need to >> make real progress on secure by default and this release is a great >> time to take the first big step. >> >> Thanks >> >> On Mon, May 31, 2021 at 5:52 AM David Handermann >> wrote: >> > >> > Thanks for kicking off the discussion Joe! >> > >> > Of the many items that could be included in the next release, securing >> the >> > default configuration as described in NIFI-8220 >> > <https://issues.apache.org/jira/browse/NIFI-8220> would be great to >> have >> > completed. Most of the elements are in place, and the current Pull >> Request >> > for NIFI-8516 <https://github.com/apache/nifi/pull/5068> is under >> review. >> > If there are any other achievable items that should be included as part >> of >> > a secure default installation for NiFi, it would be helpful to add >> > sub-tasks to NIFI-8220. The current scope is limited to a standalone >> > installation, so issues regarding clustered deployments can be handled >> > separately. If others are interested in evaluating the proposed new >> > default configuration that requires HTTPS and leverages a generated >> > username and password, feel free to provide feedback on NIFI-8516. >> > >> > Regards, >> > David Handermann >> > >> > On Thu, May 27, 2021 at 6:51 PM Otto Fowler >> wrote: >> > >> > > I think NIFI-8625 and NIFI-8461 need to be understood and addressed. >> > > >> > > >> > > > On May 27, 2021, at 13:29, Joe Witt wrote: >> > > > >> > > > Team, >> > > > >> > > > There has been a tremendous amount of work already on the 1.14 line >> as >> > > shown: >> > > > >> > > > https://issues.apache.org/jira/projects/NIFI/versions/12349644 >> > > > >> > > > These include merging the nifi registry and minifi java into the >> nifi >> > > > line itself. So when we release these things stay in sync and >> > > > maintained. The release will now produce things like Apache NiFi, >> the >> > > > Apache NiFi toolkit, Apache NiFi Registry, and Apache NiFi MiNiFi >> Java >> > > > and the Apache NiFi stateless runtime as well. There have been many >> > > > improvements to core nifi and stateless nifi now meaning we have the >> > > > traditional execution form factor and this new stateless mode. We >> can >> > > > now hot load nars from HDFS storage locations which could mean HDFS, >> > > > blob storage in the cloud, etc.. There is a lot more. >> > > > >> > > > Anyway, I wanted to start circling the wagons for a 1.14 release. >> I'm >> > > > happy to take on RM duties especially since there will be new >> elements >> > > > to the release process. >> > > > >> > > > Thanks >> > > >> > > >> >
Re: nifi windows prpblem
Thanks for providing the error message, can you include a few additional details about your system configuration? In particular, which version of Java are you using? This sounds like the following issue, which is fixed in the current main branch and will be including in the next release of NiFi: https://issues.apache.org/jira/browse/NIFI-6714 Regards, David Handermann On Sat, Jun 12, 2021 at 10:31 AM MaHmOuD El-Sayed wrote: > .bootstrap.config.log.dir=C:\Users\MaHmOuD\DOWNLO~1\COMPRE~1\NIFI-1~1.2\bin\..\\logs > org.apache.nifi.NiFi > Exception in thread "main" java.lang.reflect.InaccessibleObjectException: > Unable to make public long java.lang.ProcessImpl.pid() accessible: module > java.base does not "opens java.lang" to unnamed module @ba8d91c >
Re: [VOTE] Release Apache NiFi 1.14.0 (rc2)
+1 (non-binding) - Built from source on Ubuntu 21.04 with Azul Zulu 11.0.10 - Ran NiFi on Azul Zulu 11.0.10 and 1.8.0.282 - Configured encrypted Flow File Repository using KeyStore Key Provider and PKCS12 - Configured encrypted Flow File Swap Manager - Tested set-sensitive-properties-key and set-single-user-credentials commands - Tested flow with PutTCP and ListenTCP with TLS 1.3 - Tested flow with UnpackContent using Zip file with bzip2 compression - Tested flow with EncryptContentPGP and DecryptContentPGP - Tested NiFi Toolkit using AES-GCM encryption of nifi.properties including new repository password property - Tested NiFi Registry with bucket creation and NiFi process group version control - Test NiFi Stateless with GetFile and UnpackContent flow downloaded from NiFi process group Regards, David Handermann On Mon, Jul 12, 2021 at 6:31 AM Arpad Boda wrote: > +1 (binding) > > Built on Debian 10 using Java 11, started, designed a simple flow. > Verified hashes, signatures. > > Ps.: some tests failed, so only built without them. > > On Mon, Jul 12, 2021 at 12:38 PM Joe Gresock wrote: > > > +1 (non-binding) > > > > Verified signature and hashes > > Full build with contrib check with OpenJDK Runtime Environment (Zulu > > 8.54.0.21-CA-macosx) (build 1.8.0_292-b10) > > Upgraded a single secure nifi 1.13.2 cluster with an existing flow, which > > started successfully. > > Verified TLS 1.3 works using an existing flow with > > HandleHttpRequest/Response and InvokeHttp > > Restarted with nifi.security.autoreload.enabled=true and verified that I > > was able to replace a keystore and truststore while NiFi was running > > Upgraded a 3-node secure nifi 1.13.2 cluster with an existing flow, which > > started successfully. > > Verified that load balanced connections work as expected, before and > after > > offloading a node. > > Used the encrypt-config.sh tool from Toolkit to encrypt nifi.properties > and > > nifi-registry.properties using AES/GCM, verified both apps started > > successfully. > > Used encrypt-config.sh to migrate the encryption to the > > HASHICORP_VAULT_TRANSIT protection scheme, verified both apps started > > successfully. > > Built a DataFlow and saved to file, then ran it via Stateless NiFi, > using a > > custom ParamaterProvider, verified that parameters were correctly > provided. > > > > On Sun, Jul 11, 2021 at 1:33 PM Robert Fellows > > wrote: > > > > > +1 (non-binding) > > > > > > Verified signature and hashes > > > Full build with contrib check with OpenJDK Runtime Environment > > AdoptOpenJDK > > > (build 11.0.8+10) > > > Started new instance, verified it was secured by default. > > > Changed the username and password using bin/nifi.sh > > > set-single-user-credentials > > > Logged in with the new user/password i created > > > Fired up registry, created a bucket > > > Connected nifi to registry, versioned a flow. Imported a flow into > > another > > > process group. > > > General App usage (Firefox 89.0.2) > > > > > > Thanks for RM'ing this Joe! > > > > > > -- Rob Fellows > > > > > > On Sat, Jul 10, 2021 at 9:31 PM Mark Payne > wrote: > > > > > > > +1 (binding) > > > > > > > > Verified the hashes. > > > > Performed full build with contrib-check profile > > > > Ran all system tests > > > > Started the newly created instance, verified that it was secure by > > > default. > > > > Changed username & password and verified the behavior. > > > > Created a 2-node cluster, secured by mutual TLS and verified > behavior. > > > > > > > > Started nifi registry. Created a bucket. > > > > Pushed to the bucket from standalone instance. Then added a second > > > version > > > > of flow. > > > > Downloaded flow onto cluster and switched between versioned a few > > times. > > > > > > > > Verified behavior of new default backpressure thresholds. > > > > Built a DataFlow and saved to file, then ran it via Stateless NiFi > > using > > > > command-line parameter overrides. > > > > Started DataFlow using Stateless nifi and the kafka connector. > Verified > > > > this behavior. > > > > > > > > Encountered no issues this time. > > > > > > > > Thanks for putting the RC together, Joe! > > > > > > > > > On Jul 10, 2021, at 6:40 PM, Joe Witt wrote: > > > > > >
Re: [discuss] nifi 1.14.0
Chad, Can you provide the output of the test failures, specifically which unit tests failed and any associated error messages? That would help narrow down the potential source of the problem. Regards, David Handermann On Mon, Jul 12, 2021 at 9:05 PM Chad Zobrisky wrote: > Anyone else having issues building the RC? I'm also having the same test > failure on main. I can build if I skip tests. > > Ubuntu 20.04 > java OpenJDK Runtime Environment AdoptOpenJDK-11.0.11+9 > maven 3.6.3 > > It fails on tests, specifically nifi-database-utils. I've cleared the maven > repository, made sure my java is up to date and tried multiple times with > the same result. I had a different test failure with some slf4j issue on a > slightly older java 11 release from April, and updated to resolve that > issue. > > Thanks, > Chad > > On Mon, Jul 12, 2021 at 8:20 PM Otto Fowler > wrote: > > > I’ll take care of it. I’ll make the minimum 3.6.0 > > > > From: Joe Witt > > Reply: dev@nifi.apache.org > > Date: July 12, 2021 at 18:25:28 > > To: dev@nifi.apache.org > > Subject: Re: [discuss] nifi 1.14.0 > > > > Yeah we just need to update the latest required version . We're going > > all in on all the HTTPS requirements and such so we're probably really > > tight on required version at this point anyway. > > > > I'd not be sinking the RC for this. Just a good JIRA to work on. > > > > Thanks > > > > On Mon, Jul 12, 2021 at 12:59 PM Otto Fowler > > wrote: > > > > > > NIFI-8778 > > > > > > From: Otto Fowler > > > Reply: Otto Fowler > > > Date: July 12, 2021 at 13:56:36 > > > To: dev@nifi.apache.org > > > Subject: Re: [discuss] nifi 1.14.0 > > > > > > I see an issue, though I’m not sure if we want to cancel the RC so I’m > > > asking before voting. > > > The documentation states that we require Apache Maven 3.1.1 or newer, > but > > > the build fails with the error: > > > > > > [ERROR] Failed to execute goal > > > com.github.eirslett:frontend-maven-plugin:1.12.0:install-node-and-npm > > > (install-node-and-npm) on project nifi-web-ui: The plugin > > > com.github.eirslett:frontend-maven-plugin:1.12.0 requires Maven version > > > 3.6.0 -> [Help 1] > > > > > > I have maven version: > > > > > > Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; > > > 2018-06-17T14:33:14-04:00) > > > Maven home: /usr/local/Cellar/maven@3.5/3.5.4_1/libexec > > > Java version: 1.8.0_292, vendor: AdoptOpenJDK, runtime: > > > /Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/jre > > > > > > So, the documentation on building nifi is wrong. > > > > > > > > > > > > From: Joe Witt > > > Reply: dev@nifi.apache.org > > > Date: July 9, 2021 at 10:24:58 > > > To: dev@nifi.apache.org > > > Subject: Re: [discuss] nifi 1.14.0 > > > > > > Still working on the 1.14 RC. Has been a series of issues with the > > > build. Will be up soon hopefully! > > > > > > On Thu, Jul 8, 2021 at 1:41 PM Mark Payne > wrote: > > > > > > > > Joe, > > > > > > > > I just filed a BUG Jira [1]. It’s a bit of a corner case, but when it > > > occurs it can cause a pretty big problem. Should have a fix up very > > > shortly. Will leave it up to you whether or not you think we should get > > > this into 1.14.0. > > > > > > > > Thanks > > > > -Mark > > > > > > > > > > > > [1] https://issues.apache.org/jira/browse/NIFI-8771 > > > > > > > > > > > > On Jul 8, 2021, at 10:26 AM, Joe Witt > > joe.w...@gmail.com>> wrote: > > > > > > > > Team, > > > > > > > > The 1.14 RC1 build is underway. Hopefully I will email artifacts > > > > today. Reminder this should now generate convenience binaries for > > > > nifi, stateless nifi, minifi java, nifi registry and the associated > > > > toolkits all in a single release process which also keeps these > things > > > > in sync. > > > > > > > > Please do not tag any further fix versions to 1.14.0. Use 1.15.0 from > > > > here. I'll pull things in if RCs fail/etc. > > > > > > > > Thanks > > > > > > > > On Tue, Jul 6, 2021 at 8:21 AM Joe Witt > > joe.w...@gmail.com>>
Re: [DISCUSS] NiFi Registry -> NiFi consensus
Thanks for handling this, Matt. As one of the reviewers on the PR, I vote +1 (non-binding). Regards, David Handermann On Fri, Jul 16, 2021 at 1:20 PM Matt Burgess wrote: > All, > > We've been asked to record our consensus for the move of NiFi Registry > to the NiFi codebase in a mailing list thread for posterity. Most > discussion happened on the PR but INFRA would like a link to this > thread showing consensus from PMC members, committers, and the > community. > > I didn't put my +1 on the PR but I am in favor of moving the NiFi > Registry codebase into NiFi :) Please feel free to share your thoughts > as well. > > Thank you, > Matt >
[DISCUSS] NiFi 2.0 Release Goals
Team, With all of the excellent work that many have contributed to NiFi over the years, the code base has also accumulated some amount of technical debt. A handful of components have been marked as deprecated, and some components remain in the code base to support integration with old versions of various products. Following the principles of semantic versioning, introducing a major release would provide the opportunity to remove these deprecated and unsupported components. Rather than focusing the next major release on new features, what do you think about focusing on technical debt removal? This approach would not make for the most interesting release, but it provides the opportunity to clean up elements that involve breaking changes. Focusing on technical debt, at least three primary goals come to mind for the next major release: 1. Removal of deprecated and unmaintained components 2. Require Java 11 as the minimum supported version 3. Transition internal date and time handling to JSR 310 java.time components *Removing Deprecated Components* Removing support for older and deprecated components provides a great opportunity to improve the overall security posture when it comes to maintaining dependencies. The OWASP dependency plugin report currently generates 50 MB of HTML for questionable dependencies, many of which are related to old versions of various libraries. As a starting point, here are a handful of components and extension modules that could be targeted for removal in a major version: - PostHTTP and GetHTTP - ListenLumberjack and the entire nifi-lumberjack-bundle - ListenBeats and the entire nifi-beats-bundle - Elasticsearch 5 components - Hive 1 and 2 components *Requiring Java 11* Java 8 is now over seven years old, and NiFi has supported general compatibility with Java 11 for several years. NiFi 1.14.0 incorporated internal improvements specifically related to TLS 1.3, which allowed closing out the long-running Java 11 compatibility epic NIFI-5174. Making Java 11 the minimum required version provides the opportunity to address any lingering edge cases and put NiFi in a better position to support current Java versions. *JSR 310 for Date and Time Handling* Without making the scope too broad, transitioning internal date and time handling to use DateTimeFormatter instead of SimpleDateFormat would provide a number of advantages. The Java Time components provide much better clarity when it comes to handling localized date and time representations, and also avoid the inherent confusion of java.sql.Date extending java.util.Date. Many internal components, specifically Record-oriented processors and services, rely on date parsing, leading to confusion and various workarounds. The pattern formats of SimpleDateFormat and DateTimeFormatter are very similar, but there are a few subtle differences. Making this transition would provide a much better foundation going forward. *Conclusion* Thanks for giving this proposal some consideration. Many of you have been developing NiFi for years and I look forward to your feedback. I would be glad to put together a more formalized recommendation on Confluence and write up Jira epics if this general approach sounds agreeable to the community. Regards, David Handermann
Re: [DISCUSS] NiFi 2.0 Release Goals
Thanks to everyone who has provided feedback thus far, it is good to see general interest in moving forward with a release focusing on technical debt removal. It seems like it would be helpful to start gathering removal candidates on a Confluence wiki page, and then turning those into Jira issues. I am open to suggestions here, but it seems like creating several epics related to high level goals would be a good way to organize the work. I can start with creating a Confluence wiki page to capture some of these initial details. A release version 2.0.0 already exists in Jira, so that could be used for tagging planned issues. >From a branching and maintenance perspective, perhaps PMC members could describe how this should work? Should the main branch become the branch planned for 2.0.0 work, and does that require a vote for approval to proceed? Or should a new 2.0 branch be created and new commits applied selectively? Regards, David Handermann On Fri, Jul 23, 2021 at 11:29 AM Matt Burgess wrote: > Along with the itemized list for ancient components we should look at > updating versions of drivers, SDKs, etc. for external systems such as > Elasticsearch, Cassandra, etc. There may be breaking changes but 2.0 > is probably the right time to get things up to date to make them more > useful to more people. > > On Fri, Jul 23, 2021 at 12:21 PM Nathan Gough wrote: > > > > I'm a +1 for removing pretty much all of this stuff. There are security > > implications to keeping old dependencies around, so the more old code we > > can remove the better. I agree that eventually we need to move to > > supporting only Java 11+, and as our next release will probably be about > 4 > > - 6 months from now that doesn't seem too soon. We could potentially > break > > this in two and remove the deprecated processors and leave 1.x on Java 8, > > and finally start on 2.x which would support only Java 11. I'm unsure of > > what implications changing the date and time handling would have - for > > running systems that use long term historical logs, unexpected impacts to > > time logging could be a problem. > > > > As Joe says I think feature work will have to be dedicated to 2.x and we > > could support 1.x for security fixes for some period of time. 2.x seems > > like a gargantuan task but it's probably time to get started. Not sure > how > > we handle all open PRs and the transition between 1.x and 2.x. > > > > On Fri, Jul 23, 2021 at 10:57 AM Joe Witt wrote: > > > > > Jon > > > > > > You're right we have to be careful and you're right there are still > > > significant Java 8 users out there. But we also have to be careful > > > about security and sustainability of the codebase. If we had talked > > > about this last year when that article came out I'd have agreed it is > > > too early. Interestingly that link seems to get updated and I tried > > > [1] and found more recent data (not sure how recent). Anyway it > > > suggests Java 8 is still the top dog but we see good growth on 11. In > > > my $dayjob this aligns to what I'm seeing too. Customers didn't seem > > > to care about Java 11 until later half last year and now suddenly it > > > is all over the place. > > > > > > I think once we put out a NiFi 2.0 release we'd see rapid decrease in > > > work on the 1.x line just being blunt. We did this many years ago > > > with 0.x to 1.x and we stood behind 0.x for a while (maybe a year or > > > so) but it was purely bug fix/security related bits. We would need to > > > do something similar. But feature work would almost certainly go to > > > the 2.x line. Maybe there are other workable models but my instinct > > > suggests this is likely to follow a similar path. > > > > > > ...anyway I agree it isn't that easy of a call to dump Java 8. We > > > need to make the call in both the interests of the user base and the > > > contributor base of the community. > > > > > > [1] https://www.jetbrains.com/lp/devecosystem-2021/java/ > > > > > > > > > Thanks > > > Joe > > > > > > On Fri, Jul 23, 2021 at 7:46 AM Joe Witt wrote: > > > > > > > > Russ > > > > > > > > Yeah the flow registry is a key part of it. But also now you can > > > > download the flow definition in JSON (upload i think is there now > > > > too). Templates offered a series of challenges such as we store them > > > > in the flow definition which has made flows massive in an unintended &
Re: [DISCUSS] NiFi 2.0 Release Goals
Team, I created the following page on the Apache NiFi wiki to track proposed goals and particular focus areas. If the page should go under another section of the wiki, it can be moved. https://cwiki.apache.org/confluence/display/NIFI/NiFi+2.0+Proposed+Release+Goals The Primary Goals section is intended to cover high-level categories, and each subsection includes some initial elements that seem to fit in that category. This initial version is not intended to be exhaustive or definitive, but should keep this conversation moving forward. I can make updates based on feedback in this thread, and others with access are welcome to make direct changes. As a list of proposed goals, the purpose is not to exclude additional ideas, but to promote a focused discussion toward the next major release. I would be glad to help write up Jira issues when we get to that point. For now, more feedback is better in terms of what should be covered under the general heading of technical debt reduction. Regards, David Handermann On Fri, Jul 23, 2021 at 12:11 PM Russell Bateman wrote: > Bringing up Elastic also reminds me that the Elastic framework has just > recently transitioned out of Open Source, so to acknowledge that, maybe > some effort toward OpenSearch--I say this not understanding exactly how > this sort of thing is considered in a large-scale, world-class software > project like Apache NiFi. (I'm not a contributor, just a grateful > consumer.) > > Russ > > On 7/23/21 10:28 AM, Matt Burgess wrote: > > Along with the itemized list for ancient components we should look at > > updating versions of drivers, SDKs, etc. for external systems such as > > Elasticsearch, Cassandra, etc. There may be breaking changes but 2.0 > > is probably the right time to get things up to date to make them more > > useful to more people. > > > > On Fri, Jul 23, 2021 at 12:21 PM Nathan Gough > wrote: > >> I'm a +1 for removing pretty much all of this stuff. There are security > >> implications to keeping old dependencies around, so the more old code we > >> can remove the better. I agree that eventually we need to move to > >> supporting only Java 11+, and as our next release will probably be > about 4 > >> - 6 months from now that doesn't seem too soon. We could potentially > break > >> this in two and remove the deprecated processors and leave 1.x on Java > 8, > >> and finally start on 2.x which would support only Java 11. I'm unsure of > >> what implications changing the date and time handling would have - for > >> running systems that use long term historical logs, unexpected impacts > to > >> time logging could be a problem. > >> > >> As Joe says I think feature work will have to be dedicated to 2.x and we > >> could support 1.x for security fixes for some period of time. 2.x seems > >> like a gargantuan task but it's probably time to get started. Not sure > how > >> we handle all open PRs and the transition between 1.x and 2.x. > >> > >> On Fri, Jul 23, 2021 at 10:57 AM Joe Witt wrote: > >> > >>> Jon > >>> > >>> You're right we have to be careful and you're right there are still > >>> significant Java 8 users out there. But we also have to be careful > >>> about security and sustainability of the codebase. If we had talked > >>> about this last year when that article came out I'd have agreed it is > >>> too early. Interestingly that link seems to get updated and I tried > >>> [1] and found more recent data (not sure how recent). Anyway it > >>> suggests Java 8 is still the top dog but we see good growth on 11. In > >>> my $dayjob this aligns to what I'm seeing too. Customers didn't seem > >>> to care about Java 11 until later half last year and now suddenly it > >>> is all over the place. > >>> > >>> I think once we put out a NiFi 2.0 release we'd see rapid decrease in > >>> work on the 1.x line just being blunt. We did this many years ago > >>> with 0.x to 1.x and we stood behind 0.x for a while (maybe a year or > >>> so) but it was purely bug fix/security related bits. We would need to > >>> do something similar. But feature work would almost certainly go to > >>> the 2.x line. Maybe there are other workable models but my instinct > >>> suggests this is likely to follow a similar path. > >>> > >>> ...anyway I agree it isn't that easy of a call to dump Java 8. We > >>> need to make the call in both the interests of the user base and the &
Re: [DISCUSS] NiFi 2.0 Release Goals
Thanks for pointing out the standard NAR bundles, Mark. There are a number of components in the standard NAR bundles with particular dependencies that would make more sense in separate NARs. Reorganizing the standard NAR to components with limited dependencies and wide applicability would definitely help with future maintenance. Regards, David Handermann On Sat, Jul 24, 2021 at 10:57 AM Mark Payne wrote: > There’s also some code that exists in order to maintain backward > compatibility in the repositories. I would very much like the repositories > to contain no unnecessary code. And swap file format supports really old > formats. And the old impls of the repositories themselves, like > PersistentProvRepo instead of WriteAheadProv Repo, etc. Lots of stuff there > that can be removed. And some methods in ProcessSession that are never used > by any processor in the codebase but exists in the public API so can’t be > removed till 2.0. > > I think his is also a great time to clean up the “standard nar.” At this > point, it’s something like 70 MB. And many of the components there are not > really “standard” - things like connecting to FTP & SFTP servers, XML > processing, Jolt transform, etc. could potentially be moved into other > nars. The nifi-standard-content-viewer-1.15.0-SNAPSHOT.war is 6.9 MB is not > necessary for stateless or minifi java. Lots of things probably to > reconsider within the standard nar. > > I definitely think this is a reasonable approach, to allow for a 2.0 that > is not a huge feature release but allows the project to be simpler and more > nimble. > > Thanks > -Mark > > On Jul 24, 2021, at 10:59 AM, Mike Thomsen mikerthom...@gmail.com>> wrote: > > Russell, > > AFAICT from looking at Elastic's repos, the low level REST client is > still fine. > https://github.com/elastic/elasticsearch/blob/e5518e07f13701e3bb3dcc6842b9023966752497/client/rest/src/main/java/org/elasticsearch/client/RestClient.java > > Our Elasticsearch support is spread over two NARs at present. One uses > OkHttp and the other uses that low level Elastic REST client. > Therefore, I think we're fine on licensing for the moment. > > Mike > > On Fri, Jul 23, 2021 at 1:10 PM Russell Bateman <mailto:r...@windofkeltia.com>> wrote: > > Bringing up Elastic also reminds me that the Elastic framework has just > recently transitioned out of Open Source, so to acknowledge that, maybe > some effort toward OpenSearch--I say this not understanding exactly how > this sort of thing is considered in a large-scale, world-class software > project like Apache NiFi. (I'm not a contributor, just a grateful > consumer.) > > Russ > > On 7/23/21 10:28 AM, Matt Burgess wrote: > Along with the itemized list for ancient components we should look at > updating versions of drivers, SDKs, etc. for external systems such as > Elasticsearch, Cassandra, etc. There may be breaking changes but 2.0 > is probably the right time to get things up to date to make them more > useful to more people. > > On Fri, Jul 23, 2021 at 12:21 PM Nathan Gough thena...@gmail.com>> wrote: > I'm a +1 for removing pretty much all of this stuff. There are security > implications to keeping old dependencies around, so the more old code we > can remove the better. I agree that eventually we need to move to > supporting only Java 11+, and as our next release will probably be about 4 > - 6 months from now that doesn't seem too soon. We could potentially break > this in two and remove the deprecated processors and leave 1.x on Java 8, > and finally start on 2.x which would support only Java 11. I'm unsure of > what implications changing the date and time handling would have - for > running systems that use long term historical logs, unexpected impacts to > time logging could be a problem. > > As Joe says I think feature work will have to be dedicated to 2.x and we > could support 1.x for security fixes for some period of time. 2.x seems > like a gargantuan task but it's probably time to get started. Not sure how > we handle all open PRs and the transition between 1.x and 2.x. > > On Fri, Jul 23, 2021 at 10:57 AM Joe Witt joe.w...@gmail.com>> wrote: > > Jon > > You're right we have to be careful and you're right there are still > significant Java 8 users out there. But we also have to be careful > about security and sustainability of the codebase. If we had talked > about this last year when that article came out I'd have agreed it is > too early. Interestingly that link seems to get updated and I tried > [1] and found more recent data (not sure how recent). Anyway it > suggests Java 8 is still the top dog but we see good growth on 11. In &g
Re: [DISCUSS] NiFi 2.0 Release Goals
Chris, Thanks for the reply and recommendations. It seems like some of the work to reorganize the module structure could be done outside of a major release, but it would be great to target any breaking changes for 2.0. Perhaps a separate feature proposal on module restructuring, with the goal of supporting optimized builds, would be a helpful way to move that part of the discussion forward. Regarding updating AWS SDK to version 2, it seems like that might be possible now. I haven't taken a close look at the referencing components, so I'm not sure about the level of effort involved. Minor NiFi version updates have incorporated major new versions of dependencies. For example, NiFi 1.14 included an upgrade from Spring Framework 4 to 5. On the one hand, including the AWS SDK update as part of a major release seems helpful, but unless there are changes that break existing component properties, upgrading the AWS SDK could be worked independently. Others may have more insight into particular usage of that library. Regards, David Handermann On Sun, Jul 25, 2021 at 2:12 AM Chris Sampson wrote: > Might be worth considering refactoring the build as part of this work too, > e.g. only building the bits of the repo affected by a commit, etc. - > discussed briefly in previous threads but don't think any changes made yet. > If NARs/components are likely to be split up and refactored then such work > around the build probably makes sense to consider. > > I've a couple of PRs open that include updates to Elasticsearch versions > already, although I stopped at 7.10.2 (after which Elastic changed licence) > in case there were licence concerns. But more work can be done to tidy up > the processors, absolutely. > > AWS libraries to v2 would seem a sensible move and a refactor of those > processors as well. > > > Cheers, > > Chris Sampson > > On Sat, 24 Jul 2021, 17:47 David Handermann, > wrote: > > > Thanks for pointing out the standard NAR bundles, Mark. There are a > number > > of components in the standard NAR bundles with particular dependencies > that > > would make more sense in separate NARs. Reorganizing the standard NAR to > > components with limited dependencies and wide applicability would > > definitely help with future maintenance. > > > > Regards, > > David Handermann > > > > On Sat, Jul 24, 2021 at 10:57 AM Mark Payne > wrote: > > > > > There’s also some code that exists in order to maintain backward > > > compatibility in the repositories. I would very much like the > > repositories > > > to contain no unnecessary code. And swap file format supports really > old > > > formats. And the old impls of the repositories themselves, like > > > PersistentProvRepo instead of WriteAheadProv Repo, etc. Lots of stuff > > there > > > that can be removed. And some methods in ProcessSession that are never > > used > > > by any processor in the codebase but exists in the public API so can’t > be > > > removed till 2.0. > > > > > > I think his is also a great time to clean up the “standard nar.” At > this > > > point, it’s something like 70 MB. And many of the components there are > > not > > > really “standard” - things like connecting to FTP & SFTP servers, XML > > > processing, Jolt transform, etc. could potentially be moved into other > > > nars. The nifi-standard-content-viewer-1.15.0-SNAPSHOT.war is 6.9 MB is > > not > > > necessary for stateless or minifi java. Lots of things probably to > > > reconsider within the standard nar. > > > > > > I definitely think this is a reasonable approach, to allow for a 2.0 > that > > > is not a huge feature release but allows the project to be simpler and > > more > > > nimble. > > > > > > Thanks > > > -Mark > > > > > > On Jul 24, 2021, at 10:59 AM, Mike Thomsen > > > mikerthom...@gmail.com>> wrote: > > > > > > Russell, > > > > > > AFAICT from looking at Elastic's repos, the low level REST client is > > > still fine. > > > > > > https://github.com/elastic/elasticsearch/blob/e5518e07f13701e3bb3dcc6842b9023966752497/client/rest/src/main/java/org/elasticsearch/client/RestClient.java > > > > > > Our Elasticsearch support is spread over two NARs at present. One uses > > > OkHttp and the other uses that low level Elastic REST client. > > > Therefore, I think we're fine on licensing for the moment. > > > > > > Mike > > > > > > On Fri, Jul 23, 2021 at 1:10 PM Russell Bateman > > &l
Re: [DISCUSS] NiFi 2.0 Release Goals
Mark, Thanks for the feedback. It may be better to start a separate thread on PostHTTP, but can you provide an example flow demonstrating the performance differences between PostHTTP and InvokeHTTP? PostHTTP uses the Apache HttpComponents library, whereas InvokeHTTP uses OkHttp. NiFi 1.13.2 and 1.14.0 included major version updates for OkHttp, so it would be important to test with the most recent version. It is also worth noting that test classes for PostHTTP are now integration tests as opposed to unit tests, which means they are not executed as part of the automated builds. The NiFi documentation includes the contents of the DeprecationNotice for PostHTTP: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.14.0/org.apache.nifi.processors.standard.PostHTTP/index.html Regards, David Handermann On Tue, Jul 27, 2021 at 9:56 AM Mark Bean wrote: > I'm strongly in favor of reducing tech debt, and moving deliberately to a > 2.0 release. I have a concern with just one processor that is currently > marked as deprecated: PostHTTP. (I have not evaluated specifically any > other deprecated components; I cannot say if there are or are not similar > issues elsewhere.) I understand the rationale for deprecating this > processor in that it eliminates a processor whose functionality is > available elsewhere, specifically in the more flexible InvokeHTTP. However, > in my experience and testing, PostHTTP performs transfers with far greater > throughput than InvokeHTTP. I would not be in favor of removing PostHTTP > unless/until InvokeHTTP is refactored to increase its throughput > capability. > > Has anyone else continued to use PostHTTP over InvokeHTTP for similar > reasons? Or, is there a performance-related configuration for InvokeHTTP I > may have missed? > > Also, in order to help facilitate a smooth transition to 2.0 from a user > perspective, would it be advisable to add some sort of visual indicator in > the UI for components that are currently annotated as @Deprecated? This > might help users proactively modify their flow prior to a release that > would otherwise break it. > > > > > On Mon, Jul 26, 2021 at 5:02 PM Bryan Bende wrote: > > > With the merging of MiNiFi and registry into the NiFi repo, we've > > actually gone more towards a mono-repo where everything is released > > together. Eventually it would still be nice to have a smaller base > > distribution containing just the framework and standard NARs, but it > > is hard to tackle that until we provide a way for users to easily > > obtain the NARs in some other way. > > > > On Mon, Jul 26, 2021 at 4:20 PM Edward Armes > > wrote: > > > > > > Given the major version number shift and the spliting up of processors > > into > > > multiple NAR's. I'd like to suggest that we start individually > versioning > > > individual NARs/ bundles. > > > > > > I can see this bringing a large number of benifits including making > Nifi > > > more deployable with things RPM, but also potentially allowing those > that > > > have to deploy managed Nifi instances easier to upgrade. > > > > > > Edward > > > > > > On Mon, 26 Jul 2021, 20:42 Otto Fowler, > wrote: > > > > > > > The issue with updating the aws sdk is if it breaks any one of the > > > > processors. > > > > the Web Gateway API invoke processor for example is not using a high > > level > > > > purpose build client and may break. > > > > > > > > If we change the aws version, we need to coordinate in such a way > that > > they > > > > all > > > > can come along reasonably. > > > > IE: what happens if 1 or 2 break but the rest or OK? > > > > > > > > > > > > > > > > From: David Handermann > > > > > > > > Reply: dev@nifi.apache.org < > dev@nifi.apache.org> > > > > Date: July 26, 2021 at 09:33:42 > > > > To: dev@nifi.apache.org > > > > Subject: Re: [DISCUSS] NiFi 2.0 Release Goals > > > > > > > > Chris, > > > > > > > > Thanks for the reply and recommendations. It seems like some of the > > work to > > > > reorganize the module structure could be done outside of a major > > release, > > > > but it would be great to target any breaking changes for 2.0. > Perhaps a > > > > separate feature proposal on module restructuring, with the goal of > > > > supporting optimized builds, would be a helpful way to move that part > > of > > >
Re: [DISCUSS] NiFi 2.0 Release Goals
Thanks Mark, providing a template or comparison statistics with Java versions and component configuration details would be very helpful. If it is possible to run tests using a public API or deployable service, that would also help confirm potential differences. Showing a deprecation notice in the UI could be helpful, perhaps as a configurable option. NIFI-8650 describes a general Flow Analysis capability, and it seems like that might be one possible way to surface deprecation warnings. For something more specific to component deprecation, it seems important to find a balance between making it obvious and making it something that ends up getting ignored. Regards, David Handermann On Tue, Jul 27, 2021 at 10:28 AM Mark Bean wrote: > I'll start a new thread for PostHTTP when I get a template and/or detailed > stats. > > I know the deprecation is noted in the documentation. That's a necessary > and minimum level of notification. I was suggesting it be more obvious in > the UI. I think it would be beneficial to somehow be aware of the > deprecation status simply by looking at the flow (perhaps even on the > summary pages too), and without having to open the documentation for every > processor to confirm whether or not a component is marked as deprecated. > > Thanks, > Mark > > > On Tue, Jul 27, 2021 at 11:16 AM David Handermann < > exceptionfact...@apache.org> wrote: > > > Mark, > > > > Thanks for the feedback. It may be better to start a separate thread on > > PostHTTP, but can you provide an example flow demonstrating the > performance > > differences between PostHTTP and InvokeHTTP? > > > > PostHTTP uses the Apache HttpComponents library, whereas InvokeHTTP uses > > OkHttp. NiFi 1.13.2 and 1.14.0 included major version updates for OkHttp, > > so it would be important to test with the most recent version. It is also > > worth noting that test classes for PostHTTP are now integration tests as > > opposed to unit tests, which means they are not executed as part of the > > automated builds. > > > > The NiFi documentation includes the contents of the DeprecationNotice for > > PostHTTP: > > > > > > > https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.14.0/org.apache.nifi.processors.standard.PostHTTP/index.html > > > > Regards, > > David Handermann > > > > On Tue, Jul 27, 2021 at 9:56 AM Mark Bean wrote: > > > > > I'm strongly in favor of reducing tech debt, and moving deliberately > to a > > > 2.0 release. I have a concern with just one processor that is currently > > > marked as deprecated: PostHTTP. (I have not evaluated specifically any > > > other deprecated components; I cannot say if there are or are not > similar > > > issues elsewhere.) I understand the rationale for deprecating this > > > processor in that it eliminates a processor whose functionality is > > > available elsewhere, specifically in the more flexible InvokeHTTP. > > However, > > > in my experience and testing, PostHTTP performs transfers with far > > greater > > > throughput than InvokeHTTP. I would not be in favor of removing > PostHTTP > > > unless/until InvokeHTTP is refactored to increase its throughput > > > capability. > > > > > > Has anyone else continued to use PostHTTP over InvokeHTTP for similar > > > reasons? Or, is there a performance-related configuration for > InvokeHTTP > > I > > > may have missed? > > > > > > Also, in order to help facilitate a smooth transition to 2.0 from a > user > > > perspective, would it be advisable to add some sort of visual indicator > > in > > > the UI for components that are currently annotated as @Deprecated? This > > > might help users proactively modify their flow prior to a release that > > > would otherwise break it. > > > > > > > > > > > > > > > On Mon, Jul 26, 2021 at 5:02 PM Bryan Bende wrote: > > > > > > > With the merging of MiNiFi and registry into the NiFi repo, we've > > > > actually gone more towards a mono-repo where everything is released > > > > together. Eventually it would still be nice to have a smaller base > > > > distribution containing just the framework and standard NARs, but it > > > > is hard to tackle that until we provide a way for users to easily > > > > obtain the NARs in some other way. > > > > > > > > On Mon, Jul 26, 2021 at 4:20 PM Edward Armes > > > > > wrote: > > > > > > > > > > G
Re: [DISCUSS] NiFi 2.0 Release Goals
Thanks for the continued discussion around what can or should be removed. Having a clear migration path from version 1 to version 2 will be very important for supporting current deployments. Following the example of some other projects, one way to consider the upgrade is from the point of view of deprecation warnings. If implemented correctly, an installation of NiFi running the latest minor release of version 1, and showing no deprecation warnings, should be upgradeable to version 2 without any changes. Making this work involves accurately tagging components and methods with deprecation indicators, and providing a clear way to observe deprecation warnings. With the current version containing both deprecated components and deprecated methods in various classes, this would involve some work to get right. A simple approach might be a named logger for deprecation warnings, which would write a separate log file. A more advanced approach might involve a tool to analyze the system and flow configurations. Either way, it seems that some additional work in a minor release version is necessary before considering a major release version. One other point on compatibility: as long as the core component API definition remains backwards compatible, it should be possible to run older components. This provides a transition option for customers using components such as PostHTTP, or any others that are not actively maintained. At some point it may become necessary to break compatibility at that level, but at least for version 2, maintaining component API compatibility is key. Regards, David Handermann On Sun, Aug 1, 2021 at 10:23 AM Mark Bean wrote: > I created a JIRA ticket to investigate and improve the performance of > InvokeHTTP. It includes a flow definition for benchmarking the performance > of both PostHTTP and InvokeHTTP. > > https://issues.apache.org/jira/browse/NIFI-8968 > > Thanks, > Mark > > On Fri, Jul 30, 2021 at 6:41 PM Adam Taft wrote: > > > I'm not seeing the side thread that was going to discuss deprecation of > > PostHTTP. Has that thread started and I just don't see it? > > > > One (significant?) concern with PostHTTP is the smooth integration of > > NiFi-to-NiFi communication that is very transparently enabled with the > > ListenHTTP and PostHTTP processors. There's some special logic in there > for > > handling flowfiles that InvokeHTTP doesn't really (nor should really) > have. > > > > I know of several (large) NiFi installations that rely on the PostHTTP / > > ListenHTTP combination. It has enabled NiFi to NiFi communication for > folks > > reluctant or unable to enable site-to-site type configuration. > > > > Honestly, I don't know why we'd want to "deprecate" any processor, as > > opposed to just moving it to a new location. If these processors can be > > ported and maintained to whatever the 2.0 API looks like, there's > possibly > > little harm keeping them around. > > > > And by the way, what happened to the "marketplace" concept? Is this being > > considered for 2.0 as well? Because relocating the deprecated processors > > to an external nar might be the best solution. Losing PostHTTP entirely I > > think would be a mistake, but I'd conceptually support relocating it. > > > > Thanks, > > > > /Adam > > > > On Tue, Jul 27, 2021 at 2:11 PM Joe Witt wrote: > > > > > Looks like we just need to knock out 5 JIRAs :) [1] > > > > > > I felt like we had a label folks were using at one point but quickly > > > looking revealed nothing exciting. After this confluence page > > > stabilizes a bit we can probably knock out some JIRAs and such. > > > > > > [1] https://issues.apache.org/jira/projects/NIFI/versions/12339599 > > > > > > On Tue, Jul 27, 2021 at 1:06 PM Otto Fowler > > > wrote: > > > > > > > > I find myself wishing I had a list of all the jiras / issues that > have > > > > been put off for a 2.0 release because they required some change or > > > another > > > > :( > > > > > > > > From: Joe Witt > > > > Reply: dev@nifi.apache.org < > dev@nifi.apache.org> > > > > Date: July 27, 2021 at 12:30:35 > > > > To: dev@nifi.apache.org > > > > Subject: Re: [DISCUSS] NiFi 2.0 Release Goals > > > > > > > > A few thoughts: > > > > > > > > 1. I would love to see deprecation notices show up in the UI in > > > > various ways to help motivate users to move off things to more > > > > supportable things. That is not a prerequi
GitHub Workflow Improvements
Team, As part of ongoing efforts to improve automated testing, the following NiFi issues introduced several updates to GitHub workflows: - https://issues.apache.org/jira/browse/NIFI-9037 - https://issues.apache.org/jira/browse/NIFI-9040 - https://issues.apache.org/jira/browse/NIFI-9057 The primary ci-workflow continues to run for all pull requests and pushes to the main branch. In order to avoid unnecessary resource usage, the workflow now includes a concurrency setting that will cancel a running job on a push to the same branch. As a result, multiple updates in quick succession to a single pull request will result in only the last push continuing to build. This also impacts multiple updates to the main branch. When this occurs, GitHub cancels previous jobs for the specified branch with the following message: Canceling since a higher priority waiting request for 'branch' exists In addition to concurrency changes, there is a new system-tests workflow that runs every day at 00:00 UTC: https://github.com/apache/nifi/actions/workflows/system-tests.yml The project README includes an additional badge indicating the system-tests workflow status. The system-tests workflow runs integration tests in the nifi-system-test-suite and nifi-stateless-system-suite modules. These tests are not part of the standard continuous integration workflow, but they provide a very helpful indication of whether NiFi and NiFi stateless are functional at a system level. Reliable automated testing is vital to the continued health of the project, so thanks to everyone who contributes to the testing process! Regards, David Handermann
Re: Flood of JIRAs and presumably PRs to follow for junit-5 migration?
Mike, Thanks for moving things forward on the JUnit 5 migration. I posted a couple comments on the PR for nifi-commons (NIFI-9080) with concerns related to breaking tests and perpetuating ignored unit tests. Summarizing in this thread for general discussion, I am concerned about breaking unit tests as part of the migration process. As long as NiFi supports both JUnit 4 and JUnit 5, the migration should improve the overall testing environment. Breaking existing tests will require additional work to go back and fix, and spending the effort upgrading and reviewing test classes doesn't seem worth it if we continue marking tests as ignored. In some particular cases, it may be worthwhile to remove a test method or test class. For test methods related to performance, migrating to JUnit 5 seems like an ideal opportunity to make tests conditional on environment variables or system properties. I would be glad to help with the migration process, but it would be helpful to avoid having to revisit code multiple times to address these issues. Regards, David Handermann On Wed, Aug 25, 2021 at 2:24 PM Mike Thomsen wrote: > I broke up the tickets because it is A LOT of individual tasks that > can break the overall build and wanted to scope the work appropriately > so that people looking for a chance to contribute could snatch up a > few easy wins. > > I'm still going to take point on making the changes, but the plan > going forward is to submit PRs that bundle a bunch of tickets so that > we don't overwhelm reviewers and the CI/CD pipeline. > > Most of the changes are automated by IntelliJ Ultimate's migration > tool, which from my testing is really good at migrating about 95% of > our JUnit 4 unit and integration tests. > > Thanks, > > Mike > > On Wed, Aug 25, 2021 at 1:05 PM Joe Witt wrote: > > > > Joey > > > > I personally dont care all that much about the number of commits in PR > > - I think that rule is sort of soft already. > > > > I dont think there is any inherent value in having a single module per > > JIRA (and PR or PR commit) on this. These can be done in much coarser > > grained chunks. It will have to be to get review cycles for instance > > (much less having the Github infra to run these builds). > > > > Thanks > > Joe > > > > On Wed, Aug 25, 2021 at 9:44 AM Joey Frazee > > wrote: > > > > > > Maybe this is an exception to the single squashed commit guidance for > the initial pull? > > > > > > I assume the intent is to make incremental progress and not have a PR > with a hundred files affected, but if the different module changes > corresponded to a different commit, GH will make it easy enough to have a > draft and review each commit in isolation. > > > > > > Would that be a reasonable approach? > > > > > > -joey > > > > > > > On Aug 25, 2021, at 9:36 AM, Joe Witt wrote: > > > > > > > > Mike, > > > > > > > > Seeing a pretty stunning flood of JIRAs for 'Refactor nifi-bla to use > > > > JUnit 5' and I'm guessing we'll see the same in terms of PRs. This > is > > > > a really high administrative overhead approach to this. > > > > > > > > Why not break this into one or maybe a few JIRAs/PRs total? > > > > > > > > Thanks >
Re: Preferred approach to the JUnit 5 migration?
Mike, Thanks for raising this question. I have a couple thoughts, and I would be interested in perspectives from others. Having worked on a number of performance and stability issues related to testing, there are varying degrees of test quality throughout the project. The collective result is long-running builds, particularly when it comes to GitHub workflows. Some of these issues require focused effort to correct, which makes it difficult to keep the scope of JUnit 5 migration focused. With that being said, limiting the scope to rearranging imports and expectations does not seem worth the effort. However, using the migration as an opportunity to perform basic cleanup would be very valuable. With that in mind, migration work seems to fit into three categories: 1. Simple replacement of imports: no structural changes needed 2. Refactoring deprecated approaches, such as replacing exception expectations with assertThrows, or replacing platform-specific assumptions with JUnit 5 annotations 3. Removing ignored tests or leveraging JUnit 5 annotations for System property-based enabling For modules that fit into the first category, a larger PR seems like the best way to go, since it should not require more than a cursory review. For modules in the second and third categories, smaller sets of modules are easier to review. With the module-based subtasks that you have already created, using a commit per subtask in a larger PR is a helpful approach. Given that some modules will require more significant refactoring, handling those changes in separate PRs will make it easier to keep the review process moving. Here is what I would like to see in the migration: 1. Elimination of ignored test methods: either complete method removal or changes to use EnabledIfSystemProperty. Using common property names like "nifi.test.performance" would be helpful 2. Elimination of commented-out or bypassed test methods 3. Replacement of platform-specific assumeTrue() references with DisabledOnOs annotations 4. Refactoring of other environment-specific assumptions using utility methods that could be leveraged using EnabledIf or DisabledIf annotations There are other items that would be helpful, but this seems like an opportunity to pay down some technical debt. With these goals in mind, the module scope for pull requests will vary on a case-by-case basis, but that seems like the best use of everyone's time. Thanks for the consideration. Regards, David Handermann On Fri, Aug 27, 2021 at 11:12 AM Mike Thomsen wrote: > Some responders seem to prefer a batched migration while others have > suggested a massive PR that does all of the low-hanging fruit in one > push. I can do either, but would like to know what folks who can do > the reviews would prefer before going much deeper. > > Either way, most of the migration can be automated cleanly with > IntellJ Ultimate's migration tool. > > Thanks, > > Mike >
Re: Pull request for ValidateJSON revisit
Tim, Thanks for highlighting the updated PR, I provided some feedback and would be glad to help move it forward. Regards, David Handermann On Fri, Sep 3, 2021 at 5:18 AM Smith, Tim wrote: > The author submitted a new pull request for ValidateJSON: > https://github.com/apache/nifi/pull/5326 > > Should be ready for a re-review. > > > From: Mark Payne > Sent: Friday, August 20, 2021 9:34:17 AM > To: dev@nifi.apache.org > Subject: Re: Pull request for ValidateJSON revisit > > Tim, > > It looks like I’d done a review but then there were updates and I missed > the fact that the PR had been updated. > > My main concern was with the licensing. It looks like I thought it was MIT > but in fact it was ASL v2 (either I looked at the wrong dependency or the > license was changed). > Looking through its LICENSE and NOTICE file, it doesn’t appear that there > is anything needed in the license. > > At this point, it looks like the PR has been closed, and I cannot re-open > it (Says “The repository that submitted this pull request has been > deleted.”). Based on a quick re-review, I think the PR is okay otherwise. > If you want to open another PR I should be able to quickly review & merge. > > Thanks > -Mark > > > [1] > > > On Aug 20, 2021, at 7:23 AM, Smith, Tim wrote: > > > > The pull request for a ValidateJSON processor, NIFI-7392: > > > > https://github.com/apache/nifi/pull/4232 for > > > > has been marked as stale. From the review comments, this request was > near approval. There was an outstanding question on licensing that still > may exist. I have a similar need for this capability. Could this pull > request be revisited? I would rather not duplicate effort. > > > > > > Tim > >
Re: [ANNOUNCE] New NiFi PMC Member David Handermann
Thank you all for the congratulations! After using NiFi for several years, it is great to be able to contribute back. I look forward to continued involvement on a great project with a great group of collaborators! Regards, David Handermann On Fri, Sep 17, 2021 at 4:25 PM Joe Witt wrote: > So many good things already. Thanks David > > On Fri, Sep 17, 2021 at 12:44 PM Kevin Doran wrote: > > > Congratulations, David! > > > > > On Sep 17, 2021, at 10:09, Joe Gresock wrote: > > > > > > Congrats, David! > > > > > > On Fri, Sep 17, 2021 at 9:57 AM Marton Szasz > wrote: > > > > > >> Congratulations David! > > >> > > >> On Fri, 17 Sept 2021 at 13:07, Chris Sampson > > >> wrote: > > >>> > > >>> Congrats David, very well deserved! > > >>> > > >>> > > >>> On Fri, 17 Sept 2021 at 13:56, Bryan Bende wrote: > > >>> > > >>>> NiFi Community, > > >>>> > > >>>> On behalf of the Apache NiFI PMC, I am very pleased to announce that > > >>>> David Handermann has accepted the PMC's invitation to join the > Apache > > >>>> NiFi PMC. > > >>>> > > >>>> David has made significant improvements in a number of security > > >>>> related areas, he continues to improve the stability of our tests > and > > >>>> CI builds, and regularly helps out the community through the mailing > > >>>> lists, Slack, and PR reviews. > > >>>> > > >>>> Please join us in congratulating and welcoming David to the Apache > > NiFi > > >>>> PMC. > > >>>> > > >>>> Congratulations David! > > >>>> > > >> > > > > >
Re: Single User Authentication Question
Hi Michael, Thanks for raising the question. The Single User Authentication is intended for individual use by definition. NiFi supports integration with a number of services that provide authentication, which avoids some of the complexities surrounding localized management of multiple users. Is there a reason that one of the current integration options is not suitable for your use case? https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#multi-tenant-authorization There is an open NiFi Jira issue from several years ago that sounds similar to what you described: https://issues.apache.org/jira/browse/NIFI-1614 Apparently the associated pull request had unresolved feedback and was never merged. If you are interested in this feature, adding a comment about the desired use case would be helpful. This is not a guarantee that the feature will be implemented. The existing multi-tenant integration options cover a wide variety of deployment patterns, but feel free to highlight potential gaps. Regards, David Handermann On Wed, Sep 29, 2021 at 11:57 AM Michael Radov (RIT Alumni) wrote: > Hey, > > Hope all is well. Is there a way to have a future version of nifi include a > way to create multiple single user authentication accounts? IE-->instead of > only 1 single user auth account as that there is now, it would allow the > creation of up to 4-5 accounts and associated management so an admin can > add 4-5 individual accounts. > > How would I get this recommendation added to something that could be > developed into a future NiFi? > > Regards, > Michael Radov >
Re: [VOTE] Release Apache NiFi 1.15.0 (rc1)
-1 (binding) Reproduced the failing system tests described in NIFI-9352. Verified many other features are working as designed, but I agree that this is a blocker for the current release candidate version. Regards, David Handermann On Sat, Oct 30, 2021 at 9:26 AM Mark Payne wrote: > -1 (binding) > > I tried to run the full system test suite and ran into NIFI-9352 [1]. I > think this is a critical enough issue to vote -1 on this release. Good news > is that it's a one-line fix. > > Thanks > -Mark > > [1] https://issues.apache.org/jira/browse/NIFI-9352 > > On 2021/10/29 17:20:44, Joe Witt wrote: > > Hello, > > > > I am pleased to be calling this vote for the source release of Apache > > NiFi 1.15.0. > > > > The source zip, including signatures, digests, etc. can be found at: > > https://repository.apache.org/content/repositories/orgapachenifi-1184 > > > > The source being voted upon and the convenience binaries can be found at: > > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.15.0/ > > > > A helpful reminder on how the release candidate verification process > works: > > > https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate > > > > The Git tag is nifi-1.15.0-RC1 > > The Git commit ID is 73278e2aa0f8673792d069dbf3407faf981adc7c > > > https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=73278e2aa0f8673792d069dbf3407faf981adc7c > > > > Checksums of nifi-1.15.0-source-release.zip: > > SHA256: d7c138219569ba89765d8a16f8c727e7a6ab118744e3689006245412ece59e08 > > SHA512: > ae1a4bd5d86c276535be942bfe6dd4e93ec8ea9d80d5e4abe6ab5c192cf2da6d7a629b55a8aa1dba484a01ecdd64875ff6fc891d9305a2a63db91c794fa08489 > > > > Release artifacts are signed with the following key: > > https://people.apache.org/keys/committer/joewitt.asc > > > > KEYS file available here: > > https://dist.apache.org/repos/dist/release/nifi/KEYS > > > > 227 issues were closed/resolved for this release: > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12350382 > > > > Release note highlights can be found here: > > > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.15.0 > > > > The vote will be open for 72 hours. > > Please download the release candidate and evaluate the necessary items > > including checking hashes, signatures, build from source, and test. > > Then please vote: > > > > [ ] +1 Release this package as nifi-1.15.0 > > [ ] +0 no opinion > > [ ] -1 Do not release this package because... > > >
Re: [discuss] NiFi 1.15
Chris,
I am running through several verification steps, but I built 1.15.0 RC 3
from sources and then ran "mvn install -pl :dockermaven -P docker" to build
the Docker image. When starting with the following command, I was able to
use the generated username and password to access the running NiFi
container:
docker run --name nifi -p 8443:8443 apache/nifi:1.15.0-dockermaven
If you are still having issues, it would be helpful to provide any logs or
error messages you are seeing.
Regards,
David Handermann
On Thu, Nov 4, 2021 at 8:10 PM Kevin Doran wrote:
> Oh meant to send a link to this too:
>
> > ARG
> NIFI_BINARY_PATH=${NIFI_BINARY_PATH:-/nifi/${NIFI_VERSION}/nifi-${NIFI_VERSION}-bin.zip}
>
>
> https://github.com/apache/nifi/blob/373498445fe589e2d4855a0730fbb9127f0b4452/nifi-docker/dockerhub/Dockerfile#L30
>
> > On Nov 4, 2021, at 9:09 PM, Kevin Doran wrote:
> >
> > Hi Joe and Chris,
> >
> > Our Dockerfile that we use to build the Dockerhub image defaults to
> looking for 1.15.0 instead of NiFi-1.15.0, but it is a variable that we can
> override, so this is easy to workaround incase the release folder does
> change. Agree its nice to keep the tree structure consistent when we can.
> >
> > I’m about to do my verification and will also check the single user with
> the docker image as part of that.
> >
> > Cheers,
> > Kevin
> >
> >> On Nov 4, 2021, at 6:44 PM, Joe Witt wrote:
> >>
> >> Chris,
> >>
> >> Yeah I should have just put it in 1.15.0 instead of nifi/1.15.0.
> >> Should generally not really matter so the docker angle there is
> >> interesting. Not sure why our docker images would have any
> >> relationship to our dist/dev storage. But when I move them into the
> >> release folder I can try to ensure I place them only in 1.15.0 instead
> >> of nifi/1.15.0.
> >>
> >> That directory the prov stuff makes does linger and can be annoying so
> >> thanks for tackling that. Saw the PR. Will take a look soon if
> >> nobody else has.
> >>
> >> Will keep an eye on your findings related to docker builds not working
> >> with username/password things. Hopefully others can chime in there.
> >>
> >> Thanks
> >> Send
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> On Thu, Nov 4, 2021 at 2:06 PM Chris Sampson
> >> wrote:
> >>>
> >>> Worryingly, when I do get the Docker Image to build (further changes
> to the
> >>> Dockerfile), the auto-generated username and password from the startup
> logs
> >>> aren't being accepted for login via my browser.
> >>>
> >>> I'll try to spend a little more time looking at this (but await input
> on my
> >>> earlier question/concern also).
> >>>
> >>>
> >>> ---
> >>> *Chris Sampson*
> >>> IT Consultant
> >>> chris.samp...@naimuri.com
> >>>
> >>>
> >>> On Thu, 4 Nov 2021 at 20:47, Chris Sampson
> >>> wrote:
> >>>
> >>>> I've got most of the way through the release check process in order to
> >>>> vote for 1.15.0, but I wanted to check on a change in the distribution
> >>>> release artifacts.
> >>>>
> >>>> For 1.14.0, the Dev artifacts were located at:
> >>>> https://dist.apache.org/repos/dist/dev/nifi/1.14.0/*
> >>>> For 1.15.0, the Dev artifacts are located at:
> >>>> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.15.0/*
> >>>>
> >>>> i.e. there's been a change of directory/path from to
> >>>> nifi-.
> >>>>
> >>>> The reason I raise this is that I can no longer build a Docker Image
> using
> >>>> the dockerhub/DockerBuild.sh script because it cannot find the
> artifacts to
> >>>> download. This may not be a problem if the path change is only for
> the Dev
> >>>> artifacts, but if the same change is going to happen for the released
> >>>> artifacts, then the apache/nifi image (and presumably the
> >>>> apache/nifi-registry, apache/nifi-toolkit and any minifi) convenience
> >>>> images will no longer be possible as part of the Release, which will
> likely
> >>>> be an issue for a number of users that have deployments using these
> images.
> >>>>
> >>>> I spotted this after rebasing m
Re: [VOTE] Release Apache NiFi 1.15.0 (rc3)
+1 (binding) - Built from source on Ubuntu 21.10 and macOS 11.6 with Azul Zulu 11.0.12 and Azul Zulu 1.8.0.282 - Ran NiFi on Azul Zulu 11.0.12 and 1.8.0.282 - Verified Single User Authentication and OIDC Authentication with Keycloak - Verified updates to JWT client storage and cookie handling - Verified basic operation of NiFi Regstriy - NIFI-6617 Verified encrypted repository configuration using new properties with BCFKS and PKCS12 KeyStores - NIFI-7322 Verified SignContentPGP and VerifyContentPGP together with EncryptContentPGP and DecryptContentPGP using various properties - NIFI-8424 Verified absence of HtmlDocumentWriter warnings on startup - NIFI-8943 Built, launched, and restarted Docker container verifying persistence of random sensitive properties key - NIFI-9059 Ran simple data flow using NiFi Stateless on Java 11 - NIFI-9223 Verified ListenSyslog listens on 0.0.0.0 using default properties - NIFI-9251 Verified absence of unused generated password in NiFi Registry - NIFI-9266 Verified Azure Key Vault Secret Sensitive Property Provider configuration for nifi.properties - NIFI-9322 Verified corrected REST API documentation for OIDC and SAML resources - NIFI-9339 Verified JoltTransformJSON custom UI works as expected Thanks for managing the release process Joe! Regards, David Handermann On Thu, Nov 4, 2021 at 6:33 PM Matt Burgess wrote: > +1 Release this package as nifi-1.15.0 > > Ran through the release helper and some tests to verify various Jiras > were included. Thanks again for RM'ing Joe! > > On Wed, Nov 3, 2021 at 3:29 PM Joe Witt wrote: > > > > Hello, > > > > I am pleased to be calling this vote for the source release of Apache > > NiFi 1.15.0. > > > > The source zip, including signatures, digests, etc. can be found at: > > https://repository.apache.org/content/repositories/orgapachenifi-1186 > > > > The source being voted upon and the convenience binaries can be found at: > > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.15.0/ > > > > A helpful reminder on how the release candidate verification process > works: > > > https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate > > > > The Git tag is nifi-1.15.0-RC3 > > The Git commit ID is 7fdc07cccdc0e23d4986557a9314e36859704a3b > > > https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=7fdc07cccdc0e23d4986557a9314e36859704a3b > > > > Checksums of nifi-1.15.0-source-release.zip: > > SHA256: 56fe317248941fdbc6216ec973e6ce898d0f682a54e2d063edbabf8542f5288a > > SHA512: > 63f10d9127cf1613c29bf2306be3d7a5b931b31304920706e0df5ea2fe87b58c410efed6ac37afc38d12c65e69f14aec7afb926000fc90cc13f15c738cd15c7f > > > > Release artifacts are signed with the following key: > > https://people.apache.org/keys/committer/joewitt.asc > > > > KEYS file available here: > > https://dist.apache.org/repos/dist/release/nifi/KEYS > > > > 234 issues were closed/resolved for this release: > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12350382 > > > > Release note highlights can be found here: > > > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.15.0 > > > > The vote will be open for 72 hours. > > Please download the release candidate and evaluate the necessary items > > including checking hashes, signatures, build from source, and test. > > Then please vote: > > > > [ ] +1 Release this package as nifi-1.15.0 > > [ ] +0 no opinion > > [ ] -1 Do not release this package because... >
Re: [discuss] NiFi 1.15
Thanks for the update Chris! I agree with the suggestion to streamline the Dockerfiles for NiFi builds, there is a lot of duplication between dockermaven and dockerhub, so unifying the modules and parameterizing necessary differences would be a helpful improvement. Regards, David Handermann On Fri, Nov 5, 2021 at 9:10 AM Kevin Doran wrote: > Hi Chris, > > Thanks for the summary of your findings. > > I agree that we should clear up our process for generating Docker images > and that the process should be consistent as possible across NiFi, MiNiFi > Java, Registry, and Toolkit given they are all part of the same repository > and maven project. This will clear up confusion and improve > maintainability. > > I think both of these methods are important and useful: > > 1. building from downloaded convenience binaries for previously released > versions > 2. building from a local assembly output > > That said, I think we could probably consolidate to a single Dockerfile > that takes the binary location as an argument that supports either a URL or > local file path (or a version which could default to the current behavior > of inferring a URL location). This would let us continue to support the > flexibility we have today while maintaining a single Docker file rather > than dockermaven and dockerhub variants. If you and others on the list > agree, I can open up a Jira summarizing what needs to be done. > > Thanks, > Kevin > > > On Nov 5, 2021, at 05:00, Chris Sampson > wrote: > > > > So the good news is that I realised what I was doing wrong yesterday - > I'd > > started a local installation after building the RC, then not shut that > down > > before booting up the Docker instance, so the username/password I was > > trying to use from the Docker logs was wrong against the local > > installation, d'oh! > > > > Having corrected that this morning, I've successfully booted up and > logged > > into a Docker instance built using the RC (with my NIFI-8779 changes so I > > could build from the Dev server instead of the main Apache Archive). > > > > The reason the dockermaven build works is that it uses the locally built > > archive files (i.e. you have to do a full Maven build locally first to > > generate the ZIP files and then create teh Docker image). The > > dockerhub build actually downloads published artifacts from the Apache > > servers - to do this the Dockerfile needs to know the exact path to the > > artifacts it's trying to download, of course. > > > > There was a question recently in one of the Slack channels about whether > > both of these builds (dockerhub and dockermaven) were still valid/needed > - > > I think Joe replied that he wasn't sure (Docker not being an area he > > ventures into much, which is fair enough). It may be worth considering > > whether these two modules are both still needed or can be rationalised > and > > if the latter, which approach should be used - download from the archive > > server or build from local (both have dis/advantages, but the more > > "official" way is arguably to download from the server). > > > > Also maybe worth noting: > > * NiFi Registry only has the "build from local" Dockerfile setup > > * MiniFi (Java) has both local and archive download options, but the > latter > > cannot be used to build an image from the Dev server (the BASE_URL cannot > > be changed via a build arg/env var... this is the issue addressed by > > NIFI-8779 for NiFi) > > * NiFi Toolkit has both local and archive download options, but are > located > > in a single assembly module (instead of split into two like NiFi and > MiniFi) > > > > Assuming the "nifi/" path will be used for the actual release > > artifacts, this shouldn't be a blocker, but it's one to note/remember > when > > the time comes (assuming the "dockerhub" module is what's used to publish > > the "apache/nifi" image to Docker Hub that is). > > > > > > --- > > *Chris Sampson* > > IT Consultant > > chris.samp...@naimuri.com > > > > > > On Fri, 5 Nov 2021 at 03:34, David Handermann < > exceptionfact...@apache.org> > > wrote: > > > >> Chris, > >> > >> I am running through several verification steps, but I built 1.15.0 RC 3 > >> from sources and then ran "mvn install -pl :dockermaven -P docker" to > build > >> the Docker image. When starting with the following command, I was able > to > >> use the generated usernam
Re: [VOTE] Release Apache NiFi NAR Maven Plugin 1.3.3
+1 (binding) - Built nifi-nar-maven-plugin 1.3.3 on Azul Zulu 11.0.10 - Built NiFi using nifi-nar-maven-plugin 1.3.3 - Verified contents of selected extension-manifest.xml files Thanks for managing the release Bryan! Regards, David Handermann On Thu, Dec 2, 2021 at 8:10 AM Pierre Villard wrote: > +1 (binding) > > Le mer. 1 déc. 2021 à 20:30, Otto Fowler a > écrit : > > > +1 (non-binding) > > ran through the testing guide > > > > From: Bryan Bende > > Reply: dev@nifi.apache.org > > Date: November 30, 2021 at 15:56:38 > > To: dev@nifi.apache.org > > Subject: [VOTE] Release Apache NiFi NAR Maven Plugin 1.3.3 > > > > Hello, > > > > I am pleased to be calling this vote for the source release of Apache > > NiFi NAR Maven Plugin 1.3.3. > > > > The source zip, including signatures, digests, etc. can be found at: > > https://repository.apache.org/content/repositories/orgapachenifi-1188 > > > > The source being voted upon and the convenience binaries can be found at: > > https://dist.apache.org/repos/dist/dev/nifi/nifi-nar-maven-plugin-1.3.3/ > > > > A helpful reminder on how the release candidate verification process > works: > > > > > https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate > > > > The Git tag is nifi-nar-maven-plugin-1.3.3-RC1 > > The Git commit ID is 99f1ff7184e00cecc4763b7bcbdf0d12dfeffef2 > > > > > https://gitbox.apache.org/repos/asf?p=nifi-maven.git;a=commit;h=99f1ff7184e00cecc4763b7bcbdf0d12dfeffef2 > > > > Checksums of nifi-nar-maven-plugin-1.3.3-source-release.zip: > > SHA256: 36d6579dfdbc4e1450a13457196ff399dbd344624348a62ee5e247cd5962dc77 > > SHA512: > > > > > 0e90100d90e9dd142283aab729957e12cc6abbfdf2f86447f488bcbe5890d55564c19a0efc473307744f813bcdac2636b027ad4e5beebf1ce1ed6ee44deaccbe > > > > > > Release artifacts are signed with the following key: > > https://people.apache.org/keys/committer/bbende.asc > > > > KEYS file available here: > > https://dist.apache.org/repos/dist/release/nifi/KEYS > > > > 1 issue was closed/resolved for this release: > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12348815 > > > > Release note highlights can be found here: > > > > > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-NiFiNARMavenPluginVersion1.3.3 > > > > The vote will be open for 72 hours. Please download the release > > candidate and evaluate the necessary items including checking hashes, > > signatures, build from source, and test. Then please vote: > > > > [ ] +1 Release this package as nifi-nar-maven-plugin-1.3.3 > > [ ] +0 no opinion > > [ ] -1 Do not release this package because... > > >
Re: Nifi 1.14 user authentication using openId connect not working
Hi Ganesh.B, It looks like you are running into an issue in NiFi 1.14.0 that was resolved in 1.15.0 under the following Jira issue: https://issues.apache.org/jira/browse/NIFI-8783 The default authorizers.xml includes a definition for the SingleUserAuthorizer, which can only be used together with the SingleUserLoginIdentityProvider. The workaround for NiFi 1.14.0 is to comment or remove the following section from authorizers.xml: single-user-authorizer org.apache.nifi.authorization.single.user.SingleUserAuthorizer Removing that configuration element from authorizers.xml should allow the OIDC configuration to load as expected in NiFi 1.14.0. Regards, David Handermann On Wed, Dec 8, 2021 at 4:54 AM Ganesh, B (Nokia - IN/Bangalore) < b.gan...@nokia.com> wrote: > Hi , > > > > We are using apache nifi 1.14 . We have 3 nodes in nifi cluster , cluster > is using external zookeeper for state management. > > We are using openId connect for the user authentication . following are > the relevant configuration in nifi.properties file . > > *nifi.security.user.authorizer=managed-authorizer* > > *nifi.security.allow.anonymous.authentication=false* > > *nifi.security.user.login.identity.provider=* > > *.* > > *………..* > > *# OpenId Connect SSO Properties #* > > *nifi.security.user.oidc.discovery.url=https:// SERVER>/access/realms/nifi/.well-known/openid-configuration* > > *nifi.security.user.oidc.connect.timeout=5 secs* > > *nifi.security.user.oidc.read.timeout=5 secs* > > *nifi.security.user.oidc.client.id > <http://nifi.security.user.oidc.client.id>=nifi-client* > > *nifi.security.user.oidc.client.secret=* > > *nifi.security.user.oidc.preferred.jwsalgorithm=RS256* > > > > *But we are observing * > > *org.springframework.beans.factory.UnsatisfiedDependencyException: Error > creating bean with name > 'org.springframework.security.config.annotation.web.configuration.WebSecurityConfiguration': > Unsatisfied dependency expressed through method > 'setFilterChainProxySecurityConfigurer' parameter 1; nested exception is > org.springframework.beans.factory.BeanExpressionException: Expression > parsing failed; nested exception is > org.springframework.beans.factory.UnsatisfiedDependencyException: Error > creating bean with name > 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied > dependency expressed through method 'setJwtAuthenticationProvider' > parameter 0; nested exception is > org.springframework.beans.factory.BeanCreationException: Error creating > bean with name 'jwtAuthenticationProvider' defined in class path resource > [nifi-web-security-context.xml]: Cannot resolve reference to bean > 'authorizer' while setting constructor argument; nested exception is > org.springframework.beans.factory.BeanCreationException: Error creating > bean with name 'authorizer': FactoryBean threw exception on object > creation; nested exception is java.lang.reflect.InvocationTargetException* > > *at > org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredMethodElement.resolveMethodArguments(AutowiredAnnotationBeanPostProcessor.java:768)* > > *at > org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPostProcessor.java:720)* > > *at > org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:119)* > > *at > org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:399)* > > *at > org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1413)* > > *at > org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:601)* > > *at > org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:524)* > > *at > org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)* > > *at > org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)* > > *at > org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)* > > *at > org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)* > > *at > org.springframework.beans.factory.su
Re: discuss: do a nifi 1.15.1 release to eliminate log4shell concern
Joe, Thanks for starting this discussion. Moving forward with a 1.15.1 patch release sounds like the best path forward. Regards, David Handermann On Mon, Dec 13, 2021 at 7:49 AM Joe Witt wrote: > Team > > We still dont think we are vulnerable but this now highly risky library is > present. We have PRs to eliminate it/main is fixed. I think we should do > a 24 hour 1.15.1 release/vote for it. It will eliminate concerns for > users. We are frankly pretty close to a 1.16 release at this point as > well it seems but can circle back. > > > Any different views on 1.15.1? > > Thanks >
Re: discuss: do a nifi 1.15.1 release to eliminate log4shell concern
PR 5598 for NIFI-9474 is now merged into the main branch, which streamlines version updates to Log4j 2 dependencies. It also excludes log4j-core older than 2.15.0 from build artifacts, so this should provide a good basis for a patch release. https://github.com/apache/nifi/pull/5598 Regards, David Handermann On Mon, Dec 13, 2021 at 10:44 AM Chris Sampson wrote: > I'd agree. The discussions in Slack and separate user mailing list thread > are a reassurance for users (who read them), but a patch for the current > 1.15 branch would seem sensible for people to pick up and assuage any > remaining security concerns they may have around the library. > > That leaves 1.16 a little longer to get more good stuff merged in for the > next feature release. > > > Cheers, > > Chris Sampson > > On Mon, 13 Dec 2021, 14:19 David Handermann, > wrote: > > > Joe, > > > > Thanks for starting this discussion. Moving forward with a 1.15.1 patch > > release sounds like the best path forward. > > > > Regards, > > David Handermann > > > > On Mon, Dec 13, 2021 at 7:49 AM Joe Witt wrote: > > > > > Team > > > > > > We still dont think we are vulnerable but this now highly risky library > > is > > > present. We have PRs to eliminate it/main is fixed. I think we > should > > do > > > a 24 hour 1.15.1 release/vote for it. It will eliminate concerns for > > > users. We are frankly pretty close to a 1.16 release at this point as > > > well it seems but can circle back. > > > > > > > > > Any different views on 1.15.1? > > > > > > Thanks > > > > > >
Re: End of Life for mail-1.4.7.jar file
Hi Sanjeet, NiFi 1.14.0 included changes for the PutEmail processor to use the new Jakarta Mail 2 library as described in NIFI-8630, but I am not aware of a similar Jira issue to change the email notification service that currently references mail-1.4.7.jar. This is easier to do for the bootstrap notification service, but there are a couple other email components that also need to be updated. Addressing each of these upgrades in separate Jira issues is the best way to go. Feel free to write up a Jira issue if you are interested, otherwise this is something that will be considered as time permits. https://issues.apache.org/jira/browse/NIFI-8630 Regards, David Handermann On Tue, Dec 14, 2021 at 7:52 AM sanjeet rath wrote: > Hi > > The latest 1.15 Nifi version contain mail-1.4.7.jar file(inside > \lib\bootstrap folder). > We r using 1.12.1 version of nifi same issue is also there. > > Actually this mail-1.4.7.jar has reached end of life (latest verion was > released on 2013). > > Could someone let me know r we planing to update this jar in future > version. > > Thanks, > Sanjeet >
Re: [ANNOUNCE] New Apache NiFi Committer Margot Tien
Congratulations Margot! On Wed, Dec 15, 2021 at 2:50 PM Nathan Gough wrote: > Congrats Margot, thanks for all your contributions! > > On Wed, Dec 15, 2021 at 3:02 PM Chris Sampson > wrote: > > > Congrat Margot! > > > > --- > > *Chris Sampson* > > IT Consultant > > chris.samp...@naimuri.com > > > > > > On Wed, 15 Dec 2021 at 19:04, Pierre Villard < > pierre.villard...@gmail.com> > > wrote: > > > > > Congrats Margot! > > > > > > Le mer. 15 déc. 2021 à 20:00, Kevin Doran a écrit > : > > > > > > > Congratulations Margot! Well deserved. > > > > > > > > > On Dec 15, 2021, at 13:47, Joe Witt wrote: > > > > > > > > > > Congrats Margot! And thanks > > > > > > > > > > On Wed, Dec 15, 2021 at 11:46 AM Matt Gilman > > > > wrote: > > > > > > > > > >> Apache NiFi community, > > > > >> > > > > >> On behalf of the Apache NiFi PMC, I am very pleased to announce > that > > > > Margot > > > > >> has accepted the PMC's invitation to become a committer on the > > Apache > > > > NiFi > > > > >> project. We greatly appreciate all of Margot's hard work and > > generous > > > > >> contributions to the project. We look forward to continued > > involvement > > > > in > > > > >> the project. > > > > >> > > > > >> Margot has been contributing to NiFi and NiFi Registry for years. > > Her > > > > >> contributions have covered both back-end and front-end > improvements > > in > > > > both > > > > >> projects in addition to release verification and thoughtful PR > > > reviews. > > > > >> > > > > >> Welcome and congratulations! > > > > >> > > > > > > > > > > > > > >
Re: Issue with setting nifi.sensitive.props.key
Phil, The section of the post describing setting the sensitive properties algorithm includes an example toolkit command that can be used to change the sensitive properties algorithm: https://exceptionfactory.com/posts/2021/07/29/deciphering-apache-nifi-component-property-encryption/#setting-the-sensitive-properties-algorithm When upgrading from a previous version of NiFi, you need to start with the previous default value for the algorithm specified in nifi.properties: nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL With that value, you should be able to run the set-sensitive-properties-key command. If you want to change the algorithm to the new default of NIFI_PBKDF2_AES_GCM_256, then you can use the encrypt-config.sh toolkit command described. Regards, David Handermann On Tue, Dec 21, 2021 at 4:43 PM Joe Witt wrote: > Phil > > Not sure if this helps but DavidH wrote this > > https://exceptionfactory.com/posts/2021/07/29/deciphering-apache-nifi-component-property-encryption/#mandatory-sensitive-properties-key > > Thanks > > On Tue, Dec 21, 2021 at 3:38 PM Phil H wrote: > > > > Hi there, > > > > I am in the process of trying to upgrade from 13.2 to 15.1. I did not > have > > a sensitive props key set previously. Based on the upgrade guide, I ran > > > > nifi.sh set-sensitive-properties-key APassword > > > > When I ran nifi, it was complaining about a lack of specified algorithm. > I > > ran up a new installation of 15.1 on another machine which automatically > > specified an algorithm of NIFI_PBKDF2_AES_GCM_256. I copied this value to > > my existing install’s nifi.properties. > > > > When I run nifi now, it halts with a javax.crypto.AEADBadTagException: > mac > > check in GCM failed > > > > If I try the same set-sensitive-properties-key command again, it now > fails > > with the same ‘GCM failed’ exception. If I remove the algorithm line from > > the nifi.properties file, this command works, but then starting nifi > gives > > me an “NullPointerException: Algorithm required” > > > > Not sure what I am missing here! > > > > Help! > > > > Thanks, > > Phil >
Re: [VOTE] Release Apache NiFi 1.15.2
+1 (binding) - Verified hashes and signatures - Built from source on Ubuntu 21.10 Azul Zulu 1.8.0.312 - Ran NiFi on Azul Zulu 1.8.0.312 - Verified Single User Authentication and credentials configuration - NIFI-9483 Verified the absence of Apache Log4j 2 log4j-core in binary distribution - NIFI-9491 Verified the absence of Apache commons-logging in binary distribution - NIFI-9497 Verified Bouncy Castle libraries upgraded to 1.70 - NIFI-9505 Verified Apache Log4j 2 log4-api upgraded to 2.17.0 - NIFI-9507 Verified that SFTP processors stop Keep Alive Threads on failures and run when expected - NIFI-9509 Verified successful SFTP retrieval and renaming using AWS Transfer Family SFTP Server as well as Linux OpenSSH Server Thanks for the quick turnaround on this release Joe! Regards, David Handermann On Tue, Dec 21, 2021 at 8:19 PM Mark Payne wrote: > +1 (binding) > > Verified hashes > Verified signature > Performed full build & verified all unit tests on OS X, OpenJDK 1.8.0 u265 > Successfully ran all 128 System Tests > > Started single standalone instance and verified a few different flows > > Started a two-node secured cluster, did basic verification of permissions, > dataflow running, etc. > > Verified that the convenience binary does not contain the log4j jar in the > lib/ directory or packaged in any NAR. > Verified that the convenience binary does not contain the JndiLookup class > (shaded or otherwise) > > Thanks for putting together another RC Joe! > > Thanks > -Mark > > > On Dec 21, 2021, at 5:51 PM, Joe Witt wrote: > > > > Hello, > > > > I am pleased to be calling this vote for the source release of Apache > > NiFi 1.15.2. > > > > This vote is purely bug fix and security focused. This is a > > continuation of our efforts to promptly and thoroughly respond to > > log4shell and related concerns. > > > > The source zip, including signatures, digests, etc. can be found at: > > https://repository.apache.org/content/repositories/orgapachenifi-1193 > > > > The source being voted upon and the convenience binaries can be found at: > > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.15.2/ > > > > A helpful reminder on how the release candidate verification process > works: > > > https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate > > > > The Git tag is nifi-1.15.2-RC1 > > The Git commit ID is 1ea460b8556b07057366abb74a5552ace8946e87 > > > https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=1ea460b8556b07057366abb74a5552ace8946e87 > > > > Checksums of nifi-1.15.2-source-release.zip: > > SHA256: 29fcc35c81de80e0fe3f59044e6fbf21bcf523e656aa64914e7546e1d7705e6b > > SHA512: > cabd1f1ad4942a61df0400488d35521202598c217ad8da97dc2d5abe21136604d1f1bb3de9ceb63bb441943de2e29e3515f5cf63607080094e1418d79eb5216b > > > > Release artifacts are signed with the following key: > > https://people.apache.org/keys/committer/joewitt.asc > > > > KEYS file available here: > > https://dist.apache.org/repos/dist/release/nifi/KEYS > > > > 8 issues were closed/resolved for this release: > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12351132 > > > > Release note highlights can be found here: > > > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.15.2 > > > > Given the nature of the vote and its limited scope > > the vote will be open for 24 hours or until we have sufficient > > votes (instead of the normal 72 hours). > > > > Please download the release candidate and evaluate the necessary items > > including checking hashes, signatures, build from source, and test. > > Then please vote: > > > > [ ] +1 Release this package as nifi-1.15.2 > > [ ] +0 no opinion > > [ ] -1 Do not release this package because... > >
Re: NoRouteToHostException is coming in NIfi UI login pageafter setting proxy in jvm arguments
Hi Sanjeet,
Thank you for providing the stack trace and details of your configuration.
Setting Java System properties in custom code is not a safe or supported
operation in NiFi. As you have observed, setting system properties alters
the behavior of other components, leading to unexpected results.
If you need to support access through a proxy server, the
ProxyConfigurationService interface and StandardProxyConfigurationService
implementation provide a way to specify proxy server properties:
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-proxy-configuration-nar/1.15.2/org.apache.nifi.proxy.StandardProxyConfigurationService/index.html
Regards,
David Handermann
On Thu, Jan 13, 2022 at 12:39 PM sanjeet rath
wrote:
> Hi,
>
> I encounter one "java.net.NoRouteToHostException: No route to host (Host
> unreachable)" in *Nifi UI login page *.
> The after debugging i realised when i am setting the proxy & port in
> *System.setProperty("proxy address") & System.setProperty("port address")
> *in
> my custom processor. then this issue is appearing .
>
> The other way i also replicated when i am setting at the* jvm lable(in
> bootstrap.conf file -Dhttp.proxyHost=address) *for nifi application this
> exception is coming in nifi ui login page:, After removal of this argument
> it works fine.
>
> Could someone help me to understand what could be the issue.
>
> Nifi version: 1.12.1
> cluster : 3 node (amazon EC2 linux cluster)
>
> *Detail exception from nifi-app.log:*
>
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request GET
> /nifi-api/flow/current-user to "exampledumyserveraddress" due to
> java.net.NoRouteToHostException: No route to host (Host unreachable)
>
> 2022-01-13 12:57:23,722 WARN [Replicate Request Thread-5]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator
>
> java.net.NoRouteToHostException: No route to host (Host unreachable)
>
> at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
>
> at
> java.base/java.net
> .AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
>
> at
> java.base/java.net
> .AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
>
> at
> java.base/java.net
> .AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
>
> at
> java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
>
> at java.base/java.net.Socket.connect(Socket.java:609)
>
> at
> okhttp3.internal.platform.Platform.connectSocket(Platform.java:130)
>
> at
>
> okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:263)
>
> at
>
> okhttp3.internal.connection.RealConnection.connectTunnel(RealConnection.java:235)
>
> at
> okhttp3.internal.connection.RealConnection.connect(RealConnection.java:177)
>
> at
>
> okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.java:224)
>
> at
>
> okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.java:108)
>
> at
> okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.java:88)
>
> at
> okhttp3.internal.connection.Transmitter.newExchange(Transmitter.java:169)
>
> at
>
> okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:41)
>
> at
>
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
>
> at
>
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
>
> at
> okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94)
>
> at
>
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
>
> at
>
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
>
> at
>
> okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
>
> at
>
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
>
> at
>
> okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88)
>
> at
>
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
>
> at
>
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
>
> at
> okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:229)
>
> at okhttp3.RealCall.execute(RealCall.java:81)
>
> at
>
> org.apache.nifi
Re: NoRouteToHostException is coming in NIfi UI login pageafter setting proxy in jvm arguments
Hi Sanjeet,
That's correct. Setting the JVM arguments for proxy server access can
create the problems you observed, and it is unlikely to work as expected
when it comes to proxy access. Various NiFi components use different
methods for creating network connections. For this reason, enabling
outgoing proxy server access will be more reliable using the
ProxyConfigurationService. If there specific components that have issues
with proxy server connectivity, it would be worth creating a Jira issue for
those components. If proxy server access is limited to your custom
component, then the best approach is to integrate the
ProxyConfigurationService and determine how that should be wired to your
custom component library.
Regards,
David Handermann
On Thu, Jan 13, 2022 at 1:02 PM sanjeet rath wrote:
> Thanks a lot for the quick response.
>
> So u r suggesting we should not use proxy in jvm argument(nifi
> bootstrap.conf file) lable also as it will impact other component.like in
> my case its impacting nifi-api.
>
> Regards,
> Sanjeet
>
> On Fri, 14 Jan 2022, 12:19 am David Handermann, <
> exceptionfact...@apache.org>
> wrote:
>
> > Hi Sanjeet,
> >
> > Thank you for providing the stack trace and details of your
> configuration.
> > Setting Java System properties in custom code is not a safe or supported
> > operation in NiFi. As you have observed, setting system properties
> alters
> > the behavior of other components, leading to unexpected results.
> >
> > If you need to support access through a proxy server, the
> > ProxyConfigurationService interface and StandardProxyConfigurationService
> > implementation provide a way to specify proxy server properties:
> >
> >
> >
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-proxy-configuration-nar/1.15.2/org.apache.nifi.proxy.StandardProxyConfigurationService/index.html
> >
> > Regards,
> > David Handermann
> >
> > On Thu, Jan 13, 2022 at 12:39 PM sanjeet rath
> > wrote:
> >
> > > Hi,
> > >
> > > I encounter one "java.net.NoRouteToHostException: No route to host
> > (Host
> > > unreachable)" in *Nifi UI login page *.
> > > The after debugging i realised when i am setting the proxy & port in
> > > *System.setProperty("proxy address") & System.setProperty("port
> address")
> > > *in
> > > my custom processor. then this issue is appearing .
> > >
> > > The other way i also replicated when i am setting at the* jvm
> lable(in
> > > bootstrap.conf file -Dhttp.proxyHost=address) *for nifi application
> this
> > > exception is coming in nifi ui login page:, After removal of this
> > argument
> > > it works fine.
> > >
> > > Could someone help me to understand what could be the issue.
> > >
> > > Nifi version: 1.12.1
> > > cluster : 3 node (amazon EC2 linux cluster)
> > >
> > > *Detail exception from nifi-app.log:*
> > >
> > > o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request
> GET
> > > /nifi-api/flow/current-user to "exampledumyserveraddress" due to
> > > java.net.NoRouteToHostException: No route to host (Host unreachable)
> > >
> > > 2022-01-13 12:57:23,722 WARN [Replicate Request Thread-5]
> > > o.a.n.c.c.h.r.ThreadPoolRequestReplicator
> > >
> > > java.net.NoRouteToHostException: No route to host (Host unreachable)
> > >
> > > at java.base/java.net.PlainSocketImpl.socketConnect(Native
> > Method)
> > >
> > > at
> > > java.base/java.net
> > > .AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
> > >
> > > at
> > > java.base/java.net
> > >
> >
> .AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
> > >
> > > at
> > > java.base/java.net
> > > .AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
> > >
> > > at
> > > java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
> > >
> > > at java.base/java.net.Socket.connect(Socket.java:609)
> > >
> > > at
> > > okhttp3.internal.platform.Platform.connectSocket(Platform.java:130)
> > >
> > > at
> > >
> > >
> >
> okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:263)
> > >
> > > at
> > >
> > >
> >
> okhttp3.inter
Re: [VOTE] Release Apache NiFi 1.15.3 (rc1)
+1 (binding) - Verified hashes and signatures - Built from source on Ubuntu 20.04 using Azul Zulu 1.8.0.312 - Ran NiFi on Azul Zulu 1.8.0.312 and 11.0.13 - Verified Single User Authentication and credentials configuration - NIFI-7089 Verified successful build without SFTP test failures - NIFI-7749 Verified ListSFTP operation using authenticated HTTP proxy - NIFI-7835 Verified ListSFTP and FetchSFTP operation using authentication SOCKS proxy - NIFI-9530 Verified absence of google-pubsublite library and successful invocation of ConsumeGCPubSub - NIFI-9534 Verified Log4j 2 API libraries with version 2.17.1 in nifi-elasticsearch-5-nar - NIFI-9566 Verified Logback version 1.2.10 included in NiFi library directory - Started NiFi Registry on Java 11 and created sample bucket for testing - Ran NiFi Encrypt Config Toolkit and verified successful NiFi properties encryption Thanks Joe! Regards, David Handermann On Thu, Jan 13, 2022 at 2:44 PM Joe Witt wrote: > Hello, > > I am pleased to be calling this vote for the source release of Apache > NiFi 1.15.3. > > The source zip, including signatures, digests, etc. can be found at: > https://repository.apache.org/content/repositories/orgapachenifi-1194 > > The source being voted upon and the convenience binaries can be found at: > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.15.3/ > > A helpful reminder on how the release candidate verification process works: > > https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate > > The Git tag is nifi-1.15.3-RC1 > The Git commit ID is 753c311382005acadc16c64952d7b1eaaf2550d5 > > https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=753c311382005acadc16c64952d7b1eaaf2550d5 > > Checksums of nifi-1.15.3-source-release.zip: > SHA256: 29fcc35c81de80e0fe3f59044e6fbf21bcf523e656aa64914e7546e1d7705e6b > SHA512: > cabd1f1ad4942a61df0400488d35521202598c217ad8da97dc2d5abe21136604d1f1bb3de9ceb63bb441943de2e29e3515f5cf63607080094e1418d79eb5216b > > Release artifacts are signed with the following key: > https://people.apache.org/keys/committer/joewitt.asc > > KEYS file available here: > https://dist.apache.org/repos/dist/release/nifi/KEYS > > 11 issues were closed/resolved for this release: > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12351203 > > Release note highlights can be found here: > > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.15.3 > > The vote will be open > Please download the release candidate and evaluate the necessary items > including checking hashes, signatures, build from source, and test. > Then please vote: > > [ ] +1 Release this package as nifi-1.15.3 > [ ] +0 no opinion > [ ] -1 Do not release this package because... >
Re: Build failure on macOS
Mark, Thanks for highlighting this issue. I have not seen any automated build issues on GitHub, but that appears to be running macOS 11.6.2. I noticed littleshoot 1.1.2 includes a build profile for Netty 4.1. In light of the fact that this works on Linux and other versions of macOS, I wonder if it has something to do with the Netty native libraries. Do you have any additional output in the target directory related to that particular test class? Unfortunately it looks like LittleProxy has not been updated in several years, but it is useful in that test class for verifying connectivity through a proxy server. Regards, David Handermann On Mon, Jan 17, 2022 at 10:07 AM Mark Bean wrote: > Is anyone else having trouble building NiFi (main) on macOS? I recently > upgraded to macOS Monterey 12.1 and since then I have not been able to > build NiFi. I'm not sure if the failure started exactly coincidental with > the macOS upgrade, but it stands out as a significant recent change. > > I get an error on TestHttpClient. > > [ERROR] Error occurred in starting fork, check output in log > [ERROR] Process Exit Code: 134 > [ERROR] Crashed tests: > [ERROR] org.apache.nifi.remote.client.http.TestHttpClient > [ERROR] at > > org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:748) > [ERROR] at > > org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:305) > [ERROR] at > > org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:265) > [ERROR] at > > org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1314) > [ERROR] at > > org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1159) > [ERROR] at > > org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:932) > [ERROR] at > > org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137) > [ERROR] at > > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:210) > [ERROR] at > > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:156) > [ERROR] at > > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:148) > [ERROR] at > > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117) > [ERROR] at > > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81) > [ERROR] at > > org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:56) > [ERROR] at > > org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128) > [ERROR] at > org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:305) > [ERROR] at > org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:192) > [ERROR] at > org.apache.maven.DefaultMaven.execute(DefaultMaven.java:105) > [ERROR] at org.apache.maven.cli.MavenCli.execute(MavenCli.java:957) > [ERROR] at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:289) > [ERROR] at org.apache.maven.cli.MavenCli.main(MavenCli.java:193) > [ERROR] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > [ERROR] at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > [ERROR] at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > [ERROR] at java.lang.reflect.Method.invoke(Method.java:498) > [ERROR] at > > org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:282) > [ERROR] at > org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:225) > [ERROR] at > > org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:406) > [ERROR] at > org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:347) > > I think I've traced this back to starting on commit > 12ba579b8f6c506288458fb6cd2191ea26da2cb3. (The commit prior to that does > not have trouble building: b0b1a57b9874cbbbd57122970db96464c7dd1279) > > More specifically, the breaking change introduced > io.netty:netty-all:4.1.72.Final. Previously, it was 4.0.44.Final in the > affected module, nifi-commons/nifi-site-to-site-client.)Replacing the > dependency (transitive in org.littleshoot:littleproxy) with 4.0.44.Final > allows a successful build. > > So, the question is whether this is macOS [12.1] specific, or if there is > something else in my environment. I am using OpenJDK 1.8.0_312, maven > 3.6.3. The build is successful on Linux (Fedora 26). > > Thanks, > Mark >
Re: Nifi toolkit encrypt single properties
Hi Isha, Thanks for contacting the developers list and describing what you are attempting to accomplish. Although there are several Java interfaces and implementation classes that could be leveraged to implement a custom approach to property encryption, that may not be the most reliable method. NiFi does not consider the Sensitive Property Provider interfaces and classes part of the public API, which means they could be subject to change in minor version updates. That being said, the input and output format for encrypted property values will remain supported across versions, so you could use the NiFi classes as a starting point. NiFi stores the serialized encrypted values in a custom format containing the initialization vector, followed by a delimiter, and then the encrypted content. It is possible to use standard AES-GCM support in other languages to read the value if you have the root key available. Recent versions of NiFi introduced support for additional nifi.properties protection schemes using external services such as HashiCorp Vault and Amazon Secrets Manager, as well as other service providers. The effective result is similar in that nifi.properties contains encrypted or placeholder values, but that might provide some different options. For example, the HashiCorp Vault Key Value store implementation uses a placeholder value for the property, instead of the encrypted string. This approach yields the same placeholder value, since HashiCorp Vault stores the actual property. That kind of strategy requires additional infrastructure, so that may not be a good fit for your environment, but it might be worth considering. The most stable integration from the perspective of an external tool is the encrypt-config.sh command and associated arguments. Regards, David Handermann On Tue, Jan 25, 2022 at 2:42 AM Isha Lamboo wrote: > Hi all, > > I hope this question is appropriate for the developers list, if not, I’ll > move it to users. > > I have an Ansible role for NiFi that includes generating the NiFi > properties files from templates and variables set per cluster/host as well > as updating keystore etc. > This works very well, except when it comes to protected properties as set > by the toolkit. The toolkit wants to operate on the actual properties > files, which causes Ansible to then see differences and want to reset. > I tried with an intermediate processing dir, but then I lose the ability > to use Ansible’s –check and –diff options to see if any changes were made. > In the end, I added the encrypted values by hand to the Ansible variables > files. > > As a next step, I’m looking at whether I can use the toolkit’s java > classes in my own wrapper to allow me to pass in a master key, protection > scheme and raw value and get out the encrypted one, so that I can easily > update my encrypted values in the Ansible inventory. > However, my Java skills are somewhat limited, so I would like to ask first > if this is even a good idea. > > Is this a sensible idea or does it conflict with NiFi or encryption design > principles I’m not aware of? > > Kind regards, > > Isha Lamboo > >
Re: [VOTE] Release Apache NiFi Flow Design System 0.3.0 RC2
+1 binding - Ran build using Node 16.13.2 - Verified hashes and signatures - Verified License and Notice files Thanks Scott! Regards, David Handermann On Wed, Jan 26, 2022 at 8:17 AM Shane Ardell wrote: > +1 (non-binding). > > Performed all verification steps listed in the helper guide and ran the > demo app. > > Thanks for RMing, Scott! > > On Wed, Jan 26, 2022 at 9:12 AM Matt Gilman > wrote: > > > +1 (binding) Release this package as nifi-fds-0.3.0 > > > > Ran through release helper. Looks good. Thanks for RMing Scott! > > > > Matt > > > > On Tue, Jan 25, 2022 at 4:41 PM Scott Aslan > wrote: > > > > > Hello, > > > > > > I am pleased to be calling this vote for the source release of Apache > > NiFi > > > Flow Design System nifi-fds-0.3.0. > > > > > > The source zip, including signatures, etc. can be found at: > > > https://dist.apache.org/repos/dist/dev/nifi/nifi-fds/nifi-fds-0.3.0 > > > > > > The Git tag is nifi-0.3.0-RC2 > > > The Git commit ID is a17b6712154f1b7bf36a04cfe728b61640843807 > > > > > > > > > https://gitbox.apache.org/repos/asf?p=nifi-fds.git;a=commit;h=a17b6712154f1b7bf36a04cfe728b61640843807 > > > > > > Checksums of nifi-fds-0.3.0-source-release.zip: > > > SHA1: 92e473c2c0da9f031c47724ade9808f7572dd6bd > > > SHA256: > 200c8f6a2a7fcdfa5a65c73dfef887c0173a9bf10d73ee7ac0160a614e9202ff > > > SHA512: > > > > > > > > > 74b93eb29a5e0bfefd25de288f387308307de63cf159f0c447565615a32b041fe5af20a326528a5dc8e99bf3adc8bbdc157cfa10090d51b30f958bb6253201b9 > > > > > > Release artifacts are signed with the following key: > > > https://people.apache.org/keys/committer/scottyaslan.asc > > > > > > KEYS file available here: > > > https://dist.apache.org/repos/dist/release/nifi/KEYS > > > > > > 17 issues were closed/resolved for this release: > > > > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12345959 > > > > > > Release note highlights can be found here: > > > > > > > > > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-NiFiFlowDesignSystem0.3.0 > > > > > > The vote will be open for 72 hours. > > > Please download the release candidate and evaluate the necessary items > > > including checking hashes, signatures, build from source, and test. > Then > > > please vote: > > > > > > [ ] +1 Release this package as nifi-fds-0.3.0 > > > [ ] +0 no opinion > > > [ ] -1 Do not release this package because... > > > > > >
Re: [discuss] pulling together a NiFi 1.16
Mark, To elaborate on Joe's reply, changing the trust store configuration alters the security profile of NiFi by allowing clients with trusted certificates to access the system. Changing the key store and trust store should always occur in conjunction with changing the authorization configuration. The Single User Authorizer includes a safety check to prevent configuration in conjunction with Login Identity Providers other than the Single User Login Identity Provider. In the case of certificate authentication, however, the Login Identity Provider does not apply, since certificate authentication happens prior to application-level access. Although it might be possible to add further safety checking in the Single User Authorizer to also check the username against a particular value, this could introduce additional coupling between authentication and authorization. As this is primarily a configuration problem, and changing the trust store has a significant impact on the security profile of the system, this particular scenario does not seem like a significant concern. Regards, David Handermann On Wed, Mar 9, 2022 at 9:55 AM Joe Witt wrote: > Mark > > The single user authorizer and default setup install is just to avoid > having wide open systems by default. So if you want to make changes to > security settings and do it right you dont' use that mode. Happy to have > improvements within that scope of intent but does not sound like anything > we'd wait for. When it lands it lands. > > Thanks > > On Wed, Mar 9, 2022 at 8:49 AM Mark Bean wrote: > > > Joe, > > > > I just discovered an issue yesterday that might need attention first. I > > haven't investigated fully yet nor created a ticket because I don't yet > > fully understand it. However, it appears as though the > > single-user-authorizer may not be behaving as intended. When I updated > > nifi.properties to swap the self-signed, auto-generated keystore and > > truststore with "real" ones, single-user became _every_ user. My > suspicion > > is that any user whose browser presents a cert that was signed by a CA in > > the truststore is allowed in - without even prompting for > > username/password. > > > > It may be considered a configuration error to allow this to happen. > Still, > > this seems like extremely dangerous behavior. > > > > -Mark > > > > > > On Wed, Mar 9, 2022 at 10:42 AM Joe Witt wrote: > > > > > Team > > > > > > We appear to be at a good point to start pulling together the release > > > candidate for 1.16. > > > > > > https://issues.apache.org/jira/projects/NIFI/versions/12350741 > > > > > > I'm basically waiting for > > https://issues.apache.org/jira/browse/NIFI-9761 > > > to land then will start pulling together the release. > > > > > > Thanks > > > > > > On Mon, Feb 14, 2022 at 11:18 AM Joe Witt wrote: > > > > > > > Eduardo > > > > > > > > Getting reviewers on the UI/rest/front-end are among the toughest as > > > > there just aren't as many of those folks. > > > > > > > > The reply from Pierre was probably most telling. It looks fine but > > > > many of us would pause to merge without knowing precisely what the > > > > implications are. What happens on a taxed system with many > > > > CSs...I''ll comment on the PR. > > > > > > > > Thanks > > > > Joe > > > > > > > > On Mon, Feb 14, 2022 at 11:13 AM Eduardo Fontes > > > > wrote: > > > > > > > > > > Hi All, > > > > > > > > > > Is it possible to include > > > > https://issues.apache.org/jira/browse/NIFI-8927 > > > > > in release 1.16? > > > > > I've been asking for a review > > https://github.com/apache/nifi/pull/5247 > > > > > since AUG/2021 and I don't understand why nobody did it. It's a > > simple > > > > and > > > > > useful UI feature. > > > > > > > > > > Peace out. > > > > > Eduardo Fontes > > > > > > > > > >
Re: nifi.apache.org website
Hi Steven, Thanks for your interest! This would also be a great opportunity to streamline the HTML generation approach to use something like Hugo or similar static-site generation system. Regards, David Handermann On Wed, Mar 9, 2022 at 12:44 PM Joe Witt wrote: > Steven > > To say this is super welcome is an understatement. We definitely need the > facelift you mention and we need to get off the old website mechanism and > onto the newer one the ASF wants projects using. This would be awesome! > > Thanks! > > > On Wed, Mar 9, 2022 at 11:40 AM Steven Matison > wrote: > > > Hey Dev Team, > > > > Is there any way I could assist with modernizing the > nifi.apache.org > > website? > > > > Its 2022 and I think a facelift is in order... I have a ton of > experience > > with web-dev and super passionate about NiFi. This would be something I > > would love to be a part of. > > > > Thanks, > > > > Steven > > >
Re: [VOTE] Release Apache NiFi 1.16.0 (rc2)
-1 (binding) In light of a problem just reported with AccessToken.isExpired() determination in the StandardOauth2TokenProvider service, this issue should be corrected: https://issues.apache.org/jira/browse/NIFI-9801 Regards, David Handermann On Wed, Mar 16, 2022 at 12:05 PM Mark Payne wrote: > +1 (binding) > > - Performed full build with contib-chck > - Veified the hashes > - Ran all system tests successfully > - Started up and loaded a large flow with thousdands of processors. > Ensured that all loaded successfully. > - Ran some smoke screen tests > - Ran several tests with disconnecting nodes, updated flow, ensuring that > node could join back and properly inherited all changes > > Thanks > -Mark > > > > On Mar 15, 2022, at 4:17 PM, Joe Witt wrote: > > > > Hello, > > > > I am pleased to be calling this vote for the source release of Apache > NiFi > > 1.16.0. > > > > The source zip, including signatures, digests, etc. can be found at: > > https://repository.apache.org/content/repositories/orgapachenifi-1197 > > > > The source being voted upon and the convenience binaries can be found at: > > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.16.0/ > > > > A helpful reminder on how the release candidate verification process > works: > > > https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate > > > > The Git tag is nifi-1.16.0-RC2 > > The Git commit ID is a680b3f093fc93197036fb43c08e75f6a3f21a18 > > > https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=a680b3f093fc93197036fb43c08e75f6a3f21a18 > > > > Checksums of nifi-1.16.0-source-release.zip: > > SHA256: 02c03615df5be60958717471e40ca97d1b47457ef7355e056b385e3384cb54f2 > > SHA512: > > > 43e9baa695b9617999d88f4d5878775d37bc293747c0cd6bcb1fc77318fca07bde5c5c536cc273937593c0194dd25426a617137861832b1b3749307eb2b9fd70 > > > > Release artifacts are signed with the following key: > > https://people.apache.org/keys/committer/joewitt.asc > > > > KEYS file available here: > > https://dist.apache.org/repos/dist/release/nifi/KEYS > > > > 387 issues were closed/resolved for this release: > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12350741 > > > > Release note highlights can be found here: > > > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.16.0 > > > > The vote will be open for 72 hours. > > Please download the release candidate and evaluate the necessary items > > including checking hashes, signatures, build from source, and test. Then > > please vote: > > > > [ ] +1 Release this package as nifi-1.16.0 > > [ ] +0 no opinion > > [ ] -1 Do not release this package because... > >
[ANNOUNCE] New Apache NiFi Committer Paul Grey
Apache NiFi community, On behalf of the Apache NiFi PMC, I am very pleased to announce that Paul Grey has accepted the PMC's invitation to become a committer on the Apache NiFi project. Paul has contributed a number of pull requests and code reviews over the past year, improving project security and test stability in a number of areas. We appreciate Paul's work and look forward to continued contributions! Welcome Paul, and congratulations!
Java 17 and GitHub Workflow Updates
Development Team, Thanks to reviews from Mike Thomsen and Paul Grey, support for building the NiFi project on Java 17 is now in the main branch. The GitHub continuous integration workflow now includes a fourth environment: Ubuntu Zulu JDK 17 EN Although some dependencies may cause runtime issues with Java 17, this new build target will help highlight and correct particular issues. For the purpose of compatibility, restricted reflective access is one of the most important differences in Java 17. Java 11 flags these issues as illegal reflective access warnings, but Java 17 prevents these operations. New issues with particular components should be flagged and addressed using the standard Jira tracking process. This is also a good time to mention ongoing work related to JUnit 5. Big thanks to Mike Thomsen for working through numerous modules and multiple rounds of feedback to update a majority of components to JUnit 5! All new test code should use JUnit 5 org.junit.jupiter components. Some work still remains to convert various modules, which can be tracked through sub-tasks under the following Jira issue: https://issues.apache.org/jira/browse/NIFI-9084 Finally, shout out to Paul Grey for his efforts to improve the reliability of the NiFi system tests on GitHub. The resource-constrained GitHub runners can expose test stability problems that are difficult to track down, but thanks to Paul's work, the stability of the system tests on GitHub has improved. Regards, David Handermann
