Re: [VOTE] Release Apache NiFi MiNiFi C++ 0.99.0 (RC3)
Hello Sigs/Hashes are good. Building source on macos I'm getting a build failure. The only option selected was to skip tests otherwise left as defaults. Am following the readme and release guide. ninja: build stopped: subcommand failed. Build was unsuccessful Traceback (most recent call last): File "/Users/joe/development/build-verify/buld/nifi-minifi-cpp-0.99.0-source/bootstrap/main.py", line 62, in main_menu(minifi_options, package_manager) File "/Users/joe/development/build-verify/buld/nifi-minifi-cpp-0.99.0-source/bootstrap/cli.py", line 87, in main_menu done = main_menu_options[main_menu_prompt["sub_menu"]](minifi_options, package_manager) File "/Users/joe/development/build-verify/buld/nifi-minifi-cpp-0.99.0-source/bootstrap/cli.py", line 60, in do_one_click_build assert do_build(minifi_options, package_manager) So then I select to not use ninja and run the one click build again. It seems to run for much longer then the same output prints again. Thanks Joe On Tue, May 7, 2024 at 9:45 AM Arpad Boda wrote: > +1 (binding) > > Built on ubuntu, verified signature and checksums. All tests passed, > started and ran a simple flow with no issues. > > Nice job, team! > > On Tue, May 7, 2024 at 11:35 AM Gábor Gyimesi > wrote: > > > Hello, > > > > I am pleased to be calling this vote for the source release of Apache > NiFi > > MiNiFi C++ 0.99.0. > > > > The source tarball, some binary builds, plus signatures and digests can > be > > found at: > > https://dist.apache.org/repos/dist/dev/nifi/nifi-minifi-cpp/0.99.0/ > > > > The Git tag is minifi-cpp-0.99.0-RC3 > > The Git commit ID is d3d205f3c8a8376bbad4c8a47758d585d57c653b > > > > > https://git-wip-us.apache.org/repos/asf?p=nifi-minifi-cpp.git;a=commit;h=d3d205f3c8a8376bbad4c8a47758d585d57c653b > > > > Checksums of nifi-minifi-cpp-0.99.0-source.tar.gz: > > SHA256: c00ff309776720b295cb52ab12c1c3d11182af4f19ed0a379d58f96b16841059 > > SHA512: > > > > > d1eda2c19da8f0e073cf460bf889c8b4a1b893de4b245dc0147d1c79545d5298c74bedf585afd7091dc9474c923fc2a7075a81a6397bedce310e5838e3afa898 > > > > Release artifacts are signed with the following key: > > https://people.apache.org/keys/committer/lordgamez.asc > > > > A helpful reminder on how the release candidate verification process > works: > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=173087303 > > > > KEYS file available here: > > https://dist.apache.org/repos/dist/release/nifi/KEYS > > > > 121 issues were closed/resolved for this release: > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12321520=12353618 > > > > Release note highlights can be found here: > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65145325#ReleaseNotesMiNiFi(C++)-Versioncpp-0.99.0 > > > > The vote will be open for 72 hours. > > Please download the release candidate and evaluate the necessary items > > including checking hashes, signatures, build > > from source, and test. Then please vote: > > > > [ ] +1 Release this package as nifi-minifi-cpp-0.99.0 > > [ ] +0 no opinion > > [ ] -1 Do not release this package because... > > >
Re: [discuss] identifying key components needed updates which require deeper review/consideration
Awesome Chris and David - both of those are done and dusted! Amazing. Matt; Thanks yes for Hive it should just be simplifying/updating the Iceberg dependency on Hive to the latest or even removing if not necessary. Iceberg itself can be updated but my understanding is the 1.5.1 release has a problem and there will soon be a 1.5.2 we should wait for (someone else had a PR for that and backed it out for now). The flyway one is hopefully easy similar to the QuestDB one. We have some other dependencies in need to attention which I've not created JIRAs for: 1. Mongo DB. We're on 4.11 and the latest is 5.1 2. SNMP4J. We're on 2.8/2.7 and the latest are 3.8 for the primary lib and agent. 3. Redis. We're on 3.10 and the latest is 5.1 4. Camel Salesforce Lib. We're on 3.14 and the latest is 4.5 5. Fastcsv. We're on 2.2.2 and the latest is 3.1.0 There are a handful of others but these are the top of mind ones in scanning the results. For those wanting to see this list on your own in the root of nifi.git run: mvn -T 1.0C versions:dependency-updates-aggregate-report -DonlyUpgradable=true Then open target/site/dependency-updates-aggregate-report.html Thanks Joe On Tue, May 7, 2024 at 2:05 PM David Handermann wrote: > Joe, > > Thanks for highlighting these areas to be updated. I was able to > address the QuestDB update with a couple minor adjustments, and > submitted a PR [1]. > > Regards, > David Handermann > > [1] https://github.com/apache/nifi/pull/8773 > > On Tue, May 7, 2024 at 2:58 PM Chris Sampson > wrote: > > > > I’ll pick up the Elasticsearch dependency upgrade. > > > > I’ll try to test it again OpenSearch as well, if for no reason other > than to prove either way whether the current NiFi components will work with > the AWS service, although I may not get to do that and instead testing will > focus on Elasticsearch 8.x instances (and the existing Integration Tests > within NiFi). > > > > > > Cheers, > > > > --- > > Chris Sampson > > IT Consultant > > chris.samp...@naimuri.com > > > > > > > On 7 May 2024, at 20:54, Joe Witt wrote: > > > > > > Yeah sorry - what I wrote for the Hive one was a bit nonsensical. I > tried > > > to clarify it better in the JIRA. > > > > > > Thanks > > > > > > On Tue, May 7, 2024 at 12:38 PM Matt Burgess > wrote: > > > > > >> For [1] Hive 3 was removed in > > >> https://issues.apache.org/jira/browse/NIFI-12981, if you mean the > Hive > > >> dependency from Apache Iceberg we probably need a release from them > that > > >> brings in Hive 4 dependencies? Or do you suggest we bring the Hive > > >> components back as long as we update the version? > > >> > > >> I'll take the Flyway one in the meantime, database integrations are my > > >> blessing and my curse :) If that goes more smoothly than I expect I > can > > >> grab the QuestDB one too, will update the Jiras as appropriate. > > >> > > >> Regards, > > >> Matt > > >> > > >> On Tue, May 7, 2024 at 1:56 PM Joe Witt wrote: > > >> > > >>> Team, > > >>> > > >>> We have made massive strides going from thousands of out of date > > >> libraries > > >>> to dozens in the 2.x line vs any prior releases. And we can/should > stay > > >> on > > >>> top of these better going forward. A few key bits that are > outstanding > > >>> with no obvious immediate work underway are as follows. If anyone is > > >>> working on these, willing to work on these, please volunteer and let > us > > >>> know. For components that we cannot find active engagement on we > should > > >>> consider removal though these aren't great examples as they're indeed > > >>> common usage components. > > >>> > > >>> [1] https://issues.apache.org/jira/browse/NIFI-13173 > > >>> Hive 4 recently came out but Hive also gives us some of the most > > >>> complicated POM/library mangling and oldest dependencies to > > >> override/etc.. > > >>> Putting serious detailed focus here is very valuable from a > maintenance > > >>> point of view. While existing usage of Hive 3 likely remains for > some > > >>> time, those users are like NiFi 1.x users anyway. In any case if > there > > >> are > > >>> motivated developers interested to see our Hive components be healthy > > >>> please do dive in on this. > > >>> > > >>> [2] https://issues.apache.org/jira/browse/NIFI-13169 > > >>> The NiFI registry is a major release behind Flyway DB. We need to > stay > > >>> current given the importance of this component. > > >>> > > >>> [3] https://issues.apache.org/jira/browse/NIFI-13168 > > >>> We are a few incrementals behind and a minor release behind > QuestDB. I > > >>> tried bumping the incremental but it broke tests. > > >>> > > >>> [4] https://issues.apache.org/jira/browse/NIFI-11686 > > >>> We are a full major release behind and several incremental/minors as > > >> well. > > >>> Apparently there is some concern related to the community changes in > > >>> Elastic vs OpenSearch/etc.. but regardless we need to stay active in > > >>> maintaining this component. We really
Re: [DISCUSS] Release Timeline for NiFi 2.0.0-M3
Team, There has been a significant amount of progress on the new UI, and it is now part of the default build in the main branch. There are a couple more dependency updates to be incorporated, and I plan initiating a release candidate build for 2.0.0-M3 in the next day or two at the latest. Regards, David Handermann On Fri, May 3, 2024 at 4:02 AM Pierre Villard wrote: > > I'm starting the process for the 1.26 RC as I don't believe we're waiting > for any additional PRs there. > > Le jeu. 2 mai 2024 à 07:30, Joe Witt a écrit : > > > Matt > > > > You are referring to NIFI-12998 and it causing rebasing to be necessary for > > outstanding PRs? > > > > The rebase should be quite straight forward for most cases. > > > > If you have any questions Im happy to help. > > > > I rebased a tremendous number of times to keep up with main changing while > > the PR took several weeks to create and test and validate against previous > > builds. Also rebased repeatedly during the review and feedback stage. As > > a result I got to practice my rebase skills about 20 times across 50 or > > more commits to main during that time. > > > > On your pr branch > > > > git fetch —all > > git rebase upstream/main > > git status > > fix any conflicts > > git add . > > git rebase —continue > > > > If any conflicts it would likely be for new modules created in the pr which > > would just be moving from a path in nifi-nar-bundles to > > nifi-extension-bundles. > > > > If you have a pr in mind you have a question on for the rebase let me know. > > > > Thanks > > Joe > > > > On Wed, May 1, 2024 at 9:51 PM Matt Burgess wrote: > > > > > One thing that was mentioned was an included Jira for "providing a > > clearer > > > distinction between framework and > > > extensions," This involved moving extensions to a new module and for > > > community folks this is causing problems with any > > > new improvements in-flight before that. Seems like it needed more > > > announcement than an upcoming RC due to the impact? > > > > > > On Mon, Apr 29, 2024 at 10:22 AM David Handermann < > > > exceptionfact...@apache.org> wrote: > > > > > > > Team, > > > > > > > > We are getting closer to ready for a release candidate build. Based on > > > > some conversations with those working on the user interface, there are > > > > still a couple more items to complete this week. Getting a solid build > > > > of the new UI into this next version will be very helpful, so getting > > > > a few more of those issues resolved should put things in a good > > > > position. > > > > > > > > Regards, > > > > David Handermann > > > > > > > > On Wed, Apr 24, 2024 at 8:32 AM Joe Witt wrote: > > > > > > > > > > Also agreed there. Profile should be there to exclude perhaps but > > > > include > > > > > should be default. > > > > > > > > > > On Wed, Apr 24, 2024 at 6:30 AM David Handermann < > > > > > exceptionfact...@apache.org> wrote: > > > > > > > > > > > Thanks for the replies thus far! > > > > > > > > > > > > With the goal of including the new UI, it seems like a good time to > > > > > > move the module from an optional profile to part of the standard > > > > > > build. That would make the release candidate build and review > > process > > > > > > simpler, not requiring the optional build profile. > > > > > > > > > > > > Regards, > > > > > > David Handermann > > > > > > > > > > > > On Tue, Apr 23, 2024 at 2:42 PM Matt Gilman < > > matt.c.gil...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > > > > I agree. Including the updated UI and getting some feedback would > > > be > > > > > > great. > > > > > > > > > > > > > > On Mon, Apr 22, 2024 at 8:50 PM Joe Witt > > > wrote: > > > > > > > > > > > > > > > Id be a big fan of including the new UI. > > > > > > > > > > > > > > > > On Mon, Apr 22, 2024 at 5:45 PM Pierre Villard < > > > > > > > > pierre.villard...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Thanks David, I'm happy to take care of a 1.26 release at the > > > > same > > > > > > time. > > > > > > > > > > > > > > > > > > Regarding the M3 release, should the new UI be included and > > > > > > instructions > > > > > > > > > given on how to access it to start collecting feedback? I'm > > > > under the > > > > > > > > > impression that almost all of the work has been done, no? > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Pierre > > > > > > > > > > > > > > > > > > Le mar. 23 avr. 2024, 02:03, David Handermann < > > > > > > > > exceptionfact...@apache.org > > > > > > > > > > > > > > > > > > > a écrit : > > > > > > > > > > > > > > > > > > > Team, > > > > > > > > > > > > > > > > > > > > We are approaching 300 Jira issues resolved for version > > > > 2.0.0-M3 > > > > > > [1], > > > > > > > > > > including a number of important dependency upgrades and > > > Python > > > > > > > > > > framework improvements. > > > > > > > > > > > > > > > > > > > > There are several open pull requests that are getting close > > > to > > > > > >
Re: [discuss] identifying key components needed updates which require deeper review/consideration
Joe, Thanks for highlighting these areas to be updated. I was able to address the QuestDB update with a couple minor adjustments, and submitted a PR [1]. Regards, David Handermann [1] https://github.com/apache/nifi/pull/8773 On Tue, May 7, 2024 at 2:58 PM Chris Sampson wrote: > > I’ll pick up the Elasticsearch dependency upgrade. > > I’ll try to test it again OpenSearch as well, if for no reason other than to > prove either way whether the current NiFi components will work with the AWS > service, although I may not get to do that and instead testing will focus on > Elasticsearch 8.x instances (and the existing Integration Tests within NiFi). > > > Cheers, > > --- > Chris Sampson > IT Consultant > chris.samp...@naimuri.com > > > > On 7 May 2024, at 20:54, Joe Witt wrote: > > > > Yeah sorry - what I wrote for the Hive one was a bit nonsensical. I tried > > to clarify it better in the JIRA. > > > > Thanks > > > > On Tue, May 7, 2024 at 12:38 PM Matt Burgess wrote: > > > >> For [1] Hive 3 was removed in > >> https://issues.apache.org/jira/browse/NIFI-12981, if you mean the Hive > >> dependency from Apache Iceberg we probably need a release from them that > >> brings in Hive 4 dependencies? Or do you suggest we bring the Hive > >> components back as long as we update the version? > >> > >> I'll take the Flyway one in the meantime, database integrations are my > >> blessing and my curse :) If that goes more smoothly than I expect I can > >> grab the QuestDB one too, will update the Jiras as appropriate. > >> > >> Regards, > >> Matt > >> > >> On Tue, May 7, 2024 at 1:56 PM Joe Witt wrote: > >> > >>> Team, > >>> > >>> We have made massive strides going from thousands of out of date > >> libraries > >>> to dozens in the 2.x line vs any prior releases. And we can/should stay > >> on > >>> top of these better going forward. A few key bits that are outstanding > >>> with no obvious immediate work underway are as follows. If anyone is > >>> working on these, willing to work on these, please volunteer and let us > >>> know. For components that we cannot find active engagement on we should > >>> consider removal though these aren't great examples as they're indeed > >>> common usage components. > >>> > >>> [1] https://issues.apache.org/jira/browse/NIFI-13173 > >>> Hive 4 recently came out but Hive also gives us some of the most > >>> complicated POM/library mangling and oldest dependencies to > >> override/etc.. > >>> Putting serious detailed focus here is very valuable from a maintenance > >>> point of view. While existing usage of Hive 3 likely remains for some > >>> time, those users are like NiFi 1.x users anyway. In any case if there > >> are > >>> motivated developers interested to see our Hive components be healthy > >>> please do dive in on this. > >>> > >>> [2] https://issues.apache.org/jira/browse/NIFI-13169 > >>> The NiFI registry is a major release behind Flyway DB. We need to stay > >>> current given the importance of this component. > >>> > >>> [3] https://issues.apache.org/jira/browse/NIFI-13168 > >>> We are a few incrementals behind and a minor release behind QuestDB. I > >>> tried bumping the incremental but it broke tests. > >>> > >>> [4] https://issues.apache.org/jira/browse/NIFI-11686 > >>> We are a full major release behind and several incremental/minors as > >> well. > >>> Apparently there is some concern related to the community changes in > >>> Elastic vs OpenSearch/etc.. but regardless we need to stay active in > >>> maintaining this component. We really should get to the latest main. > >>> > >>> Thanks to anyone who can help tackle these and make progress both in > >> terms > >>> of doing the upgrade/update work, validating tests, fixing licenses, but > >>> also reviews. > >>> > >>> Thanks > >>> Joe > >>> > >> >
Re: [discuss] removal of nifi-external module in 2.x/deprecate in 1.x
Joe, Thanks for highlighting the Kafka Connect external module. I concur with your evaluation, I have not observed community usage or questions, so it seems best to remove from the main branch. Regards, David Handermann On Tue, May 7, 2024 at 2:18 PM Otto Fowler wrote: > > I would say nuke it, as you are saying. I would also say you should cross > post this to users@ as they will have relevant input > > > On May 7, 2024 at 1:32:11 PM, Joe Witt (joew...@apache.org) wrote: > > Team, > > The only component remaining is kafka-connect in the nifi-external module. > As noted in [1] it does not see much active maintenance and I never see in > the apache community any questions/usage related either. Kafka is > currently on 3.7.0 and this component is Kafka 2.6 for example. > > We should not carry this forward in the 2.x line. To be clear Kafka > remains a super important component to interact with and our Kafka > components within NiFi do see maintenance including an active PR under > review right now. But this is our last 'nifi-external' remaining and is > not maintained. > > I'll file a PR under this JIRA [1] and its child JIRA for removal on 2x and > deprecation on 1x but wanted to flag this here in case there is some > substantial usage and/or maintenance/update coming soon. Even if desire > remains to keep it alive that should be considered to occur in another repo > or by another group that leverages it - but please let me know if there is > apache community usage/mx I'm not aware of. > > Thoughts? > > [1] https://issues.apache.org/jira/browse/NIFI-13171 > > Thanks > Joe
Re: [discuss] identifying key components needed updates which require deeper review/consideration
I’ll pick up the Elasticsearch dependency upgrade. I’ll try to test it again OpenSearch as well, if for no reason other than to prove either way whether the current NiFi components will work with the AWS service, although I may not get to do that and instead testing will focus on Elasticsearch 8.x instances (and the existing Integration Tests within NiFi). Cheers, --- Chris Sampson IT Consultant chris.samp...@naimuri.com > On 7 May 2024, at 20:54, Joe Witt wrote: > > Yeah sorry - what I wrote for the Hive one was a bit nonsensical. I tried > to clarify it better in the JIRA. > > Thanks > > On Tue, May 7, 2024 at 12:38 PM Matt Burgess wrote: > >> For [1] Hive 3 was removed in >> https://issues.apache.org/jira/browse/NIFI-12981, if you mean the Hive >> dependency from Apache Iceberg we probably need a release from them that >> brings in Hive 4 dependencies? Or do you suggest we bring the Hive >> components back as long as we update the version? >> >> I'll take the Flyway one in the meantime, database integrations are my >> blessing and my curse :) If that goes more smoothly than I expect I can >> grab the QuestDB one too, will update the Jiras as appropriate. >> >> Regards, >> Matt >> >> On Tue, May 7, 2024 at 1:56 PM Joe Witt wrote: >> >>> Team, >>> >>> We have made massive strides going from thousands of out of date >> libraries >>> to dozens in the 2.x line vs any prior releases. And we can/should stay >> on >>> top of these better going forward. A few key bits that are outstanding >>> with no obvious immediate work underway are as follows. If anyone is >>> working on these, willing to work on these, please volunteer and let us >>> know. For components that we cannot find active engagement on we should >>> consider removal though these aren't great examples as they're indeed >>> common usage components. >>> >>> [1] https://issues.apache.org/jira/browse/NIFI-13173 >>> Hive 4 recently came out but Hive also gives us some of the most >>> complicated POM/library mangling and oldest dependencies to >> override/etc.. >>> Putting serious detailed focus here is very valuable from a maintenance >>> point of view. While existing usage of Hive 3 likely remains for some >>> time, those users are like NiFi 1.x users anyway. In any case if there >> are >>> motivated developers interested to see our Hive components be healthy >>> please do dive in on this. >>> >>> [2] https://issues.apache.org/jira/browse/NIFI-13169 >>> The NiFI registry is a major release behind Flyway DB. We need to stay >>> current given the importance of this component. >>> >>> [3] https://issues.apache.org/jira/browse/NIFI-13168 >>> We are a few incrementals behind and a minor release behind QuestDB. I >>> tried bumping the incremental but it broke tests. >>> >>> [4] https://issues.apache.org/jira/browse/NIFI-11686 >>> We are a full major release behind and several incremental/minors as >> well. >>> Apparently there is some concern related to the community changes in >>> Elastic vs OpenSearch/etc.. but regardless we need to stay active in >>> maintaining this component. We really should get to the latest main. >>> >>> Thanks to anyone who can help tackle these and make progress both in >> terms >>> of doing the upgrade/update work, validating tests, fixing licenses, but >>> also reviews. >>> >>> Thanks >>> Joe >>> >>
Re: [discuss] identifying key components needed updates which require deeper review/consideration
Yeah sorry - what I wrote for the Hive one was a bit nonsensical. I tried to clarify it better in the JIRA. Thanks On Tue, May 7, 2024 at 12:38 PM Matt Burgess wrote: > For [1] Hive 3 was removed in > https://issues.apache.org/jira/browse/NIFI-12981, if you mean the Hive > dependency from Apache Iceberg we probably need a release from them that > brings in Hive 4 dependencies? Or do you suggest we bring the Hive > components back as long as we update the version? > > I'll take the Flyway one in the meantime, database integrations are my > blessing and my curse :) If that goes more smoothly than I expect I can > grab the QuestDB one too, will update the Jiras as appropriate. > > Regards, > Matt > > On Tue, May 7, 2024 at 1:56 PM Joe Witt wrote: > > > Team, > > > > We have made massive strides going from thousands of out of date > libraries > > to dozens in the 2.x line vs any prior releases. And we can/should stay > on > > top of these better going forward. A few key bits that are outstanding > > with no obvious immediate work underway are as follows. If anyone is > > working on these, willing to work on these, please volunteer and let us > > know. For components that we cannot find active engagement on we should > > consider removal though these aren't great examples as they're indeed > > common usage components. > > > > [1] https://issues.apache.org/jira/browse/NIFI-13173 > > Hive 4 recently came out but Hive also gives us some of the most > > complicated POM/library mangling and oldest dependencies to > override/etc.. > > Putting serious detailed focus here is very valuable from a maintenance > > point of view. While existing usage of Hive 3 likely remains for some > > time, those users are like NiFi 1.x users anyway. In any case if there > are > > motivated developers interested to see our Hive components be healthy > > please do dive in on this. > > > > [2] https://issues.apache.org/jira/browse/NIFI-13169 > > The NiFI registry is a major release behind Flyway DB. We need to stay > > current given the importance of this component. > > > > [3] https://issues.apache.org/jira/browse/NIFI-13168 > > We are a few incrementals behind and a minor release behind QuestDB. I > > tried bumping the incremental but it broke tests. > > > > [4] https://issues.apache.org/jira/browse/NIFI-11686 > > We are a full major release behind and several incremental/minors as > well. > > Apparently there is some concern related to the community changes in > > Elastic vs OpenSearch/etc.. but regardless we need to stay active in > > maintaining this component. We really should get to the latest main. > > > > Thanks to anyone who can help tackle these and make progress both in > terms > > of doing the upgrade/update work, validating tests, fixing licenses, but > > also reviews. > > > > Thanks > > Joe > > >
Re: [discuss] identifying key components needed updates which require deeper review/consideration
For [1] Hive 3 was removed in https://issues.apache.org/jira/browse/NIFI-12981, if you mean the Hive dependency from Apache Iceberg we probably need a release from them that brings in Hive 4 dependencies? Or do you suggest we bring the Hive components back as long as we update the version? I'll take the Flyway one in the meantime, database integrations are my blessing and my curse :) If that goes more smoothly than I expect I can grab the QuestDB one too, will update the Jiras as appropriate. Regards, Matt On Tue, May 7, 2024 at 1:56 PM Joe Witt wrote: > Team, > > We have made massive strides going from thousands of out of date libraries > to dozens in the 2.x line vs any prior releases. And we can/should stay on > top of these better going forward. A few key bits that are outstanding > with no obvious immediate work underway are as follows. If anyone is > working on these, willing to work on these, please volunteer and let us > know. For components that we cannot find active engagement on we should > consider removal though these aren't great examples as they're indeed > common usage components. > > [1] https://issues.apache.org/jira/browse/NIFI-13173 > Hive 4 recently came out but Hive also gives us some of the most > complicated POM/library mangling and oldest dependencies to override/etc.. > Putting serious detailed focus here is very valuable from a maintenance > point of view. While existing usage of Hive 3 likely remains for some > time, those users are like NiFi 1.x users anyway. In any case if there are > motivated developers interested to see our Hive components be healthy > please do dive in on this. > > [2] https://issues.apache.org/jira/browse/NIFI-13169 > The NiFI registry is a major release behind Flyway DB. We need to stay > current given the importance of this component. > > [3] https://issues.apache.org/jira/browse/NIFI-13168 > We are a few incrementals behind and a minor release behind QuestDB. I > tried bumping the incremental but it broke tests. > > [4] https://issues.apache.org/jira/browse/NIFI-11686 > We are a full major release behind and several incremental/minors as well. > Apparently there is some concern related to the community changes in > Elastic vs OpenSearch/etc.. but regardless we need to stay active in > maintaining this component. We really should get to the latest main. > > Thanks to anyone who can help tackle these and make progress both in terms > of doing the upgrade/update work, validating tests, fixing licenses, but > also reviews. > > Thanks > Joe >
Re: [discuss] removal of nifi-external module in 2.x/deprecate in 1.x
I would say nuke it, as you are saying. I would also say you should cross post this to users@ as they will have relevant input On May 7, 2024 at 1:32:11 PM, Joe Witt (joew...@apache.org) wrote: Team, The only component remaining is kafka-connect in the nifi-external module. As noted in [1] it does not see much active maintenance and I never see in the apache community any questions/usage related either. Kafka is currently on 3.7.0 and this component is Kafka 2.6 for example. We should not carry this forward in the 2.x line. To be clear Kafka remains a super important component to interact with and our Kafka components within NiFi do see maintenance including an active PR under review right now. But this is our last 'nifi-external' remaining and is not maintained. I'll file a PR under this JIRA [1] and its child JIRA for removal on 2x and deprecation on 1x but wanted to flag this here in case there is some substantial usage and/or maintenance/update coming soon. Even if desire remains to keep it alive that should be considered to occur in another repo or by another group that leverages it - but please let me know if there is apache community usage/mx I'm not aware of. Thoughts? [1] https://issues.apache.org/jira/browse/NIFI-13171 Thanks Joe
Re: ...not the most recent version of this FlowFile within this session...
Yes, what you described is what was happening, Mark. I didn't display all of the code to the session methods, and I did re-read the in-coming flowfile for different purposes than I had already read and written it. So, I wasn't helpful enough. In the end, however, I had forgotten, immediately after the call to session.putAllAttributes(), to update the resulting flowfile for passing to session.transfer(). That solved it for 1.1.2 which wasn't necessary for 1.13.2 or later versions. Being helpful, the newer versions made me a spoiled, entitled child and I will repent immediately. Thanks, guys! DevOps are happy they don't have to upgrade the customers to NiFi 1.13.2. (In a way, I'm unhappy about that, but...). Best regards, Russ On 5/7/24 11:53, Mark Payne wrote: The call to session.putAttribute would throw an Exception because you provided an outdated version of the flowFile (did not capturing the result of calling session.write) Now, as NiFi matured, we found that: (a) for more complex processors that aren’t just a series of sequential steps it becomes difficult to manage all of that bookkeeping. (b) it was not intuitive to require this (c) the ProcessSession already had more or less what it needed in order to determine what the most up-to-date version of the FlowFile was. So we updated the ProcessSession to automatically grab the latest version of the FlowFile for these methods. But since you’re trying to run an old version, you’ll need to make sure that you capture all of those outputs and always keep track of the most recent version of a FlowFile.
[discuss] identifying key components needed updates which require deeper review/consideration
Team, We have made massive strides going from thousands of out of date libraries to dozens in the 2.x line vs any prior releases. And we can/should stay on top of these better going forward. A few key bits that are outstanding with no obvious immediate work underway are as follows. If anyone is working on these, willing to work on these, please volunteer and let us know. For components that we cannot find active engagement on we should consider removal though these aren't great examples as they're indeed common usage components. [1] https://issues.apache.org/jira/browse/NIFI-13173 Hive 4 recently came out but Hive also gives us some of the most complicated POM/library mangling and oldest dependencies to override/etc.. Putting serious detailed focus here is very valuable from a maintenance point of view. While existing usage of Hive 3 likely remains for some time, those users are like NiFi 1.x users anyway. In any case if there are motivated developers interested to see our Hive components be healthy please do dive in on this. [2] https://issues.apache.org/jira/browse/NIFI-13169 The NiFI registry is a major release behind Flyway DB. We need to stay current given the importance of this component. [3] https://issues.apache.org/jira/browse/NIFI-13168 We are a few incrementals behind and a minor release behind QuestDB. I tried bumping the incremental but it broke tests. [4] https://issues.apache.org/jira/browse/NIFI-11686 We are a full major release behind and several incremental/minors as well. Apparently there is some concern related to the community changes in Elastic vs OpenSearch/etc.. but regardless we need to stay active in maintaining this component. We really should get to the latest main. Thanks to anyone who can help tackle these and make progress both in terms of doing the upgrade/update work, validating tests, fixing licenses, but also reviews. Thanks Joe
Re: ...not the most recent version of this FlowFile within this session...
Russell,
May not be too bad to fix :)
Any time you use a method on ProcessSession to modify a FlowFile in some way
(such as putAttribute, putAllAttributes, write, removeAttribute,
removeAllAttributes, etc.) the ProcessSession returns to you a new FlowFile. So
you *should* write code such as:
```
FlowFile flowFile = session.get();
If (flowFile == null) {
return;
}
flowFile = session.write(flowFile, new OutputStreamCallback() {
…
});
flowFile = session.putAttribute(flowFile, “greeting”, “hello”);
session.transfer(flowFile);
```
Note here that calls to session.write, session.putAttribute are capturing the
return value and holding on to that.
That was necessary in older versions. If you instead wrote:
```
flowFile = session.get();
session.write(flowFile, new OutputStreamCallback() {
…
});
session.putAttribute(flowFile, “greeting”, “hello”);
```
The call to session.putAttribute would throw an Exception because you provided
an outdated version of the flowFile (did not capturing the result of calling
session.write)
Now, as NiFi matured, we found that:
(a) for more complex processors that aren’t just a series of sequential steps
it becomes difficult to manage all of that bookkeeping.
(b) it was not intuitive to require this
(c) the ProcessSession already had more or less what it needed in order to
determine what the most up-to-date version of the FlowFile was.
So we updated the ProcessSession to automatically grab the latest version of
the FlowFile for these methods. But since you’re trying to run an old version,
you’ll need to make sure that you capture all of those outputs and always keep
track of the most recent version of a FlowFile.
You’ll also need to ensure that you’re not calling any methods, etc. that
didn’t exist in the 1.1.2 API :)
Thanks
-Mark
> On May 7, 2024, at 1:38 PM, Russell Bateman wrote:
>
> In /pom.xml/ I specify using NiFi framework libraries from 1.13.2.
>
> There are peculiar reasons (that we are trying to fix) that inhibit us from
> moving forward as all of our customer machines are running 1.1.2. (Don't
> shoot me, I'm not DevOps, but just the guy who writes custom processors.)
>
> I have custom processor code that, built as noted using 1.13.2, runs
> _perfectly_ on any version of NiFi from 1.13.2 to 1.25.0. When loaded on
> 1.1.2, one processor fails to process a very simple flowfile with the error
> message below.
>
> I'm wondering if there's a dance I can do to tweak one or more of what I
> suspect are the calls where this is happening (in onTrigger()):
>
> FlowFile result = session.*write*( flowfile, new StreamCallback()
> session.*read*( flowfile, new InputStreamCallback()
> session.*removeAllAttributes*( result, keys );
> session.*putAllAttributes*( result, attributes );
> session.*transfer*( result, SUCCESS );
>
>
> Error message:
>
> 10:32:47 MDTERRORc78b2d48-79a6-39ae-1454-4ee57bbf8928
> ExcerptWarehouseResponse[id=c78b2d48-79a6-39ae-1454-4ee57bbf8928] \
> failed to process session due to
> org.apache.nifi.processor.exception.FlowFileHandlingException: \
> StandardFlowFileRecord[uuid=cfb2a124-4b07-46bd-a70e-3848502b3df1,claim=StandardContentClaim
> \
> [resourceClaim=StandardResourceClaim[id=1715094869203-1, container=default,
> section=1], offset=0,
> length=1077766],offset=0,name=30575811037248611,size=1077766] \
> is not the most recent version of this FlowFile within this session
> (StandardProcessSession[id=3923]): \
> org.apache.nifi.processor.exception.FlowFileHandlingException:
> StandardFlowFileRecord[uuid=cfb2a124-4b07-46bd-a70e-3848502b3df1,claim=StandardContentClaim
> \
> [resourceClaim=StandardResourceClaim[id=1715094869203-1, container=default,
> section=1], offset=0,
> length=1077766],offset=0,name=30575811037248611,size=1077766] \
> is not the most recent version of this FlowFile within this session
> (StandardProcessSession[id=3923])
>
> 10:32:47 MDTWARNINGc78b2d48-79a6-39ae-1454-4ee57bbf8928
> ExcerptWarehouseResponse[id=c78b2d48-79a6-39ae-1454-4ee57bbf8928] Processor
> Administratively Yielded for 1 sec due to processing failure
>
>
> I know this is old stuff and I'm prepared be told that I'm out of luck or
> that the solution is unfeasible or would take too much effort to dig through.
> I'm just hoping that someone remembers something quick off the top of the
> head and can give advice.
>
> Thanks,
>
> Russ
>
>
>
>
>
Re: ...not the most recent version of this FlowFile within this session...
Russ FlowFile result = session.*write*( flowfile, new StreamCallback() session.*read*( flowfile, new InputStreamCallback() session.*removeAllAttributes*( result, keys ); session.*putAllAttributes*( result, attributes ); session.*transfer*( result, SUCCESS ); The session.read passes in the same original flowfile reference that you send in to the write. I'm guessing that is not what you intended. You probably want session.read(result, newInputStream Thanks Joe On Tue, May 7, 2024 at 10:38 AM Russell Bateman wrote: > In /pom.xml/ I specify using NiFi framework libraries from 1.13.2. > > There are peculiar reasons (that we are trying to fix) that inhibit us > from moving forward as all of our customer machines are running 1.1.2. > (Don't shoot me, I'm not DevOps, but just the guy who writes custom > processors.) > > I have custom processor code that, built as noted using 1.13.2, runs > _perfectly_ on any version of NiFi from 1.13.2 to 1.25.0. When loaded on > 1.1.2, one processor fails to process a very simple flowfile with the > error message below. > > I'm wondering if there's a dance I can do to tweak one or more of what I > suspect are the calls where this is happening (in onTrigger()): > > FlowFile result = session.*write*( flowfile, new StreamCallback() >session.*read*( flowfile, new InputStreamCallback() >session.*removeAllAttributes*( result, keys ); >session.*putAllAttributes*( result, attributes ); >session.*transfer*( result, SUCCESS ); > > > Error message: > > 10:32:47 MDTERRORc78b2d48-79a6-39ae-1454-4ee57bbf8928 > ExcerptWarehouseResponse[id=c78b2d48-79a6-39ae-1454-4ee57bbf8928] \ > failed to process session due to > org.apache.nifi.processor.exception.FlowFileHandlingException: \ > StandardFlowFileRecord[uuid=cfb2a124-4b07-46bd-a70e-3848502b3df1,claim=StandardContentClaim > \ > [resourceClaim=StandardResourceClaim[id=1715094869203-1, > container=default, section=1], offset=0, > length=1077766],offset=0,name=30575811037248611,size=1077766] \ > is not the most recent version of this FlowFile within this session > (StandardProcessSession[id=3923]): \ > org.apache.nifi.processor.exception.FlowFileHandlingException: > StandardFlowFileRecord[uuid=cfb2a124-4b07-46bd-a70e-3848502b3df1,claim=StandardContentClaim > \ > [resourceClaim=StandardResourceClaim[id=1715094869203-1, > container=default, section=1], offset=0, > length=1077766],offset=0,name=30575811037248611,size=1077766] \ > is not the most recent version of this FlowFile within this session > (StandardProcessSession[id=3923]) > > 10:32:47 MDTWARNINGc78b2d48-79a6-39ae-1454-4ee57bbf8928 > ExcerptWarehouseResponse[id=c78b2d48-79a6-39ae-1454-4ee57bbf8928] > Processor Administratively Yielded for 1 sec due to processing failure > > > I know this is old stuff and I'm prepared be told that I'm out of luck > or that the solution is unfeasible or would take too much effort to dig > through. I'm just hoping that someone remembers something quick off the > top of the head and can give advice. > > Thanks, > > Russ > > > > > >
...not the most recent version of this FlowFile within this session...
In /pom.xml/ I specify using NiFi framework libraries from 1.13.2. There are peculiar reasons (that we are trying to fix) that inhibit us from moving forward as all of our customer machines are running 1.1.2. (Don't shoot me, I'm not DevOps, but just the guy who writes custom processors.) I have custom processor code that, built as noted using 1.13.2, runs _perfectly_ on any version of NiFi from 1.13.2 to 1.25.0. When loaded on 1.1.2, one processor fails to process a very simple flowfile with the error message below. I'm wondering if there's a dance I can do to tweak one or more of what I suspect are the calls where this is happening (in onTrigger()): FlowFile result = session.*write*( flowfile, new StreamCallback() session.*read*( flowfile, new InputStreamCallback() session.*removeAllAttributes*( result, keys ); session.*putAllAttributes*( result, attributes ); session.*transfer*( result, SUCCESS ); Error message: 10:32:47 MDTERRORc78b2d48-79a6-39ae-1454-4ee57bbf8928 ExcerptWarehouseResponse[id=c78b2d48-79a6-39ae-1454-4ee57bbf8928] \ failed to process session due to org.apache.nifi.processor.exception.FlowFileHandlingException: \ StandardFlowFileRecord[uuid=cfb2a124-4b07-46bd-a70e-3848502b3df1,claim=StandardContentClaim \ [resourceClaim=StandardResourceClaim[id=1715094869203-1, container=default, section=1], offset=0, length=1077766],offset=0,name=30575811037248611,size=1077766] \ is not the most recent version of this FlowFile within this session (StandardProcessSession[id=3923]): \ org.apache.nifi.processor.exception.FlowFileHandlingException: StandardFlowFileRecord[uuid=cfb2a124-4b07-46bd-a70e-3848502b3df1,claim=StandardContentClaim \ [resourceClaim=StandardResourceClaim[id=1715094869203-1, container=default, section=1], offset=0, length=1077766],offset=0,name=30575811037248611,size=1077766] \ is not the most recent version of this FlowFile within this session (StandardProcessSession[id=3923]) 10:32:47 MDTWARNINGc78b2d48-79a6-39ae-1454-4ee57bbf8928 ExcerptWarehouseResponse[id=c78b2d48-79a6-39ae-1454-4ee57bbf8928] Processor Administratively Yielded for 1 sec due to processing failure I know this is old stuff and I'm prepared be told that I'm out of luck or that the solution is unfeasible or would take too much effort to dig through. I'm just hoping that someone remembers something quick off the top of the head and can give advice. Thanks, Russ
[discuss] removal of nifi-external module in 2.x/deprecate in 1.x
Team, The only component remaining is kafka-connect in the nifi-external module. As noted in [1] it does not see much active maintenance and I never see in the apache community any questions/usage related either. Kafka is currently on 3.7.0 and this component is Kafka 2.6 for example. We should not carry this forward in the 2.x line. To be clear Kafka remains a super important component to interact with and our Kafka components within NiFi do see maintenance including an active PR under review right now. But this is our last 'nifi-external' remaining and is not maintained. I'll file a PR under this JIRA [1] and its child JIRA for removal on 2x and deprecation on 1x but wanted to flag this here in case there is some substantial usage and/or maintenance/update coming soon. Even if desire remains to keep it alive that should be considered to occur in another repo or by another group that leverages it - but please let me know if there is apache community usage/mx I'm not aware of. Thoughts? [1] https://issues.apache.org/jira/browse/NIFI-13171 Thanks Joe
Re: [VOTE] Release Apache NiFi MiNiFi C++ 0.99.0 (RC3)
+1 (binding) Built on ubuntu, verified signature and checksums. All tests passed, started and ran a simple flow with no issues. Nice job, team! On Tue, May 7, 2024 at 11:35 AM Gábor Gyimesi wrote: > Hello, > > I am pleased to be calling this vote for the source release of Apache NiFi > MiNiFi C++ 0.99.0. > > The source tarball, some binary builds, plus signatures and digests can be > found at: > https://dist.apache.org/repos/dist/dev/nifi/nifi-minifi-cpp/0.99.0/ > > The Git tag is minifi-cpp-0.99.0-RC3 > The Git commit ID is d3d205f3c8a8376bbad4c8a47758d585d57c653b > > https://git-wip-us.apache.org/repos/asf?p=nifi-minifi-cpp.git;a=commit;h=d3d205f3c8a8376bbad4c8a47758d585d57c653b > > Checksums of nifi-minifi-cpp-0.99.0-source.tar.gz: > SHA256: c00ff309776720b295cb52ab12c1c3d11182af4f19ed0a379d58f96b16841059 > SHA512: > > d1eda2c19da8f0e073cf460bf889c8b4a1b893de4b245dc0147d1c79545d5298c74bedf585afd7091dc9474c923fc2a7075a81a6397bedce310e5838e3afa898 > > Release artifacts are signed with the following key: > https://people.apache.org/keys/committer/lordgamez.asc > > A helpful reminder on how the release candidate verification process works: > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=173087303 > > KEYS file available here: > https://dist.apache.org/repos/dist/release/nifi/KEYS > > 121 issues were closed/resolved for this release: > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12321520=12353618 > > Release note highlights can be found here: > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65145325#ReleaseNotesMiNiFi(C++)-Versioncpp-0.99.0 > > The vote will be open for 72 hours. > Please download the release candidate and evaluate the necessary items > including checking hashes, signatures, build > from source, and test. Then please vote: > > [ ] +1 Release this package as nifi-minifi-cpp-0.99.0 > [ ] +0 no opinion > [ ] -1 Do not release this package because... >
[VOTE] Release Apache NiFi MiNiFi C++ 0.99.0 (RC3)
Hello, I am pleased to be calling this vote for the source release of Apache NiFi MiNiFi C++ 0.99.0. The source tarball, some binary builds, plus signatures and digests can be found at: https://dist.apache.org/repos/dist/dev/nifi/nifi-minifi-cpp/0.99.0/ The Git tag is minifi-cpp-0.99.0-RC3 The Git commit ID is d3d205f3c8a8376bbad4c8a47758d585d57c653b https://git-wip-us.apache.org/repos/asf?p=nifi-minifi-cpp.git;a=commit;h=d3d205f3c8a8376bbad4c8a47758d585d57c653b Checksums of nifi-minifi-cpp-0.99.0-source.tar.gz: SHA256: c00ff309776720b295cb52ab12c1c3d11182af4f19ed0a379d58f96b16841059 SHA512: d1eda2c19da8f0e073cf460bf889c8b4a1b893de4b245dc0147d1c79545d5298c74bedf585afd7091dc9474c923fc2a7075a81a6397bedce310e5838e3afa898 Release artifacts are signed with the following key: https://people.apache.org/keys/committer/lordgamez.asc A helpful reminder on how the release candidate verification process works: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=173087303 KEYS file available here: https://dist.apache.org/repos/dist/release/nifi/KEYS 121 issues were closed/resolved for this release: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12321520=12353618 Release note highlights can be found here: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65145325#ReleaseNotesMiNiFi(C++)-Versioncpp-0.99.0 The vote will be open for 72 hours. Please download the release candidate and evaluate the necessary items including checking hashes, signatures, build from source, and test. Then please vote: [ ] +1 Release this package as nifi-minifi-cpp-0.99.0 [ ] +0 no opinion [ ] -1 Do not release this package because...
