NiFi automation
Hi, I'm currently working on building a CI/CD for NiFi in my workplace. My goal is to provide the developers with a ready-to-work platform without the need for manual changes (import templates, configure registry and add plugins) from their side. I hoped you can tell me if there is a way to specify in NiFi code to make those changes. Thanks in advance, Tomer
Java 11 build support is live!
Apache NiFi Developers, Apache NiFi can now be built with Java 11! Thanks to those that helped with the review and committing of several PRs to get us to this point. For all contributions going forward, developers, reviewers, and committers need to make sure that PRs are verified by building with and running on Java 8 and 11. To assist in this process, the Linux-based Travis CI build has been updated to perform an en_US locale-based build of NiFi using AdoptOpenJDK 11.0.4 in addition to the three locale-based builds (en_US, fr_FR, ja_JP) on OpenJDK 8. Apache NiFi still has a minimum requirement of Java 8 and as long as that requirement exists the released convenience binaries will be built with Java 8. When building locally with Java 11, on all operating systems, a minimum JDK version of 11.0.3 is required for a successful build. If you are building on OSX and use AdoptOpenJDK 11.0.4, there's an issue [4] [5] with native libraries. If you run into this issue, please downgrade your AdoptOpenJDK 11 installation to 11.0.3. A updated release of AdoptOpenJDK 11.0.4 that fixes the native library issues is forthcoming. There is a comment [6] on AdoptOpenJDK issue 1211 that describes the downgrade process. Work still remains regarding Java 11. Some of the components that NiFi uses are not completely compatible with Java 11, and JIRAs will be created or linked to the Java 11 parent JIRA [1] to track the remaining changes. Several PRs not explicitly related to Java 11 were created and merged to master ahead of NIFI-5176 [2] (PR 3404 [3]), and are listed as blockers on NIFI-5176. All blockers have been resolved, as well as NIFI-5176. If you encounter an issue with the Java 11 build, or running the resulting build on Java 11, please check NIFI-5174 to see if a JIRA for the issue has already been filed, and create one if needed. [1] https://issues.apache.org/jira/browse/NIFI-5174 [2] https://issues.apache.org/jira/browse/NIFI-5176 [3] https://github.com/apache/nifi/pull/3404 [4] https://github.com/AdoptOpenJDK/openjdk-build/issues/1206 [5] https://github.com/AdoptOpenJDK/openjdk-build/issues/1211 [6] https://github.com/AdoptOpenJDK/openjdk-build/issues/1211#issuecomment-521392147
Re: [EXT] Airflow and NiFi
i blame spell check for ‘complimentary’ On Thu, Aug 15, 2019 at 4:11 PM Joe Witt wrote: > Love it. > > Airflow I see as an orchestrator of making other systems do things. Nifi > i see as very focused on acquiring, manipulating, and routing data between > systems. > > In short, I believe theyre complimentary in a given architecture. Someone > always tries to use one to do both of the things but each is better at > their sweet spot. > > Thanks > > On Thu, Aug 15, 2019 at 4:07 PM Mike Thomsen > wrote: > >> Yeah, pretty much. >> >> On Thu, Aug 15, 2019 at 4:02 PM Peter Wicks (pwicks) >> wrote: >> >> > User: "I want to change my production flow while it's running, you know, >> > mid stream just route my content to a completely different flow, re-run >> > data mid run through a new set of processors, fork it on the fly, you >> know, >> > whatever I want anytime I want " >> > >> > NiFi: "Yeah, I can do that" >> > >> > Everyone Else: " I'm sorry... what?" >> > >> > -Original Message- >> > From: Mike Thomsen >> > Sent: Thursday, August 15, 2019 2:00 PM >> > To: dev@nifi.apache.org >> > Subject: [EXT] Airflow and NiFi >> > >> > Does anyone see any areas where the two can complement each other and >> > where we might want to give users the ability to offload processing to >> > Airflow? >> > Curious since our poking around the docs lead us to conclude it was >> > probably more "Airflow vs Spark+Oozie" than really competing with NiFi. >> > >> >
Re: Java 11 build support is live!
This is awesome work that will be crucial for a large part of the community moving forward Jeff. Thanks for taking the lead on this and slogging through a lot of difficult challenges. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Aug 15, 2019, at 11:04 AM, Jeff wrote: > > Apache NiFi Developers, > > Apache NiFi can now be built with Java 11! Thanks to those that helped > with the review and committing of several PRs to get us to this point. > > For all contributions going forward, developers, reviewers, and committers > need to make sure that PRs are verified by building with and running on > Java 8 and 11. > > To assist in this process, the Linux-based Travis CI build has been updated > to perform an en_US locale-based build of NiFi using AdoptOpenJDK 11.0.4 in > addition to the three locale-based builds (en_US, fr_FR, ja_JP) on OpenJDK > 8. > > Apache NiFi still has a minimum requirement of Java 8 and as long as that > requirement exists the released convenience binaries will be built with > Java 8. > > When building locally with Java 11, on all operating systems, a minimum JDK > version of 11.0.3 is required for a successful build. If you are building > on OSX and use AdoptOpenJDK 11.0.4, there's an issue [4] [5] with native > libraries. If you run into this issue, please downgrade your AdoptOpenJDK > 11 installation to 11.0.3. A updated release of AdoptOpenJDK 11.0.4 that > fixes the native library issues is forthcoming. There is a comment [6] on > AdoptOpenJDK issue 1211 that describes the downgrade process. > > Work still remains regarding Java 11. Some of the components that NiFi > uses are not completely compatible with Java 11, and JIRAs will be created > or linked to the Java 11 parent JIRA [1] to track the remaining changes. > > Several PRs not explicitly related to Java 11 were created and merged to > master ahead of NIFI-5176 [2] (PR 3404 [3]), and are listed as blockers on > NIFI-5176. All blockers have been resolved, as well as NIFI-5176. > > If you encounter an issue with the Java 11 build, or running the resulting > build on Java 11, please check NIFI-5174 to see if a JIRA for the issue has > already been filed, and create one if needed. > > [1] https://issues.apache.org/jira/browse/NIFI-5174 > [2] https://issues.apache.org/jira/browse/NIFI-5176 > [3] https://github.com/apache/nifi/pull/3404 > [4] https://github.com/AdoptOpenJDK/openjdk-build/issues/1206 > [5] https://github.com/AdoptOpenJDK/openjdk-build/issues/1211 > [6] > https://github.com/AdoptOpenJDK/openjdk-build/issues/1211#issuecomment-521392147
Re: Java 11 build support is live!
Yes, at the risk of adding nothing more than a "me too," I nevertheless wish to add my thanks. Good job! On 8/15/19 12:04 PM, Jeff wrote: Apache NiFi Developers, Apache NiFi can now be built with Java 11! Thanks to those that helped with the review and committing of several PRs to get us to this point. For all contributions going forward, developers, reviewers, and committers need to make sure that PRs are verified by building with and running on Java 8 and 11. To assist in this process, the Linux-based Travis CI build has been updated to perform an en_US locale-based build of NiFi using AdoptOpenJDK 11.0.4 in addition to the three locale-based builds (en_US, fr_FR, ja_JP) on OpenJDK 8. Apache NiFi still has a minimum requirement of Java 8 and as long as that requirement exists the released convenience binaries will be built with Java 8. When building locally with Java 11, on all operating systems, a minimum JDK version of 11.0.3 is required for a successful build. If you are building on OSX and use AdoptOpenJDK 11.0.4, there's an issue [4] [5] with native libraries. If you run into this issue, please downgrade your AdoptOpenJDK 11 installation to 11.0.3. A updated release of AdoptOpenJDK 11.0.4 that fixes the native library issues is forthcoming. There is a comment [6] on AdoptOpenJDK issue 1211 that describes the downgrade process. Work still remains regarding Java 11. Some of the components that NiFi uses are not completely compatible with Java 11, and JIRAs will be created or linked to the Java 11 parent JIRA [1] to track the remaining changes. Several PRs not explicitly related to Java 11 were created and merged to master ahead of NIFI-5176 [2] (PR 3404 [3]), and are listed as blockers on NIFI-5176. All blockers have been resolved, as well as NIFI-5176. If you encounter an issue with the Java 11 build, or running the resulting build on Java 11, please check NIFI-5174 to see if a JIRA for the issue has already been filed, and create one if needed. [1] https://issues.apache.org/jira/browse/NIFI-5174 [2] https://issues.apache.org/jira/browse/NIFI-5176 [3] https://github.com/apache/nifi/pull/3404 [4] https://github.com/AdoptOpenJDK/openjdk-build/issues/1206 [5] https://github.com/AdoptOpenJDK/openjdk-build/issues/1211 [6] https://github.com/AdoptOpenJDK/openjdk-build/issues/1211#issuecomment-521392147
RE: [EXT] Airflow and NiFi
User: "I want to change my production flow while it's running, you know, mid stream just route my content to a completely different flow, re-run data mid run through a new set of processors, fork it on the fly, you know, whatever I want anytime I want " NiFi: "Yeah, I can do that" Everyone Else: " I'm sorry... what?" -Original Message- From: Mike Thomsen Sent: Thursday, August 15, 2019 2:00 PM To: dev@nifi.apache.org Subject: [EXT] Airflow and NiFi Does anyone see any areas where the two can complement each other and where we might want to give users the ability to offload processing to Airflow? Curious since our poking around the docs lead us to conclude it was probably more "Airflow vs Spark+Oozie" than really competing with NiFi.
Re: [EXT] Airflow and NiFi
Yeah, pretty much. On Thu, Aug 15, 2019 at 4:02 PM Peter Wicks (pwicks) wrote: > User: "I want to change my production flow while it's running, you know, > mid stream just route my content to a completely different flow, re-run > data mid run through a new set of processors, fork it on the fly, you know, > whatever I want anytime I want " > > NiFi: "Yeah, I can do that" > > Everyone Else: " I'm sorry... what?" > > -Original Message- > From: Mike Thomsen > Sent: Thursday, August 15, 2019 2:00 PM > To: dev@nifi.apache.org > Subject: [EXT] Airflow and NiFi > > Does anyone see any areas where the two can complement each other and > where we might want to give users the ability to offload processing to > Airflow? > Curious since our poking around the docs lead us to conclude it was > probably more "Airflow vs Spark+Oozie" than really competing with NiFi. >
Re: Airflow and NiFi
We use NiFi for all our ETL, and have done so very happily for several years (including persuading partner organisations to adopt it); we have recently started using Airflow for scheduling periodic machine learning algorithm execution and retraining. My £0.02 - we probably could do everything we currently do in Airflow using NiFi, but for our Airflow use cases, things like task duration are interesting metrics to monitor and I don't immediately know how we would do this with NiFi. Plus the different user communities make two separate tools no bad thing. On Thu, 15 Aug 2019 at 21:00, Mike Thomsen wrote: > > Does anyone see any areas where the two can complement each other and where > we might want to give users the ability to offload processing to Airflow? > Curious since our poking around the docs lead us to conclude it was > probably more "Airflow vs Spark+Oozie" than really competing with NiFi.
Re: NiFi automation
The easiest way to do something like that would be to derive a new Docker image from our image. You can put whatever custom features you want from new NARs, to templates, to a whole flow.xml.gz in there. On Thu, Aug 15, 2019 at 7:32 AM Tomer Leibovich wrote: > Hi, > > I'm currently working on building a CI/CD for NiFi in my workplace. > My goal is to provide the developers with a ready-to-work platform without > the need for manual changes (import templates, configure registry and add > plugins) from their side. > > I hoped you can tell me if there is a way to specify in NiFi code to make > those changes. > > Thanks in advance, > Tomer >
Re: [EXT] Airflow and NiFi
Love it. Airflow I see as an orchestrator of making other systems do things. Nifi i see as very focused on acquiring, manipulating, and routing data between systems. In short, I believe theyre complimentary in a given architecture. Someone always tries to use one to do both of the things but each is better at their sweet spot. Thanks On Thu, Aug 15, 2019 at 4:07 PM Mike Thomsen wrote: > Yeah, pretty much. > > On Thu, Aug 15, 2019 at 4:02 PM Peter Wicks (pwicks) > wrote: > > > User: "I want to change my production flow while it's running, you know, > > mid stream just route my content to a completely different flow, re-run > > data mid run through a new set of processors, fork it on the fly, you > know, > > whatever I want anytime I want " > > > > NiFi: "Yeah, I can do that" > > > > Everyone Else: " I'm sorry... what?" > > > > -Original Message- > > From: Mike Thomsen > > Sent: Thursday, August 15, 2019 2:00 PM > > To: dev@nifi.apache.org > > Subject: [EXT] Airflow and NiFi > > > > Does anyone see any areas where the two can complement each other and > > where we might want to give users the ability to offload processing to > > Airflow? > > Curious since our poking around the docs lead us to conclude it was > > probably more "Airflow vs Spark+Oozie" than really competing with NiFi. > > >
Re: [EXT] Airflow and NiFi
Nifi is used for ETL and focus on transformations and data movement. Airflow is good for orchestration. Both tools have scheduling in common. Airflow is good at where the data is already available and you want to control jobs. If your focus is more like the data integertion, manipulation and transformation and movement then NiFi is a great choice. No doubt there is some overlap but ultimately it comes down to your use case. Thanks, Muazma On Thu, Aug 15, 2019, 1:15 PM Joe Witt wrote: > i blame spell check for ‘complimentary’ > > On Thu, Aug 15, 2019 at 4:11 PM Joe Witt wrote: > > > Love it. > > > > Airflow I see as an orchestrator of making other systems do things. Nifi > > i see as very focused on acquiring, manipulating, and routing data > between > > systems. > > > > In short, I believe theyre complimentary in a given architecture. > Someone > > always tries to use one to do both of the things but each is better at > > their sweet spot. > > > > Thanks > > > > On Thu, Aug 15, 2019 at 4:07 PM Mike Thomsen > > wrote: > > > >> Yeah, pretty much. > >> > >> On Thu, Aug 15, 2019 at 4:02 PM Peter Wicks (pwicks) > > >> wrote: > >> > >> > User: "I want to change my production flow while it's running, you > know, > >> > mid stream just route my content to a completely different flow, > re-run > >> > data mid run through a new set of processors, fork it on the fly, you > >> know, > >> > whatever I want anytime I want " > >> > > >> > NiFi: "Yeah, I can do that" > >> > > >> > Everyone Else: " I'm sorry... what?" > >> > > >> > -Original Message- > >> > From: Mike Thomsen > >> > Sent: Thursday, August 15, 2019 2:00 PM > >> > To: dev@nifi.apache.org > >> > Subject: [EXT] Airflow and NiFi > >> > > >> > Does anyone see any areas where the two can complement each other and > >> > where we might want to give users the ability to offload processing to > >> > Airflow? > >> > Curious since our poking around the docs lead us to conclude it was > >> > probably more "Airflow vs Spark+Oozie" than really competing with > NiFi. > >> > > >> > > >
