This should be brought up in a new thread (and will do later) given that the original decision to not re-publish these processors was down to space issues. Is it worth of the back of 2.0 to re-think the idea of re-publishing a new version of a processor every single time a release is done, given that we are likely to have more processors as time goes this is somewhat delaying the inevitable in that perspective.
The overheads of doing so are not lost on me in the short term, but longer term I suspect it might help us. As an aside I've noticed that those processors are not listed in the docs on the website, which could limit usage especially for any one looking for an easy to use implementation of Apache Tika. Edward On Thu, Jan 20, 2022 at 7:41 PM Michal Tomaszewski < [email protected]> wrote: > Thank you very much for explanation. > > Regards, > Mike > > -----Original Message----- > From: Joe Witt <[email protected]> > Sent: czwartek, 20 stycznia 2022 19:36 > To: [email protected] > Subject: Re: Nifi 1.15.2 and 1.15.3 compilation errors > > Mike > > Yeah a bit more than two years ago we stopped including the nifi-kite-nar > by default. > https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance > > That means the processors in it are no longer included by default. > Since that time though we still build and published the nar to maven > artifact repos. > > What we've done now is we've removed the source from our repo and will not > publish the nar going forward. That said you'll be able to keep using the > already published nars to do this as you need. Until we release a nifi 2.0 > you should be safe from them breaking. > > But as Chris Sampsson just replied you almost certainly want to > discontinue using them anyway and switch to using the record based > processors. They are far more efficient, far more powerful, and are > extremely actively developed and maintained. > > Thanks > > On Thu, Jan 20, 2022 at 11:32 AM Chris Sampson <[email protected]> > wrote: > > > > If the kite nar is refined then that processor would indeed no longer be > present. > > > > A good alternative would be to use a Record based processor such as > ConvertRecord with a CSVReader and an AvroWriter. Such processors and > controller services typically offer much more powerful options than the > older processors such as those in the kite nar. > > > > > > Cheers, > > > > Chris Sampson > > > > On Thu, 20 Jan 2022, 18:08 Michal Tomaszewski, < > [email protected]> wrote: > >> > >> Joe, > >> > >> Does it mean ConvertCSVToAvro will disappear too? > >> > >> Or you discontinue only a part of org.apache.nifi.processors.kite > schema and the rest will be somehow moved to other schemas? > >> > >> > >> > >> Regards, > >> > >> Mike > >> > >> > >> > >> > >> > >> From: Joe Witt <[email protected]> > >> Sent: środa, 19 stycznia 2022 15:57 > >> To: [email protected] > >> Subject: Re: Nifi 1.15.2 and 1.15.3 compilation errors > >> > >> > >> > >> We should discontinue Kite processors from the build at all in 1.16.0. > >> > >> > >> > >> Ill file a Jira > >> > >> > >> > >> Thanks > >> > >> > >> > >> On Wed, Jan 19, 2022 at 7:47 AM David Handermann < > [email protected]> wrote: > >> > >> Mike, > >> > >> > >> > >> Thanks for the additional details. According to the original output, > the problem appears to be in the nifi-kite-processors module. The > nifi-kite-processors module includes a dependency on hive-hcatalog-core, > the version of which is based on the hive.version property. We should be > able to add an exclusion for log4j-core within that dependency definition, > which will allow it to compile with Hive 3.1.2. For the moment, if you > don't need the Kite processors, you could skip building nifi-kite-nar, or > you could manually add the following exclusion: > >> > >> <exclusion> > >> > >> <groupId>org.apache.logging.log4j</groupId> > >> > >> <artifactId>log4j-core</artifactId> > >> > >> <exclusion> > >> > >> Regards, > >> > >> David Handermann > >> > >> > >> > >> On Wed, Jan 19, 2022 at 8:27 AM Michal Tomaszewski < > [email protected]> wrote: > >> > >> Compilation with -Dhive.version=3.1.0 makes the problem. > >> mvn -e -X -T C2.0 clean install -DskipTests -Dhive.version=3.1.0 is > >> not successful. > >> We also tested -Dhive.version=3.1.2, that is most recent on Apache Hive > site. Also without success. > >> > >> Please note that without this parameter probably hive2 version is > compiled into nifi and HiveStreaming is not working then. > >> And hive 3 is most popular novadays. > >> > >> Regards, > >> Mike > >> > >> -----Original Message----- > >> From: Bryan Bende <[email protected]> > >> Sent: Wednesday, January 19, 2022 3:03 PM > >> To: [email protected] > >> Subject: Re: Nifi 1.15.2 and 1.15.3 compilation errors > >> > >> We did ban all use of log4j-core in favor of log4j-to-slf4j... > >> > >> https://issues.apache.org/jira/browse/NIFI-9483 > >> > >> With your build specifying the -D versions, it created a different > classpath during your build than what existed during the normal build of > the release. > >> > >> We should try to fix any issues like this though, so if you can figure > out which dependency path is the issue, the fix is to add some exclusions > and then to ensure that log4j-to-slf4j dependency is added. > >> > >> On Wed, Jan 19, 2022 at 8:48 AM Michal Tomaszewski < > [email protected]> wrote: > >> > > >> > OK, but there is information that log4j-core:jar:2.17.1 is blocked: > >> > > >> > [WARNING] Rule 0: > org.apache.maven.plugins.enforcer.BannedDependencies failed with message: > >> > > >> > Found Banned Dependency: > >> > org.apache.logging.log4j:log4j-core:jar:2.17.1 > >> > > >> > while this version is not impacted by log4j security vulnerability: > >> > > >> > https://logging.apache.org/log4j/2.x/security.html > >> > > >> > > >> > > >> > So for sure it shouldn’t be banned. > >> > > >> > > >> > > >> > Regards, > >> > > >> > Mike > >> > > >> > From: Pierre Villard <[email protected]> > >> > Sent: Wednesday, January 19, 2022 1:11 PM > >> > To: [email protected] > >> > Subject: Re: Nifi 1.15.2 and 1.15.3 compilation errors > >> > > >> > > >> > > >> > We recently changed the build to fail if it's including problematic > transitive dependencies for log4j. I think those are coming with the > specific versions of HBase/Hive you're using. Not a maven expert but I > guess there is a way to exclude the step "enforce-banned-dependencies" from > the build. > >> > > >> > > >> > > >> > Le mer. 19 janv. 2022 à 12:57, Michal Tomaszewski < > [email protected]> a écrit : > >> > > >> > Hi Edward, > >> > > >> > Full mvn dependency:tree log enclosed. > >> > > >> > We’re compiling using command: > >> > > >> > mvn -e -X -T C2.0 clean install -DskipTests -Dhive.version=3.1.0 > >> > -Dhbase.version=2.0.2 > >> > > >> > > >> > > >> > Regards, > >> > > >> > Mike > >> > > >> > > >> > > >> > From: Edward Armes <[email protected]> > >> > Sent: Wednesday, January 19, 2022 12:34 PM > >> > To: [email protected] > >> > Subject: Re: Nifi 1.15.2 and 1.15.3 compilation errors > >> > > >> > > >> > > >> > Hi Mike, > >> > > >> > > >> > > >> > Can you give the full output you get from mvn dependency:tree as we > 15.2 and 3 was done to remove log4j dependences from Nifi as a safety > measure due the recent log4shell issues. > >> > > >> > > >> > > >> > Thanks > >> > > >> > > >> > > >> > Edward > >> > > >> > > >> > > >> > On Wed, Jan 19, 2022 at 10:24 AM Michal Tomaszewski < > [email protected]> wrote: > >> > > >> > Hi, > >> > > >> > We are trying to compile nifi from sources. > >> > > >> > There is no problem with 1.15.1 compilation but in case of 1.15.2 and > 1.15.3 there are compilation errors. > >> > > >> > > >> > > >> > Can you suggest how to solve this problem? > >> > > >> > > >> > > >> > mvn dependency:tree has no errors: > >> > > >> > > >> > > >> > > >> > > >> > $ tail log1.txt > >> > > >> > [INFO] nifi-toolkit-flowfile-repo ......................... SUCCESS > >> > [ > >> > 0.005 s] > >> > > >> > [INFO] nifi-toolkit-flowanalyzer .......................... SUCCESS > >> > [ > >> > 0.008 s] > >> > > >> > [INFO] nifi-toolkit-assembly .............................. SUCCESS > >> > [ > >> > 0.463 s] > >> > > >> > [INFO] nifi-toolkit-api ................................... SUCCESS > >> > [ > >> > 0.023 s] > >> > > >> > [INFO] > >> > ------------------------------------------------------------------- > >> > --- > >> > -- > >> > > >> > [INFO] BUILD SUCCESS > >> > > >> > [INFO] > >> > ------------------------------------------------------------------- > >> > --- > >> > -- > >> > > >> > [INFO] Total time: 02:41 min > >> > > >> > [INFO] Finished at: 2022-01-19T08:51:46+01:00 > >> > > >> > > >> > > >> > > >> > > >> > Compilation of branch rel/nifi-1.15.3: > >> > > >> > > >> > > >> > [DEBUG] Adding ERROR message due to exception > >> > > >> > org.apache.maven.enforcer.rule.api.EnforcerRuleException: Found > >> > Banned > >> > Dependency: org.apache.logging.log4j:log4j-core:jar:2.17.1 > >> > > >> > Use 'mvn dependency:tree' to locate the source of the banned > dependencies. > >> > > >> > at > >> > org.apache.maven.plugins.enforcer.AbstractBanDependencies.execute > >> > (AbstractBanDependencies.java:113) > >> > > >> > at org.apache.maven.plugins.enforcer.EnforceMojo.execute > >> > (EnforceMojo.java:200) > >> > > >> > at > >> > org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo > >> > (DefaultBuildPluginManager.java:137) > >> > > >> > at org.apache.maven.lifecycle.internal.MojoExecutor.execute > >> > (MojoExecutor.java:210) > >> > > >> > at org.apache.maven.lifecycle.internal.MojoExecutor.execute > >> > (MojoExecutor.java:156) > >> > > >> > at org.apache.maven.lifecycle.internal.MojoExecutor.execute > >> > (MojoExecutor.java:148) > >> > > >> > at > >> > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildPro > >> > jec t (LifecycleModuleBuilder.java:117) > >> > > >> > at > >> > org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThre > >> > ade dBuilder$1.call (MultiThreadedBuilder.java:196) > >> > > >> > at > >> > org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThre > >> > ade dBuilder$1.call (MultiThreadedBuilder.java:186) > >> > > >> > at java.util.concurrent.FutureTask.run (FutureTask.java:266) > >> > > >> > at java.util.concurrent.Executors$RunnableAdapter.call > >> > (Executors.java:511) > >> > > >> > at java.util.concurrent.FutureTask.run (FutureTask.java:266) > >> > > >> > at java.util.concurrent.ThreadPoolExecutor.runWorker > >> > (ThreadPoolExecutor.java:1149) > >> > > >> > at java.util.concurrent.ThreadPoolExecutor$Worker.run > >> > (ThreadPoolExecutor.java:624) > >> > > >> > at java.lang.Thread.run (Thread.java:748) > >> > > >> > [WARNING] Rule 0: > org.apache.maven.plugins.enforcer.BannedDependencies failed with message: > >> > > >> > Found Banned Dependency: > >> > org.apache.logging.log4j:log4j-core:jar:2.17.1 > >> > > >> > Use 'mvn dependency:tree' to locate the source of the banned > dependencies. > >> > > >> > [INFO] > >> > > >> > > >> > > >> > …. > >> > > >> > > >> > > >> > [INFO] > >> > ------------------------------------------------------------------- > >> > --- > >> > -- > >> > > >> > [ERROR] Failed to execute goal > >> > org.apache.maven.plugins:maven-enforcer-plugin:3.0.0:enforce > >> > (enforce-banned-dependencies) on project nifi-kite-processors: Some > >> > Enforcer rules have failed. Look above for specific messages > >> > explaining why the rule failed. -> [Help 1] > >> > > >> > org.apache.maven.lifecycle.LifecycleExecutionException: Failed to > execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.0.0:enforce > (enforce-banned-dependencies) on project nifi-kite-processors: Some > Enforcer rules have failed. Look above for specific messages explaining why > the rule failed. > >> > > >> > at org.apache.maven.lifecycle.internal.MojoExecutor.execute > >> > (MojoExecutor.java:215) > >> > > >> > at org.apache.maven.lifecycle.internal.MojoExecutor.execute > >> > (MojoExecutor.java:156) > >> > > >> > at org.apache.maven.lifecycle.internal.MojoExecutor.execute > >> > (MojoExecutor.java:148) > >> > > >> > at > >> > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildPro > >> > jec t (LifecycleModuleBuilder.java:117) > >> > > >> > at > >> > org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThre > >> > ade dBuilder$1.call (MultiThreadedBuilder.java:196) > >> > > >> > at > >> > org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThre > >> > ade dBuilder$1.call (MultiThreadedBuilder.java:186) > >> > > >> > at java.util.concurrent.FutureTask.run (FutureTask.java:266) > >> > > >> > at java.util.concurrent.Executors$RunnableAdapter.call > >> > (Executors.java:511) > >> > > >> > at java.util.concurrent.FutureTask.run (FutureTask.java:266) > >> > > >> > at java.util.concurrent.ThreadPoolExecutor.runWorker > >> > (ThreadPoolExecutor.java:1149) > >> > > >> > at java.util.concurrent.ThreadPoolExecutor$Worker.run > >> > (ThreadPoolExecutor.java:624) > >> > > >> > at java.lang.Thread.run (Thread.java:748) > >> > > >> > Caused by: org.apache.maven.plugin.MojoExecutionException: Some > Enforcer rules have failed. Look above for specific messages explaining why > the rule failed. > >> > > >> > at org.apache.maven.plugins.enforcer.EnforceMojo.execute > >> > (EnforceMojo.java:255) > >> > > >> > at > >> > org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo > >> > (DefaultBuildPluginManager.java:137) > >> > > >> > at org.apache.maven.lifecycle.internal.MojoExecutor.execute > >> > (MojoExecutor.java:210) > >> > > >> > at org.apache.maven.lifecycle.internal.MojoExecutor.execute > >> > (MojoExecutor.java:156) > >> > > >> > at org.apache.maven.lifecycle.internal.MojoExecutor.execute > >> > (MojoExecutor.java:148) > >> > > >> > at > >> > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildPro > >> > jec t (LifecycleModuleBuilder.java:117) > >> > > >> > at > >> > org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThre > >> > ade dBuilder$1.call (MultiThreadedBuilder.java:196) > >> > > >> > at > >> > org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThre > >> > ade dBuilder$1.call (MultiThreadedBuilder.java:186) > >> > > >> > at java.util.concurrent.FutureTask.run (FutureTask.java:266) > >> > > >> > at java.util.concurrent.Executors$RunnableAdapter.call > >> > (Executors.java:511) > >> > > >> > at java.util.concurrent.FutureTask.run (FutureTask.java:266) > >> > > >> > at java.util.concurrent.ThreadPoolExecutor.runWorker > >> > (ThreadPoolExecutor.java:1149) > >> > > >> > at java.util.concurrent.ThreadPoolExecutor$Worker.run > >> > (ThreadPoolExecutor.java:624) > >> > > >> > at java.lang.Thread.run (Thread.java:748) > >> > > >> > [ERROR] > >> > > >> > [ERROR] > >> > > >> > [ERROR] For more information about the errors and possible solutions, > please read the following articles: > >> > > >> > [ERROR] [Help 1] > >> > http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionExcep > >> > tio > >> > n > >> > > >> > [ERROR] > >> > > >> > [ERROR] After correcting the problems, you can resume the build > >> > with the command > >> > > >> > [ERROR] mvn <args> -rf :nifi-kite-processors > >> > > >> > > >> > > >> > Regards, > >> > > >> > Mike >
