Juan, I'm not sure if NIFI-6912 [1] will help or not, but the imminent 1.11.0 release will have extra JARs (ADLS, Azure, AWS, etc.) in the Hive 3 bundle. Since you're using the Hive 1 bundle, then to include such dependencies you'd have to add them to the nifi-hive-nar/pom.xml and build a custom NAR manually.
Regards, Matt [1] https://issues.apache.org/jira/browse/NIFI-6912 On Thu, Jan 23, 2020 at 11:58 AM FABIAN Juan-antonio < [email protected]> wrote: > Hi, > > I've migrated one flow from using HDFS to use Minio as the storage layer. > Basically I deleted the PutHDFS processors, and now I only rely on > PutS3Object. After writing to Minio, I got stuck with PutHiveQL. To make it > work, I had to setup a HiveConnectionPool. I'm using the provided > hive-site.xml, for NiFi to be aware of our Hive metastore. > > Just for you to know, I started a NiFi cluster somewhere else, and > configured git as the flow provider, so I edited the "old" flow inside the > "new" NiFi deployment. When deploying NiFi, I manually copied the > hive-site.xml to a known location, and it's working fine, apparently. > > Even there's no PutHDFS in the "new" NiFi flow, I'm getting this error > from the PutHiveQL processor: > > 2020-01-23 16:46:09,402 ERROR [Timer-Driven Process Thread-7] > o.apache.nifi.processors.hive.PutHiveQL > PutHiveQL[id=bff6add0-cdbc-3c2f-b79f-b3a2438139c1] Failed to process > session due to org.apache.nifi.processor.exception.ProcessException: Failed > to process > StandardFlowFileRecord[uuid=3a089972-3b10-46bc-a880-bbf9cb322a6b,claim=StandardContentClaim > [resourceClaim=StandardResourceClaim[id=1579621741830-937, > container=default, section=937], offset=852885, > length=176],offset=0,name=3441009295635454,size=176] due to > java.sql.SQLException: org.apache.spark.sql.AnalysisException: > org.apache.hadoop.hive.ql.metadata.HiveException: > MetaException(message:java.lang.RuntimeException: > java.lang.ClassNotFoundException: Class > org.apache.hadoop.fs.adl.AdlFileSystem not found);: > org.apache.nifi.processor.exception.ProcessException: Failed to process > StandardFlowFileRecord[uuid=3a089972-3b10-46bc-a880-bbf9cb322a6b,claim=StandardContentClaim > [resourceClaim=StandardResourceClaim[id=1579621741830-937, > container=default, section=937], offset=852885, > length=176],offset=0,name=3441009295635454,size=176] due to > java.sql.SQLException: org.apache.spark.sql.AnalysisException: > org.apache.hadoop.hive.ql.metadata.HiveException: > MetaException(message:java.lang.RuntimeException: > java.lang.ClassNotFoundException: Class > org.apache.hadoop.fs.adl.AdlFileSystem not found); > org.apache.nifi.processor.exception.ProcessException: Failed to process > StandardFlowFileRecord[uuid=3a089972-3b10-46bc-a880-bbf9cb322a6b,claim=StandardContentClaim > [resourceClaim=StandardResourceClaim[id=1579621741830-937, > container=default, section=937], offset=852885, > length=176],offset=0,name=3441009295635454,size=176] due to > java.sql.SQLException: org.apache.spark.sql.AnalysisException: > org.apache.hadoop.hive.ql.metadata.HiveException: > MetaException(message:java.lang.RuntimeException: > java.lang.ClassNotFoundException: Class > org.apache.hadoop.fs.adl.AdlFileSystem not found); > at > org.apache.nifi.processor.util.pattern.ExceptionHandler.lambda$createOnGroupError$2(ExceptionHandler.java:226) > at > org.apache.nifi.processor.util.pattern.ExceptionHandler.lambda$createOnError$1(ExceptionHandler.java:179) > at > org.apache.nifi.processor.util.pattern.ExceptionHandler$OnError.lambda$andThen$0(ExceptionHandler.java:54) > at > org.apache.nifi.processor.util.pattern.ExceptionHandler$OnError.lambda$andThen$0(ExceptionHandler.java:54) > at > org.apache.nifi.processor.util.pattern.ExceptionHandler.execute(ExceptionHandler.java:148) > at > org.apache.nifi.processors.hive.PutHiveQL.lambda$new$4(PutHiveQL.java:223) > at org.apache.nifi.processor.util.pattern.Put.putFlowFiles(Put.java:59) > at org.apache.nifi.processor.util.pattern.Put.onTrigger(Put.java:102) > at > org.apache.nifi.processors.hive.PutHiveQL.lambda$onTrigger$6(PutHiveQL.java:289) > at > org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114) > at > org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrigger(RollbackOnFailure.java:184) > at org.apache.nifi.processors.hive.PutHiveQL.onTrigger(PutHiveQL.java:289) > at > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165) > at > org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203) > at > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.sql.SQLException: org.apache.spark.sql.AnalysisException: > org.apache.hadoop.hive.ql.metadata.HiveException: > MetaException(message:java.lang.RuntimeException: > java.lang.ClassNotFoundException: Class > org.apache.hadoop.fs.adl.AdlFileSystem not found); > at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:296) > at > org.apache.hive.jdbc.HivePreparedStatement.execute(HivePreparedStatement.java:98) > at > org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172) > at > org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172) > at > org.apache.nifi.processors.hive.PutHiveQL.lambda$null$3(PutHiveQL.java:251) > at > org.apache.nifi.processor.util.pattern.ExceptionHandler.execute(ExceptionHandler.java:127) > ... 17 common frames omitted > > Somewhere on the Internet I've read about some jars. So I've copied the > required jars to /usr/lib/hdinsight-datalake, but since none of my > processors need this, I think it was not necessary (I'm not manually > referencing them). However, the flow is not working. The jars I copied are: > > adls2-oauth2-token-provider-1.0.jar > hadoop-azure-datalake-2.7.3.2.6.5.10-2.jar > okhttp-2.7.5.jar > azure-data-lake-store-sdk-2.2.5.jar > jackson-core-2.7.8.jar > okio-1.6.0.jar > > Any help or insight is really appreciated, I'm just starting my NiFi > journey. > > Best, > > > *Juan A. Fabián Simón* > > Data Engineer > > Alstom > > Calle Martínez Villergas 49, ed. V - 28027 Madrid - Spain > > Office: +34 91 384 89 00 > > Email: <[email protected]> > [email protected] > > *www.alstom.com <http://www.alstom.com>* > > * <https://twitter.com/Alstom> > <https://www.linkedin.com/company/alstom/> > <https://www.facebook.com/ALSTOM/> <https://www.instagram.com/alstom> > <https://www.youtube.com/user/Alstom> * > > > ------------------------------ > CONFIDENTIALITY : This e-mail and any attachments are confidential and may > be privileged. If you are not a named recipient, please notify the sender > immediately and do not disclose the contents to another person, use it for > any purpose or store or copy the information in any medium. >
