Yari, NiFi was coded and built against Apache Hive 1.2.1. Some of the API, files, and folders had changed between Hive 1.1.0 and Hive 1.2.1, such as the org.apache.hadoop.hive.ql.io.filters package being added in 1.2.0. Bringing in Apache ORC won't work for the same reason, it was split from Hive after 1.2.0. In fact in Hive 1.1.0 I don't think there is even a hive-orc JAR, it is just part of another Hive JAR. Some classes didn't get moved to Apache ORC from Hive until 2.0, and some got moved to different packages (hence your missing classes errors below).
I think the bottom line is that you'd probably have to change the source code in order to get NiFi to build against an older Hive version. Regards, Matt On Fri, Sep 9, 2016 at 8:50 AM, Yari Marchetti <[email protected]> wrote: > Thanks, Andre. This is the command I'm using: > > mvn install -DskipTests -Pcloudera -Dhadoop.version=2.6.0-cdh5.5.2 > -Dhive.version=1.1.0-cdh5.5.2 > > Adding as dependency: > > <dependency> > <groupId>org.apache.orc</groupId> > <artifactId>orc-core</artifactId> > <version>1.1.0</version> > </dependency> > > Yari > > > On 9 September 2016 at 13:23, Andre <[email protected]> wrote: >> >> Yari, >> >> Can you please confirm what command line are you using to compile? >> >> Happy yo try to reproduce. >> >> Cheers >> >> On Fri, Sep 9, 2016 at 8:49 PM, Yari Marchetti >> <[email protected]> wrote: >>> >>> Hi Peter/Andre, >>> I cherrypicked the commit and added the dependency on: >>> >>> <dependency> >>> <groupId>org.apache.orc</groupId> >>> <artifactId>orc-core</artifactId> >>> <version>1.1.0</version> >>> </dependency> >>> >>> (tried also with version 1.2.0) but stil no luck, as I keep on getting >>> compilation errors: >>> >>> [ERROR] >>> /nifi/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/hadoop/hive/ql/io/orc/OrcFlowFileWriter.java:[200,20] >>> error: cannot find symbol >>> [ERROR] symbol: variable OrcUtils >>> [ERROR] location: class OrcFlowFileWriter >>> [ERROR] >>> /nifi/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/hadoop/hive/ql/io/orc/OrcFlowFileWriter.java:[203,20] >>> error: cannot find symbol >>> [ERROR] symbol: variable OrcUtils >>> [ERROR] location: class OrcFlowFileWriter >>> [ERROR] >>> /nifi/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/hadoop/hive/ql/io/orc/OrcFlowFileWriter.java:[313,45] >>> error: cannot find symbol >>> [ERROR] symbol: method loadClass(String) >>> [ERROR] location: class JavaUtils >>> [ERROR] >>> /nifi/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/hadoop/hive/ql/io/orc/OrcFlowFileWriter.java:[494,21] >>> error: an enum switch case label must be the unqualified name of an >>> enumeration constant >>> [ERROR] >>> /nifi/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/hadoop/hive/ql/io/orc/OrcFlowFileWriter.java:[689,43] >>> error: cannot find symbol >>> [ERROR] symbol: variable BloomFilter >>> [ERROR] location: class OrcProto >>> [ERROR] >>> /nifi/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/hadoop/hive/ql/io/orc/OrcFlowFileWriter.java:[690,43] >>> error: cannot find symbol >>> [ERROR] symbol: variable BloomFilterIndex >>> [ERROR] location: class OrcProto >>> [ERROR] >>> /nifi/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/hadoop/hive/ql/io/orc/OrcFlowFileWriter.java:[691,87] >>> error: cannot find symbol >>> [ERROR] symbol: variable BLOOM_FILTER >>> [ERROR] location: class Kind >>> [ERROR] >>> /nifi/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/hadoop/hive/ql/io/orc/OrcFlowFileWriter.java:[692,34] >>> error: cannot find symbol >>> [ERROR] symbol: class BloomFilterIO >>> [ERROR] location: class TreeWriter >>> [ERROR] >>> /nifi/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/hadoop/hive/ql/io/orc/OrcFlowFileWriter.java:[726,52] >>> error: incompatible types: PositionedOutputStream cannot be converted to >>> OutStream >>> [ERROR] >>> /nifi/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/hadoop/hive/ql/io/orc/OrcFlowFileWriter.java:[728,50] >>> error: incompatible types: PositionedOutputStream cannot be converted to >>> OutStream >>> [ERROR] >>> /nifi/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/hadoop/hive/ql/io/orc/OrcFlowFileWriter.java:[804,19] >>> error: cannot find symbol >>> [ERROR] symbol: method setWriterTimezone(String) >>> [ERROR] location: variable builder of type Builder >>> [ERROR] >>> /nifi/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/hadoop/hive/ql/io/orc/NiFiOrcUtils.java:[450,32] >>> error: cannot find symbol >>> >>> do you have any idea? >>> >>> Thanks, >>> Yari >>> >>> On 8 September 2016 at 20:42, Peter Wicks (pwicks) <[email protected]> >>> wrote: >>>> >>>> Also, ORC File support was pulled out into its own library on the HIVE >>>> side. >>>> >>>> If you are willing to compile and run your own version you might need to >>>> include orc-core as a MVN dependency: >>>> https://mvnrepository.com/artifact/org.apache.orc/orc-core/1.2.0. >>>> >>>> >>>> >>>> >>>> >>>> From: Andre [mailto:[email protected]] >>>> Sent: Thursday, September 08, 2016 4:51 AM >>>> To: [email protected] >>>> Subject: Re: Nifi 1.0.0 compatibility with Hive 1.1.0 >>>> >>>> >>>> >>>> Yari, >>>> >>>> >>>> >>>> Is there any chance you can cherry pick commit >>>> 80224e3e5ed7ee7b09c4985a920a7fa393bff26c and try again? >>>> >>>> >>>> >>>> Post 1.0.0 there have been some changes to streamline compilation using >>>> vendor provided libraries. >>>> >>>> >>>> >>>> Cheers >>>> >>>> >>>> >>>> On Thu, Sep 8, 2016 at 8:44 PM, Yari Marchetti >>>> <[email protected]> wrote: >>>> >>>> Hello, >>>> >>>> I'd like to use Nifi 1.0.0 with Hive 1.1.0 (on CDH 5.5.2) but after some >>>> investigation I realised that the hive-jdbc driver included in Nifi is >>>> incompatible with the Hive version we're using (1.1.0 on CDH 5.5.2) as I'm >>>> getting the error: >>>> >>>> >>>> >>>> org.apache.hive.jdbc.HiveConnection Error opening session >>>> >>>> org.apache.thrift.TApplicationException: Required field >>>> 'client_protocol' is unset! Struct:TOpenSessionReq(client_protocol:null, >>>> configuration:{use:database=unifieddata}) >>>> >>>> >>>> >>>> So I just tried to recompile Nifi using the Cloudera profile 5.5.2 but >>>> compilation is failing: >>>> >>>> >>>> >>>> [ERROR] Failed to execute goal >>>> org.apache.maven.plugins:maven-compiler-plugin:3.2:compile >>>> (default-compile) >>>> on project nifi-hive-processors: Compilation failure: Compilation failure: >>>> >>>> [ERROR] >>>> /home/matteo/git/nifi/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/hadoop/hive/ql/io/orc/NiFiOrcUtils.java:[26,43] >>>> error: package org.apache.hadoop.hive.ql.io.filters does not exist >>>> >>>> [ERROR] >>>> /home/matteo/git/nifi/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/hadoop/hive/ql/io/orc/OrcFlowFileWriter.java:[45,43] >>>> error: package org.apache.hadoop.hive.ql.io.filters does not exist >>>> >>>> [ERROR] >>>> /home/matteo/git/nifi/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/hadoop/hive/ql/io/orc/OrcFlowFileWriter.java:[643,24] >>>> error: cannot find symbol >>>> >>>> [ERROR] symbol: class BloomFilterIO >>>> >>>> [ERROR] location: class TreeWriter >>>> >>>> [ERROR] >>>> /home/matteo/git/nifi/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/hadoop/hive/ql/io/orc/OrcFlowFileWriter.java:[645,30] >>>> error: cannot find symbol >>>> >>>> [ERROR] symbol: class BloomFilterIndex >>>> >>>> [ERROR] location: class OrcProto >>>> >>>> [ERROR] >>>> /home/matteo/git/nifi/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/hadoop/hive/ql/io/orc/OrcFlowFileWriter.java:[646,30] >>>> error: cannot find symbol >>>> >>>> [ERROR] symbol: class BloomFilter >>>> >>>> [ERROR] location: class OrcProto >>>> >>>> [ERROR] >>>> /home/matteo/git/nifi/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/hadoop/hive/ql/io/orc/NiFiOrcUtils.java:[450,32] >>>> error: cannot find symbol >>>> >>>> [ERROR] symbol: variable BloomFilterIO >>>> >>>> [ERROR] location: class NiFiOrcUtils >>>> >>>> [ERROR] >>>> /home/matteo/git/nifi/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/hadoop/hive/ql/io/orc/OrcFlowFileWriter.java:[200,20] >>>> error: cannot find symbol >>>> >>>> [ERROR] symbol: variable OrcUtils >>>> >>>> [ERROR] location: class OrcFlowFileWriter >>>> >>>> >>>> >>>> >>>> >>>> Is there any way to get Nifi to work with Hive 1.1.0 and CDH 5.5.2? >>>> >>>> >>>> >>>> Thanks, >>>> >>>> Yari >>>> >>>> >>> >>> >> >
