Re: Adding JAR in Hive classpath

[Sathi Chowdhury]

I had a similar problem with one of out udf libs using commons-lang3…
The only way it worked was to create a fat jar with shaded class of and place 
it under hdfs ….


How you will shade , an example below
<configuration>
        <transformers>
        <transformer 
implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
        </transformers>
        <createDependencyReducedPom>false</createDependencyReducedPom>
        <relocations>
        <relocation>
        <!-- MR in CDH5.1 puts Guava 11 on the classpath, which breaks 
DQDataTypes -->
        <pattern>com.google.common</pattern>
        <shadedPattern>org.shaded.google.common</shadedPattern>
        </relocation>
        <relocation>
        <pattern>org.apache.commons</pattern>
        <shadedPattern>org.shaded.apache.commons</shadedPattern>
        </relocation>
        </relocations>
        <artifactSet>
        <includes>
        <include>x.y.analytics:z-libs</include> is your lib jar
        <include>com.google.guava:guava</include>
        <include>org.apache.commons:commons-lang3</include>
        <include>commons-io:commons-io</include>
        <include>commons-logging:commons-logging</include>
        <include>commons-codec:commons-codec</include>
        <include>org.apache.commons:commons-math3</include>
        <include>commons-cli:commons-cli</include>
        <include>commons-lang:commons-lang</include>

        </includes>
        </artifactSet>
        </configuration>
Additional note
If you are running this with oozie place this is oozie user lib area e….user 
lib is where your workflow.xml is need to have a subfolder “lib” and keep it 
under this lib folder
HTH
Sathi
From: Akansha Jain
Reply-To: "user@hive.apache.org<mailto:user@hive.apache.org>"
Date: Friday, September 4, 2015 at 2:58 PM
To: "user@hive.apache.org<mailto:user@hive.apache.org>"
Subject: Adding JAR in Hive classpath


Hi All,

I am facing an issue with the Hive classpath. I have written a UDAF which is 
using common maths 3.3 version. So, while creating temporary function I first 
add common maths 3.3 and then UDAF jar and create temporary function.

There is another version of common math 3.1 present under HADOOP_HOME/lib 
directory.

Now the problem is even after adding common maths 3.3 in Hive classpath (by ADD 
JAR ..) , Hive is picking common maths 3.1 version from HADOOP_HOME/lib folder. 
How do I remove 3.1 version from classpath.

I tried using DELETE JAR ... but it doesnt work.

Is there any way, I can force Hive to pick my version and not the one with 
Hadoop lib. Any help is appreciated.

Thanks

AJ​