Hi, Having two different versions of the same jar is not good. I hope those are not put manually.
./oozie/oozie-server/webapps/oozie/WEB-INF/lib/httpclient-4.2.5.jar ./oozie/oozie-server/webapps/oozie/WEB-INF/lib/httpclient-4.3.jar ./oozie/oozie-server/webapps/oozie/WEB-INF/lib/httpcore-4.3.jar ./oozie/oozie-server/webapps/oozie/WEB-INF/lib/httpcore-4.2.4.jar --------------- I am using an higher version of the HDP 2.4.2.0-130. Here are the results: cd /usr/hdp/2.4.2.0-130/ find . -name httpcore* ./hadoop-mapreduce/httpcore-4.2.5.jar ./hive/lib/httpcore-4.4.jar ./zookeeper/lib/httpcore-4.2.3.jar ./hbase/lib/httpcore-4.2.5.jar ./oozie/libtools/httpcore-4.3.jar ./oozie/lib/httpcore-4.3.jar ./oozie/libserver/httpcore-4.3.jar ./hadoop/client/httpcore.jar ./hadoop/client/httpcore-4.2.5.jar ./hadoop/lib/httpcore-4.2.5.jar ./hadoop-yarn/lib/httpcore-4.2.5.jar ./falcon/client/lib/httpcore-4.2.4.jar find . -name httpclient* ./hadoop-mapreduce/httpclient-4.2.5.jar ./hive/lib/httpclient-4.4.jar ./zookeeper/lib/httpclient-4.2.3.jar ./hbase/lib/httpclient-4.2.5.jar ./oozie/lib/httpclient-4.3.jar ./oozie/libserver/httpclient-4.3.jar ./hadoop/client/httpclient-4.2.5.jar ./hadoop/client/httpclient.jar ./hadoop/lib/httpclient-4.2.5.jar ./hadoop-yarn/lib/httpclient-4.2.5.jar ./falcon/client/lib/httpclient-4.2.5.jar Thanks, Abhishek On Mon, Apr 18, 2016 at 5:53 PM, Morgrim Murdargent <[email protected] > wrote: > Re Abhishek. > > HDP version : > 2.3.2.0-2950 > > ### > # pwd > /usr/hdp/2.3.2.0-2950 > [root@inbdfot21 2.3.2.0-2950]# find . -name httpcore* > ./hadoop-yarn/lib/httpcore-4.2.5.jar > ./hadoop-httpfs/webapps/webhdfs/WEB-INF/lib/httpcore-4.2.5.jar > ./hadoop-mapreduce/httpcore-4.2.5.jar > ./hive/lib/httpcore-4.4.jar > ./hadoop/client/httpcore.jar > ./hadoop/client/httpcore-4.2.5.jar > ./hadoop/lib/httpcore-4.2.5.jar > ./slider/lib/httpcore-4.2.5.jar > ./hbase/lib/httpcore-4.2.5.jar > ./zookeeper/lib/httpcore-4.2.3.jar > ./storm/extlib-daemon/httpcore-4.2.5.jar > ./falcon/client/lib/httpcore-4.2.4.jar > ./oozie/oozie-server/webapps/oozie/WEB-INF/lib/httpcore-4.3.jar > ./oozie/oozie-server/webapps/oozie/WEB-INF/lib/httpcore-4.2.4.jar > ./oozie/lib/httpcore-4.2.4.jar > ./oozie/libserver/httpcore-4.3.jar > ./oozie/libserver/httpcore-4.2.4.jar > ./oozie/libtools/httpcore-4.4.jar > ./oozie/share/lib/sqoop/httpcore-4.2.4.jar > ./oozie/share/lib/hive/httpcore-4.2.4.jar > ./oozie/share/lib/pig/httpcore-4.2.4.jar > ./oozie/share/lib/oozie/httpcore-4.2.4.jar > ./oozie/share/lib/distcp/httpcore-4.2.4.jar > ./oozie/share/lib/spark/httpcore-4.2.4.jar > ./oozie/share/lib/hcatalog/httpcore-4.2.4.jar > ./oozie/share/lib/mapreduce-streaming/httpcore-4.2.4.jar > ./oozie/share/lib/hive2/httpcore-4.4.jar > ./ranger-storm-plugin/lib/httpcore-4.2.5.jar > ### > > #### > find . -name httpclient* > ./hadoop-yarn/lib/httpclient-4.2.5.jar > ./hadoop-httpfs/webapps/webhdfs/WEB-INF/lib/httpclient-4.2.5.jar > ./hadoop-mapreduce/httpclient-4.2.5.jar > ./hive/lib/httpclient-4.4.jar > ./hadoop/client/httpclient-4.2.5.jar > ./hadoop/client/httpclient.jar > ./hadoop/lib/httpclient-4.2.5.jar > ./slider/lib/httpclient-4.2.5.jar > ./hbase/lib/httpclient-4.2.5.jar > ./zookeeper/lib/httpclient-4.2.3.jar > ./storm/extlib-daemon/httpclient-4.2.5.jar > ./falcon/client/lib/httpclient-4.2.5.jar > ./oozie/oozie-server/webapps/oozie/WEB-INF/lib/httpclient-4.2.5.jar > ./oozie/oozie-server/webapps/oozie/WEB-INF/lib/httpclient-4.3.jar > ./oozie/lib/httpclient-4.2.5.jar > ./oozie/libserver/httpclient-4.2.5.jar > ./oozie/libserver/httpclient-4.3.jar > ./oozie/libtools/httpclient-4.2.5.jar > ./oozie/share/lib/sqoop/httpclient-4.2.5.jar > ./oozie/share/lib/hive/httpclient-4.2.5.jar > ./oozie/share/lib/pig/httpclient-4.2.5.jar > ./oozie/share/lib/oozie/httpclient-4.2.5.jar > ./oozie/share/lib/distcp/httpclient-4.2.5.jar > ./oozie/share/lib/spark/httpclient-4.2.5.jar > ./oozie/share/lib/hcatalog/httpclient-4.2.5.jar > ./oozie/share/lib/mapreduce-streaming/httpclient-4.2.5.jar > ./oozie/share/lib/hive2/httpclient-4.4.jar > ./ranger-storm-plugin/lib/httpclient-4.2.5.jar > ### > > Best regards. > > Morgrim. > > > > > > On Mon, Apr 18, 2016 at 2:07 PM, Abhishek Bafna <[email protected]> > wrote: > > > Hi, > > > > As you have mentioned you are using HDP in one of the mail chain, Please > > execute the below commands on the machine which have oozie server running > > and see the versions of the jars and path they are coming from: > > > > cd /use/hdp/<version> > > find . -name httpcore* > > find . -name httpclient* > > > > Thanks, > > Abhishek > > > > On Mon, Apr 18, 2016 at 3:46 PM, Morgrim Murdargent < > > [email protected] > > > wrote: > > > > > Hello Abhishek ! > > > > > > Thank you for your answer. > > > > > > Hm, this is not clear for me. > > > > > > ### > > > The httpcore-4.2.5 and httpclient-4.2.5 version is coming from the > hadoop > > > classpath not from the oozie. Because Java action executes the main() > > > method from the launcher mapper only. You can check the oozie httpc* > > > version from the share lib with below command. BTW, none of these are > > > getting used here. > > > ### > > > > > > I'm pretty sure these two libs are coming from the oozie folder and not > > the > > > hadoop classpath. Because by replacing both http-core-4.2.5 and > > > httpclient-4.2.5 with their 4.4 version in the oozie folder in the > oozie > > > sharelibs, it solved the problem at 100%. > > > > > > Best regards. > > > > > > Gwenael Le Barzic > > > > > > On Fri, Apr 15, 2016 at 4:40 PM, Abhishek Bafna <[email protected]> > > > wrote: > > > > > > > Hi Morgrim, > > > > > > > > The httpcore-4.2.5 and httpclient-4.2.5 version is coming from the > > hadoop > > > > classpath not from the oozie. Because Java action executes the main() > > > > method from the launcher mapper only. You can check the oozie httpc* > > > > version from the share lib with below command. BTW, none of these are > > > > getting used here. > > > > > > > > hdfs dfs -ls /user/oozie/share/lib/lib_*/* | grep httpc > > > > > > > > I guess your java action needs the httpc* jars, so they must be > present > > > in > > > > the pom. Copy these to the lib\ directory of your workflow and set > the > > ' > > > > oozie.launcher.mapreduce.user.classpath.first' to 'true' in your java > > > > action 'configuration' element. > > > > > > > > Also note that, when you set the property, > > > > oozie.action.sharelib.for.java=hive2, you are loading the hive2 > > > > configuration in the classpath, which is not required. > > > > > > > > You should always provide the custom jars required by the application > > in > > > > the workflow application's lib/ directory. > > > > > > > > Check this for more information: > > > > https://issues.apache.org/jira/browse/OOZIE-2066 > > > > > > > > Thanks, > > > > Abhishek > > > > > > > > On Thu, Apr 7, 2016 at 9:46 PM, Morgrim Murdargent < > > > > [email protected]> > > > > wrote: > > > > > > > > > Hey Aaron. > > > > > > > > > > Thank you for your answer. > > > > > > > > > > Ok for the moment the solution I'm trying is to replace the libs > > > > > httpcore-4.2.5 and httpclient-4.2.5 in the oozie folder of the > oozie > > > > > sharelibs by the version 4.4. > > > > > My workflow ended successfully. > > > > > > > > > > I'm doing additionnal tests with the other workflows that I have on > > my > > > > > cluster to see if they are impacted. > > > > > > > > > > To the Oozie developpers, may you help me on this topic please ? > > > > > > > > > > Best regards. > > > > > > > > > > Morgrim. > > > > > > > > > > On Thu, Apr 7, 2016 at 4:29 PM, Aaron.Dossett < > > > [email protected]> > > > > > wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > > 1. Am I right ? Is the oozie folder of the oozie sharelibs > loaded > > > in > > > > > the > > > > > > > classpath in case of a java action ? > > > > > > > > > > > > This has given me all sorts of problems too, please see a > > discussion > > > I > > > > > had > > > > > > on the Hortonworks Community Connection: > > > > > > > > > > > > > > > > > > > > > https://community.hortonworks.com/questions/25796/oozie-share-lib-for-oozie > > > > > > -contains-a-lot-of-jar-fi.html > > > > > > > > > > > > As far as I can tell there is no way around this. If anyone on > the > > > > list > > > > > > has a suggestion, I would love to hear it. > > > > > > > > > > > > Thanks, Aaron > > > > > > > > > > > > On 4/7/16, 9:09 AM, "Morgrim Murdargent" < > [email protected]> > > > > > wrote: > > > > > > > > > > > > >Hello ! > > > > > > > > > > > > > >I have an oozie workflow with a java action. > > > > > > >In this java action, I need the following version of httpclient > > and > > > > > > >httpcore : > > > > > > >- httpclient : 4.4 > > > > > > >- httpcore : 4.4 > > > > > > > > > > > > > >I found that in the oozie sharelibs, only the hive2 folder has > > these > > > > > > >versions of httpclient and httpcore. > > > > > > >So I added to my workflow the following property : > > > > > > >oozie.action.sharelib.for.java=hive2 > > > > > > > > > > > > > >It solved the first part of my problem with httpclient but did > not > > > > with > > > > > > >httpcore. > > > > > > >When I check the classpath, I can see two versions for each lib > > > > httpcore > > > > > > >and httpclient : > > > > > > >- httpclient : 4.2.5 and 4.4 > > > > > > >- httpcore : 4.2.5 and 4.4 > > > > > > > > > > > > > >By some chances, the httpclient-4.4 is loaded first, so it > solved > > my > > > > > > >problem with httpclient. > > > > > > >But unfortunately, httpcore-4.2.5 is loaded before the version > > 4.4, > > > > so I > > > > > > >got an error message : > > > > > > >### > > > > > > >Failing Oozie Launcher, Main class > > > > > > >[org.apache.oozie.action.hadoop.JavaMain], main() threw > exception, > > > > > > >java.lang.NoSuchFieldError: INSTANCE > > > > > > >org.apache.oozie.action.hadoop.JavaMainException: > > > > > > >java.lang.NoSuchFieldError: INSTANCE > > > > > > >### > > > > > > > > > > > > > >I didn't find neither the httpclient-4.2.5 nor the httpcore > 4.2.5 > > in > > > > > hive2 > > > > > > >folder of oozie sharelib. > > > > > > >But I found them in the oozie folder of oozie sharelib. > > > > > > > > > > > > > >And when we do java action in oozie workflow, the oozie folder > is > > > > > > >automatically loaded in the classpath. > > > > > > > > > > > > > >My questions are the following : > > > > > > >1. Am I right ? Is the oozie folder of the oozie sharelibs > loaded > > in > > > > the > > > > > > >classpath in case of a java action ? > > > > > > >2. How can I have the latest versions of the libs loaded first > in > > my > > > > > > >classpath ? > > > > > > >3. Can I load in the classpath only the libs contained in hive2 > > > > folder ? > > > > > > >If > > > > > > >yes, how ? > > > > > > >4. Can I safely replace the version 4.2.5 of httpclient and > > httpcore > > > > in > > > > > > >the > > > > > > >oozie folder of the oozie sharelibs by the version 4.4 ? > > > > > > >5. Can I do that for all the folders of the oozie sharelibs ? > Why > > do > > > > we > > > > > > >have different versions of libraries in each folder of the oozie > > > > > > >sharelibs ? > > > > > > > > > > > > > >Thank you in advanced for your help ! > > > > > > > > > > > > > >Best regards. > > > > > > > > > > > > > >Morgrim. > > > > > > > > > > > > > > > > > > > > > > > > > > >
