Re Abhishe. I checked also the HDP 2.3.4 and it seems that these two libs have disappeared from the oozie folder in the oozier sharelibs.
So I guess my problem is HDP 2.3.2 in fact. Just another quick question then, do you know if oozie 4.2.0 (the one embedded in HDP2.3.2) used these two libs from the oozie folder of the oozie sharelibs ? I mean the httpcore-4.2.5 and the httpclient-4.2.5 ? Best regards. Morgrim On Mon, Apr 18, 2016 at 4:45 PM, Abhishek Bafna <[email protected]> wrote: > Hi, > > Having two different versions of the same jar is not good. I hope those are > not put manually. > > ./oozie/oozie-server/webapps/oozie/WEB-INF/lib/httpclient-4.2.5.jar > ./oozie/oozie-server/webapps/oozie/WEB-INF/lib/httpclient-4.3.jar > > ./oozie/oozie-server/webapps/oozie/WEB-INF/lib/httpcore-4.3.jar > ./oozie/oozie-server/webapps/oozie/WEB-INF/lib/httpcore-4.2.4.jar > > --------------- > I am using an higher version of the HDP 2.4.2.0-130. Here are the results: > > cd /usr/hdp/2.4.2.0-130/ > > find . -name httpcore* > ./hadoop-mapreduce/httpcore-4.2.5.jar > ./hive/lib/httpcore-4.4.jar > ./zookeeper/lib/httpcore-4.2.3.jar > ./hbase/lib/httpcore-4.2.5.jar > ./oozie/libtools/httpcore-4.3.jar > ./oozie/lib/httpcore-4.3.jar > ./oozie/libserver/httpcore-4.3.jar > ./hadoop/client/httpcore.jar > ./hadoop/client/httpcore-4.2.5.jar > ./hadoop/lib/httpcore-4.2.5.jar > ./hadoop-yarn/lib/httpcore-4.2.5.jar > ./falcon/client/lib/httpcore-4.2.4.jar > > find . -name httpclient* > ./hadoop-mapreduce/httpclient-4.2.5.jar > ./hive/lib/httpclient-4.4.jar > ./zookeeper/lib/httpclient-4.2.3.jar > ./hbase/lib/httpclient-4.2.5.jar > ./oozie/lib/httpclient-4.3.jar > ./oozie/libserver/httpclient-4.3.jar > ./hadoop/client/httpclient-4.2.5.jar > ./hadoop/client/httpclient.jar > ./hadoop/lib/httpclient-4.2.5.jar > ./hadoop-yarn/lib/httpclient-4.2.5.jar > ./falcon/client/lib/httpclient-4.2.5.jar > > Thanks, > Abhishek > > On Mon, Apr 18, 2016 at 5:53 PM, Morgrim Murdargent < > [email protected] > > wrote: > > > Re Abhishek. > > > > HDP version : > > 2.3.2.0-2950 > > > > ### > > # pwd > > /usr/hdp/2.3.2.0-2950 > > [root@inbdfot21 2.3.2.0-2950]# find . -name httpcore* > > ./hadoop-yarn/lib/httpcore-4.2.5.jar > > ./hadoop-httpfs/webapps/webhdfs/WEB-INF/lib/httpcore-4.2.5.jar > > ./hadoop-mapreduce/httpcore-4.2.5.jar > > ./hive/lib/httpcore-4.4.jar > > ./hadoop/client/httpcore.jar > > ./hadoop/client/httpcore-4.2.5.jar > > ./hadoop/lib/httpcore-4.2.5.jar > > ./slider/lib/httpcore-4.2.5.jar > > ./hbase/lib/httpcore-4.2.5.jar > > ./zookeeper/lib/httpcore-4.2.3.jar > > ./storm/extlib-daemon/httpcore-4.2.5.jar > > ./falcon/client/lib/httpcore-4.2.4.jar > > ./oozie/oozie-server/webapps/oozie/WEB-INF/lib/httpcore-4.3.jar > > ./oozie/oozie-server/webapps/oozie/WEB-INF/lib/httpcore-4.2.4.jar > > ./oozie/lib/httpcore-4.2.4.jar > > ./oozie/libserver/httpcore-4.3.jar > > ./oozie/libserver/httpcore-4.2.4.jar > > ./oozie/libtools/httpcore-4.4.jar > > ./oozie/share/lib/sqoop/httpcore-4.2.4.jar > > ./oozie/share/lib/hive/httpcore-4.2.4.jar > > ./oozie/share/lib/pig/httpcore-4.2.4.jar > > ./oozie/share/lib/oozie/httpcore-4.2.4.jar > > ./oozie/share/lib/distcp/httpcore-4.2.4.jar > > ./oozie/share/lib/spark/httpcore-4.2.4.jar > > ./oozie/share/lib/hcatalog/httpcore-4.2.4.jar > > ./oozie/share/lib/mapreduce-streaming/httpcore-4.2.4.jar > > ./oozie/share/lib/hive2/httpcore-4.4.jar > > ./ranger-storm-plugin/lib/httpcore-4.2.5.jar > > ### > > > > #### > > find . -name httpclient* > > ./hadoop-yarn/lib/httpclient-4.2.5.jar > > ./hadoop-httpfs/webapps/webhdfs/WEB-INF/lib/httpclient-4.2.5.jar > > ./hadoop-mapreduce/httpclient-4.2.5.jar > > ./hive/lib/httpclient-4.4.jar > > ./hadoop/client/httpclient-4.2.5.jar > > ./hadoop/client/httpclient.jar > > ./hadoop/lib/httpclient-4.2.5.jar > > ./slider/lib/httpclient-4.2.5.jar > > ./hbase/lib/httpclient-4.2.5.jar > > ./zookeeper/lib/httpclient-4.2.3.jar > > ./storm/extlib-daemon/httpclient-4.2.5.jar > > ./falcon/client/lib/httpclient-4.2.5.jar > > ./oozie/oozie-server/webapps/oozie/WEB-INF/lib/httpclient-4.2.5.jar > > ./oozie/oozie-server/webapps/oozie/WEB-INF/lib/httpclient-4.3.jar > > ./oozie/lib/httpclient-4.2.5.jar > > ./oozie/libserver/httpclient-4.2.5.jar > > ./oozie/libserver/httpclient-4.3.jar > > ./oozie/libtools/httpclient-4.2.5.jar > > ./oozie/share/lib/sqoop/httpclient-4.2.5.jar > > ./oozie/share/lib/hive/httpclient-4.2.5.jar > > ./oozie/share/lib/pig/httpclient-4.2.5.jar > > ./oozie/share/lib/oozie/httpclient-4.2.5.jar > > ./oozie/share/lib/distcp/httpclient-4.2.5.jar > > ./oozie/share/lib/spark/httpclient-4.2.5.jar > > ./oozie/share/lib/hcatalog/httpclient-4.2.5.jar > > ./oozie/share/lib/mapreduce-streaming/httpclient-4.2.5.jar > > ./oozie/share/lib/hive2/httpclient-4.4.jar > > ./ranger-storm-plugin/lib/httpclient-4.2.5.jar > > ### > > > > Best regards. > > > > Morgrim. > > > > > > > > > > > > On Mon, Apr 18, 2016 at 2:07 PM, Abhishek Bafna <[email protected]> > > wrote: > > > > > Hi, > > > > > > As you have mentioned you are using HDP in one of the mail chain, > Please > > > execute the below commands on the machine which have oozie server > running > > > and see the versions of the jars and path they are coming from: > > > > > > cd /use/hdp/<version> > > > find . -name httpcore* > > > find . -name httpclient* > > > > > > Thanks, > > > Abhishek > > > > > > On Mon, Apr 18, 2016 at 3:46 PM, Morgrim Murdargent < > > > [email protected] > > > > wrote: > > > > > > > Hello Abhishek ! > > > > > > > > Thank you for your answer. > > > > > > > > Hm, this is not clear for me. > > > > > > > > ### > > > > The httpcore-4.2.5 and httpclient-4.2.5 version is coming from the > > hadoop > > > > classpath not from the oozie. Because Java action executes the main() > > > > method from the launcher mapper only. You can check the oozie httpc* > > > > version from the share lib with below command. BTW, none of these are > > > > getting used here. > > > > ### > > > > > > > > I'm pretty sure these two libs are coming from the oozie folder and > not > > > the > > > > hadoop classpath. Because by replacing both http-core-4.2.5 and > > > > httpclient-4.2.5 with their 4.4 version in the oozie folder in the > > oozie > > > > sharelibs, it solved the problem at 100%. > > > > > > > > Best regards. > > > > > > > > Gwenael Le Barzic > > > > > > > > On Fri, Apr 15, 2016 at 4:40 PM, Abhishek Bafna < > [email protected]> > > > > wrote: > > > > > > > > > Hi Morgrim, > > > > > > > > > > The httpcore-4.2.5 and httpclient-4.2.5 version is coming from the > > > hadoop > > > > > classpath not from the oozie. Because Java action executes the > main() > > > > > method from the launcher mapper only. You can check the oozie > httpc* > > > > > version from the share lib with below command. BTW, none of these > are > > > > > getting used here. > > > > > > > > > > hdfs dfs -ls /user/oozie/share/lib/lib_*/* | grep httpc > > > > > > > > > > I guess your java action needs the httpc* jars, so they must be > > present > > > > in > > > > > the pom. Copy these to the lib\ directory of your workflow and set > > the > > > ' > > > > > oozie.launcher.mapreduce.user.classpath.first' to 'true' in your > java > > > > > action 'configuration' element. > > > > > > > > > > Also note that, when you set the property, > > > > > oozie.action.sharelib.for.java=hive2, you are loading the hive2 > > > > > configuration in the classpath, which is not required. > > > > > > > > > > You should always provide the custom jars required by the > application > > > in > > > > > the workflow application's lib/ directory. > > > > > > > > > > Check this for more information: > > > > > https://issues.apache.org/jira/browse/OOZIE-2066 > > > > > > > > > > Thanks, > > > > > Abhishek > > > > > > > > > > On Thu, Apr 7, 2016 at 9:46 PM, Morgrim Murdargent < > > > > > [email protected]> > > > > > wrote: > > > > > > > > > > > Hey Aaron. > > > > > > > > > > > > Thank you for your answer. > > > > > > > > > > > > Ok for the moment the solution I'm trying is to replace the libs > > > > > > httpcore-4.2.5 and httpclient-4.2.5 in the oozie folder of the > > oozie > > > > > > sharelibs by the version 4.4. > > > > > > My workflow ended successfully. > > > > > > > > > > > > I'm doing additionnal tests with the other workflows that I have > on > > > my > > > > > > cluster to see if they are impacted. > > > > > > > > > > > > To the Oozie developpers, may you help me on this topic please ? > > > > > > > > > > > > Best regards. > > > > > > > > > > > > Morgrim. > > > > > > > > > > > > On Thu, Apr 7, 2016 at 4:29 PM, Aaron.Dossett < > > > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > 1. Am I right ? Is the oozie folder of the oozie sharelibs > > loaded > > > > in > > > > > > the > > > > > > > > classpath in case of a java action ? > > > > > > > > > > > > > > This has given me all sorts of problems too, please see a > > > discussion > > > > I > > > > > > had > > > > > > > on the Hortonworks Community Connection: > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://community.hortonworks.com/questions/25796/oozie-share-lib-for-oozie > > > > > > > -contains-a-lot-of-jar-fi.html > > > > > > > > > > > > > > As far as I can tell there is no way around this. If anyone on > > the > > > > > list > > > > > > > has a suggestion, I would love to hear it. > > > > > > > > > > > > > > Thanks, Aaron > > > > > > > > > > > > > > On 4/7/16, 9:09 AM, "Morgrim Murdargent" < > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > > >Hello ! > > > > > > > > > > > > > > > >I have an oozie workflow with a java action. > > > > > > > >In this java action, I need the following version of > httpclient > > > and > > > > > > > >httpcore : > > > > > > > >- httpclient : 4.4 > > > > > > > >- httpcore : 4.4 > > > > > > > > > > > > > > > >I found that in the oozie sharelibs, only the hive2 folder has > > > these > > > > > > > >versions of httpclient and httpcore. > > > > > > > >So I added to my workflow the following property : > > > > > > > >oozie.action.sharelib.for.java=hive2 > > > > > > > > > > > > > > > >It solved the first part of my problem with httpclient but did > > not > > > > > with > > > > > > > >httpcore. > > > > > > > >When I check the classpath, I can see two versions for each > lib > > > > > httpcore > > > > > > > >and httpclient : > > > > > > > >- httpclient : 4.2.5 and 4.4 > > > > > > > >- httpcore : 4.2.5 and 4.4 > > > > > > > > > > > > > > > >By some chances, the httpclient-4.4 is loaded first, so it > > solved > > > my > > > > > > > >problem with httpclient. > > > > > > > >But unfortunately, httpcore-4.2.5 is loaded before the version > > > 4.4, > > > > > so I > > > > > > > >got an error message : > > > > > > > >### > > > > > > > >Failing Oozie Launcher, Main class > > > > > > > >[org.apache.oozie.action.hadoop.JavaMain], main() threw > > exception, > > > > > > > >java.lang.NoSuchFieldError: INSTANCE > > > > > > > >org.apache.oozie.action.hadoop.JavaMainException: > > > > > > > >java.lang.NoSuchFieldError: INSTANCE > > > > > > > >### > > > > > > > > > > > > > > > >I didn't find neither the httpclient-4.2.5 nor the httpcore > > 4.2.5 > > > in > > > > > > hive2 > > > > > > > >folder of oozie sharelib. > > > > > > > >But I found them in the oozie folder of oozie sharelib. > > > > > > > > > > > > > > > >And when we do java action in oozie workflow, the oozie folder > > is > > > > > > > >automatically loaded in the classpath. > > > > > > > > > > > > > > > >My questions are the following : > > > > > > > >1. Am I right ? Is the oozie folder of the oozie sharelibs > > loaded > > > in > > > > > the > > > > > > > >classpath in case of a java action ? > > > > > > > >2. How can I have the latest versions of the libs loaded first > > in > > > my > > > > > > > >classpath ? > > > > > > > >3. Can I load in the classpath only the libs contained in > hive2 > > > > > folder ? > > > > > > > >If > > > > > > > >yes, how ? > > > > > > > >4. Can I safely replace the version 4.2.5 of httpclient and > > > httpcore > > > > > in > > > > > > > >the > > > > > > > >oozie folder of the oozie sharelibs by the version 4.4 ? > > > > > > > >5. Can I do that for all the folders of the oozie sharelibs ? > > Why > > > do > > > > > we > > > > > > > >have different versions of libraries in each folder of the > oozie > > > > > > > >sharelibs ? > > > > > > > > > > > > > > > >Thank you in advanced for your help ! > > > > > > > > > > > > > > > >Best regards. > > > > > > > > > > > > > > > >Morgrim. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
