Re Abhishek. HDP version : 2.3.2.0-2950
### # pwd /usr/hdp/2.3.2.0-2950 [root@inbdfot21 2.3.2.0-2950]# find . -name httpcore* ./hadoop-yarn/lib/httpcore-4.2.5.jar ./hadoop-httpfs/webapps/webhdfs/WEB-INF/lib/httpcore-4.2.5.jar ./hadoop-mapreduce/httpcore-4.2.5.jar ./hive/lib/httpcore-4.4.jar ./hadoop/client/httpcore.jar ./hadoop/client/httpcore-4.2.5.jar ./hadoop/lib/httpcore-4.2.5.jar ./slider/lib/httpcore-4.2.5.jar ./hbase/lib/httpcore-4.2.5.jar ./zookeeper/lib/httpcore-4.2.3.jar ./storm/extlib-daemon/httpcore-4.2.5.jar ./falcon/client/lib/httpcore-4.2.4.jar ./oozie/oozie-server/webapps/oozie/WEB-INF/lib/httpcore-4.3.jar ./oozie/oozie-server/webapps/oozie/WEB-INF/lib/httpcore-4.2.4.jar ./oozie/lib/httpcore-4.2.4.jar ./oozie/libserver/httpcore-4.3.jar ./oozie/libserver/httpcore-4.2.4.jar ./oozie/libtools/httpcore-4.4.jar ./oozie/share/lib/sqoop/httpcore-4.2.4.jar ./oozie/share/lib/hive/httpcore-4.2.4.jar ./oozie/share/lib/pig/httpcore-4.2.4.jar ./oozie/share/lib/oozie/httpcore-4.2.4.jar ./oozie/share/lib/distcp/httpcore-4.2.4.jar ./oozie/share/lib/spark/httpcore-4.2.4.jar ./oozie/share/lib/hcatalog/httpcore-4.2.4.jar ./oozie/share/lib/mapreduce-streaming/httpcore-4.2.4.jar ./oozie/share/lib/hive2/httpcore-4.4.jar ./ranger-storm-plugin/lib/httpcore-4.2.5.jar ### #### find . -name httpclient* ./hadoop-yarn/lib/httpclient-4.2.5.jar ./hadoop-httpfs/webapps/webhdfs/WEB-INF/lib/httpclient-4.2.5.jar ./hadoop-mapreduce/httpclient-4.2.5.jar ./hive/lib/httpclient-4.4.jar ./hadoop/client/httpclient-4.2.5.jar ./hadoop/client/httpclient.jar ./hadoop/lib/httpclient-4.2.5.jar ./slider/lib/httpclient-4.2.5.jar ./hbase/lib/httpclient-4.2.5.jar ./zookeeper/lib/httpclient-4.2.3.jar ./storm/extlib-daemon/httpclient-4.2.5.jar ./falcon/client/lib/httpclient-4.2.5.jar ./oozie/oozie-server/webapps/oozie/WEB-INF/lib/httpclient-4.2.5.jar ./oozie/oozie-server/webapps/oozie/WEB-INF/lib/httpclient-4.3.jar ./oozie/lib/httpclient-4.2.5.jar ./oozie/libserver/httpclient-4.2.5.jar ./oozie/libserver/httpclient-4.3.jar ./oozie/libtools/httpclient-4.2.5.jar ./oozie/share/lib/sqoop/httpclient-4.2.5.jar ./oozie/share/lib/hive/httpclient-4.2.5.jar ./oozie/share/lib/pig/httpclient-4.2.5.jar ./oozie/share/lib/oozie/httpclient-4.2.5.jar ./oozie/share/lib/distcp/httpclient-4.2.5.jar ./oozie/share/lib/spark/httpclient-4.2.5.jar ./oozie/share/lib/hcatalog/httpclient-4.2.5.jar ./oozie/share/lib/mapreduce-streaming/httpclient-4.2.5.jar ./oozie/share/lib/hive2/httpclient-4.4.jar ./ranger-storm-plugin/lib/httpclient-4.2.5.jar ### Best regards. Morgrim. On Mon, Apr 18, 2016 at 2:07 PM, Abhishek Bafna <[email protected]> wrote: > Hi, > > As you have mentioned you are using HDP in one of the mail chain, Please > execute the below commands on the machine which have oozie server running > and see the versions of the jars and path they are coming from: > > cd /use/hdp/<version> > find . -name httpcore* > find . -name httpclient* > > Thanks, > Abhishek > > On Mon, Apr 18, 2016 at 3:46 PM, Morgrim Murdargent < > [email protected] > > wrote: > > > Hello Abhishek ! > > > > Thank you for your answer. > > > > Hm, this is not clear for me. > > > > ### > > The httpcore-4.2.5 and httpclient-4.2.5 version is coming from the hadoop > > classpath not from the oozie. Because Java action executes the main() > > method from the launcher mapper only. You can check the oozie httpc* > > version from the share lib with below command. BTW, none of these are > > getting used here. > > ### > > > > I'm pretty sure these two libs are coming from the oozie folder and not > the > > hadoop classpath. Because by replacing both http-core-4.2.5 and > > httpclient-4.2.5 with their 4.4 version in the oozie folder in the oozie > > sharelibs, it solved the problem at 100%. > > > > Best regards. > > > > Gwenael Le Barzic > > > > On Fri, Apr 15, 2016 at 4:40 PM, Abhishek Bafna <[email protected]> > > wrote: > > > > > Hi Morgrim, > > > > > > The httpcore-4.2.5 and httpclient-4.2.5 version is coming from the > hadoop > > > classpath not from the oozie. Because Java action executes the main() > > > method from the launcher mapper only. You can check the oozie httpc* > > > version from the share lib with below command. BTW, none of these are > > > getting used here. > > > > > > hdfs dfs -ls /user/oozie/share/lib/lib_*/* | grep httpc > > > > > > I guess your java action needs the httpc* jars, so they must be present > > in > > > the pom. Copy these to the lib\ directory of your workflow and set the > ' > > > oozie.launcher.mapreduce.user.classpath.first' to 'true' in your java > > > action 'configuration' element. > > > > > > Also note that, when you set the property, > > > oozie.action.sharelib.for.java=hive2, you are loading the hive2 > > > configuration in the classpath, which is not required. > > > > > > You should always provide the custom jars required by the application > in > > > the workflow application's lib/ directory. > > > > > > Check this for more information: > > > https://issues.apache.org/jira/browse/OOZIE-2066 > > > > > > Thanks, > > > Abhishek > > > > > > On Thu, Apr 7, 2016 at 9:46 PM, Morgrim Murdargent < > > > [email protected]> > > > wrote: > > > > > > > Hey Aaron. > > > > > > > > Thank you for your answer. > > > > > > > > Ok for the moment the solution I'm trying is to replace the libs > > > > httpcore-4.2.5 and httpclient-4.2.5 in the oozie folder of the oozie > > > > sharelibs by the version 4.4. > > > > My workflow ended successfully. > > > > > > > > I'm doing additionnal tests with the other workflows that I have on > my > > > > cluster to see if they are impacted. > > > > > > > > To the Oozie developpers, may you help me on this topic please ? > > > > > > > > Best regards. > > > > > > > > Morgrim. > > > > > > > > On Thu, Apr 7, 2016 at 4:29 PM, Aaron.Dossett < > > [email protected]> > > > > wrote: > > > > > > > > > Hi, > > > > > > > > > > > 1. Am I right ? Is the oozie folder of the oozie sharelibs loaded > > in > > > > the > > > > > > classpath in case of a java action ? > > > > > > > > > > This has given me all sorts of problems too, please see a > discussion > > I > > > > had > > > > > on the Hortonworks Community Connection: > > > > > > > > > > > > > > > https://community.hortonworks.com/questions/25796/oozie-share-lib-for-oozie > > > > > -contains-a-lot-of-jar-fi.html > > > > > > > > > > As far as I can tell there is no way around this. If anyone on the > > > list > > > > > has a suggestion, I would love to hear it. > > > > > > > > > > Thanks, Aaron > > > > > > > > > > On 4/7/16, 9:09 AM, "Morgrim Murdargent" <[email protected]> > > > > wrote: > > > > > > > > > > >Hello ! > > > > > > > > > > > >I have an oozie workflow with a java action. > > > > > >In this java action, I need the following version of httpclient > and > > > > > >httpcore : > > > > > >- httpclient : 4.4 > > > > > >- httpcore : 4.4 > > > > > > > > > > > >I found that in the oozie sharelibs, only the hive2 folder has > these > > > > > >versions of httpclient and httpcore. > > > > > >So I added to my workflow the following property : > > > > > >oozie.action.sharelib.for.java=hive2 > > > > > > > > > > > >It solved the first part of my problem with httpclient but did not > > > with > > > > > >httpcore. > > > > > >When I check the classpath, I can see two versions for each lib > > > httpcore > > > > > >and httpclient : > > > > > >- httpclient : 4.2.5 and 4.4 > > > > > >- httpcore : 4.2.5 and 4.4 > > > > > > > > > > > >By some chances, the httpclient-4.4 is loaded first, so it solved > my > > > > > >problem with httpclient. > > > > > >But unfortunately, httpcore-4.2.5 is loaded before the version > 4.4, > > > so I > > > > > >got an error message : > > > > > >### > > > > > >Failing Oozie Launcher, Main class > > > > > >[org.apache.oozie.action.hadoop.JavaMain], main() threw exception, > > > > > >java.lang.NoSuchFieldError: INSTANCE > > > > > >org.apache.oozie.action.hadoop.JavaMainException: > > > > > >java.lang.NoSuchFieldError: INSTANCE > > > > > >### > > > > > > > > > > > >I didn't find neither the httpclient-4.2.5 nor the httpcore 4.2.5 > in > > > > hive2 > > > > > >folder of oozie sharelib. > > > > > >But I found them in the oozie folder of oozie sharelib. > > > > > > > > > > > >And when we do java action in oozie workflow, the oozie folder is > > > > > >automatically loaded in the classpath. > > > > > > > > > > > >My questions are the following : > > > > > >1. Am I right ? Is the oozie folder of the oozie sharelibs loaded > in > > > the > > > > > >classpath in case of a java action ? > > > > > >2. How can I have the latest versions of the libs loaded first in > my > > > > > >classpath ? > > > > > >3. Can I load in the classpath only the libs contained in hive2 > > > folder ? > > > > > >If > > > > > >yes, how ? > > > > > >4. Can I safely replace the version 4.2.5 of httpclient and > httpcore > > > in > > > > > >the > > > > > >oozie folder of the oozie sharelibs by the version 4.4 ? > > > > > >5. Can I do that for all the folders of the oozie sharelibs ? Why > do > > > we > > > > > >have different versions of libraries in each folder of the oozie > > > > > >sharelibs ? > > > > > > > > > > > >Thank you in advanced for your help ! > > > > > > > > > > > >Best regards. > > > > > > > > > > > >Morgrim. > > > > > > > > > > > > > > > > > > > >
