Adding to Rohini. I also created a Jira for the same few months back: https://issues.apache.org/jira/browse/MAPREDUCE-4568
Roman, We are not using Bigtop. Can you please try after removing the "share/lib/sqoop/hsqldb-1.8.0.10.jar"? Just to make sure that is the root cause. One more thing, we need to make sure whether this version (Oozie 3.3) will support Hadoop 2.x. I'm sure without fixing MAPREDUCE-4568, it will be hard to support that newer hadoop version. As the JIRA discussions show that we are not close to agreement yet on MR fix. Need more input from others about this issue. Regards, Mohammad ________________________________ From: Rohini Palaniswamy <[email protected]> To: [email protected] Cc: Mohammad Islam <[email protected]>; "[email protected]" <[email protected]>; [email protected] Sent: Sunday, November 25, 2012 8:21 PM Subject: Re: [VOTE] Release Oozie 3.3.0 (candidate 0) Context on that issue is in https://issues.apache.org/jira/browse/MAPREDUCE-4549. Failing with that error in hadoop has been reverted in 0.23.x line but not in 2.0.x line. Regards, Rohini On Wed, Nov 21, 2012 at 6:17 PM, Roman Shaposhnik <[email protected]> wrote: > -1 based on Bigtop testing against Hadoop 2.0.2 and Hadoop 1.1.0 Sqoop > and Hive wf fail to execute. > > Here's how to reproduce: > > 1. On your linux box hook up 2 Bigtop repos to your system: > > http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Repository/ > http://bigtop01.cloudera.org:8080/job/Bigtop-git/ > e.g. on Ubuntu Lucid you'd run: > # curl > http://bigtop01.cloudera.org:8080/job/Bigtop-git/label=lucid/lastSuccessfulBuild/artifact/output/bigtop.list > > /etc/apt/sources.list.d/bigtop1.list > # curl > http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Repository/label=lucid/lastSuccessfulBuild/artifact/repo/bigtop.list > > /etc/apt/sources.list.d/bigtop2.list > # apt-get update > on CentOS 5 you'd run: > # curl > http://bigtop01.cloudera.org:8080/job/Bigtop-git/label=centos5/lastSuccessfulBuild/artifact/output/bigtop.repo > > /etc/yum.repos.d/bigtop1.repo > # curl > http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Repository/label=centos5/lastSuccessfulBuild/artifact/repo/bigtop.repo > > /etc/yum.repos.d/bigtop2.repo > > 2. Install Hadoop, Sqoop and Oozie in pseudo distributed mode: > ubuntu# apt-get install -y hadoop-conf-pseudo sqoop oozie > RedHat# yum install -y sqoop hadoop-conf-pseudo oozie > > 3. Init and start the services > # service hadoop-hdfs-namenode init > # service oozie init > # sudo -u hdfs hadoop fs -chmod -R 777 / > # for i in hadoop-hdfs-namenode hadoop-hdfs-datanode > hadoop-yarn-resourcemanager hadoop-yarn-nodemanager > hadoop-mapreduce-historyserver ; do service $i start ; done > # service oozie restart > > 4. Make sure that Oozie is up and running > # oozie admin -version -oozie http://localhost:11000/oozie > > 5. Install examples and try running a workflow > # cd /tmp > # tar xzvf /usr/share/doc/oozie*/oozie-examples.tar.gz > # hadoop fs -mkdir -p /user/oozie/share/lib/sqoop > # hadoop fs -mkdir -p /user/root > # hadoop fs -put examples /user/root/examples > # hadoop fs -put /usr/lib/sqoop/*.jar /usr/lib/sqoop/lib/*.jar > /user/oozie/share/lib/sqoop > # oozie job -DnameNode=hdfs://localhost:8020 > -DjobTracker=localhost:8032 -config examples/apps/sqoop/job.properties > -run -oozie http://localhost:11000/oozie > > At this point the workflow would fail with the following: > > eption: cache file (mapreduce.job.cache.files) scheme: "hdfs", host: > "localhost", port: 8020, file: > "/user/oozie/share/lib/sqoop/hsqldb-1.8.0.10.jar", conflicts with > cache file (mapreduce.job.cache.files) > > hdfs://localhost:8020/tmp/hadoop-yarn/staging/root/.staging/job_1353549235835_0004/libjars/hsqldb-1.8.0.10.jar > at > org.apache.hadoop.mapreduce.v2.util.MRApps.parseDistributedCacheArtifacts(MRApps.java:338) > at > org.apache.hadoop.mapreduce.v2.util.MRApps.setupDistributedCache(MRApps.java:273) > at > org.apache.hadoop.mapred.YARNRunner.createApplicationSubmissionContext(YARNRunner.java:419) > at > org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:288) > at > org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:391) > at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218) > at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1367) > at org.apache.hadoop.mapreduce.Job.submit(Job.java:1215) > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1236) > at > org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:141) > at > org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:202) > at > org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:465) > at > org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:403) > at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:476) > at org.apache.sqoop.Sqoop.run(Sqoop.java:145) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181) > at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220) > at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229) > at org.apache.sqoop.Sqoop.main(Sqoop.java:238) > at > org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:205) > at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:174) > at > org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37) > at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:47) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:473) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:400) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:335) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1367) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152) > > Intercepting System.exit(1) > > <<< Invocation of Main class completed <<< > > Thanks, > Roman. > > On Wed, Nov 21, 2012 at 12:00 PM, Mohammad Islam <[email protected]> > wrote: > > Dear Oozie community, > > > > The release candidate 0 for for Oozie 3.3.0 is available. > > > > > > Oozie 3.3.0 has the following new features: > > 1. Bulk Monitoring API - Consolidated view of jobs > > 2. Eliminate redundancies in xml through global section. > > 3. Add formal parameters to XML for early valiadation > > > > 4. Visualize color coded job DAG at runtime. > > > > 5. Load Hbase/HCat credentials in Job conf > > 6. Support direct map-reduce job submission through Oozie CLI without > workflow XML > > 7. Add support for multiple/configurable sharelibs for each action type > > > > > > > > In addition, it includes several improvements for performance and > stability > > and several bug fixes. Detail release log could be found at: > > http://people.apache.org/~kamrul/oozie-3.3.0-rc0/release-log.txt > > > > > > Keys used to sign the release are available at > > http://www.apache.org/dist/oozie/KEYS > > > > Please download, test, and try it out: > > http://people.apache.org/~kamrul/oozie-3.3.0-rc0/ > > > > The release, md5 signature, gpg signature, and rat report can all be > found > > at the above URL. > > > > Vote closes in 3 days . > > > > Regards, > > > > Mohammad >
