By default, only sqoop sharelib is loaded for sqoop actions. Add hcatalog sharelib as well - http://oozie.apache.org/docs/4.2.0/WorkflowFunctionalSpec.html#a17.1_Action _Share_Library_Override_since_Oozie_3.3
-Shwetha On 21/07/15 12:45 pm, "Venkatesan Ramachandran" <[email protected]> wrote: >Hi All, > >I'm running the following Sqoop command to write to HCATALOG on the >command >line and it works as expected; > >sqoop import --connect jdbc:mysql://c6402/test --table simple --username >sqoop_user --password-file /user/ambari-qa/datastore/testdb_password >--num-mappers 2 --split-by id --hcatalog-database default --hcatalog-table >simple --create-hcatalog-table > >But when it run it via Oozie Sqoop action as below, it throws class not >found exception for org/apache/hive/hcatalog/mapreduce/HCatOutputFormat. > >I have all the hcatalog jars in the oozie shared lib in the following >path: > >-rw-r--r-- 3 oozie hdfs 1997485 2015-07-21 01:08 >/user/oozie/share/lib/lib_20150721010816/hcatalog/ant-1.9.1.jar >-rw-r--r-- 3 oozie hdfs 18336 2015-07-21 01:09 >/user/oozie/share/lib/lib_20150721010816/hcatalog/ant-launcher-1.9.1.jar >-rw-r--r-- 3 oozie hdfs 346729 2015-07-21 01:09 >/user/oozie/share/lib/lib_20150721010816/hcatalog/apache-log4j-extras-1.1. >jar >-rw-r--r-- 3 oozie hdfs 110600 2015-07-21 01:08 >/user/oozie/share/lib/lib_20150721010816/hcatalog/bonecp-0.8.0.RELEASE.jar >-rw-r--r-- 3 oozie hdfs 339666 2015-07-21 01:08 >/user/oozie/share/lib/lib_20150721010816/hcatalog/datanucleus-api-jdo-3.2. >6.jar >-rw-r--r-- 3 oozie hdfs 294321 2015-07-21 01:08 >/user/oozie/share/lib/lib_20150721010816/hcatalog/hive-common-1.2.1.2.3.0. >0-2557.jar >-rw-r--r-- 3 oozie hdfs 257633 2015-07-21 01:09 >/user/oozie/share/lib/lib_20150721010816/hcatalog/hive-hcatalog-core-1.2.1 >.2.3.0.0-2557.jar >-rw-r--r-- 3 oozie hdfs 54328 2015-07-21 01:08 >/user/oozie/share/lib/lib_20150721010816/hcatalog/hive-hcatalog-server-ext >ensions-1.2.1.2.3.0.0-2557.jar >-rw-r--r-- 3 oozie hdfs 5505255 2015-07-21 01:09 >/user/oozie/share/lib/lib_20150721010816/hcatalog/hive-metastore-1.2.1.2.3 >.0.0-2557.jar >-rw-r--r-- 3 oozie hdfs 916926 2015-07-21 01:08 >/user/oozie/share/lib/lib_20150721010816/hcatalog/hive-serde-1.2.1.2.3.0.0 >-2557.jar >-rw-r--r-- 3 oozie hdfs 108030 2015-07-21 01:08 >/user/oozie/share/lib/lib_20150721010816/hcatalog/hive-webhcat-java-client >-1.2.1.2.3.0.0-2557.jar >-rw-r--r-- 3 oozie hdfs 232248 2015-07-21 01:08 >/user/oozie/share/lib/lib_20150721010816/hcatalog/jackson-core-asl-1.9.13. >jar >-rw-r--r-- 3 oozie hdfs 780664 2015-07-21 01:09 >/user/oozie/share/lib/lib_20150721010816/hcatalog/jackson-mapper-asl-1.9.1 >3.jar >-rw-r--r-- 3 oozie hdfs 201124 2015-07-21 01:09 >/user/oozie/share/lib/lib_20150721010816/hcatalog/jdo-api-3.0.1.jar >-rw-r--r-- 3 oozie hdfs 570478 2015-07-21 01:08 >/user/oozie/share/lib/lib_20150721010816/hcatalog/joda-time-2.1.jar >-rw-r--r-- 3 oozie hdfs 313686 2015-07-21 01:08 >/user/oozie/share/lib/lib_20150721010816/hcatalog/libfb303-0.9.2.jar >-rw-r--r-- 3 oozie hdfs 481535 2015-07-21 01:08 >/user/oozie/share/lib/lib_20150721010816/hcatalog/log4j-1.2.16.jar >-rw-r--r-- 3 oozie hdfs 16134 2015-07-21 01:08 >/user/oozie/share/lib/lib_20150721010816/hcatalog/oozie-sharelib-hcatalog- >4.2.0.2.3.0.0-2557.jar >-rw-r--r-- 3 oozie hdfs 19827 2015-07-21 01:08 >/user/oozie/share/lib/lib_20150721010816/hcatalog/opencsv-2.3.jar >-rw-r--r-- 3 oozie hdfs 2796935 2015-07-21 01:09 >/user/oozie/share/lib/lib_20150721010816/hcatalog/parquet-hadoop-bundle-1. >6.0.jar >-rw-r--r-- 3 oozie hdfs 26176 2015-07-21 01:08 >/user/oozie/share/lib/lib_20150721010816/hcatalog/slf4j-api-1.6.6.jar >-rw-r--r-- 3 oozie hdfs 9711 2015-07-21 01:08 >/user/oozie/share/lib/lib_20150721010816/hcatalog/slf4j-log4j12-1.6.6.jar > > >SQOOP command and the exception > >Sqoop command arguments : > import > --connect > jdbc:mysql://c6402/test > --table > simple > --username > sqoop_user > --password-file > /user/ambari-qa/datastore/testdb_password > --num-mappers > 2 > --split-by > id > --hcatalog-database > default > --hcatalog-table > simple > --create-hcatalog-table >Fetching child yarn jobs >tag id : oozie-a6b97cd85c4eba81b5653acf1dd341ad >2015-07-21 06:38:17,708 INFO [main] client.RMProxy >(RMProxy.java:createRMProxy(98)) - Connecting to ResourceManager at >c6402.ambari.apache.org/192.168.64.102:8050 >Child yarn jobs are found - >================================================================= > >>>> Invoking Sqoop command line now >>> > >5433 [main] WARN org.apache.sqoop.tool.SqoopTool - $SQOOP_CONF_DIR >has not been set in the environment. Cannot check for additional >configuration. >2015-07-21 06:38:18,204 WARN [main] tool.SqoopTool >(SqoopTool.java:loadPluginsFromConfDir(177)) - $SQOOP_CONF_DIR has not >been set in the environment. Cannot check for additional >configuration. >5467 [main] INFO org.apache.sqoop.Sqoop - Running Sqoop version: >1.4.6.2.3.0.0-2557 >2015-07-21 06:38:18,238 INFO [main] sqoop.Sqoop >(Sqoop.java:<init>(92)) - Running Sqoop version: 1.4.6.2.3.0.0-2557 >5668 [main] WARN org.apache.sqoop.ConnFactory - $SQOOP_CONF_DIR has >not been set in the environment. Cannot check for additional >configuration. >2015-07-21 06:38:18,439 WARN [main] sqoop.ConnFactory >(ConnFactory.java:loadManagersFromConfDir(273)) - $SQOOP_CONF_DIR has >not been set in the environment. Cannot check for additional >configuration. >5797 [main] INFO org.apache.sqoop.manager.MySQLManager - Preparing >to use a MySQL streaming resultset. >2015-07-21 06:38:18,568 INFO [main] manager.MySQLManager >(MySQLManager.java:initOptionDefaults(69)) - Preparing to use a MySQL >streaming resultset. >5797 [main] INFO org.apache.sqoop.tool.CodeGenTool - Beginning code >generation >2015-07-21 06:38:18,568 INFO [main] tool.CodeGenTool >(CodeGenTool.java:generateORM(92)) - Beginning code generation >6492 [main] INFO org.apache.sqoop.manager.SqlManager - Executing SQL >statement: SELECT t.* FROM `simple` AS t LIMIT 1 >2015-07-21 06:38:19,263 INFO [main] manager.SqlManager >(SqlManager.java:execute(757)) - Executing SQL statement: SELECT t.* >FROM `simple` AS t LIMIT 1 >6542 [main] INFO org.apache.sqoop.manager.SqlManager - Executing SQL >statement: SELECT t.* FROM `simple` AS t LIMIT 1 >2015-07-21 06:38:19,313 INFO [main] manager.SqlManager >(SqlManager.java:execute(757)) - Executing SQL statement: SELECT t.* >FROM `simple` AS t LIMIT 1 >6560 [main] INFO org.apache.sqoop.orm.CompilationManager - >HADOOP_MAPRED_HOME is /usr/hdp/2.3.0.0-2557/hadoop-mapreduce >2015-07-21 06:38:19,331 INFO [main] orm.CompilationManager >(CompilationManager.java:findHadoopJars(94)) - HADOOP_MAPRED_HOME is >/usr/hdp/2.3.0.0-2557/hadoop-mapreduce >9744 [main] INFO org.apache.sqoop.orm.CompilationManager - Writing >jar file: >/tmp/sqoop-yarn/compile/457f0736705f75f68537447520d1a916/simple.jar >2015-07-21 06:38:22,515 INFO [main] orm.CompilationManager >(CompilationManager.java:jar(330)) - Writing jar file: >/tmp/sqoop-yarn/compile/457f0736705f75f68537447520d1a916/simple.jar >9797 [main] WARN org.apache.sqoop.manager.MySQLManager - It looks >like you are importing from mysql. >2015-07-21 06:38:22,568 WARN [main] manager.MySQLManager >(MySQLManager.java:importTable(107)) - It looks like you are importing >from mysql. >9797 [main] WARN org.apache.sqoop.manager.MySQLManager - This >transfer can be faster! Use the --direct >2015-07-21 06:38:22,568 WARN [main] manager.MySQLManager >(MySQLManager.java:importTable(108)) - This transfer can be faster! >Use the --direct >9797 [main] WARN org.apache.sqoop.manager.MySQLManager - option to >exercise a MySQL-specific fast path. >2015-07-21 06:38:22,568 WARN [main] manager.MySQLManager >(MySQLManager.java:importTable(109)) - option to exercise a >MySQL-specific fast path. >9797 [main] INFO org.apache.sqoop.manager.MySQLManager - Setting >zero DATETIME behavior to convertToNull (mysql) >2015-07-21 06:38:22,568 INFO [main] manager.MySQLManager >(MySQLManager.java:checkDateTimeBehavior(189)) - Setting zero DATETIME >behavior to convertToNull (mysql) >9813 [main] INFO org.apache.sqoop.mapreduce.ImportJobBase - >Beginning import of simple >2015-07-21 06:38:22,584 INFO [main] mapreduce.ImportJobBase >(ImportJobBase.java:runImport(235)) - Beginning import of simple >2015-07-21 06:38:22,585 INFO [main] Configuration.deprecation >(Configuration.java:warnOnceIfDeprecated(1173)) - mapred.job.tracker >is deprecated. Instead, use mapreduce.jobtracker.address >2015-07-21 06:38:22,594 INFO [main] Configuration.deprecation >(Configuration.java:warnOnceIfDeprecated(1173)) - mapred.jar is >deprecated. Instead, use mapreduce.job.jar > ><<< Invocation of Main class completed <<< > >Failing Oozie Launcher, Main class >[org.apache.oozie.action.hadoop.SqoopMain], main() threw exception, >org/apache/hive/hcatalog/mapreduce/HCatOutputFormat >java.lang.NoClassDefFoundError: >org/apache/hive/hcatalog/mapreduce/HCatOutputFormat > at >org.apache.sqoop.mapreduce.DataDrivenImportJob.getOutputFormatClass(DataDr >ivenImportJob.java:178) > at >org.apache.sqoop.mapreduce.ImportJobBase.configureOutputFormat(ImportJobBa >se.java:94) > at >org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:259) > at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:673) > at >org.apache.sqoop.manager.MySQLManager.importTable(MySQLManager.java:118) > at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:497) > at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605) > at org.apache.sqoop.Sqoop.run(Sqoop.java:143) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179) > at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218) > at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227) > at org.apache.sqoop.Sqoop.main(Sqoop.java:236) > at >org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:197) > at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:177) > at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:47) > at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:46) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at >sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java: >62) > at >sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm >pl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at >org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:236) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at >org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation. >java:1657) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) >Caused by: java.lang.ClassNotFoundException: >org.apache.hive.hcatalog.mapreduce.HCatOutputFormat > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > ... 30 more > >Oozie Launcher failed, finishing Hadoop job gracefully > >Oozie Launcher, uploading action data to HDFS sequence file: >hdfs://c6401.ambari.apache.org:8020/user/ambari-qa/oozie-oozi/0000005-1507 >21024948042-oozie-oozi-W/sqoop-node--sqoop/action-data.seq >2015-07-21 ><http://c6401.ambari.apache.org:8020/user/ambari-qa/oozie-oozi/0000005-150 >721024948042-oozie-oozi-W/sqoop-node--sqoop/action-data.seq2015-07-21> >06:38:23,062 INFO [main] zlib.ZlibFactory >(ZlibFactory.java:<clinit>(49)) - Successfully loaded & initialized >native-zlib library >2015-07-21 06:38:23,063 INFO [main] compress.CodecPool >(CodecPool.java:getCompressor(153)) - Got brand-new compressor >[.deflate] > >Oozie Launcher ends
