Hi All,
I'm running the following Sqoop command to write to HCATALOG on the command
line and it works as expected;
sqoop import --connect jdbc:mysql://c6402/test --table simple --username
sqoop_user --password-file /user/ambari-qa/datastore/testdb_password
--num-mappers 2 --split-by id --hcatalog-database default --hcatalog-table
simple --create-hcatalog-table
But when it run it via Oozie Sqoop action as below, it throws class not
found exception for org/apache/hive/hcatalog/mapreduce/HCatOutputFormat.
I have all the hcatalog jars in the oozie shared lib in the following path:
-rw-r--r-- 3 oozie hdfs 1997485 2015-07-21 01:08
/user/oozie/share/lib/lib_20150721010816/hcatalog/ant-1.9.1.jar
-rw-r--r-- 3 oozie hdfs 18336 2015-07-21 01:09
/user/oozie/share/lib/lib_20150721010816/hcatalog/ant-launcher-1.9.1.jar
-rw-r--r-- 3 oozie hdfs 346729 2015-07-21 01:09
/user/oozie/share/lib/lib_20150721010816/hcatalog/apache-log4j-extras-1.1.jar
-rw-r--r-- 3 oozie hdfs 110600 2015-07-21 01:08
/user/oozie/share/lib/lib_20150721010816/hcatalog/bonecp-0.8.0.RELEASE.jar
-rw-r--r-- 3 oozie hdfs 339666 2015-07-21 01:08
/user/oozie/share/lib/lib_20150721010816/hcatalog/datanucleus-api-jdo-3.2.6.jar
-rw-r--r-- 3 oozie hdfs 294321 2015-07-21 01:08
/user/oozie/share/lib/lib_20150721010816/hcatalog/hive-common-1.2.1.2.3.0.0-2557.jar
-rw-r--r-- 3 oozie hdfs 257633 2015-07-21 01:09
/user/oozie/share/lib/lib_20150721010816/hcatalog/hive-hcatalog-core-1.2.1.2.3.0.0-2557.jar
-rw-r--r-- 3 oozie hdfs 54328 2015-07-21 01:08
/user/oozie/share/lib/lib_20150721010816/hcatalog/hive-hcatalog-server-extensions-1.2.1.2.3.0.0-2557.jar
-rw-r--r-- 3 oozie hdfs 5505255 2015-07-21 01:09
/user/oozie/share/lib/lib_20150721010816/hcatalog/hive-metastore-1.2.1.2.3.0.0-2557.jar
-rw-r--r-- 3 oozie hdfs 916926 2015-07-21 01:08
/user/oozie/share/lib/lib_20150721010816/hcatalog/hive-serde-1.2.1.2.3.0.0-2557.jar
-rw-r--r-- 3 oozie hdfs 108030 2015-07-21 01:08
/user/oozie/share/lib/lib_20150721010816/hcatalog/hive-webhcat-java-client-1.2.1.2.3.0.0-2557.jar
-rw-r--r-- 3 oozie hdfs 232248 2015-07-21 01:08
/user/oozie/share/lib/lib_20150721010816/hcatalog/jackson-core-asl-1.9.13.jar
-rw-r--r-- 3 oozie hdfs 780664 2015-07-21 01:09
/user/oozie/share/lib/lib_20150721010816/hcatalog/jackson-mapper-asl-1.9.13.jar
-rw-r--r-- 3 oozie hdfs 201124 2015-07-21 01:09
/user/oozie/share/lib/lib_20150721010816/hcatalog/jdo-api-3.0.1.jar
-rw-r--r-- 3 oozie hdfs 570478 2015-07-21 01:08
/user/oozie/share/lib/lib_20150721010816/hcatalog/joda-time-2.1.jar
-rw-r--r-- 3 oozie hdfs 313686 2015-07-21 01:08
/user/oozie/share/lib/lib_20150721010816/hcatalog/libfb303-0.9.2.jar
-rw-r--r-- 3 oozie hdfs 481535 2015-07-21 01:08
/user/oozie/share/lib/lib_20150721010816/hcatalog/log4j-1.2.16.jar
-rw-r--r-- 3 oozie hdfs 16134 2015-07-21 01:08
/user/oozie/share/lib/lib_20150721010816/hcatalog/oozie-sharelib-hcatalog-4.2.0.2.3.0.0-2557.jar
-rw-r--r-- 3 oozie hdfs 19827 2015-07-21 01:08
/user/oozie/share/lib/lib_20150721010816/hcatalog/opencsv-2.3.jar
-rw-r--r-- 3 oozie hdfs 2796935 2015-07-21 01:09
/user/oozie/share/lib/lib_20150721010816/hcatalog/parquet-hadoop-bundle-1.6.0.jar
-rw-r--r-- 3 oozie hdfs 26176 2015-07-21 01:08
/user/oozie/share/lib/lib_20150721010816/hcatalog/slf4j-api-1.6.6.jar
-rw-r--r-- 3 oozie hdfs 9711 2015-07-21 01:08
/user/oozie/share/lib/lib_20150721010816/hcatalog/slf4j-log4j12-1.6.6.jar
SQOOP command and the exception
Sqoop command arguments :
import
--connect
jdbc:mysql://c6402/test
--table
simple
--username
sqoop_user
--password-file
/user/ambari-qa/datastore/testdb_password
--num-mappers
2
--split-by
id
--hcatalog-database
default
--hcatalog-table
simple
--create-hcatalog-table
Fetching child yarn jobs
tag id : oozie-a6b97cd85c4eba81b5653acf1dd341ad
2015-07-21 06:38:17,708 INFO [main] client.RMProxy
(RMProxy.java:createRMProxy(98)) - Connecting to ResourceManager at
c6402.ambari.apache.org/192.168.64.102:8050
Child yarn jobs are found -
=================================================================
>>> Invoking Sqoop command line now >>>
5433 [main] WARN org.apache.sqoop.tool.SqoopTool - $SQOOP_CONF_DIR
has not been set in the environment. Cannot check for additional
configuration.
2015-07-21 06:38:18,204 WARN [main] tool.SqoopTool
(SqoopTool.java:loadPluginsFromConfDir(177)) - $SQOOP_CONF_DIR has not
been set in the environment. Cannot check for additional
configuration.
5467 [main] INFO org.apache.sqoop.Sqoop - Running Sqoop version:
1.4.6.2.3.0.0-2557
2015-07-21 06:38:18,238 INFO [main] sqoop.Sqoop
(Sqoop.java:<init>(92)) - Running Sqoop version: 1.4.6.2.3.0.0-2557
5668 [main] WARN org.apache.sqoop.ConnFactory - $SQOOP_CONF_DIR has
not been set in the environment. Cannot check for additional
configuration.
2015-07-21 06:38:18,439 WARN [main] sqoop.ConnFactory
(ConnFactory.java:loadManagersFromConfDir(273)) - $SQOOP_CONF_DIR has
not been set in the environment. Cannot check for additional
configuration.
5797 [main] INFO org.apache.sqoop.manager.MySQLManager - Preparing
to use a MySQL streaming resultset.
2015-07-21 06:38:18,568 INFO [main] manager.MySQLManager
(MySQLManager.java:initOptionDefaults(69)) - Preparing to use a MySQL
streaming resultset.
5797 [main] INFO org.apache.sqoop.tool.CodeGenTool - Beginning code generation
2015-07-21 06:38:18,568 INFO [main] tool.CodeGenTool
(CodeGenTool.java:generateORM(92)) - Beginning code generation
6492 [main] INFO org.apache.sqoop.manager.SqlManager - Executing SQL
statement: SELECT t.* FROM `simple` AS t LIMIT 1
2015-07-21 06:38:19,263 INFO [main] manager.SqlManager
(SqlManager.java:execute(757)) - Executing SQL statement: SELECT t.*
FROM `simple` AS t LIMIT 1
6542 [main] INFO org.apache.sqoop.manager.SqlManager - Executing SQL
statement: SELECT t.* FROM `simple` AS t LIMIT 1
2015-07-21 06:38:19,313 INFO [main] manager.SqlManager
(SqlManager.java:execute(757)) - Executing SQL statement: SELECT t.*
FROM `simple` AS t LIMIT 1
6560 [main] INFO org.apache.sqoop.orm.CompilationManager -
HADOOP_MAPRED_HOME is /usr/hdp/2.3.0.0-2557/hadoop-mapreduce
2015-07-21 06:38:19,331 INFO [main] orm.CompilationManager
(CompilationManager.java:findHadoopJars(94)) - HADOOP_MAPRED_HOME is
/usr/hdp/2.3.0.0-2557/hadoop-mapreduce
9744 [main] INFO org.apache.sqoop.orm.CompilationManager - Writing
jar file: /tmp/sqoop-yarn/compile/457f0736705f75f68537447520d1a916/simple.jar
2015-07-21 06:38:22,515 INFO [main] orm.CompilationManager
(CompilationManager.java:jar(330)) - Writing jar file:
/tmp/sqoop-yarn/compile/457f0736705f75f68537447520d1a916/simple.jar
9797 [main] WARN org.apache.sqoop.manager.MySQLManager - It looks
like you are importing from mysql.
2015-07-21 06:38:22,568 WARN [main] manager.MySQLManager
(MySQLManager.java:importTable(107)) - It looks like you are importing
from mysql.
9797 [main] WARN org.apache.sqoop.manager.MySQLManager - This
transfer can be faster! Use the --direct
2015-07-21 06:38:22,568 WARN [main] manager.MySQLManager
(MySQLManager.java:importTable(108)) - This transfer can be faster!
Use the --direct
9797 [main] WARN org.apache.sqoop.manager.MySQLManager - option to
exercise a MySQL-specific fast path.
2015-07-21 06:38:22,568 WARN [main] manager.MySQLManager
(MySQLManager.java:importTable(109)) - option to exercise a
MySQL-specific fast path.
9797 [main] INFO org.apache.sqoop.manager.MySQLManager - Setting
zero DATETIME behavior to convertToNull (mysql)
2015-07-21 06:38:22,568 INFO [main] manager.MySQLManager
(MySQLManager.java:checkDateTimeBehavior(189)) - Setting zero DATETIME
behavior to convertToNull (mysql)
9813 [main] INFO org.apache.sqoop.mapreduce.ImportJobBase -
Beginning import of simple
2015-07-21 06:38:22,584 INFO [main] mapreduce.ImportJobBase
(ImportJobBase.java:runImport(235)) - Beginning import of simple
2015-07-21 06:38:22,585 INFO [main] Configuration.deprecation
(Configuration.java:warnOnceIfDeprecated(1173)) - mapred.job.tracker
is deprecated. Instead, use mapreduce.jobtracker.address
2015-07-21 06:38:22,594 INFO [main] Configuration.deprecation
(Configuration.java:warnOnceIfDeprecated(1173)) - mapred.jar is
deprecated. Instead, use mapreduce.job.jar
<<< Invocation of Main class completed <<<
Failing Oozie Launcher, Main class
[org.apache.oozie.action.hadoop.SqoopMain], main() threw exception,
org/apache/hive/hcatalog/mapreduce/HCatOutputFormat
java.lang.NoClassDefFoundError:
org/apache/hive/hcatalog/mapreduce/HCatOutputFormat
at
org.apache.sqoop.mapreduce.DataDrivenImportJob.getOutputFormatClass(DataDrivenImportJob.java:178)
at
org.apache.sqoop.mapreduce.ImportJobBase.configureOutputFormat(ImportJobBase.java:94)
at
org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:259)
at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:673)
at
org.apache.sqoop.manager.MySQLManager.importTable(MySQLManager.java:118)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:497)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
at
org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:197)
at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:177)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:47)
at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:46)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:236)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.ClassNotFoundException:
org.apache.hive.hcatalog.mapreduce.HCatOutputFormat
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 30 more
Oozie Launcher failed, finishing Hadoop job gracefully
Oozie Launcher, uploading action data to HDFS sequence file:
hdfs://c6401.ambari.apache.org:8020/user/ambari-qa/oozie-oozi/0000005-150721024948042-oozie-oozi-W/sqoop-node--sqoop/action-data.seq
2015-07-21
<http://c6401.ambari.apache.org:8020/user/ambari-qa/oozie-oozi/0000005-150721024948042-oozie-oozi-W/sqoop-node--sqoop/action-data.seq2015-07-21>
06:38:23,062 INFO [main] zlib.ZlibFactory
(ZlibFactory.java:<clinit>(49)) - Successfully loaded & initialized
native-zlib library
2015-07-21 06:38:23,063 INFO [main] compress.CodecPool
(CodecPool.java:getCompressor(153)) - Got brand-new compressor
[.deflate]
Oozie Launcher ends
