Thanks Shwetha. Adding hcatalog,hive,hive2 to oozie.action.sharelib.for.sqoop
in job configuration makes it move beyond the class not found exception.
However, now Sqoop throws *java.io.IOException:
NoSuchObjectException(message:default.simple table not found) *even though
I have manually created the table in default.simple.
How does the Sqoop action (or oozie) know how to talk to HCAT/metastore
server? is there some other config I'm missing:
*Sqoop action workflow.xml*<workflow-app xmlns="uri:oozie:workflow:0.2"
name="sqoop-wf">
<start to="sqoop-node"/>
<action name="sqoop-node">
<sqoop xmlns="uri:oozie:sqoop-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>${queueName}</value>
</property>
</configuration>
<command>
import --connect jdbc:mysql://c6402/test --table simple --username
sqoop_user --password-file /user/ambari-qa/datastore/testdb_password
--num-mappers 2 --split-by id --hcatalog-database default --hcatalog-table
simple
</command>
</sqoop>
<ok to="end"/>
<error to="fail"/>
</action>
<kill name="fail">
<message>Sqoop failed, error
message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<end name="end"/>
</workflow-app>
*Exception*:
Sqoop command arguments :
import
--connect
jdbc:mysql://c6402/test
--table
simple
--username
sqoop_user
--password-file
/user/ambari-qa/datastore/testdb_password
--num-mappers
2
--split-by
id
--hcatalog-database
default
--hcatalog-table
simple
7511 [main] INFO org.apache.sqoop.mapreduce.hcat.SqoopHCatUtilities
- Database column names projected : [id, name, value, modified_ts]
2015-07-21 08:18:43,718 INFO [main] hcat.SqoopHCatUtilities
(SqoopHCatUtilities.java:initDBColumnInfo(519)) - Database column
names projected : [id, name, value, modified_ts]
7511 [main] INFO org.apache.sqoop.mapreduce.hcat.SqoopHCatUtilities
- Database column name - info map :
modified_ts : [Type : 93,Precision : 19,Scale : 0]
name : [Type : 12,Precision : 20,Scale : 0]
id : [Type : 4,Precision : 11,Scale : 0]
value : [Type : 4,Precision : 11,Scale : 0]
2015-07-21 08:18:43,718 INFO [main] hcat.SqoopHCatUtilities
(SqoopHCatUtilities.java:initDBColumnInfo(530)) - Database column name
- info map :
modified_ts : [Type : 93,Precision : 19,Scale : 0]
name : [Type : 12,Precision : 20,Scale : 0]
id : [Type : 4,Precision : 11,Scale : 0]
value : [Type : 4,Precision : 11,Scale : 0]
2015-07-21 08:18:44,244 INFO [main] metastore.HiveMetaStore
(HiveMetaStore.java:newRawStore(589)) - 0: Opening raw store with
implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
2015-07-21 08:18:44,286 INFO [main] metastore.ObjectStore
(ObjectStore.java:initialize(289)) - ObjectStore, initialize called
2015-07-21 08:18:44,682 INFO [main] DataNucleus.Persistence
(Log4JLogger.java:info(77)) - Property
hive.metastore.integral.jdo.pushdown unknown - will be ignored
2015-07-21 08:18:44,682 INFO [main] DataNucleus.Persistence
(Log4JLogger.java:info(77)) - Property datanucleus.cache.level2
unknown - will be ignored
2015-07-21 08:18:49,025 INFO [main] metastore.ObjectStore
(ObjectStore.java:getPMF(370)) - Setting MetaStore object pin classes
with
hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
2015-07-21 08:18:51,339 INFO [main] DataNucleus.Datastore
(Log4JLogger.java:info(77)) - The class
"org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as
"embedded-only" so does not have its own datastore table.
2015-07-21 08:18:51,341 INFO [main] DataNucleus.Datastore
(Log4JLogger.java:info(77)) - The class
"org.apache.hadoop.hive.metastore.model.MOrder" is tagged as
"embedded-only" so does not have its own datastore table.
2015-07-21 08:18:53,638 INFO [main] DataNucleus.Datastore
(Log4JLogger.java:info(77)) - The class
"org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as
"embedded-only" so does not have its own datastore table.
2015-07-21 08:18:53,638 INFO [main] DataNucleus.Datastore
(Log4JLogger.java:info(77)) - The class
"org.apache.hadoop.hive.metastore.model.MOrder" is tagged as
"embedded-only" so does not have its own datastore table.
2015-07-21 08:18:54,116 INFO [main] metastore.MetaStoreDirectSql
(MetaStoreDirectSql.java:<init>(139)) - Using direct SQL, underlying
DB is DERBY
2015-07-21 08:18:54,120 INFO [main] metastore.ObjectStore
(ObjectStore.java:setConf(272)) - Initialized ObjectStore
2015-07-21 08:18:54,303 WARN [main] metastore.ObjectStore
(ObjectStore.java:checkSchema(6658)) - Version information not found
in metastore. hive.metastore.schema.verification is not enabled so
recording the schema version 1.2.0
2015-07-21 08:18:54,508 WARN [main] metastore.ObjectStore
(ObjectStore.java:getDatabase(568)) - Failed to get database default,
returning NoSuchObjectException
2015-07-21 08:18:54,694 INFO [main] metastore.HiveMetaStore
(HiveMetaStore.java:createDefaultRoles_core(663)) - Added admin role
in metastore
2015-07-21 08:18:54,701 INFO [main] metastore.HiveMetaStore
(HiveMetaStore.java:createDefaultRoles_core(672)) - Added public role
in metastore
2015-07-21 08:18:54,813 INFO [main] metastore.HiveMetaStore
(HiveMetaStore.java:addAdminUsers_core(712)) - No user is added in
admin role, since config is empty
2015-07-21 08:18:54,980 INFO [main] metastore.HiveMetaStore
(HiveMetaStore.java:logInfo(746)) - 0: get_databases:
NonExistentDatabaseUsedForHealthCheck
2015-07-21 08:18:54,981 INFO [main] HiveMetaStore.audit
(HiveMetaStore.java:logAuditEvent(371)) -
ugi=ambari-qa ip=unknown-ip-addr cmd=get_databases:
NonExistentDatabaseUsedForHealthCheck
2015-07-21 08:18:55,019 INFO [main] metastore.HiveMetaStore
(HiveMetaStore.java:logInfo(746)) - 0: get_table : db=default
tbl=simple
2015-07-21 08:18:55,022 INFO [main] HiveMetaStore.audit
(HiveMetaStore.java:logAuditEvent(371)) -
ugi=ambari-qa ip=unknown-ip-addr cmd=get_table : db=default tbl=simple
18847 [main] ERROR org.apache.sqoop.tool.ImportTool - Encountered
IOException running import job: java.io.IOException:
NoSuchObjectException(message:default.simple table not found)
at
org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:97)
at
org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:51)
at
org.apache.sqoop.mapreduce.hcat.SqoopHCatUtilities.configureHCat(SqoopHCatUtilities.java:343)
at
org.apache.sqoop.mapreduce.hcat.SqoopHCatUtilities.configureImportOutputFormat(SqoopHCatUtilities.java:783)
at
org.apache.sqoop.mapreduce.ImportJobBase.configureOutputFormat(ImportJobBase.java:98)
at
org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:259)
at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:673)
at
org.apache.sqoop.manager.MySQLManager.importTable(MySQLManager.java:118)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:497)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
at
org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:197)
at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:177)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:47)
at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:46)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:236)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: NoSuchObjectException(message:default.simple table not found)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_core(HiveMetaStore.java:1808)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveMetaStore.java:1778)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
at com.sun.proxy.$Proxy20.get_table(Unknown Source)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1208)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:152)
at com.sun.proxy.$Proxy21.getTable(Unknown Source)
at org.apache.hive.hcatalog.common.HCatUtil.getTable(HCatUtil.java:180)
at
org.apache.hive.hcatalog.mapreduce.InitializeInput.getInputJobInfo(InitializeInput.java:105)
at
org.apache.hive.hcatalog.mapreduce.InitializeInput.setInput(InitializeInput.java:86)
at
org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:95)
... 32 more
