Makes sense. Let me try On Tuesday, July 21, 2015, Shwetha Shivalingamurthy < [email protected]> wrote:
> Hive-site.xml should be in the class path like > https://github.com/apache/oozie/blob/master/examples/src/main/apps/hcatalog > /workflow.xml > > Alternatively, you can probably add it to hive sharelib, I am not sure > > -Shwetha > > On 21/07/15 1:57 pm, "Venkatesan Ramachandran" <[email protected] > <javascript:;>> > wrote: > > >Thanks Shwetha. Adding hcatalog,hive,hive2 to > >oozie.action.sharelib.for.sqoop > >in job configuration makes it move beyond the class not found exception. > > > >However, now Sqoop throws *java.io.IOException: > >NoSuchObjectException(message:default.simple table not found) *even though > >I have manually created the table in default.simple. > > > >How does the Sqoop action (or oozie) know how to talk to HCAT/metastore > >server? is there some other config I'm missing: > > > > > > > >*Sqoop action workflow.xml*<workflow-app xmlns="uri:oozie:workflow:0.2" > >name="sqoop-wf"> > > <start to="sqoop-node"/> > > <action name="sqoop-node"> > > <sqoop xmlns="uri:oozie:sqoop-action:0.2"> > > <job-tracker>${jobTracker}</job-tracker> > > <name-node>${nameNode}</name-node> > > <configuration> > > <property> > > <name>mapred.job.queue.name</name> > > <value>${queueName}</value> > > </property> > > </configuration> > > <command> > > import --connect jdbc:mysql://c6402/test --table simple --username > >sqoop_user --password-file /user/ambari-qa/datastore/testdb_password > >--num-mappers 2 --split-by id --hcatalog-database default --hcatalog-table > >simple > > </command> > > </sqoop> > > <ok to="end"/> > > <error to="fail"/> > > </action> > > > > <kill name="fail"> > > <message>Sqoop failed, error > >message[${wf:errorMessage(wf:lastErrorNode())}]</message> > > </kill> > > <end name="end"/> > ></workflow-app> > > > >*Exception*: > > > >Sqoop command arguments : > > import > > --connect > > jdbc:mysql://c6402/test > > --table > > simple > > --username > > sqoop_user > > --password-file > > /user/ambari-qa/datastore/testdb_password > > --num-mappers > > 2 > > --split-by > > id > > --hcatalog-database > > default > > --hcatalog-table > > simple > > > > > > > >7511 [main] INFO org.apache.sqoop.mapreduce.hcat.SqoopHCatUtilities > >- Database column names projected : [id, name, value, modified_ts] > >2015-07-21 08:18:43,718 INFO [main] hcat.SqoopHCatUtilities > >(SqoopHCatUtilities.java:initDBColumnInfo(519)) - Database column > >names projected : [id, name, value, modified_ts] > >7511 [main] INFO org.apache.sqoop.mapreduce.hcat.SqoopHCatUtilities > >- Database column name - info map : > > modified_ts : [Type : 93,Precision : 19,Scale : 0] > > name : [Type : 12,Precision : 20,Scale : 0] > > id : [Type : 4,Precision : 11,Scale : 0] > > value : [Type : 4,Precision : 11,Scale : 0] > > > >2015-07-21 08:18:43,718 INFO [main] hcat.SqoopHCatUtilities > >(SqoopHCatUtilities.java:initDBColumnInfo(530)) - Database column name > >- info map : > > modified_ts : [Type : 93,Precision : 19,Scale : 0] > > name : [Type : 12,Precision : 20,Scale : 0] > > id : [Type : 4,Precision : 11,Scale : 0] > > value : [Type : 4,Precision : 11,Scale : 0] > > > >2015-07-21 08:18:44,244 INFO [main] metastore.HiveMetaStore > >(HiveMetaStore.java:newRawStore(589)) - 0: Opening raw store with > >implemenation class:org.apache.hadoop.hive.metastore.ObjectStore > >2015-07-21 08:18:44,286 INFO [main] metastore.ObjectStore > >(ObjectStore.java:initialize(289)) - ObjectStore, initialize called > >2015-07-21 08:18:44,682 INFO [main] DataNucleus.Persistence > >(Log4JLogger.java:info(77)) - Property > >hive.metastore.integral.jdo.pushdown unknown - will be ignored > >2015-07-21 08:18:44,682 INFO [main] DataNucleus.Persistence > >(Log4JLogger.java:info(77)) - Property datanucleus.cache.level2 > >unknown - will be ignored > >2015-07-21 08:18:49,025 INFO [main] metastore.ObjectStore > >(ObjectStore.java:getPMF(370)) - Setting MetaStore object pin classes > >with > >hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partit > >ion,Database,Type,FieldSchema,Order" > >2015-07-21 08:18:51,339 INFO [main] DataNucleus.Datastore > >(Log4JLogger.java:info(77)) - The class > >"org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as > >"embedded-only" so does not have its own datastore table. > >2015-07-21 08:18:51,341 INFO [main] DataNucleus.Datastore > >(Log4JLogger.java:info(77)) - The class > >"org.apache.hadoop.hive.metastore.model.MOrder" is tagged as > >"embedded-only" so does not have its own datastore table. > >2015-07-21 08:18:53,638 INFO [main] DataNucleus.Datastore > >(Log4JLogger.java:info(77)) - The class > >"org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as > >"embedded-only" so does not have its own datastore table. > >2015-07-21 08:18:53,638 INFO [main] DataNucleus.Datastore > >(Log4JLogger.java:info(77)) - The class > >"org.apache.hadoop.hive.metastore.model.MOrder" is tagged as > >"embedded-only" so does not have its own datastore table. > >2015-07-21 08:18:54,116 INFO [main] metastore.MetaStoreDirectSql > >(MetaStoreDirectSql.java:<init>(139)) - Using direct SQL, underlying > >DB is DERBY > >2015-07-21 08:18:54,120 INFO [main] metastore.ObjectStore > >(ObjectStore.java:setConf(272)) - Initialized ObjectStore > >2015-07-21 08:18:54,303 WARN [main] metastore.ObjectStore > >(ObjectStore.java:checkSchema(6658)) - Version information not found > >in metastore. hive.metastore.schema.verification is not enabled so > >recording the schema version 1.2.0 > >2015-07-21 08:18:54,508 WARN [main] metastore.ObjectStore > >(ObjectStore.java:getDatabase(568)) - Failed to get database default, > >returning NoSuchObjectException > >2015-07-21 08:18:54,694 INFO [main] metastore.HiveMetaStore > >(HiveMetaStore.java:createDefaultRoles_core(663)) - Added admin role > >in metastore > >2015-07-21 08:18:54,701 INFO [main] metastore.HiveMetaStore > >(HiveMetaStore.java:createDefaultRoles_core(672)) - Added public role > >in metastore > >2015-07-21 08:18:54,813 INFO [main] metastore.HiveMetaStore > >(HiveMetaStore.java:addAdminUsers_core(712)) - No user is added in > >admin role, since config is empty > >2015-07-21 08:18:54,980 INFO [main] metastore.HiveMetaStore > >(HiveMetaStore.java:logInfo(746)) - 0: get_databases: > >NonExistentDatabaseUsedForHealthCheck > >2015-07-21 08:18:54,981 INFO [main] HiveMetaStore.audit > >(HiveMetaStore.java:logAuditEvent(371)) - > >ugi=ambari-qa ip=unknown-ip-addr cmd=get_databases: > >NonExistentDatabaseUsedForHealthCheck > >2015-07-21 08:18:55,019 INFO [main] metastore.HiveMetaStore > >(HiveMetaStore.java:logInfo(746)) - 0: get_table : db=default > >tbl=simple > >2015-07-21 08:18:55,022 INFO [main] HiveMetaStore.audit > >(HiveMetaStore.java:logAuditEvent(371)) - > >ugi=ambari-qa ip=unknown-ip-addr cmd=get_table : db=default > tbl=simple > >18847 [main] ERROR org.apache.sqoop.tool.ImportTool - Encountered > >IOException running import job: java.io.IOException: > >NoSuchObjectException(message:default.simple table not found) > > at > >org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputForma > >t.java:97) > > at > >org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputForma > >t.java:51) > > at > >org.apache.sqoop.mapreduce.hcat.SqoopHCatUtilities.configureHCat(SqoopHCat > >Utilities.java:343) > > at > >org.apache.sqoop.mapreduce.hcat.SqoopHCatUtilities.configureImportOutputFo > >rmat(SqoopHCatUtilities.java:783) > > at > >org.apache.sqoop.mapreduce.ImportJobBase.configureOutputFormat(ImportJobBa > >se.java:98) > > at > >org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:259) > > at > org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:673) > > at > >org.apache.sqoop.manager.MySQLManager.importTable(MySQLManager.java:118) > > at > org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:497) > > at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605) > > at org.apache.sqoop.Sqoop.run(Sqoop.java:143) > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > > at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179) > > at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218) > > at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227) > > at org.apache.sqoop.Sqoop.main(Sqoop.java:236) > > at > >org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:197) > > at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:177) > > at > org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:47) > > at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:46) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > >sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java: > >62) > > at > >sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm > >pl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:497) > > at > >org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:236) > > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) > > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453) > > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAs(Subject.java:422) > > at > >org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation. > >java:1657) > > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > >Caused by: NoSuchObjectException(message:default.simple table not found) > > at > >org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_core(H > >iveMetaStore.java:1808) > > at > >org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveMe > >taStore.java:1778) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > >sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java: > >62) > > at > >sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm > >pl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:497) > > at > >org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHand > >ler.java:107) > > at com.sun.proxy.$Proxy20.get_table(Unknown Source) > > at > >org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStor > >eClient.java:1208) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > >sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java: > >62) > > at > >sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm > >pl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:497) > > at > >org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMe > >taStoreClient.java:152) > > at com.sun.proxy.$Proxy21.getTable(Unknown Source) > > at > org.apache.hive.hcatalog.common.HCatUtil.getTable(HCatUtil.java:180) > > at > >org.apache.hive.hcatalog.mapreduce.InitializeInput.getInputJobInfo(Initial > >izeInput.java:105) > > at > >org.apache.hive.hcatalog.mapreduce.InitializeInput.setInput(InitializeInpu > >t.java:86) > > at > >org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputForma > >t.java:95) > > ... 32 more > >
