Hi ,
I am currently working on a experimental feature to read data from HIVE
using HadoopInputFormatIO but facing an error(stating that the table is not
found though it is present).
Would appreciate any pointers / suggestions !
Runner : data-flow runner
Code snippet :
PipelineOptions options = PipelineOptionsFactory.*fromArgs*(args).create();
Pipeline p = Pipeline.*create*(options);
Configuration conf = *getConfiguration*();
PCollection<KV<Long, String>> hiveData = p.apply(*HadoopInputFormatIO*
.<Long, String>read().withConfiguration(conf));
The *getConfiguration() *method goes as below:
Configuration conf = *new* Configuration();
conf.set("datanucleus.schema.autoCreateTables","true");
Job job = *new* *Job**(**conf**, **"HCat"**)*;
HCatInputFormat.*setInput*(job, *null*, "data_table");
job.setMapperClass(Map.*class*);
job.setNumReduceTasks(0);
job.setInputFormatClass(HCatInputFormat.*class*);
job.setMapOutputKeyClass(IntWritable.*class*);
job.setMapOutputValueClass(IntWritable.*class*);
job.setOutputKeyClass(WritableComparable.*class*);
job.setOutputValueClass(DefaultHCatRecord.*class*);
conf.setClass("mapreduce.job.inputformat.class",
HCatInputFormat.*class*, InputFormat.*class*);
conf.setClass("key.class", WritableComparable.*class*, Writable.*class*
);
conf.setClass("value.class", DefaultHCatRecord.*class*, HCatRecord.
*class*);
*return* conf;
Exception stack trace :
java.io.IOException: NoSuchObjectException(message:default.data_table table
not found)
at
org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:97)
at
org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:51)
at
org.apache.beam.examples.TestNew.getConfiguration(TestNew.java:150)
at org.apache.beam.examples.TestNew.testHiveIO(TestNew.java:84)
at org.apache.beam.examples.TestNew.main(TestNew.java:51)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:293)
at java.lang.Thread.run(Thread.java:745)
Caused by: NoSuchObjectException(message:default.data_table table not found)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_core(HiveMetaStore.java:1937)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveMetaStore.java:1890)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:140)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99)
at com.sun.proxy.$Proxy67.get_table(Unknown Source)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1321)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:150)
at com.sun.proxy.$Proxy68.getTable(Unknown Source)
at
org.apache.hive.hcatalog.common.HCatUtil.getTable(HCatUtil.java:180)
at
org.apache.hive.hcatalog.mapreduce.InitializeInput.getInputJobInfo(InitializeInput.java:105)
at
org.apache.hive.hcatalog.mapreduce.InitializeInput.setInput(InitializeInput.java:88)
at
org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:95)
... 10 more
