Hi,
I'm facing a problem accessing Hive tables that have the "date" data type in
Pig. I get the "java.lang.NoSuchMethodError:
org.joda.time.DateTime.<init>(IIIII)V"
The relevant YARN logs for the job below:
2014-11-08 22:45:17,132 INFO [main]
org.apache.hadoop.conf.Configuration.deprecation: mapred.task.id is deprecated.
Instead, use mapreduce.task.attempt.id
2014-11-08 22:45:17,163 INFO [main]
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader:
Current split being processed
org.apache.hive.hcatalog.mapreduce.HCatSplit@4248d35a
2014-11-08 22:45:17,278 INFO [main]
org.apache.hive.hcatalog.mapreduce.InternalUtil: Initializing
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe with properties
{name=default.testdate4, numFiles=1, field.delim=,, columns.types=date,
serialization.format=,, columns=dt, rawDataSize=0, numRows=0,
serialization.lib=org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe,
COLUMN_STATS_ACCURATE=true, totalSize=77, serialization.null.format=\N,
transient_lastDdlTime=1415395168}
2014-11-08 22:45:17,364 INFO [main] org.apache.pig.data.SchemaTupleBackend: Key
[pig.schematuple] was not set... will not generate code.
2014-11-08 22:45:17,411 INFO [main]
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map:
Aliases being processed per job phase (AliasName[line,offset]): M: A[1,4] C: R:
2014-11-08 22:45:17,442 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error
running child : java.lang.NoSuchMethodError:
org.joda.time.DateTime.<init>(IIIII)V
at
org.apache.hive.hcatalog.pig.PigHCatUtil.extractPigObject(PigHCatUtil.java:420)
at
org.apache.hive.hcatalog.pig.PigHCatUtil.transformToTuple(PigHCatUtil.java:456)
at
org.apache.hive.hcatalog.pig.PigHCatUtil.transformToTuple(PigHCatUtil.java:374)
at
org.apache.hive.hcatalog.pig.HCatBaseLoader.getNext(HCatBaseLoader.java:64)
at
org.apache.hive.hcatalog.pig.HCatLoader.getNext(HCatLoader.java:59)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.nextKeyValue(PigRecordReader.java:211)
at
org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:533)
at
org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
at
org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at
org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
I ran the following Pig code:
pig -useHCatalog
A = load 'testdate4' using org.apache.hive.hcatalog.pig.HCatLoader();
dump A;
The "testdate4" hive table details are as follows:
0: jdbc:hive2://hdpbox.musigma:10000> desc testdate4;
+-------------+---------------+-------------+
| col_name | data_type | comment |
+-------------+---------------+-------------+
| dt | date | |
+-------------+---------------+-------------+
0: jdbc:hive2://*****.*****:10000> select * from testdate4;
+---------------+
| testdate4.dt |
+---------------+
| 2011-09-01 |
| 2012-08-11 |
| 1992-10-18 |
| 1996-03-20 |
| 2007-07-23 |
| 2008-04-15 |
| 2014-12-30 |
+---------------+
Could it be related to the joda-time version as discussed in
https://issues.apache.org/jira/browse/PIG-3953
Any ideas on how I can resolve this?
Version details:
Hive - 0.13.0.2.1.7.0-784
Pig - Pig 0.12.1.2.1.7.0-784
HDP - 2.1
P.S. I can't use "org.apache.hcatalog.pig.HCatLoader()" either as it does not
support the "date type" (ERROR 1200: Type date not present)
Gufran Pathan| +91 7760913355|
Disclaimer: http://www.mu-sigma.com/disclaimer.html
