Hi all,
(Running Hive 12.0)
I have two tables and both are stored as ORC. I attempted to insert via select
from tbl1 to tbl2 using 'BETWEEN' in my where clause to narrow down some dates.
Something like so:
"Insert into tbl1 select col1, col2 from tbl2 where col1 between 2 and 4"
I kept hitting the error pasted below. So, I switched to a different approach
to see if it would work:
"Insert into tbl1 select col1,col2 from tbl2 where col1>=2 and col1<=4"
Hit the same error. When I just use "where col1>=2" in the where clause the
insert will run fine.
Is this expected?
2014-02-26 11:22:53,755 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
mapreduce.job.end-notification.max.retry.interval; Ignoring.
2014-02-26 11:22:53,782 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
mapreduce.job.end-notification.max.attempts; Ignoring.
2014-02-26 11:22:53,902 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
hadoop-metrics2.properties
2014-02-26 11:22:53,930 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSinkAdapter: Sink ganglia started
2014-02-26 11:22:53,975 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at
10 second(s).
2014-02-26 11:22:53,975 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system
started
2014-02-26 11:22:53,985 INFO [main] org.apache.hadoop.mapred.YarnChild:
Executing with tokens:
2014-02-26 11:22:53,985 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind:
mapreduce.job, Service: job_1392147432508_1108, Ident:
(org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@249c2715)
2014-02-26 11:22:54,057 INFO [main] org.apache.hadoop.mapred.YarnChild:
Sleeping for 0ms before retrying again. Got null now.
2014-02-26 11:22:54,352 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
mapreduce.job.end-notification.max.retry.interval; Ignoring.
2014-02-26 11:22:54,363 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
mapreduce.job.end-notification.max.attempts; Ignoring.
2014-02-26 11:22:54,409 INFO [main] org.apache.hadoop.mapred.YarnChild:
mapreduce.cluster.local.dir for child:
/hdfs/01/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/02/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/03/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/04/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/05/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/06/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/07/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/08/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/09/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/10/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/11/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108,/hdfs/12/hadoop/yarn/local/usercache/myusername/appcache/application_1392147432508_1108
2014-02-26 11:22:54,481 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
mapreduce.job.end-notification.max.retry.interval; Ignoring.
2014-02-26 11:22:54,486 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
mapreduce.job.end-notification.max.attempts; Ignoring.
2014-02-26 11:22:54,542 INFO [main]
org.apache.hadoop.conf.Configuration.deprecation: mapred.task.id is deprecated.
Instead, use mapreduce.task.attempt.id
2014-02-26 11:22:54,542 INFO [main]
org.apache.hadoop.conf.Configuration.deprecation: mapred.task.is.map is
deprecated. Instead, use mapreduce.task.ismap
2014-02-26 11:22:54,543 INFO [main]
org.apache.hadoop.conf.Configuration.deprecation: mapred.local.dir is
deprecated. Instead, use mapreduce.cluster.local.dir
2014-02-26 11:22:54,543 INFO [main]
org.apache.hadoop.conf.Configuration.deprecation: mapred.cache.localFiles is
deprecated. Instead, use mapreduce.job.cache.local.files
2014-02-26 11:22:54,543 INFO [main]
org.apache.hadoop.conf.Configuration.deprecation: mapred.job.id is deprecated.
Instead, use mapreduce.job.id
2014-02-26 11:22:54,544 INFO [main]
org.apache.hadoop.conf.Configuration.deprecation: mapred.tip.id is deprecated.
Instead, use mapreduce.task.id
2014-02-26 11:22:54,544 INFO [main]
org.apache.hadoop.conf.Configuration.deprecation: mapred.task.partition is
deprecated. Instead, use mapreduce.task.partition
2014-02-26 11:22:54,545 INFO [main]
org.apache.hadoop.conf.Configuration.deprecation: job.local.dir is deprecated.
Instead, use mapreduce.job.local.dir
2014-02-26 11:22:54,546 INFO [main]
org.apache.hadoop.conf.Configuration.deprecation: mapred.cache.localArchives is
deprecated. Instead, use mapreduce.job.cache.local.archives
2014-02-26 11:22:54,710 INFO [main]
org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated.
Instead, use dfs.metrics.session-id
2014-02-26 11:22:55,007 INFO [main] org.apache.hadoop.mapred.Task: Using
ResourceCalculatorProcessTree : [ ]
2014-02-26 11:22:55,316 INFO [main] org.apache.hadoop.mapred.MapTask:
Processing split:
Paths:/apps/hive/warehouse/edw.db/penopp_fsm/000000_0:0+116456645,/apps/hive/warehouse/edw.db/penopp_fsm/000001_0:0+116484938,/apps/hive/warehouse/edw.db/penopp_fsm/000002_0:0+116473509InputFormatClass:
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
2014-02-26 11:22:55,370 INFO [main] org.apache.hadoop.hive.ql.exec.Utilities:
<PERFLOG method=deserializePlan>
2014-02-26 11:22:55,370 INFO [main] org.apache.hadoop.hive.ql.exec.Utilities:
Deserializing MapWork via kryo
2014-02-26 11:22:55,574 INFO [main] org.apache.hadoop.hive.ql.exec.Utilities:
</PERFLOG method=deserializePlan start=1393431775370 end=1393431775574
duration=204>
2014-02-26 11:22:55,708 INFO [main]
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat: included column ids =
0,1,2,3,4,5,6
2014-02-26 11:22:55,708 INFO [main]
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat: included columns names =
col1,col2,col3,col4,col5,col6,col7
2014-02-26 11:22:55,728 INFO [main]
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat: ORC pushdown predicate: leaf-0
= (BETWEEN col7 201301 201312)
expr = leaf-0
2014-02-26 11:22:55,897 ERROR [main]
org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:myusername (auth:SIMPLE) cause:java.io.IOException:
java.lang.reflect.InvocationTargetException
2014-02-26 11:22:55,898 WARN [main] org.apache.hadoop.mapred.YarnChild:
Exception running child : java.io.IOException:
java.lang.reflect.InvocationTargetException
at
org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
at
org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
at
org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:343)
at
org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.<init>(HadoopShimsSecure.java:290)
at
org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getRecordReader(HadoopShimsSecure.java:404)
at
org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:556)
at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:167)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:408)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:329)
... 11 more
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.planReadPartialDataStreams(RecordReaderImpl.java:1962)
at
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.readPartialDataStreams(RecordReaderImpl.java:2118)
at
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.readStripe(RecordReaderImpl.java:1775)
at
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.advanceStripe(RecordReaderImpl.java:2148)
at
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.advanceToNextRow(RecordReaderImpl.java:2190)
at
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.<init>(RecordReaderImpl.java:130)
at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rows(ReaderImpl.java:335)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$OrcRecordReader.<init>(OrcInputFormat.java:100)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:205)
at
org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.<init>(CombineHiveRecordReader.java:65)
... 16 more
2014-02-26 11:22:55,902 INFO [main] org.apache.hadoop.mapred.Task: Runnning
cleanup for the task
