I have multiple tables, and was trying to recursively join them. The first join
finishes with no error but not the second
i am not sure if it have something to do using hive tables as input to the next
join... because every time I use external tables that i created there is no
error at all, it finishes the job.
in hive.log
2013-08-30 17:06:32,145 WARN shims.HadoopShimsSecure
(Hadoop23Shims.java:getTaskAttemptLogUrl(48)) - Can't fetch tasklog:
TaskLogServlet is not supported in MR2 mode.
I set this property to fix this problem but still no change...
<property>
<name>hive.exec.show.job.failure.debug.info</name>
<value>false</value>
</property>
task tracker log
2013-08-30 18:01:58,200 FATAL [IPC Server handler 25 on 47684]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task:
attempt_1377878075746_0001_m_000025_0 - exited : java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row {............}
at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:161)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:399)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error
while processing row {.............}
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:565)
at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143)
... 8 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.io.IOException: Spill failed
at
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:304)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:83)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:546)
... 9 more
Caused by: java.io.IOException: Spill failed
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.checkSpillException(MapTask.java:1489)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1011)
at org.apache.hadoop.mapred.MapTask$OldOutputCollector.collect(MapTask.java:550)
at
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:289)
... 15 more
Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not
find any valid local directory for
attempt_1377878075746_0001_m_000025_0_spill_1.out
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:385)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
at
org.apache.hadoop.mapred.YarnOutputFiles.getSpillFileForWrite(YarnOutputFiles.java:159)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1522)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$600(MapTask.java:813)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1459)
the syntax i used is
CREATE TABLE Join_Two ROW FORMAT DELIMITED FIELDS TERMINATED BY '9' LINES
TERMINATED BY '\n' STORED AS TEXTFILE as select /*+MAPJOIN(JOIN_TWO)*/ * From
Join_one fact LEFT OUTER JOIN Dim2 dim WHERE fact.id = dim.id;
I am using
Hadoop 0.23.1
Hive 0.10.0
MR2/Yarn
I have 9 data nodes and 1 name node. The tables are not big tables, I am sure
it is not a size problem.
