On 8/11/14, 1:48 PM, karthik Srivasthava wrote:
Hi,
Below was my log.. I couldnt find where the error is. Can you please point
out what caused my error...
The correct items to post back would be hive.tez.container.size,
hive.tez.java.opts and the value of io.sort.mb.
It does look like you have a small container/wrong -Xmx and a very big
sort buffer.
Cheers,
Gopal
Map 1: -/- Reducer 2: 0/20
Map 1: 0/138 Reducer 2: 0/20
Map 1: 0/138 Reducer 2: 0/20
Map 1: 0/138 Reducer 2: 0/20
Map 1: 0/138 Reducer 2: 0/20
Map 1: 0/138 Reducer 2: 0/20
Map 1: 0/138 Reducer 2: 0/20
Map 1: 0/138 Reducer 2: 0/20
Status: Failed
Vertex failed, vertexName=Map 1, vertexId=vertex_1407300075548_0189_1_01,
diagno
stics=[Task failed, taskId=task_1407300075548_0189_1_01_000001,
diagnostics=[Att
emptID:attempt_1407300075548_0189_1_01_000001_0 Info:Error:
java.lang.RuntimeExc
eption: java.lang.OutOfMemoryError: Java heap space
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java
:174)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOPro
cessorRuntimeTask.java:307)
at
org.apache.hadoop.mapred.YarnTezDagChild$5.run(YarnTezDagChild.java:5
64)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInforma
tion.java:1594)
at
org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:55
3)
Caused by: java.lang.OutOfMemoryError: Java heap space
at
org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter.<i
nit>(DefaultSorter.java:137)
at
org.apache.tez.runtime.library.output.OnFileSortedOutput.start(OnFile
SortedOutput.java:103)
at
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordP
rocessor.java:86)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java
:164)
... 6 more
, AttemptID:attempt_1407300075548_0189_1_01_000001_1 Info:Error:
java.lang.Runti
meException: java.lang.OutOfMemoryError: Java heap space
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java
:174)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOPro
cessorRuntimeTask.java:307)
at
org.apache.hadoop.mapred.YarnTezDagChild$5.run(YarnTezDagChild.java:5
64)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInforma
tion.java:1594)
at
org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:55
3)
Caused by: java.lang.OutOfMemoryError: Java heap space
at
org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter.<i
nit>(DefaultSorter.java:137)
at
org.apache.tez.runtime.library.output.OnFileSortedOutput.start(OnFile
SortedOutput.java:103)
at
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordP
rocessor.java:86)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java
:164)
... 6 more
Container released by application,
AttemptID:attempt_1407300075548_0189_1_01_000
001_2 Info:Error: java.lang.RuntimeException: java.lang.OutOfMemoryError:
Java h
eap space
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java
:174)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOPro
cessorRuntimeTask.java:307)
at
org.apache.hadoop.mapred.YarnTezDagChild$5.run(YarnTezDagChild.java:5
64)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInforma
tion.java:1594)
at
org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:55
3)
Caused by: java.lang.OutOfMemoryError: Java heap space
at
org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter.<i
nit>(DefaultSorter.java:137)
at
org.apache.tez.runtime.library.output.OnFileSortedOutput.start(OnFile
SortedOutput.java:103)
at
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordP
rocessor.java:86)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java
:164)
... 6 more
Container released by application,
AttemptID:attempt_1407300075548_0189_1_01_000
001_3 Info:Error: java.lang.RuntimeException: java.lang.OutOfMemoryError:
Java h
eap space
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java
:174)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOPro
cessorRuntimeTask.java:307)
at
org.apache.hadoop.mapred.YarnTezDagChild$5.run(YarnTezDagChild.java:5
64)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInforma
tion.java:1594)
at
org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:55
3)
Caused by: java.lang.OutOfMemoryError: Java heap space
at
org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter.<i
nit>(DefaultSorter.java:137)
at
org.apache.tez.runtime.library.output.OnFileSortedOutput.start(OnFile
SortedOutput.java:103)
at
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordP
rocessor.java:86)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java
:164)
... 6 more
], Vertex failed as one or more tasks failed. failedTasks:1]
Vertex killed, vertexName=Reducer 2,
vertexId=vertex_1407300075548_0189_1_00, di
agnostics=[Vertex received Kill while in RUNNING state., Vertex killed as
other
vertex failed. failedTasks:0]
DAG failed due to vertex failure. failedVertices:1 killedVertices:1
FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.tez.T
ezTask
