I am a newbie.
Isn't clear to me if I should post this type of issue to dev or user, so I
decided to post to [email protected]
We have a reproducible script that works with -x mapreduce, fails with -x
tez
Failure while running
task:org.apache.pig.backend.executionengine.ExecException: ERROR 0: Input
from vertex scope-108 is missing
[snip]
pigstats.tez.TezPigScriptStats - Script Statistics:
HadoopVersion: 2.5.0-cdh5.3.2
PigVersion: 0.15.0-SNAPSHOT
TezVersion: 0.6.0
UserId: root
FileName: foo.pig
StartedAt: 2015-03-27 08:47:40
FinishedAt: 2015-03-27 08:49:08
Features: HASH_JOIN,GROUP_BY,FILTER,UNION
Failed!
Q: Is this a known issue?
Q: What kind of diagnostics/info can I provide to assist?
Michael
full backtrace
, progress=TotalTasks: 497 Succeeded: 24 Running: 0 Failed: 1 Killed: 472
FailedTaskAttempts: 11, diagnostics=Vertex failed, vertexName=scope-122,
vertexId=vertex_1427151599024_0195_1_08, diagnostics=[Task failed,
taskId=task_1427151599024_0195_1_08_000001, diagnostics=[TaskAttempt 0
failed, info=[Error: Failure while running
task:org.apache.pig.backend.executionengine.ExecException: ERROR 0: Input
from vertex scope-108 is missing
at
org.apache.pig.backend.hadoop.executionengine.tez.plan.udf.ReadScalarsTez.attachInputs(ReadScalarsTez.java:77)
at
org.apache.pig.backend.hadoop.executionengine.tez.runtime.PigProcessor.initializeInputs(PigProcessor.java:293)
at
org.apache.pig.backend.hadoop.executionengine.tez.runtime.PigProcessor.run(PigProcessor.java:183)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:324)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:176)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
], TaskAttempt 1 failed, info=[Error: Failure while running
task:org.apache.pig.backend.executionengine.ExecException: ERROR 0: Input
from vertex scope-108 is missing
at
org.apache.pig.backend.hadoop.executionengine.tez.plan.udf.ReadScalarsTez.attachInputs(ReadScalarsTez.java:77)
at
org.apache.pig.backend.hadoop.executionengine.tez.runtime.PigProcessor.initializeInputs(PigProcessor.java:293)
at
org.apache.pig.backend.hadoop.executionengine.tez.runtime.PigProcessor.run(PigProcessor.java:183)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:324)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:176)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
], TaskAttempt 2 failed, info=[Error: Failure while running
task:org.apache.pig.backend.executionengine.ExecException: ERROR 0: Input
from vertex scope-108 is missing
at
org.apache.pig.backend.hadoop.executionengine.tez.plan.udf.ReadScalarsTez.attachInputs(ReadScalarsTez.java:77)
at
org.apache.pig.backend.hadoop.executionengine.tez.runtime.PigProcessor.initializeInputs(PigProcessor.java:293)
at
org.apache.pig.backend.hadoop.executionengine.tez.runtime.PigProcessor.run(PigProcessor.java:183)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:324)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:176)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
], TaskAttempt 3 failed, info=[Container
container_1427151599024_0195_01_000010 finished with diagnostics set to
[Container preempted internally]]], Vertex failed as one or more tasks
failed. failedTasks:1, Vertex vertex_1427151599024_0195_1_08 [scope-122]
killed/failed due to:null]
Vertex killed, vertexName=scope-129,
vertexId=vertex_1427151599024_0195_1_09, diagnostics=[Vertex received Kill
while in RUNNING state., Vertex killed as other vertex failed.
failedTasks:0, Vertex vertex_1427151599024_0195_1_09 [scope-129]
killed/failed due to:null]
Vertex killed, vertexName=scope-130,
vertexId=vertex_1427151599024_0195_1_10, diagnostics=[Vertex received Kill
while in RUNNING state., Vertex killed as other vertex failed.
failedTasks:0, Vertex vertex_1427151599024_0195_1_10 [scope-130]
killed/failed due to:null]
Vertex killed, vertexName=scope-128,
vertexId=vertex_1427151599024_0195_1_07, diagnostics=[Vertex received Kill
while in RUNNING state., Vertex killed as other vertex failed.
failedTasks:0, Vertex vertex_1427151599024_0195_1_07 [scope-128]
killed/failed due to:null]
Vertex killed, vertexName=scope-126,
vertexId=vertex_1427151599024_0195_1_05, diagnostics=[Vertex received Kill
while in RUNNING state., Vertex killed as other vertex failed.
failedTasks:0, Vertex vertex_1427151599024_0195_1_05 [scope-126]
killed/failed due to:null]
Vertex killed, vertexName=scope-125,
vertexId=vertex_1427151599024_0195_1_06, diagnostics=[Vertex received Kill
while in RUNNING state., Vertex killed as other vertex failed.
failedTasks:0, Vertex vertex_1427151599024_0195_1_06 [scope-125]
killed/failed due to:null]
Vertex killed, vertexName=scope-115,
vertexId=vertex_1427151599024_0195_1_04, diagnostics=[Vertex received Kill
while in RUNNING state., Vertex killed as other vertex failed.
failedTasks:0, Vertex vertex_1427151599024_0195_1_04 [scope-115]
killed/failed due to:null]
DAG failed due to vertex failure. failedVertices:1 killedVertices:6,
counters=Counters: 43
org.apache.tez.common.counters.DAGCounter
NUM_FAILED_TASKS=11
NUM_KILLED_TASKS=36
NUM_SUCCEEDED_TASKS=24
TOTAL_LAUNCHED_TASKS=51
OTHER_LOCAL_TASKS=1
DATA_LOCAL_TASKS=21
File System Counters
FILE_BYTES_READ=187910630
FILE_BYTES_WRITTEN=3859160973
FILE_READ_OPS=0
FILE_LARGE_READ_OPS=0
FILE_WRITE_OPS=0
HDFS_BYTES_READ=2619256965
HDFS_BYTES_WRITTEN=0
HDFS_READ_OPS=21
HDFS_LARGE_READ_OPS=0
HDFS_WRITE_OPS=0
org.apache.tez.common.counters.TaskCounter
SPILLED_RECORDS=45455352
NUM_SHUFFLED_INPUTS=3
NUM_FAILED_SHUFFLE_INPUTS=0
GC_TIME_MILLIS=12756
CPU_MILLISECONDS=1366940
PHYSICAL_MEMORY_BYTES=15069728768
VIRTUAL_MEMORY_BYTES=37774110720
COMMITTED_HEAP_BYTES=18567659520
INPUT_RECORDS_PROCESSED=36917387
OUTPUT_RECORDS=83870869
OUTPUT_LARGE_RECORDS=0
OUTPUT_BYTES=8898320232
OUTPUT_BYTES_WITH_OVERHEAD=8241464266
OUTPUT_BYTES_PHYSICAL=3671993306
ADDITIONAL_SPILLS_BYTES_WRITTEN=62138713
ADDITIONAL_SPILLS_BYTES_READ=187158971
ADDITIONAL_SPILL_COUNT=18
SHUFFLE_BYTES=233406
SHUFFLE_BYTES_DECOMPRESSED=493440
SHUFFLE_BYTES_TO_MEM=233406
SHUFFLE_BYTES_TO_DISK=0
SHUFFLE_BYTES_DISK_DIRECT=0
SHUFFLE_PHASE_TIME=689
FIRST_EVENT_RECEIVED=228
LAST_EVENT_RECEIVED=235
org.apache.hadoop.mapreduce.TaskCounter
COMBINE_INPUT_RECORDS=20829242
COMBINE_OUTPUT_RECORDS=23476241
2015-03-27 08:49:07,439 [PigTezLauncher-0] INFO
org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is
deprecated. Instead, use fs.defaultFS
2015-03-27 08:49:07,597 [PigTezLauncher-0] INFO
org.apache.pig.tools.pigstats.JobStats - using output size reader:
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.FileBasedOutputSizeReader
2015-03-27 08:49:08,220 [main] INFO
org.apache.pig.tools.pigstats.tez.TezPigScriptStats - Script Statistics:
HadoopVersion: 2.5.0-cdh5.3.2
PigVersion: 0.15.0-SNAPSHOT
TezVersion: 0.6.0
UserId: root
FileName: foo.pig
StartedAt: 2015-03-27 08:47:40
FinishedAt: 2015-03-27 08:49:08
Features: HASH_JOIN,GROUP_BY,FILTER,UNION
Failed!
DAG PigLatin:foo.pig-0_scope-0:
ApplicationId: job_1427151599024_0195
TotalLaunchedTasks: 51
FileBytesRead: 187910630
FileBytesWritten: 3859160973
HdfsBytesRead: 2619256965
HdfsBytesWritten: 0
Input(s):
Failed to read data from "pig.XXXX"
Successfully read 1000 records (171668 bytes) from: "pig.YYYY_sample"
Successfully read 13438146 records (483689478 bytes) from: "pig.ZZZZ"
Output(s):
Failed to produce result in "hdfs://
XXXX.YYYY.com:8020/tmp/temp-498428122/tmp-576253744"
2015-03-27 08:49:08,302 [main] ERROR org.apache.pig.tools.grunt.Grunt -
ERROR 1066: Unable to open iterator for alias cntallclaims_patient
Details at logfile: /root/PIG/pig_1427460455726.log
2015-03-27 08:49:08,322 [main] INFO org.apache.pig.Main - Pig script
completed in 1 minute, 32 seconds and 719 milliseconds (92719 ms)
2015-03-27 08:49:08,344 [main] INFO
org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher - Shutting
down thread pool
2015-03-27 08:49:08,394 [Thread-37] INFO
org.apache.pig.backend.hadoop.executionengine.tez.TezSessionManager -
Shutting down Tez session org.apache.tez.client.TezClient@37bdce24
2015-03-27 08:49:08,395 [Thread-37] INFO org.apache.tez.client.TezClient -
Shutting down Tez Session, sessionName=PigLatin:foo.pig,
applicationId=application_1427151599024_0195
[root@ludwig PIG]#
