[
https://issues.apache.org/jira/browse/TEZ-2317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14497163#comment-14497163
]
Bikas Saha commented on TEZ-2317:
---------------------------------
bq. Optimize by not sending a commit go/no-go request if there is no hdfs
output (DataSink) involved. In the above case, it is always intermediate output
Fix in Pig
bq. Handle the commit go/no-go request after processing events in the event
queue. May be something like ask the task to come back after some time
In this jira
bq. We saw that for 3058 KilledTaskAttempts TA_KILL_REQUEST events was 383519.
This is way high.
That is because each canCommit request from the task was resulting in a kill
event being enqueued. Not killing (in this jira) will fix that.
bq.In the attached AM-taskkill.log which has grepped statements for a single
task that was killed, it has 327 repeats of below message. Need to see why so
much and fix that.
The log happens for each canCommit call from the task that gets denied because
the AM task state is not running. Can change to debug in this patch. The pig
processor is calling canCommit every 100ms.
> Successful task attempts getting killed
> ---------------------------------------
>
> Key: TEZ-2317
> URL: https://issues.apache.org/jira/browse/TEZ-2317
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Rohini Palaniswamy
> Assignee: Bikas Saha
> Fix For: 0.7.0
>
> Attachments: AM-taskkill.log
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
