[
https://issues.apache.org/jira/browse/PIG-4069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136710#comment-14136710
]
Rohini Palaniswamy commented on PIG-4069:
-----------------------------------------
Had [~sseth] investigate the job logs. His response to my mail as below
For a job which does just load and limit, there are two vertices
- Load vertex (parallelism=27, vertex id=scope-3)
- Limit vertex (parallelism=1, vertex id=scope-5)
I set tez.shuffle-vertex-manager.min-src-fraction and max fraction to 0.00001
for the limit vertex. The edge connecting them uses unordered output and input.
Few issues:
- The second vertex task was launched only after launching all first vertex
tasks even if src fraction was set to such low value.
Sidd: Tasks for the second vertex will get scheduled the moment the specified
fraction of tasks on the source vertex complete. However, all tasks on the
previous vertex are already scheduled, and with a higher execution priority -
so they will execute first.
A couple of ways to get past this.
1) To control the way the tasks on the first vertex are scheduled via a custom
plugin. Essentially - wait for feedback, and launch tasks only when they are
required. This may be the easier approach.
2) Invert priorities (avoid DAGSchedulerNaturalOrder). This can get complicated
really fast, especially, when multiple edges are involved, and has led to
daedlocks in the past.
- Even if second vertex completes, the DAG has to wait till the source tasks
of first vertex complete.
Sidd: This is expected behaviour. A vertex does not complete till all it's
tasks complete, and the DAG does not complete till all vertices complete. We'll
have to introduce the ability to KILL tasks with state SUCCESS, or just mark
unlaunched tasks as SUCCESSFUL for this to work.
- Another thing I am seeing is that there are two attempts launched for first
vertex. The second attempts are after the second vertex completes.
Sidd: This is a bug. The ShuffleManager is supposed to ignore all errors from
the fetchers which it could not shutdown if the processor is complete. It's not
doing this in certain circumstances. Should be fairly easy to fix.
My expectation was that once the destination vertex completes, src vertex will
be terminated without issues and job will finish fast. But that is not the
case. In fact it took more time because of the retry attempts. Have attached
the whole application log.
Sidd: See notes above about expected behaviour - and the requirement to be able
to mark a task as KILLED+SUCCEEDED. I'd prefer avoiding special casing DAG
completion on leaf vertex completion - since we don't really know much about
what is being generated by the intermediate vertices.
I will create jiras once one of you confirm the issues or clarify the expected
behaviour.
> Limit reduce task should start as soon as one map task finishes
> ---------------------------------------------------------------
>
> Key: PIG-4069
> URL: https://issues.apache.org/jira/browse/PIG-4069
> Project: Pig
> Issue Type: Sub-task
> Components: tez
> Reporter: Rohini Palaniswamy
> Fix For: 0.14.0
>
> Attachments: PIG-4069-1.patch
>
>
> Set very low values for
> ShuffleVertexManager.TEZ_AM_SHUFFLE_VERTEX_MANAGER_MIN_SRC_FRACTION and
> ShuffleVertexManager.TEZ_AM_SHUFFLE_VERTEX_MANAGER_MAX_SRC_FRACTION in case
> of LIMIT job not following an order by so that the reduce task starts as soon
> as 1 map task finishes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
