[
https://issues.apache.org/jira/browse/HIVE-22966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17052118#comment-17052118
]
Panagiotis Garefalakis edited comment on HIVE-22966 at 3/5/20, 1:20 PM:
------------------------------------------------------------------------
Just to be clear here: waiting time has nothing to do with starvation as all
tasks eventually complete within a vertex no matter how long they wait.
Across vertices, a vertex can not really affect the completion of another as
their priorities are different (so no starvation there either).
In this patch we are using wait time as a proxy of a long-waiting task which is
a really weak assumption – it might work for a query but it might not work for
another based on the task runtime distributions – a better approach would be
to use input/split size stats to make more involved decisions when prioritising
tasks which requires significantly more work.
Other than that, with this patch we are assigning resources to tasks in a more
Fair manner so it does not hurt – but to avoid tasks with long-tails executed
at the end there is more work involved.
That said it is a useful patch to have.
was (Author: pgaref):
Just to be clear here: waiting time has nothing to do with starvation as all
tasks eventually complete within a vertex no matter how long they wait.
Across vertices, a vertex can not really affect the completion of another as
their priorities are different (so no starvation there either).
In this patch we are using wait time as a proxy of a long-waiting task which is
a really weak assumption – it might work for a query but it might not work for
another based on the task runtime distributions – a better approach would be
to use input/split size stats to make more involved decisions when prioritising
tasks which requires significantly more work.
Other than that, with this patch we are assigning resources to tasks in a more
Fair manner so it does not hurt – but to avoid tasks with long-tails executed
at the end there is more work involved.
+1 from me well
> LLAP: Consider including waitTime for comparing attempts in same vertex
> -----------------------------------------------------------------------
>
> Key: HIVE-22966
> URL: https://issues.apache.org/jira/browse/HIVE-22966
> Project: Hive
> Issue Type: Improvement
> Components: llap
> Reporter: Rajesh Balamohan
> Assignee: Rajesh Balamohan
> Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-22966.3.patch, HIVE-22966.4.patch
>
>
> When attempts are compared within same vertex, it should pick up the attempt
> with longest wait time to avoid starvation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
