[
https://issues.apache.org/jira/browse/HIVE-23238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17086481#comment-17086481
]
Panagiotis Garefalakis commented on HIVE-23238:
-----------------------------------------------
Hey [~gopalv] thanks for the comments! Ignoring the number of completedTasks
when preempting could indeed cause queries with lots of tasks that are near
completion to be victimised in favour of shorter running ones.
To avoid such cases, in [^HIVE-23238.02.patch] I added a completion percentage
comparison that preserves the most amount of work completed instead (queries
closer to completion will be preempted last).
Task locality is another interesting property to take into account during
preemption, but I believe this kind of decisions could also be taken into
account from the AM side (when sending tasks to the TaskExecutorService) e.g.,
critical tasks with data locality could always be made Guaranteed.
> FIX PreemptionQueueComparator edge cases
> ----------------------------------------
>
> Key: HIVE-23238
> URL: https://issues.apache.org/jira/browse/HIVE-23238
> Project: Hive
> Issue Type: Improvement
> Reporter: Panagiotis Garefalakis
> Assignee: Panagiotis Garefalakis
> Priority: Major
> Labels: pull-request-available
> Fix For: llap
>
> Attachments: HIVE-23238.01.patch, HIVE-23238.02.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Properly handle preemption comparator edge cases where tasks are same type
> and have the same number or upstream tasks.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
