[
https://issues.apache.org/jira/browse/HIVE-23238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17086162#comment-17086162
]
Gopal Vijayaraghavan commented on HIVE-23238:
---------------------------------------------
Minor license nit.
This (and the original code) has a possible problem scenario where a reducer
with 1009 reducers with 1 pending to finish is going to be punished vs a task
which has 199 out of 199.
As in, the assumption of upstream tasks ignores pending tasks vs total upstream
tasks. The one which has 1008 finished might actually have pulled & merged data
locally, which makes it a bad candidate to preempt (for lost work).
I need to think a bit through this about whether the preemption should care
about host locality pinning or not (i.e a task which has no locality will go to
another host, while a pinned task will retry itself on the same host).
> FIX PreemptionQueueComparator edge cases
> ----------------------------------------
>
> Key: HIVE-23238
> URL: https://issues.apache.org/jira/browse/HIVE-23238
> Project: Hive
> Issue Type: Improvement
> Reporter: Panagiotis Garefalakis
> Assignee: Panagiotis Garefalakis
> Priority: Major
> Labels: pull-request-available
> Fix For: llap
>
> Attachments: HIVE-23238.01.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Properly handle preemption comparator edge cases where tasks are same type
> and have the same number or upstream tasks.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
