[
https://issues.apache.org/jira/browse/HIVE-24061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan reassigned HIVE-24061:
---------------------------------------
Assignee: Rajesh Balamohan
> Improve llap task scheduling for better cache hit rate
> -------------------------------------------------------
>
> Key: HIVE-24061
> URL: https://issues.apache.org/jira/browse/HIVE-24061
> Project: Hive
> Issue Type: Improvement
> Reporter: Rajesh Balamohan
> Assignee: Rajesh Balamohan
> Priority: Major
> Labels: perfomance, pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> TaskInfo is initialized with the "requestTime and locality delay". When lots
> of vertices are in the same level, "taskInfo" details would be available
> upfront. By the time, it gets to scheduling, "requestTime + localityDelay"
> won't be higher than current time. Due to this, it misses scheduling delay
> details and ends up choosing random node. This ends up missing cache hits and
> reads data from remote storage.
> E.g Observed this pattern in Q75 of tpcds.
> Related lines of interest in scheduler:
> [https://github.com/apache/hive/blob/master/llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
>
> |https://github.com/apache/hive/blob/master/llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java]
> {code:java}
> boolean shouldDelayForLocality =
> request.shouldDelayForLocality(schedulerAttemptTime);
> ..
> ..
> boolean shouldDelayForLocality(long schedulerAttemptTime) {
> return localityDelayTimeout > schedulerAttemptTime;
> }
> {code}
>
> Ideally, "localityDelayTimeout" should be adjusted based on it's first
> scheduling opportunity.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
