Rajesh Balamohan created HIVE-24061:

             Summary: Improve llap task scheduling for better cache hit rate 
                 Key: HIVE-24061
                 URL: https://issues.apache.org/jira/browse/HIVE-24061
             Project: Hive
          Issue Type: Improvement
            Reporter: Rajesh Balamohan

TaskInfo is initialized with the "requestTime and locality delay". When lots of 
vertices are in the same level, "taskInfo" details would be available upfront. 
By the time, it gets to scheduling, "requestTime + localityDelay" won't be 
higher than current time. Due to this, it misses scheduling delay details and 
ends up choosing random node. This ends up missing cache hits and reads data 
from remote storage.

E.g Observed this pattern in Q75 of tpcds.

Related lines of interest in scheduler: 
   boolean shouldDelayForLocality = 
    boolean shouldDelayForLocality(long schedulerAttemptTime) {
      return localityDelayTimeout > schedulerAttemptTime;

Ideally, "localityDelayTimeout" should be adjusted based on it's first 
scheduling opportunity.

This message was sent by Atlassian Jira

Reply via email to