Zoltán Borók-Nagy created IMPALA-11539:
------------------------------------------
Summary: Intra-node skew for HDFS scans
Key: IMPALA-11539
URL: https://issues.apache.org/jira/browse/IMPALA-11539
Project: IMPALA
Issue Type: Bug
Components: Backend
Reporter: Zoltán Borók-Nagy
Before IMPALA-9655 scan ranges were statically assigned to intra-node fragment
instances based on Least-Processing Time algorithm:
https://github.com/apache/impala/blame/a7866a94578be6289bbac31686de4d9032ad9261/be/src/scheduling/scheduler.cc#L499-L501
>From IMPALA-9655 we use dynamic intra-node load balancing for HDFS scans. It
>means fragment instances have a shared queue of scan ranges and the fragment
>instances grab the next scan range to be read from this queue.
IMPALA-9655 got rid of the LPT-algorithm which means the scan ranges are in a
random order in the queue. This can lead to a skew if there are large scan
ranges at the end.
We could mix the above two by using a priority queue for the scan ranges, so
each fragment instance would grab the largest scan range in the queue. This
could further mitigate intra-node skewing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
