Github user jinxing64 commented on the issue:
https://github.com/apache/spark/pull/17533
Yes, I did the test in my cluster. In highly-skew stage, the time cost can
be reduced significantly. Tasks are scheduled with locality preference. But in
current code, input size of tasks are not taken into consideration. Think about
this scenario:
1. There are 9 partitions(0~8) in the ShuffledRDD and size of partition-8
is 8 times of the previous 8 partitions. (Lets assume that time cost of task
has linear relation with the size of input and time cost of first 8 tasks is 1
and the time cost of the last task is 8.)
2. Tasks are scheduled on 2 executors.
In current code, the tasks are scheduled in serial order and task for
partition-8 will be the last one to launch and the time cost is 12.
With this change, task for partition-8 will be scheduled first and the time
cost will be reduced to 8.
This change is related to SPARK-19100. In my prod env, skew situations
happens mostly on ShuffledRDD. Thus this pr proposes to consider the size of
input from ShuffledRDD when scheduling. This change can bring benefit when skew
situations and won't have negative impact on performance in other scenarios.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
