Github user jinxing64 commented on the issue:
https://github.com/apache/spark/pull/18388
Thanks a lot for quick reply :)
Yes, this patch doesn't guarantee avoiding the OOM on shuffle service when
all reducers are opening the blocks at the same time. But we can alleviate this
by adjusting `spark.reducer.maxSizeInFlight`. `ShuffleBlockFetcherIterator`
will break the blocks in several requests(see
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala#L240).
e.g. 500 blocks can be split into 20 requests, which will be send one by one
to shuffle service. If memory cost is high on shuffle service, following
requests will be rejected. In normal case this is pretty useful.
Also if the the `OpenBlocks` is rejected, reducer can sleep for a random
duration, say 2s~5s. Thus help to avoid all reducer open blocks at the same
time.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
