GitHub user jinxing64 opened a pull request:
https://github.com/apache/spark/pull/16989
[SPARK-19659] Fetch big blocks to disk when shuffle-read.
## What changes were proposed in this pull request?
Currently the whole block is fetched into memory(off heap by default) when
shuffle-read. A block is defined by (shuffleId, mapId, reduceId). Thus it can
be large when skew situations. If OOM happens during shuffle read, job will be
killed and users will be notified to "Consider boosting
spark.yarn.executor.memoryOverhead". Adjusting parameter and allocating more
memory can resolve the OOM. However the approach is not perfectly suitable for
production environment, especially for data warehouse.
Using Spark SQL as data engine in warehouse, users hope to have a unified
parameter(e.g. memory) but less resource wasted(resource is allocated but not
used),
It's not always easy to predict skew situations, when happen, it make sense
to fetch remote blocks to disk for shuffle-read, rather than
kill the job because of OOM.
Fetching big blocks to disk is mentioned in SPARK-3019. In this pr, when
sendRequest in `ShuffleBlockFetcherIterator`, check if `bytesInFlight` is over
`maxBytesInFlight`, if so, fetch remote blocks to disk by sending
`StreamRequest`. By setting `spark.reducer.maxSizeInFlight` properly, user can
avoid OOM without causing performance degradation.
## How was this patch tested?
Not added yet.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jinxing64/spark SPARK-19659
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16989.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16989
----
commit 21f6da36b127956bf35da088f1ecfeb55b307f3e
Author: jinxing <jinxing6...@126.com>
Date: 2017-02-18T15:00:50Z
[SPARK-19659] Fetch big blocks to disk when shuffle-read.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
