GitHub user raajay opened a pull request:
https://github.com/apache/spark/pull/18683
[SPARK-21474][CORE] Make number of parallel fetches from a reducer
configurable
## What changes were proposed in this pull request?
Currently the number of parallel fetches is hard-coded to 5. As a result
the size of each fetch request is fixed at 1/5th of maxSizeInFlights. Since,
chunks are requested in bursts of fetchRequests; the size of a burst to a
single shuffle service can be high for large maxSizeInFlight
Introduce a new configuration parameter
"spark.reducer.numParallelFetchRequets" to make it configurable.
## How was this patch tested?
Not tested.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/raajay/spark
raajay-configure-parallel-requests
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/18683.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #18683
----
commit 9bf1f9fd5aa08ef19653a4cfa52a8eccc64dc18b
Author: Raajay Viswanathan <raaja...@gmail.com>
Date: 2017-07-19T16:49:25Z
Make num parallel fetches from a reducer configurable
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
