GitHub user vincent-grosbois opened a pull request:
https://github.com/apache/spark/pull/21933
[SPARK-24917] make chunk size configurable
Add an option in Spark configuration to change the chunk size (which is by
default 4 Mb).
This would allow to bypass the issue mentionned in SPARK-24917 by allowing
users to define larger chunks
## What changes were proposed in this pull request?
I'm introducing a configuration parameter to define the chunk size
## How was this patch tested?
not tested as it's a trivial test. I can write a test if needed
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/vincent-grosbois/spark chunk_size
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21933.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21933
----
commit 0251bd517e7fd3e695cb8366ffa03de8c9e2900b
Author: vincent-grosbois <vincent.grosbois@...>
Date: 2018-07-31T15:21:24Z
[SPARK-24917] make chunk size configurable
Add an option in Spark configuration to change the chunk size (which is by
default 4 Mb).
This would allow to bypass the issue mentionned in SPARK-24917 when
fetching large partitions (a bit less than 2 Gb)
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
