[
https://issues.apache.org/jira/browse/SPARK-20074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16618061#comment-16618061
]
Kevin English commented on SPARK-20074:
---------------------------------------
I found this issue from external content that indicated that this limits IO
write block sizes, for instance for parquet files following a N:1
re-partitioning. Can someone confirm that being able to radically increase
this value would reduce spilling when aggregating a large number of small files?
> Make buffer size in unsafe external sorter configurable
> -------------------------------------------------------
>
> Key: SPARK-20074
> URL: https://issues.apache.org/jira/browse/SPARK-20074
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 2.0.1
> Reporter: Sital Kedia
> Priority: Major
>
> Currently, it is hardcoded to 32kb, see -
> https://github.com/sitalkedia/spark/blob/master/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java#L123
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
