Sital Kedia created SPARK-20014:
-----------------------------------
Summary: Optimize mergeSpillsWithFileStream method
Key: SPARK-20014
URL: https://issues.apache.org/jira/browse/SPARK-20014
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 2.0.0
Reporter: Sital Kedia
When the individual partition size in a spill is small,
mergeSpillsWithTransferTo method does many small disk ios which is really
inefficient. One way to improve the performance will be to use
mergeSpillsWithFileStream method by turning off transfer to and using buffered
file read/write to improve the io throughput.
However, the current implementation of mergeSpillsWithFileStream does not do a
buffer read/write of the files and in addition to that it unnecessarily
opens/close the output files for each partitions.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
