[
https://issues.apache.org/jira/browse/SPARK-17839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shixiong Zhu resolved SPARK-17839.
----------------------------------
Resolution: Fixed
Assignee: Sital Kedia
Fix Version/s: 2.1.0
> Use Nio's directbuffer instead of BufferedInputStream in order to avoid
> additional copy from os buffer cache to user buffer
> ----------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-17839
> URL: https://issues.apache.org/jira/browse/SPARK-17839
> Project: Spark
> Issue Type: Improvement
> Components: Shuffle
> Affects Versions: 2.0.1
> Reporter: Sital Kedia
> Assignee: Sital Kedia
> Priority: Minor
> Fix For: 2.1.0
>
>
> Currently we use BufferedInputStream to read the shuffle file which copies
> the file content from os buffer cache to the user buffer. This adds
> additional latency in reading the spill files. We made a change to use java
> nio's direct buffer to read the spill files and for certain jobs spilling
> significant amount of data, we see between 5 - 7% speedup.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
