Shixiong Zhu resolved SPARK-17839.
       Resolution: Fixed
         Assignee: Sital Kedia
    Fix Version/s: 2.1.0

> Use Nio's directbuffer instead of BufferedInputStream in order to avoid 
> additional copy from os buffer cache to user buffer 
> ----------------------------------------------------------------------------------------------------------------------------
>                 Key: SPARK-17839
>                 URL: https://issues.apache.org/jira/browse/SPARK-17839
>             Project: Spark
>          Issue Type: Improvement
>          Components: Shuffle
>    Affects Versions: 2.0.1
>            Reporter: Sital Kedia
>            Assignee: Sital Kedia
>            Priority: Minor
>             Fix For: 2.1.0
> Currently we use BufferedInputStream to read the shuffle file which copies 
> the file content from os buffer cache to the user buffer. This adds 
> additional latency in reading the spill files. We made a change to use java 
> nio's direct buffer to read the spill files and for certain jobs spilling 
> significant amount of data, we see between 5 - 7% speedup.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to