[
https://issues.apache.org/jira/browse/SPARK-21113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-21113:
------------------------------------
Assignee: Apache Spark
> Support for read ahead input stream to amortize disk IO cost in the Spill
> reader
> --------------------------------------------------------------------------------
>
> Key: SPARK-21113
> URL: https://issues.apache.org/jira/browse/SPARK-21113
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 2.0.2
> Reporter: Sital Kedia
> Assignee: Apache Spark
> Priority: Minor
>
> Profiling some of our big jobs, we see that around 30% of the time is being
> spent in reading the spill files from disk. In order to amortize the disk IO
> cost, the idea is to implement a read ahead input stream which which
> asynchronously reads ahead from the underlying input stream when specified
> amount of data has been read from the current buffer. It does it by
> maintaining two buffer - active buffer and read ahead buffer. Active buffer
> contains data which should be returned when a read() call is issued. The read
> ahead buffer is used to asynchronously read from the underlying input stream
> and once the current active buffer is exhausted, we flip the two buffers so
> that we can start reading from the read ahead buffer without being blocked in
> disk I/O.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
