[
https://issues.apache.org/jira/browse/SPARK-24160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenchen Fan resolved SPARK-24160.
---------------------------------
Resolution: Fixed
Fix Version/s: 2.4.0
Issue resolved by pull request 21219
[https://github.com/apache/spark/pull/21219]
> ShuffleBlockFetcherIterator should fail if it receives zero-size blocks
> -----------------------------------------------------------------------
>
> Key: SPARK-24160
> URL: https://issues.apache.org/jira/browse/SPARK-24160
> Project: Spark
> Issue Type: Improvement
> Components: Shuffle
> Affects Versions: 2.3.0
> Reporter: Josh Rosen
> Assignee: Josh Rosen
> Priority: Major
> Fix For: 2.4.0
>
>
> In the shuffle layer, we guarantee that zero-size blocks will never be
> requested (a block containing zero records is always 0 bytes in size and is
> marked as empty such that it will never be legitimately requested by
> executors). However, we failed to take advantage of this in the shuffle-read
> path: the existing code did not explicitly check whether blocks are
> non-zero-size.
>
> We should add `buf.size != 0` checks to ShuffleBlockFetcherIterator to take
> advantage of this invariant and prevent potential data loss / corruption
> issues.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
