Josh Rosen created SPARK-15736:
----------------------------------
Summary: Gracefully handle loss of cached RDDs' on-disk files
Key: SPARK-15736
URL: https://issues.apache.org/jira/browse/SPARK-15736
Project: Spark
Issue Type: Bug
Components: Block Manager
Reporter: Josh Rosen
Assignee: Josh Rosen
If an RDD partition is cached on disk and the on-disk file is lost, then reads
of that cached partition will fail and the missing partition is supposed to be
recomputed by a new task attempt. However, the current behavior is to
repeatedly re-attempt the read on the same machine without performing any
recomputation, which leads to a complete job failure.
In order to fix this problem, the executor with the missing file needs to
properly mark the corresponding block as missing so that it stops advertising
itself as a cache location for that block.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
