Tathagata Das created SPARK-10210:
-------------------------------------
Summary: Exception "Could not compute split, block input-XXX not
found" after streaming app driver recovers from checkpoint
Key: SPARK-10210
URL: https://issues.apache.org/jira/browse/SPARK-10210
Project: Spark
Issue Type: Bug
Components: Streaming
Affects Versions: 1.4.1
Reporter: Tathagata Das
Assignee: Tathagata Das
Priority: Critical
When write ahead log is not enabled, a recovered streaming driver still tries
to run jobs using pre-failure block ids, and fails as the block do not exists
in-memory any more (and cannot be recovered as receiver WAL is not enabled).
This occurs because the driver-side WAL of ReceivedBlockTracker is recovers
that past block information, and ReceiveInputDStream creates BlockRDDs even if
those blocks do not exist.
The solution is to filter out block ids that do not exist before creating the
BlockRDD.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
