Dibyendu Bhattacharya created SPARK-10694:
---------------------------------------------
Summary: Prevent Data Loss in Spark Streaming when used with
OFF_HEAP ExternalBlockStore (Tachyon)
Key: SPARK-10694
URL: https://issues.apache.org/jira/browse/SPARK-10694
Project: Spark
Issue Type: Bug
Components: Streaming
Affects Versions: 1.5.0
Reporter: Dibyendu Bhattacharya
If Streaming application stores the blocks OFF_HEAP, it may not need any WAL
like feature to recover from Driver failure. As long as the writing of blocks
to Tachyon from Streaming receiver is durable, it should be recoverable from
Tachyon directly on Driver failure.
This can solve the issue of expensive WAL write and duplicating the blocks both
in MEMORY and also WAL and also guarantee end to end No-Data-Loss channel using
OFF_HEAP store.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
