[
https://issues.apache.org/jira/browse/SPARK-12306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15054995#comment-15054995
]
Liwei Lin commented on SPARK-12306:
-----------------------------------
Sorry for the inconvenience; I've added some detailed description, [~srowen]
would you mind reopen it? Thanks. :-)
> Add an option to ignore BlockRDD partition data loss
> ----------------------------------------------------
>
> Key: SPARK-12306
> URL: https://issues.apache.org/jira/browse/SPARK-12306
> Project: Spark
> Issue Type: Improvement
> Components: Streaming
> Affects Versions: 1.5.2
> Reporter: Liwei Lin
>
> Currently in Spark Streaming, a Receiver stores the received data into some
> BlockManager and then later the data will be used by a BlockRDD. If this
> BlockManager were to lost because of some failure, then this BlockRDD would
> throw a SparkException saying "Could not compute split, block not found".
> In most cases this is the right thing to do. However, in a streaming scenario
> where it can tolerant small pieces of data loss, maybe just move on silently
> -- instead of throwing an exception -- is more preferable.
> This issue proposes to add such an "spark.streaming.ignoreBlockNotFound"
> option, which defaults to false, to tell whether to throw an exception or
> just move on when a block is not found.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
