[
https://issues.apache.org/jira/browse/SPARK-2985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14224555#comment-14224555
]
Tathagata Das commented on SPARK-2985:
--------------------------------------
This is by design. If you are using the receiver.store(<single record>) API to
store one record at a time, the block generator is used to divide the stream of
records into blocks. While this interface is simple (user does not have to
worry about blocking), it does not provide any control over when the data has
been persisted reliably. Instead if you use receiver.store(<multiple records>),
that function returns only when the data has been replicated (if replication is
enabled, it is enabled by default). So your receiver needs to be implemented
accordingly - either use simple API with no feedback, or harder API with full
control.
> Buffered data in BlockGenerator gets lost when receiver crashes
> ---------------------------------------------------------------
>
> Key: SPARK-2985
> URL: https://issues.apache.org/jira/browse/SPARK-2985
> Project: Spark
> Issue Type: Bug
> Components: Streaming
> Affects Versions: 1.0.0
> Reporter: dai zhiyuan
> Priority: Critical
>
> If recevierTracker crashes,the buffer data of BlockGenerator will be lost.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
