[jira] [Commented] (SPARK-17632) make console sink and other sinks work with 'recoverFromCheckpointLocation' option enabled
[
https://issues.apache.org/jira/browse/SPARK-17632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512567#comment-15512567
]
Chuanlei Ni commented on SPARK-17632:
-
hi, [~tdas]
could you help me review this pr?
> make console sink and other sinks work with 'recoverFromCheckpointLocation'
> option enabled
> --
>
> Key: SPARK-17632
> URL: https://issues.apache.org/jira/browse/SPARK-17632
> Project: Spark
> Issue Type: Bug
> Components: SQL, Streaming
>Affects Versions: 2.0.0
>Reporter: Chuanlei Ni
>Priority: Trivial
>
> val (useTempCheckpointLocation, recoverFromCheckpointLocation) =
> if (source == "console") {
> (true, false)
> } else {
> (false, true)
> }
> I think 'source' sink should work with 'recoverFromCheckpoint' option
> enabled. If user specified checkpointLocation, we may lost the processed
> data, however it is acceptable since this sink used for test mainly.
> And other sink can be work with 'useTempCheckpointLocation' enabled, which
> means replaying all the data in kafka. If user mean it, it is ok.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-17632) make console sink and other sinks work with 'recoverFromCheckpointLocation' option enabled
[
https://issues.apache.org/jira/browse/SPARK-17632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512554#comment-15512554
]
Chuanlei Ni commented on SPARK-17632:
-
pr is as below
https://github.com/apache/spark/pull/15195
> make console sink and other sinks work with 'recoverFromCheckpointLocation'
> option enabled
> --
>
> Key: SPARK-17632
> URL: https://issues.apache.org/jira/browse/SPARK-17632
> Project: Spark
> Issue Type: Bug
> Components: SQL, Streaming
>Affects Versions: 2.0.0
>Reporter: Chuanlei Ni
>Priority: Trivial
>
> val (useTempCheckpointLocation, recoverFromCheckpointLocation) =
> if (source == "console") {
> (true, false)
> } else {
> (false, true)
> }
> I think 'source' sink should work with 'recoverFromCheckpoint' option
> enabled. If user specified checkpointLocation, we may lost the processed
> data, however it is acceptable since this sink used for test mainly.
> And other sink can be work with 'useTempCheckpointLocation' enabled, which
> means replaying all the data in kafka. If user mean it, it is ok.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-17632) make console sink and other sinks work with 'recoverFromCheckpointLocation' option enabled
[
https://issues.apache.org/jira/browse/SPARK-17632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512552#comment-15512552
]
Apache Spark commented on SPARK-17632:
--
User 'chuanlei' has created a pull request for this issue:
https://github.com/apache/spark/pull/15195
> make console sink and other sinks work with 'recoverFromCheckpointLocation'
> option enabled
> --
>
> Key: SPARK-17632
> URL: https://issues.apache.org/jira/browse/SPARK-17632
> Project: Spark
> Issue Type: Bug
> Components: SQL, Streaming
>Affects Versions: 2.0.0
>Reporter: Chuanlei Ni
>Priority: Trivial
>
> val (useTempCheckpointLocation, recoverFromCheckpointLocation) =
> if (source == "console") {
> (true, false)
> } else {
> (false, true)
> }
> I think 'source' sink should work with 'recoverFromCheckpoint' option
> enabled. If user specified checkpointLocation, we may lost the processed
> data, however it is acceptable since this sink used for test mainly.
> And other sink can be work with 'useTempCheckpointLocation' enabled, which
> means replaying all the data in kafka. If user mean it, it is ok.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
