Chuanlei Ni created SPARK-17632:
-----------------------------------
Summary: make console sink and other sinks work with
'recoverFromCheckpointLocation' option enabled
Key: SPARK-17632
URL: https://issues.apache.org/jira/browse/SPARK-17632
Project: Spark
Issue Type: Bug
Components: SQL, Streaming
Affects Versions: 2.0.0
Reporter: Chuanlei Ni
Priority: Trivial
val (useTempCheckpointLocation, recoverFromCheckpointLocation) =
if (source == "console") {
(true, false)
} else {
(false, true)
}
I think 'source' sink should work with 'recoverFromCheckpoint' option enabled.
If user specified checkpointLocation, we may lost the processed data, however
it is acceptable since this sink used for test mainly.
And other sink can be work with 'useTempCheckpointLocation' enabled, which
means replaying all the data in kafka. If user mean it, it is ok.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
