Xi Liu created SPARK-3505:
-----------------------------
Summary: Augmenting SparkStreaming updateStateByKey API with
timestamp
Key: SPARK-3505
URL: https://issues.apache.org/jira/browse/SPARK-3505
Project: Spark
Issue Type: Improvement
Components: Streaming
Affects Versions: 1.0.0
Reporter: Xi Liu
Priority: Minor
Fix For: 1.1.0
The current updateStateByKey API in Spark Streaming does not expose timestamp
to the application.
In our use case, the application need to know the batch timestamp to decide
whether to keep the state or not. And we do not want to use real system time
because we want to decouple the two (because the same code base is used for
streaming and offline processing).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
