[
https://issues.apache.org/jira/browse/SPARK-3505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xi Liu closed SPARK-3505.
-------------------------
Resolution: Won't Fix
Close this issue for now. Will re-open later when I find time to work on it.
> Augmenting SparkStreaming updateStateByKey API with timestamp
> -------------------------------------------------------------
>
> Key: SPARK-3505
> URL: https://issues.apache.org/jira/browse/SPARK-3505
> Project: Spark
> Issue Type: Improvement
> Components: Streaming
> Affects Versions: 1.0.0
> Reporter: Xi Liu
> Priority: Minor
>
> The current updateStateByKey API in Spark Streaming does not expose timestamp
> to the application.
> In our use case, the application need to know the batch timestamp to decide
> whether to keep the state or not. And we do not want to use real system time
> because we want to decouple the two (because the same code base is used for
> streaming and offline processing).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
