[
https://issues.apache.org/jira/browse/SPARK-19932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liwei Lin updated SPARK-19932:
------------------------------
Summary: Disallow a case that might case OOM for steaming deduplication
(was: Also save event time into StateStore for certain cases)
> Disallow a case that might case OOM for steaming deduplication
> --------------------------------------------------------------
>
> Key: SPARK-19932
> URL: https://issues.apache.org/jira/browse/SPARK-19932
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 2.1.0
> Reporter: Liwei Lin
>
> {code}
> spark
> .readStream // schema: (word, eventTime), like ("a", 10),
> ("a", 11), ("b", 12) ...
> ...
> .withWatermark("eventTime", "10 seconds")
> .dropDuplicates("word") // note: "eventTime" is not part of the key
> columns
> ...
> {code}
> As shown above, right now if watermark is specified for a streaming
> dropDuplicates query, but not specified as the key columns, then we'll still
> get the correct answer, but the state just keeps growing and will never get
> cleaned up.
> The reason is, the watermark attribute is not part of the key of the state
> store in this case. We're not saving event time information in the state
> store.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
