Jungtaek Lim created SPARK-36480:
------------------------------------
Summary: SessionWindowStateStoreSaveExec should not filter input
rows against watermark
Key: SPARK-36480
URL: https://issues.apache.org/jira/browse/SPARK-36480
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 3.2.0
Reporter: Jungtaek Lim
SessionWindowStateStoreSaveExec receives all sessions including existing
sessions into input rows and stores as they are. That said, we should not
filter out input rows before storing into state store, but we do.
Fortunately it hasn't showed any actual problem due to the nature how we deal
with watermark against micro-batch and it seems hard to come up with the broken
case, but it should be better to fix it before someone succeeds to touch the
possible edge case.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
