Jungtaek Lim created SPARK-55147:
------------------------------------
Summary: Improve the retrieval operation in new stream-stream
stateful operator to scope the timestamp range for time-interval join
Key: SPARK-55147
URL: https://issues.apache.org/jira/browse/SPARK-55147
Project: Spark
Issue Type: Task
Components: Structured Streaming
Affects Versions: 4.2.0
Reporter: Jungtaek Lim
With SPARK-55144, the state format of the primary store would support for
"prefix scan + range scan on event time". That allows us to perform the
retrieval operation of time interval join more efficiently, via only scanning
through the timestamp range which is constructed with "time interval condition".
This ticket tracks the effort of modifying the state store API and state
manager, and modifying stream-stream join operator to integrate the change.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
