[
https://issues.apache.org/jira/browse/SPARK-51779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim resolved SPARK-51779.
----------------------------------
Fix Version/s: 4.1.0
Resolution: Fixed
Issue resolved by pull request 50572
[https://github.com/apache/spark/pull/50572]
> Use virtual column families for stream-stream join
> --------------------------------------------------
>
> Key: SPARK-51779
> URL: https://issues.apache.org/jira/browse/SPARK-51779
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 4.1.0
> Reporter: Zeyu Chen
> Assignee: Zeyu Chen
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.1.0
>
>
> Currently, the streaming join implementation in Spark uses four distinct
> state stores to perform stream-stream join operations. The complexity in
> managing multiple state stores concurrently poses a lot of stability issues
> and pain points in debugging.
> Since virtual column families have been introduced for RocksDB state stores,
> we can use column families to combine these four stores into one state store.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
