[
https://issues.apache.org/jira/browse/SPARK-60348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim resolved SPARK-60348.
----------------------------------
Fix Version/s: 4.3.0
Resolution: Fixed
Issue resolved by pull request 59521
[https://github.com/apache/spark/pull/59521]
> Fix state data source change feed dropping list state values after the first
> ----------------------------------------------------------------------------
>
> Key: SPARK-60348
> URL: https://issues.apache.org/jira/browse/SPARK-60348
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 4.1.0, 4.2.0, 5.0.0
> Reporter: Kavpreet Grewal
> Assignee: Kavpreet Grewal
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.3.0
>
>
> Since SPARK-54118, {{ListState.put}} and {{appendList}} write all of their
> values in a single changelog record, but
> {{RocksDBStateStoreChangeDataReader}} decodes only the first value of each
> record. So reading list state with {{readChangeFeed}} silently drops every
> value after the first. For example, {{put(Array("a", "b", "c"))}} shows up
> only as {{update a}}. The stored state is not affected.
> We should return one row per value: the first value keeps the record's change
> type and the rest are returned as {{append}}, which matches the change feed
> from before SPARK-54118.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
