Kavpreet Grewal created SPARK-60348:
---------------------------------------
Summary: Fix state data source change feed dropping list state
values after the first
Key: SPARK-60348
URL: https://issues.apache.org/jira/browse/SPARK-60348
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 4.2.0, 4.1.0, 5.0.0
Reporter: Kavpreet Grewal
Since SPARK-54118, {{ListState.put}} and {{appendList}} write all of their
values in a single changelog record, but {{RocksDBStateStoreChangeDataReader}}
decodes only the first value of each record. So reading list state with
{{readChangeFeed}} silently drops every value after the first. For example,
{{put(Array("a", "b", "c"))}} shows up only as {{update a}}. The stored state
is not affected.
We should return one row per value: the first value keeps the record's change
type and the rest are returned as {{append}}, which matches the change feed
from before SPARK-54118.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
