Jing Zhan created SPARK-47331: --------------------------------- Summary: Serialization using case classes/primitives/POJO based on SQL encoder for Arbitrary State API v2. Key: SPARK-47331 URL: https://issues.apache.org/jira/browse/SPARK-47331 Project: Spark Issue Type: Task Components: Structured Streaming Affects Versions: 4.0.0 Reporter: Jing Zhan
In the new operator for arbitrary state-v2, we cannot rely on the session/encoder being available since the initialization for the various state instances happens on the executors. Also, we can only support limited state types with the available encoders. Hence, for the state serialization, we propose to serialize primitives/case classes/POJO with SQL encoder. Leveraging SQL encoder can speed up the serialization. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org