Jing Zhan created SPARK-47331:
---------------------------------
Summary: Serialization using case classes/primitives/POJO based on
SQL encoder for Arbitrary State API v2.
Key: SPARK-47331
URL: https://issues.apache.org/jira/browse/SPARK-47331
Project: Spark
Issue Type: Task
Components: Structured Streaming
Affects Versions: 4.0.0
Reporter: Jing Zhan
In the new operator for arbitrary state-v2, we cannot rely on the
session/encoder being available since the initialization for the various state
instances happens on the executors. Also, we can only support limited state
types with the available encoders. Hence, for the state serialization, we
propose to serialize primitives/case classes/POJO with SQL encoder. Leveraging
SQL encoder can speed up the serialization.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
