Arsenii Krasikov created SPARK-54392:
----------------------------------------
Summary: Optimize JVM-Python communication for TWS initial state
Key: SPARK-54392
URL: https://issues.apache.org/jira/browse/SPARK-54392
Project: Spark
Issue Type: Task
Components: Structured Streaming
Affects Versions: 4.2.0
Reporter: Arsenii Krasikov
* group multiple keys into one arrow batch;
generally will have much less batches in case of high keys cardinality.
* do not group `init_data` and `input_data` in batch0: instead of it serialize
`init_data` first, and then `input_data`;
in worst case we're going to have one more chunk by not grouping them together,
but winning by having much simpler logic on python side.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
