[
https://issues.apache.org/jira/browse/SPARK-54392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-54392:
-----------------------------------
Labels: pull-request-available (was: )
> Optimize JVM-Python communication for TWS initial state
> -------------------------------------------------------
>
> Key: SPARK-54392
> URL: https://issues.apache.org/jira/browse/SPARK-54392
> Project: Spark
> Issue Type: Task
> Components: Structured Streaming
> Affects Versions: 4.2.0
> Reporter: Arsenii Krasikov
> Priority: Major
> Labels: pull-request-available
>
> * group multiple keys into one arrow batch;
> generally will have much less batches in case of high keys cardinality.
> * do not group `init_data` and `input_data` in batch0: instead of it
> serialize `init_data` first, and then `input_data`;
> in worst case we're going to have one more chunk by not grouping them
> together, but winning by having much simpler logic on python side.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
