[
https://issues.apache.org/jira/browse/SPARK-54392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim resolved SPARK-54392.
----------------------------------
Fix Version/s: 4.2.0
Resolution: Fixed
Issue resolved by pull request 53122
[https://github.com/apache/spark/pull/53122]
> Optimize JVM-Python communication for TWS initial state
> -------------------------------------------------------
>
> Key: SPARK-54392
> URL: https://issues.apache.org/jira/browse/SPARK-54392
> Project: Spark
> Issue Type: Task
> Components: Structured Streaming
> Affects Versions: 4.2.0
> Reporter: Arsenii Krasikov
> Assignee: Arsenii Krasikov
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.2.0
>
>
> * group multiple keys into one arrow batch;
> generally will have much less batches in case of high keys cardinality.
> * do not group `init_data` and `input_data` in batch0: instead of it
> serialize `init_data` first, and then `input_data`;
> in worst case we're going to have one more chunk by not grouping them
> together, but winning by having much simpler logic on python side.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
