[
https://issues.apache.org/jira/browse/SPARK-58584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruifeng Zheng updated SPARK-58584:
----------------------------------
Summary: Optimize ML implementations for Spark Connect server driver CPU
and memory usage (was: Optimize ML implementations for Spark Connect server
memory usage)
> Optimize ML implementations for Spark Connect server driver CPU and memory
> usage
> --------------------------------------------------------------------------------
>
> Key: SPARK-58584
> URL: https://issues.apache.org/jira/browse/SPARK-58584
> Project: Spark
> Issue Type: Improvement
> Components: ML
> Affects Versions: 5.0.0
> Reporter: Ruifeng Zheng
> Priority: Major
>
> Umbrella ticket for reducing memory use of Spark ML implementations in
> long-lived Spark Connect servers.
> Track opportunities to:
> - reuse immutable singleton objects;
> - avoid retaining or eagerly creating algorithm-specific auxiliary state;
> - avoid allocating or serializing large aggregation zero values; and
> - use compact representations and avoid unnecessary driver-side collections
> or broadcasts.
> Discussion:
> https://docs.google.com/document/d/1w91foXGwfUcBjj3g3nWbrhcZzBG3Q8aLgqGkcW8XPd4/edit
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
