Ruifeng Zheng created SPARK-58584:
-------------------------------------
Summary: Optimize ML implementations for Spark Connect server
memory usage
Key: SPARK-58584
URL: https://issues.apache.org/jira/browse/SPARK-58584
Project: Spark
Issue Type: Improvement
Components: ML
Affects Versions: 5.0.0
Reporter: Ruifeng Zheng
Umbrella ticket for reducing memory use of Spark ML implementations in
long-lived Spark Connect servers.
Track opportunities to:
- reuse immutable singleton objects;
- avoid retaining or eagerly creating algorithm-specific auxiliary state;
- avoid allocating or serializing large aggregation zero values; and
- use compact representations and avoid unnecessary driver-side collections or
broadcasts.
Discussion:
https://docs.google.com/document/d/1w91foXGwfUcBjj3g3nWbrhcZzBG3Q8aLgqGkcW8XPd4/edit
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
