[
https://issues.apache.org/jira/browse/SPARK-58584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruifeng Zheng updated SPARK-58584:
----------------------------------
Description:
Umbrella ticket for reducing memory use of Spark ML implementations in
long-lived Spark Connect servers.
Track opportunities to:
- reuse immutable singleton objects;
- avoid retaining or eagerly creating algorithm-specific auxiliary state;
- avoid allocating or serializing large aggregation zero values; and
- use compact representations and avoid unnecessary driver-side collections or
broadcasts.
was:
Umbrella ticket for reducing memory use of Spark ML implementations in
long-lived Spark Connect servers.
Track opportunities to:
- reuse immutable singleton objects;
- avoid retaining or eagerly creating algorithm-specific auxiliary state;
- avoid allocating or serializing large aggregation zero values; and
- use compact representations and avoid unnecessary driver-side collections or
broadcasts; and
- move computation, collection, and post-processing from the driver to
executors while keeping driver logic simple.
Discussion:
https://docs.google.com/document/d/1w91foXGwfUcBjj3g3nWbrhcZzBG3Q8aLgqGkcW8XPd4/edit
> Optimize ML implementations to reduce Spark Connect server pressure
> -------------------------------------------------------------------
>
> Key: SPARK-58584
> URL: https://issues.apache.org/jira/browse/SPARK-58584
> Project: Spark
> Issue Type: Improvement
> Components: Connect, ML
> Affects Versions: 5.0.0
> Reporter: Ruifeng Zheng
> Priority: Major
>
> Umbrella ticket for reducing memory use of Spark ML implementations in
> long-lived Spark Connect servers.
> Track opportunities to:
> - reuse immutable singleton objects;
> - avoid retaining or eagerly creating algorithm-specific auxiliary state;
> - avoid allocating or serializing large aggregation zero values; and
> - use compact representations and avoid unnecessary driver-side collections
> or broadcasts.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
