[
https://issues.apache.org/jira/browse/SPARK-58584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruifeng Zheng updated SPARK-58584:
----------------------------------
Description:
Umbrella ticket for reducing memory use, serialization overhead, and redundant
computation in Spark ML implementations running on long-lived Spark Connect
servers.
Track opportunities to:
- reuse immutable singleton objects;
- avoid retaining or eagerly creating algorithm-specific auxiliary state;
- avoid allocating or serializing large aggregation zero values;
- reduce UDF closure sizes to lower driver memory use from retained query-plan
object graphs and serialized payloads; and
- use compact representations and avoid unnecessary driver-side collections or
broadcasts.
was:
Umbrella ticket for reducing memory use of Spark ML implementations in
long-lived Spark Connect servers.
Track opportunities to:
- reuse immutable singleton objects;
- avoid retaining or eagerly creating algorithm-specific auxiliary state;
- avoid allocating or serializing large aggregation zero values;
- reduce UDF closure sizes to lower driver memory use from retained query-plan
object graphs and serialized payloads; and
- use compact representations and avoid unnecessary driver-side collections or
broadcasts.
> Optimize ML implementations to reduce Spark Connect server pressure
> -------------------------------------------------------------------
>
> Key: SPARK-58584
> URL: https://issues.apache.org/jira/browse/SPARK-58584
> Project: Spark
> Issue Type: Improvement
> Components: Connect, ML
> Affects Versions: 5.0.0
> Reporter: Ruifeng Zheng
> Priority: Major
>
> Umbrella ticket for reducing memory use, serialization overhead, and
> redundant computation in Spark ML implementations running on long-lived Spark
> Connect servers.
> Track opportunities to:
> - reuse immutable singleton objects;
> - avoid retaining or eagerly creating algorithm-specific auxiliary state;
> - avoid allocating or serializing large aggregation zero values;
> - reduce UDF closure sizes to lower driver memory use from retained
> query-plan object graphs and serialized payloads; and
> - use compact representations and avoid unnecessary driver-side collections
> or broadcasts.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
