[
https://issues.apache.org/jira/browse/SPARK-62036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Zhilko updated SPARK-62036:
--------------------------------
Description:
On multi-zone clusters, the driver and executors of one application often land
in different zones. Traffic between zones (driver-executor RPC,
executor-executor block transfer, and push-based remote shuffle services such
as Apache Celeborn) is slower and, on public clouds, billed per GB.
Today users work around this with executor {{podAffinity}} to the driver's
zone. Shuffle-service settings that depend on the zone, such as Celeborn's
{{spark.celeborn.tags.tagsExpr}}, can't be set that way: the zone is only known
after the driver is scheduled, so each application has to look up its own
node's zone at runtime.
Proposal: the operator picks one zone per application attempt before it creates
the driver.
* Zone choice: a {{spark.operator/zone}} annotation pins an explicit zone.
Otherwise the operator picks the configured zone with the fewest active pinned
applications.
* Placement: it adds a node selector on the configured topology key (default
{{topology.kubernetes.io/zone}}) to the driver and executors, using
{{spark.kubernetes.\{driver,executor\}.node.selector.*}}.
* Spark conf: it optionally fills operator-configured conf templates with the
chosen zone, e.g. {{spark.celeborn.tags.tagsExpr=az-\{zone\}}}. Keys the
application sets itself are left alone.
* Status: it records the zone in the current attempt's status. Rebuilding the
spec keeps the same zone, and a restart picks again.
* Scope: it is off by default. Applications queued through Kueue are left to
Kueue.
> Support pinning a SparkApplication attempt to a single topology zone
> --------------------------------------------------------------------
>
> Key: SPARK-62036
> URL: https://issues.apache.org/jira/browse/SPARK-62036
> Project: Spark
> Issue Type: Improvement
> Components: Kubernetes
> Affects Versions: kubernetes-operator-2.0.0
> Reporter: Alex Zhilko
> Priority: Major
>
> On multi-zone clusters, the driver and executors of one application often
> land in different zones. Traffic between zones (driver-executor RPC,
> executor-executor block transfer, and push-based remote shuffle services such
> as Apache Celeborn) is slower and, on public clouds, billed per GB.
> Today users work around this with executor {{podAffinity}} to the driver's
> zone. Shuffle-service settings that depend on the zone, such as Celeborn's
> {{spark.celeborn.tags.tagsExpr}}, can't be set that way: the zone is only
> known after the driver is scheduled, so each application has to look up its
> own node's zone at runtime.
> Proposal: the operator picks one zone per application attempt before it
> creates the driver.
> * Zone choice: a {{spark.operator/zone}} annotation pins an explicit zone.
> Otherwise the operator picks the configured zone with the fewest active
> pinned applications.
> * Placement: it adds a node selector on the configured topology key (default
> {{topology.kubernetes.io/zone}}) to the driver and executors, using
> {{spark.kubernetes.\{driver,executor\}.node.selector.*}}.
> * Spark conf: it optionally fills operator-configured conf templates with the
> chosen zone, e.g. {{spark.celeborn.tags.tagsExpr=az-\{zone\}}}. Keys the
> application sets itself are left alone.
> * Status: it records the zone in the current attempt's status. Rebuilding the
> spec keeps the same zone, and a restart picks again.
> * Scope: it is off by default. Applications queued through Kueue are left to
> Kueue.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]