[
https://issues.apache.org/jira/browse/SPARK-60778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60778:
-----------------------------------
Labels: pull-request-available (was: )
> [K8S] ExecutorPodsAllocator can reuse a PVC still mounted by a live executor
> when no new snapshot arrives in an allocation round
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60778
> URL: https://issues.apache.org/jira/browse/SPARK-60778
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
> Affects Versions: 4.1.1
> Reporter: Guang Yang
> Priority: Major
> Labels: pull-request-available
>
> h3. Summary
>
>
>
>
>
>
>
> When {{spark.kubernetes.driver.ownPersistentVolumeClaim=true}} and
> {{spark.kubernetes.driver.reusePersistentVolumeClaim=true}} (both default
> true), ExecutorPodsAllocator can give a new executor pod a PVC that a
> *running* executor is still
> using. With the default ReadWriteOncePod access mode, the new pod can never
> be scheduled. It stays Pending (scheduler reason {{VolumeRestrictions}})
> until dynamic allocation deletes it or the application ends. Several pending
> pods can end up
> pointing at the same claim.
>
>
>
> h3. Root cause
>
>
>
>
>
> In {{ExecutorPodsAllocator.onNewSnapshots}}, the set of PVCs in use
> ({{k8sKnownPVCNames}}) is built only from the snapshots delivered in the
> current batch:
>
>
> {code:scala}
>
> val aggregatedPods = if (snapshots.size <= 1) {
>
> snapshots.headOption.map(_.executorPods).getOrElse(Map.empty)
>
>
>
> } else { ... merge snapshots ... }
>
>
>
> ...
>
>
>
> val k8sKnownPVCNames: Set[String] = {
>
>
>
> aggregatedPods.valuesIterator.foreach { state => /* collect claimName of
> each PVC volume */ }
>
>
> }
>
>
>
> {code}
>
>
>
>
>
> This set is passed to {{requestNewExecutors}} ->
> {{getReusablePVCs(applicationId, pvcsInUse)}}. That method treats any
> driver-owned PVC as reusable if it is not in {{pvcsInUse}} and is older than
> {{podCreationTimeout}}.
>
>
> There are two gaps.
>
>
>
>
>
>
>
> # *Allocation rounds with no new snapshot.*
> {{ExecutorPodsSnapshotsStoreImpl.SnapshotsSubscriber}} runs every
> {{spark.kubernetes.allocation.batch.delay}} (default 1s). It drains the
> snapshot buffer and calls {{onNewSnapshots}} even when the
> buffer is empty. Executor counting still uses {{lastSnapshot}}, so the
> allocator can decide to scale up in that round. But {{aggregatedPods}} is
> empty, so {{pvcsInUse}} is empty. Every driver-owned PVC older than
> {{podCreationTimeout}} then
> looks reusable, including PVCs mounted by Running executors.
>
>
>
> # *Executors requested but not yet in any snapshot.* A pod created in round
> N that reused PVC X shows up in a snapshot only after the watch or poll
> delivers it. In round N+1, X is again missing from {{pvcsInUse}}, so it can
> be handed out a
> second time.
> SPARK-41388 and SPARK-49385 (PR #47867) reduced related races by skipping
> PVCs younger than {{podCreationTimeout}}. That only protects recently created
> PVCs. In a long-running application, nearly every PVC is older than the
> timeout, so
> neither change covers this case.
> h3. Observed in production (Spark 4.1.1, dynamic allocation, on-demand
> executor PVC)
> Configuration:
> {code}
>
> spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.options.claimName=OnDemand
>
> spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.options.sizeLimit=100Gi
>
> spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.mount.path=/mnt/spark-local
> spark.dynamicAllocation.enabled=true
> # own/reuse PVC left at default (true); access mode default ReadWriteOncePod
> {code}
> What we saw on one application after it had run for more than 10 minutes
> and scaled up again:
> * {{<app>-exec-1}} had been Running on {{<app>-exec-1-pvc-0}} for over a
> minute. The PVC was 17 minutes old, older than {{podCreationTimeout}}.
> * The allocator created {{<app>-exec-11}} with the same claim
> {{<app>-exec-1-pvc-0}}.
> * Seven pods in total referenced {{exec-1-pvc-0}}. Two other claims had
> four pods each. The application had 18 duplicate Pending executor pods and
> 636 driver-owned PVCs.
> * The duplicate pods failed scheduling with {{VolumeRestrictions}} and
> never ran. They were cleaned up only when dynamic allocation scaled down or
> the application finished.
> Effects beyond the stuck pods:
> * The application gets fewer executors than it asked for.
> * The cluster autoscaler keeps provisioning nodes for pods that can never
> run.
> * A batch scheduler (YuniKorn in our case) re-checks these pods against
> every node on every cycle. Across many applications, this starved other
> applications' pods.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
