[
https://issues.apache.org/jira/browse/SPARK-34519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Dyagilev updated SPARK-34519:
-------------------------------------
Attachment: spark_backoff.png
> ExecutorPodsAllocator use exponential backoff strategy when request executor
> pod failed
> ---------------------------------------------------------------------------------------
>
> Key: SPARK-34519
> URL: https://issues.apache.org/jira/browse/SPARK-34519
> Project: Spark
> Issue Type: Improvement
> Components: Kubernetes
> Affects Versions: 3.0.1
> Environment: spark 3.0.1
> kubernetes 1.18.8
> Reporter: Fengyu Cao
> Priority: Minor
> Attachments: spark_backoff.png
>
>
> # create a resouce quota `kubectl create quota test --hard=cpu=20,memory=60G`
> # submit an application request more than quota `spark-submit
> --executor-cores 5 --executor-memory 10G --num-executors 10 <spark-pi.py>`
> # seems `ExecutorPodsAllocator: Going to request 5 executors from
> Kubernetes.` print every second
> `spark.kubernetes.allocation.batch.delay` default is 1s, which good enough
> when allocation succeeded, but exponential backoff maybe an better choice
> when alloction failed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
