Filip Darmanovic created SPARK-55397:
----------------------------------------
Summary: The waitToReusePersistentVolumeClaim feature assumes only
one PVC per executor
Key: SPARK-55397
URL: https://issues.apache.org/jira/browse/SPARK-55397
Project: Spark
Issue Type: Improvement
Components: Kubernetes
Affects Versions: 3.5.8, 4.1.0, 3.5.2
Reporter: Filip Darmanovic
Spark supports attaching multiple disks to a worker, which is very useful for
increasing throughput, however, if the PVC re-use feature is enabled and the
podAllocOnPVC flag is on, meaning that the driver should wait to re-use PVCs,
the max number of PVCs is limited to the max number of workers.
So what happens in practice is that if you ask for 4 workers with 4 PVCs each,
Spark will assign 4 PVCs to the first worker and then stop spawning new workers.
I think we should just keep track of the number of PVCs spawned so far and wait
for the ones that are not bound.
I'm willing to work on this PR if I can get help with integration tests.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
