Tongwei created SPARK-60804:
-------------------------------
Summary: Fix off-by-one error in Kubernetes executor pod
allocation maximum
Key: SPARK-60804
URL: https://issues.apache.org/jira/browse/SPARK-60804
Project: Spark
Issue Type: Bug
Components: Kubernetes
Affects Versions: 5.0.0
Reporter: Tongwei
spark.kubernetes.allocation.maximum specifies the maximum number of executor
pod creation attempts during an application lifecycle. However,
ExecutorPodsAllocator increments the executor ID before checking newExecutorId
>= podAllocationMaximum, allowing only N-1 attempts for a limit of N.
With the direct allocator, spark.kubernetes.allocation.maximum=1,
spark.executor.instances=1 and spark.dynamicAllocation.enabled=false, the first
allocation throws "Exceed the pod creation limit: 1" before creating any
executor pod. It should allow one creation attempt.
The fix checks the limit before incrementing the counter, allowing exactly N
attempts and preventing integer overflow. Regression tests cover limits of 1, 3
and Int.MaxValue, including repeated attempts after reaching the limit. All 32
tests in ExecutorPodsAllocatorSuite pass.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
