dongjoon-hyun opened a new pull request, #59289:
URL: https://github.com/apache/spark/pull/59289
### What changes were proposed in this pull request?
This PR aims to add an integration test for `ExecutorResizePlugin`.
The test runs `JavaSparkSQLCli` with `SELECT reflect('java.lang.Thread',
'sleep', 600000L)` so that the executor stays alive while the plugin works.
The plugin uses the following configurations:
- `spark.kubernetes.executor.resizeInterval=5s`
- `spark.kubernetes.executor.resizeThreshold=0.01`, so that any running
executor exceeds it.
- `spark.kubernetes.executor.resizeFactor=0.1`
- `spark.kubernetes.executor.resizeMaxMemory=1536m`, which is lower than the
first increased limit (`1408Mi * 1.1`). So, the first resize reaches the
maximum.
The test verifies the following:
- The driver log shows that executor 1 is resized to `1536Mi`. A later log
line shows that the next resize is skipped because the maximum was reached.
- The executor pod spec has `1536Mi` as both the memory request and the
memory limit.
- The container status reports `1536Mi` as the memory limit. This means the
kubelet applied the resize in place.
- The container restart count is `0`.
The test is skipped when the `metrics.k8s.io` API is unavailable. It is also
skipped in the `KubernetesSuite` subclasses like `VolcanoSuite`, because the
resize does not depend on the scheduler.
### Why are the changes needed?
`ExecutorResizePlugin` has only unit tests with a mocked `KubernetesClient`.
This PR adds a real K8s cluster test to cover the following:
- Reading memory usage through the `metrics.k8s.io` API.
- Patching the `resize` subresource.
- The in-place resize by the kubelet.
The CI now provides the `metrics.k8s.io` API via SPARK-60059 (#59281).
### Does this PR introduce _any_ user-facing change?
No. This is a test-only change.
### How was this patch tested?
Pass the CIs.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
