This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-4.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.1 by this push:
new 8fb5285810ac [SPARK-54366][INFRA] Add `free_disk_space` step to K8s
integration test GitHub Action job
8fb5285810ac is described below
commit 8fb5285810acb6329e57b6a4a81460484c0b807c
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sat Nov 15 15:42:50 2025 -0800
[SPARK-54366][INFRA] Add `free_disk_space` step to K8s integration test
GitHub Action job
### What changes were proposed in this pull request?
This PR aims to add `free_disk_space` step to K8s integration test GitHub
Action job.
### Why are the changes needed?
The K8s integration test CI is flaky due to `No space left on device` error.
- https://github.com/apache/spark/actions/runs/19354883389/job/55448531341
```
[info] 25/11/14 21:27:02 ERROR TaskSchedulerImpl: Lost executor 4 on
10.244.0.67: Unable to create executor due to
/var/data/spark-163899dd-08da-4b76-b71d-c428207a3bdf/spark-1e9d976f-69b3-4274-af03-300cfc4d6fd5/-14621403551763155568738_cache
-> ./software.amazon.awssdk_bundle-2.29.52.jar:
No space left on device
```
Like the other four GitHub Action jobs, `free_disk_space_container` will
mitigate this situation in this job.
**BEFORE**
```
$ git grep 'free_disk_space$'
.github/workflows/build_and_test.yml: ./dev/free_disk_space
.github/workflows/release.yml: ./dev/free_disk_space
```
**AFTER**
```
$ git grep 'free_disk_space$'
.github/workflows/build_and_test.yml: ./dev/free_disk_space
.github/workflows/build_and_test.yml: ./dev/free_disk_space
.github/workflows/release.yml: ./dev/free_disk_space
```
### Does this PR introduce _any_ user-facing change?
No, this is a test infra change.
### How was this patch tested?
Pass the CIs and check the log. The following is the log result of this PR.
-
https://github.com/dongjoon-hyun/spark/actions/runs/19395758483/job/55495933312
**BEFORE CLEANUP**
```
+ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 72G 54G 18G 76% /
tmpfs 7.9G 84K 7.9G 1% /dev/shm
tmpfs 3.2G 1.1M 3.2G 1% /run
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/sda16 881M 62M 758M 8% /boot
/dev/sda15 105M 6.2M 99M 6% /boot/efi
/dev/sdb1 74G 4.1G 66G 6% /mnt
tmpfs 1.6G 12K 1.6G 1% /run/user/1001
```
**AFTER CLEANUP**
```
+ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 72G 21G 52G 29% /
tmpfs 7.9G 84K 7.9G 1% /dev/shm
tmpfs 3.2G 1.1M 3.2G 1% /run
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/sda16 881M 62M 758M 8% /boot
/dev/sda15 105M 6.2M 99M 6% /boot/efi
/dev/sdb1 74G 4.1G 66G 6% /mnt
tmpfs 1.6G 12K 1.6G 1% /run/user/1001
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #53077 from dongjoon-hyun/SPARK-54366.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 0311f44e33e5cf8ba60ccc330de3df4f688f5847)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.github/workflows/build_and_test.yml | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/.github/workflows/build_and_test.yml
b/.github/workflows/build_and_test.yml
index 161164dc5866..f0d2acbacff7 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -1314,6 +1314,11 @@ jobs:
key: k8s-integration-coursier-${{ hashFiles('**/pom.xml',
'**/plugins.sbt') }}
restore-keys: |
k8s-integration-coursier-
+ - name: Free up disk space
+ run: |
+ if [ -f ./dev/free_disk_space ]; then
+ ./dev/free_disk_space
+ fi
- name: Install Java ${{ inputs.java }}
uses: actions/setup-java@v4
with:
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
