This is an automated email from the ASF dual-hosted git repository.
chengpan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kyuubi.git
The following commit(s) were added to refs/heads/master by this push:
new 5f99ec8d6 [KYUUBI #6047] [CI] Free up disk space
5f99ec8d6 is described below
commit 5f99ec8d6a9336d0bd2b3f9db83d0deaacf71e35
Author: Cheng Pan <[email protected]>
AuthorDate: Mon Feb 5 12:12:15 2024 +0800
[KYUUBI #6047] [CI] Free up disk space
# :mag: Description
## Issue References ๐
I saw [the K8s
IT](https://github.com/apache/kyuubi/actions/runs/7776268590/job/21203214430?pr=6045)
failed with
```
Error: No space left on device :
'/home/runner/runners/2.312.0/_diag/pages/131e7057-723d-4281-b296-b876237f4318_21774d93-c106-57cd-810b-f0e6bf96d865_1.log'
```
## Describe Your Solution ๐ง
Borrow the solution from Apache Spark, just uninstall the un-used large
pre-installed packages.
## Types of changes :bookmark:
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
## Test Plan ๐งช
Wait CI report
---
# Checklist ๐
- [x] This patch was not authored or co-authored using [Generative
Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6047 from pan3793/ci-space.
Closes #6047
272df2790 [Cheng Pan] nit
8c78115a4 [Cheng Pan] nit
a9c8b035c [Cheng Pan] nit
ab4c3e461 [Cheng Pan] keep jdk8
9d1ec4e44 [Cheng Pan] extend
47f90b95e [Cheng Pan] nit
30b82cdf0 [Cheng Pan] fix
352689f89 [Cheng Pan] [CI] Free up disk space for K8s IT
Authored-by: Cheng Pan <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
---
.github/scripts/free_disk_space.sh | 55 +++++++++++++++++++++++++++
.github/workflows/gluten.yml | 4 ++
.github/workflows/master.yml | 24 ++++++++++--
.github/workflows/publish-snapshot-docker.yml | 5 ++-
.github/workflows/publish-snapshot-nexus.yml | 5 ++-
5 files changed, 85 insertions(+), 8 deletions(-)
diff --git a/.github/scripts/free_disk_space.sh
b/.github/scripts/free_disk_space.sh
new file mode 100755
index 000000000..b51dba8aa
--- /dev/null
+++ b/.github/scripts/free_disk_space.sh
@@ -0,0 +1,55 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# This script is inspired by Apache Spark
+
+echo "=================================="
+echo "Free up disk space on CI system"
+echo "=================================="
+
+echo "Listing 100 largest packages"
+dpkg-query -Wf '${Installed-Size}\t${Package}\n' | sort -n | tail -n 100
+df -h
+
+echo "Removing large packages"
+sudo rm -rf /usr/share/dotnet/
+sudo rm -rf /usr/share/php/
+sudo rm -rf /usr/local/graalvm/
+sudo rm -rf /usr/local/.ghcup/
+sudo rm -rf /usr/local/share/powershell
+sudo rm -rf /usr/local/share/chromium
+sudo rm -rf /usr/local/lib/android
+sudo rm -rf /usr/local/lib/node_modules
+
+sudo rm -rf /opt/az
+sudo rm -rf /opt/hostedtoolcache/CodeQL
+sudo rm -rf /opt/hostedtoolcache/go
+sudo rm -rf /opt/hostedtoolcache/node
+
+sudo apt-get remove --purge -y '^aspnet.*'
+sudo apt-get remove --purge -y '^dotnet-.*'
+sudo apt-get remove --purge -y '^llvm-.*'
+sudo apt-get remove --purge -y 'php.*'
+sudo apt-get remove --purge -y '^temurin-\d{n,}.*'
+sudo apt-get remove --purge -y snapd google-chrome-stable
microsoft-edge-stable firefox
+sudo apt-get remove --purge -y azure-cli google-cloud-sdk mono-devel msbuild
powershell libgl1-mesa-dri
+sudo apt-get autoremove --purge -y
+sudo apt-get clean
+
+df -h
diff --git a/.github/workflows/gluten.yml b/.github/workflows/gluten.yml
index 45f4edf54..d04c1ff21 100644
--- a/.github/workflows/gluten.yml
+++ b/.github/workflows/gluten.yml
@@ -30,6 +30,8 @@ jobs:
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v4
+ - name: Free up disk space
+ run: ./.github/scripts/free_disk_space.sh
- name: Tune Runner VM
uses: ./.github/actions/tune-runner-vm
- name: Update and Upgrade
@@ -79,6 +81,8 @@ jobs:
spark: [ '3.4', '3.3' ]
steps:
- uses: actions/checkout@v4
+ - name: Free up disk space
+ run: ./.github/scripts/free_disk_space.sh
- name: Tune Runner VM
uses: ./.github/actions/tune-runner-vm
- name: Update and Upgrade
diff --git a/.github/workflows/master.yml b/.github/workflows/master.yml
index b1083cfa2..42ac6e37c 100644
--- a/.github/workflows/master.yml
+++ b/.github/workflows/master.yml
@@ -87,6 +87,8 @@ jobs:
SPARK_LOCAL_IP: localhost
steps:
- uses: actions/checkout@v4
+ - name: Free up disk space
+ run: ./.github/scripts/free_disk_space.sh
- name: Tune Runner VM
uses: ./.github/actions/tune-runner-vm
- name: Setup JDK ${{ matrix.java }}
@@ -144,6 +146,8 @@ jobs:
- '3.4'
steps:
- uses: actions/checkout@v4
+ - name: Free up disk space
+ run: ./.github/scripts/free_disk_space.sh
- name: Tune Runner VM
uses: ./.github/actions/tune-runner-vm
- name: Setup JDK ${{ matrix.java }}
@@ -200,6 +204,8 @@ jobs:
comment: 'verify-on-flink-1.18-binary'
steps:
- uses: actions/checkout@v4
+ - name: Free up disk space
+ run: ./.github/scripts/free_disk_space.sh
- name: Tune Runner VM
uses: ./.github/actions/tune-runner-vm
- name: Setup JDK ${{ matrix.java }}
@@ -256,6 +262,8 @@ jobs:
comment: 'verify-on-hive-2.1-cdh6-binary'
steps:
- uses: actions/checkout@v4
+ - name: Free up disk space
+ run: ./.github/scripts/free_disk_space.sh
- name: Tune Runner VM
uses: ./.github/actions/tune-runner-vm
- name: Setup JDK ${{ matrix.java }}
@@ -302,6 +310,8 @@ jobs:
comment: [ "normal" ]
steps:
- uses: actions/checkout@v4
+ - name: Free up disk space
+ run: ./.github/scripts/free_disk_space.sh
- name: Tune Runner VM
uses: ./.github/actions/tune-runner-vm
- name: Setup JDK ${{ matrix.java }}
@@ -341,6 +351,8 @@ jobs:
comment: [ "normal" ]
steps:
- uses: actions/checkout@v4
+ - name: Free up disk space
+ run: ./.github/scripts/free_disk_space.sh
- name: Tune Runner VM
uses: ./.github/actions/tune-runner-vm
- name: Setup JDK ${{ matrix.java }}
@@ -375,6 +387,8 @@ jobs:
SPARK_LOCAL_IP: localhost
steps:
- uses: actions/checkout@v4
+ - name: Free up disk space
+ run: ./.github/scripts/free_disk_space.sh
- name: Tune Runner VM
uses: ./.github/actions/tune-runner-vm
- name: Setup JDK 8
@@ -400,8 +414,9 @@ jobs:
name: Kyuubi Server On Kubernetes Integration Test
runs-on: ubuntu-22.04
steps:
- - name: Checkout
- uses: actions/checkout@v4
+ - uses: actions/checkout@v4
+ - name: Free up disk space
+ run: ./.github/scripts/free_disk_space.sh
# https://github.com/docker/build-push-action
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
@@ -472,8 +487,9 @@ jobs:
name: Spark Engine On Kubernetes Integration Test
runs-on: ubuntu-22.04
steps:
- - name: Checkout
- uses: actions/checkout@v4
+ - uses: actions/checkout@v4
+ - name: Free up disk space
+ run: ./.github/scripts/free_disk_space.sh
- name: Cache Engine Archives
uses: ./.github/actions/cache-engine-archives
- name: Setup Minikube
diff --git a/.github/workflows/publish-snapshot-docker.yml
b/.github/workflows/publish-snapshot-docker.yml
index 0a73dcc2d..a7df6135f 100644
--- a/.github/workflows/publish-snapshot-docker.yml
+++ b/.github/workflows/publish-snapshot-docker.yml
@@ -27,8 +27,9 @@ jobs:
if: ${{ startsWith(github.repository, 'apache/') }}
runs-on: ubuntu-22.04
steps:
- - name: Checkout
- uses: actions/checkout@v4
+ - uses: actions/checkout@v4
+ - name: Free up disk space
+ run: ./.github/scripts/free_disk_space.sh
- name: Set up QEMU
uses: docker/setup-qemu-action@v3
- name: Set up Docker Buildx
diff --git a/.github/workflows/publish-snapshot-nexus.yml
b/.github/workflows/publish-snapshot-nexus.yml
index 64dd1a690..2be08301f 100644
--- a/.github/workflows/publish-snapshot-nexus.yml
+++ b/.github/workflows/publish-snapshot-nexus.yml
@@ -42,10 +42,11 @@ jobs:
- branch: branch-1.8
profiles: -Pflink-provided,spark-provided,hive-provided,spark-3.4
steps:
- - name: Checkout repository
- uses: actions/checkout@v4
+ - uses: actions/checkout@v4
with:
ref: ${{ matrix.branch }}
+ - name: Free up disk space
+ run: ./.github/scripts/free_disk_space.sh
- name: Setup JDK 8
uses: actions/setup-java@v4
with:
