This is an automated email from the ASF dual-hosted git repository.
liuxun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/main by this push:
new 391edb8ce9 [#7101] Improvement(docker-image): fix ranger docker image
build failure in CI (#7112)
391edb8ce9 is described below
commit 391edb8ce932ccd66e8e87c2e2912f6202aec9a0
Author: Eric Chang <[email protected]>
AuthorDate: Tue May 20 22:23:07 2025 +0800
[#7101] Improvement(docker-image): fix ranger docker image build failure in
CI (#7112)
Tasks should be done before merging this PR:
- [x] @xunliu should publish `apache/gravitino-ci:ranger-0.1.2` and
`apache/gravitino-ci:hive-0.1.19` docker image to Docker Hub.
- [x] revert e53011809bf5cd4bc3b252d6f0f208d0ed683a17 (commit to test
newly built docker image)
- [x] upgrade image version in `build.gradle.kts`.
- [x] add new release notes in `docs/docker-image-details.md`.
- [x] Make sure CI passed.
### What changes were proposed in this pull request?
Make building ranger-related docker image works again.
This PR includes these changes:
In `dev/docker/ranger/ranger-dependency.sh`:
Tweak some settings to build ranger in Github CI.
In `.github/workflows/docker-image.yml`:
- remove unnecessary GitHub access token input field in
`.github/workflows/docker-image.yml`, because this is for downloading
ranger from `datastrato's` repository, and now this is not needed.
- Github Action: allow setting docker repo name in workflow_dispatch
(default is `apache`), this is needed because developers can use their
own Gravitino repo with Github Action to publish docker image to their
Docker repo.
In `dev/docker/build-docker.sh`:
- Remove `--use` option while creating `gravitino-builder`, because this
will globally affect other builds outside of Gravitino project. Instead,
we use `--builder` to specify builder for every `docker buildx build`
command.
```
docker buildx build --builder ${BUILDER_NAME}
```
### Why are the changes needed?
Without this, the ranger, hive docker image for CI can not be built.
Fix: #7101
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
This PR is tested by following steps:
1. Build image
[unknowntpo/gravitino-ci:ranger-ranger-docker-github-ci](https://hub.docker.com/layers/unknowntpo/gravitino-ci/ranger-ranger-docker-github-ci/images/sha256-bdf0064583251a2dd0af7801ec4690a8d21160e61b86422c17e64f260dd1a536)
in [Github
Action](https://github.com/unknowntpo/gravitino/actions/runs/14963769539)
2. Build image
[unknowntpo/gravitino-ci:hive-ranger-docker-github-ci](https://hub.docker.com/layers/unknowntpo/gravitino-ci/hive-ranger-docker-github-ci/images/sha256-4759d88061f4b4911dff6d6b01b0e62a7aed2fd2b9146cb17b37d9802f7ef666)
in [Github
Action](https://github.com/unknowntpo/gravitino/actions/runs/14963715516)
3. Add commit 5458dc7e7b4dca2eb29dd6e423ddde60cdbd0e87 ,
710998b1f5e0f6b5bff5761f48bfaf804c30c379 and use image above to run IT
tests in Github Action on pull request.
---
.github/workflows/docker-image.yml | 45 ++++++++---------
build.gradle.kts | 4 +-
dev/docker/build-docker.sh | 10 ++--
dev/docker/ranger/docker-compose.ranger-build.yml | 61 +++++++++++++++++++++++
dev/docker/ranger/ranger-dependency.sh | 32 ++++++------
docs/docker-image-details.md | 5 ++
6 files changed, 111 insertions(+), 46 deletions(-)
diff --git a/.github/workflows/docker-image.yml
b/.github/workflows/docker-image.yml
index 03bd77946c..4bd63eed73 100644
--- a/.github/workflows/docker-image.yml
+++ b/.github/workflows/docker-image.yml
@@ -19,11 +19,15 @@ on:
- 'gravitino-playground:hive'
- 'gravitino-playground:ranger'
- 'gravitino-iceberg-rest-server'
+ docker_repo_name:
+ description: 'Docker repository name (default is apache)'
+ required: false
+ default: 'apache'
+ type: string
version:
description: 'Docker version to apply to this image'
required: true
type: string
-
username:
description: 'Docker username'
required: true
@@ -32,11 +36,6 @@ on:
description: 'Publish Docker token'
required: true
type: string
- pat:
- description: 'GitHub private access token'
- required: true
- type: string
-
publish-latest-tag:
description: 'Whether to update the latest tag. This operation is only
applicable to official releases and should not be used for Release Candidate
(RC).'
required: false
@@ -55,50 +54,50 @@ jobs:
run: |
if [ "${{ github.event.inputs.image }}" == "gravitino-ci:hive" ];
then
echo "image_type=hive" >> $GITHUB_ENV
- echo "image_name=apache/gravitino-ci" >> $GITHUB_ENV
+ echo "image_name=${{ github.event.inputs.docker_repo_name
}}/gravitino-ci" >> $GITHUB_ENV
echo "tag_name=hive" >> $GITHUB_ENV
elif [ "${{ github.event.inputs.image }}" ==
"gravitino-ci:kerberos-hive" ]; then
echo "image_type=kerberos-hive" >> $GITHUB_ENV
- echo "image_name=apache/gravitino-ci" >> $GITHUB_ENV
+ echo "image_name=${{ github.event.inputs.docker_repo_name
}}/gravitino-ci" >> $GITHUB_ENV
echo "tag_name=kerberos-hive" >> $GITHUB_ENV
elif [ "${{ github.event.inputs.image }}" == "gravitino-ci:trino" ];
then
echo "image_type=trino" >> $GITHUB_ENV
- echo "image_name=apache/gravitino-ci" >> $GITHUB_ENV
+ echo "image_name=${{ github.event.inputs.docker_repo_name
}}/gravitino-ci" >> $GITHUB_ENV
echo "tag_name=trino" >> $GITHUB_ENV
elif [ "${{ github.event.inputs.image }}" == "gravitino-ci:doris" ];
then
echo "image_type=doris" >> $GITHUB_ENV
- echo "image_name=apache/gravitino-ci" >> $GITHUB_ENV
+ echo "image_name=${{ github.event.inputs.docker_repo_name
}}/gravitino-ci" >> $GITHUB_ENV
echo "tag_name=doris" >> $GITHUB_ENV
elif [ "${{ github.event.inputs.image }}" == "gravitino-ci:ranger"
]; then
echo "image_type=ranger" >> $GITHUB_ENV
- echo "image_name=apache/gravitino-ci" >> $GITHUB_ENV
+ echo "image_name=${{ github.event.inputs.docker_repo_name
}}/gravitino-ci" >> $GITHUB_ENV
echo "tag_name=ranger" >> $GITHUB_ENV
elif [ "${{ github.event.inputs.image }}" == "gravitino" ]; then
echo "image_type=gravitino" >> $GITHUB_ENV
- echo "image_name=apache/gravitino" >> $GITHUB_ENV
- # `apache/gravitino` is the default image name, didn't need to tag
alias name
+ echo "image_name=${{ github.event.inputs.docker_repo_name
}}/gravitino" >> $GITHUB_ENV
+ # `${{ github.event.inputs.docker_repo_name }}/gravitino` is the
default image name, didn"t need to tag alias name
elif [ "${{ github.event.inputs.image }}" ==
"gravitino-playground:trino" ]; then
echo "image_type=trino" >> $GITHUB_ENV
- echo "image_name=apache/gravitino-playground" >> $GITHUB_ENV
+ echo "image_name=${{ github.event.inputs.docker_repo_name
}}/gravitino-playground" >> $GITHUB_ENV
echo "tag_name=trino" >> $GITHUB_ENV
elif [ "${{ github.event.inputs.image }}" ==
"gravitino-playground:hive" ]; then
echo "image_type=hive" >> $GITHUB_ENV
- echo "image_name=apache/gravitino-playground" >> $GITHUB_ENV
+ echo "image_name=${{ github.event.inputs.docker_repo_name
}}/gravitino-playground" >> $GITHUB_ENV
echo "tag_name=hive" >> $GITHUB_ENV
elif [ "${{ github.event.inputs.image }}" ==
"gravitino-playground:ranger" ]; then
echo "image_type=ranger" >> $GITHUB_ENV
- echo "image_name=apache/gravitino-playground" >> $GITHUB_ENV
+ echo "image_name=${{ github.event.inputs.docker_repo_name
}}/gravitino-playground" >> $GITHUB_ENV
echo "tag_name=ranger" >> $GITHUB_ENV
elif [ "${{ github.event.inputs.image }}" ==
"gravitino-iceberg-rest-server" ]; then
echo "image_type=iceberg-rest-server" >> $GITHUB_ENV
- echo "image_name=apache/gravitino-iceberg-rest" >> $GITHUB_ENV
+ echo "image_name=${{ github.event.inputs.docker_repo_name
}}/gravitino-iceberg-rest" >> $GITHUB_ENV
fi
-
+
if [ "${{ github.event.inputs.publish-latest-tag }}" == "true" ];
then
echo "publish_latest=true" >> $GITHUB_ENV
else
echo "publish_latest=false" >> $GITHUB_ENV
- fi
+ fi
- name: Check publish Docker token
run: |
@@ -127,18 +126,16 @@ jobs:
distribution: 'temurin'
- name: Build and Push the Docker image
- run: |
+ run: |
sudo rm -rf /usr/local/lib/android
sudo rm -rf /opt/hostedtoolcache/CodeQL
-
+
if [[ -n "${tag_name}" ]]; then
full_tag_name="${tag_name}-${{ github.event.inputs.version }}"
else
full_tag_name="${{ github.event.inputs.version }}"
fi
-
- export PRIVATE_ACCESS_TOKEN=${{ github.event.inputs.pat }}
-
+
if [[ "${publish_latest}" == "true" ]]; then
echo "Publish tag ${full_tag_name}, and update latest too."
./dev/docker/build-docker.sh --platform all --type ${image_type}
--image ${image_name} --tag ${full_tag_name} --latest
diff --git a/build.gradle.kts b/build.gradle.kts
index 7165687131..3ad2bdde85 100644
--- a/build.gradle.kts
+++ b/build.gradle.kts
@@ -172,11 +172,11 @@ allprojects {
param.environment("PROJECT_VERSION", project.version)
// Gravitino CI Docker image
- param.environment("GRAVITINO_CI_HIVE_DOCKER_IMAGE",
"apache/gravitino-ci:hive-0.1.18")
+ param.environment("GRAVITINO_CI_HIVE_DOCKER_IMAGE",
"apache/gravitino-ci:hive-0.1.19")
param.environment("GRAVITINO_CI_KERBEROS_HIVE_DOCKER_IMAGE",
"apache/gravitino-ci:kerberos-hive-0.1.5")
param.environment("GRAVITINO_CI_DORIS_DOCKER_IMAGE",
"apache/gravitino-ci:doris-0.1.5")
param.environment("GRAVITINO_CI_TRINO_DOCKER_IMAGE",
"apache/gravitino-ci:trino-0.1.6")
- param.environment("GRAVITINO_CI_RANGER_DOCKER_IMAGE",
"apache/gravitino-ci:ranger-0.1.1")
+ param.environment("GRAVITINO_CI_RANGER_DOCKER_IMAGE",
"apache/gravitino-ci:ranger-0.1.2")
param.environment("GRAVITINO_CI_KAFKA_DOCKER_IMAGE",
"apache/kafka:3.7.0")
param.environment("GRAVITINO_CI_LOCALSTACK_DOCKER_IMAGE",
"localstack/localstack:latest")
diff --git a/dev/docker/build-docker.sh b/dev/docker/build-docker.sh
index a2cef6c8f1..c48607e615 100755
--- a/dev/docker/build-docker.sh
+++ b/dev/docker/build-docker.sh
@@ -130,20 +130,20 @@ if echo "${builders}" | grep -q "${BUILDER_NAME}"; then
echo "BuildKit builder '${BUILDER_NAME}' already exists."
else
echo "BuildKit builder '${BUILDER_NAME}' does not exist."
- docker buildx create --driver-opt env.BUILDKIT_STEP_LOG_MAX_SIZE=10000000
--platform linux/amd64,linux/arm64 --use --name ${BUILDER_NAME}
+ docker buildx create --driver-opt env.BUILDKIT_STEP_LOG_MAX_SIZE=10000000
--platform linux/amd64,linux/arm64 --name ${BUILDER_NAME}
fi
cd ${script_dir}/${component_type}
if [[ "${platform_type}" == "all" ]]; then
if [ ${build_latest} -eq 1 ]; then
- docker buildx build --no-cache --pull --platform=linux/amd64,linux/arm64
${build_args} --push --progress plain -f Dockerfile -t ${image_name}:latest -t
${image_name}:${tag_name} .
+ docker buildx build --builder ${BUILDER_NAME} --no-cache --pull
--platform=linux/amd64,linux/arm64 ${build_args} --push --progress plain -f
Dockerfile -t ${image_name}:latest -t ${image_name}:${tag_name} .
else
- docker buildx build --no-cache --pull --platform=linux/amd64,linux/arm64
${build_args} --push --progress plain -f Dockerfile -t
${image_name}:${tag_name} .
+ docker buildx build --builder ${BUILDER_NAME} --no-cache --pull
--platform=linux/amd64,linux/arm64 ${build_args} --push --progress plain -f
Dockerfile -t ${image_name}:${tag_name} .
fi
else
if [ ${build_latest} -eq 1 ]; then
- docker buildx build --no-cache --pull --platform=${platform_type}
${build_args} --output type=docker --progress plain -f Dockerfile -t
${image_name}:latest -t ${image_name}:${tag_name} .
+ docker buildx build --builder ${BUILDER_NAME} --no-cache --pull
--platform=${platform_type} ${build_args} --output type=docker --progress plain
-f Dockerfile -t ${image_name}:latest -t ${image_name}:${tag_name} .
else
- docker buildx build --no-cache --pull --platform=${platform_type}
${build_args} --output type=docker --progress plain -f Dockerfile -t
${image_name}:${tag_name} .
+ docker buildx build --builder ${BUILDER_NAME} --no-cache --pull
--platform=${platform_type} ${build_args} --output type=docker --progress plain
-f Dockerfile -t ${image_name}:${tag_name} .
fi
fi
diff --git a/dev/docker/ranger/docker-compose.ranger-build.yml
b/dev/docker/ranger/docker-compose.ranger-build.yml
new file mode 100644
index 0000000000..77bd191edf
--- /dev/null
+++ b/dev/docker/ranger/docker-compose.ranger-build.yml
@@ -0,0 +1,61 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# This file is copied from apache/ranger at commit [RANGER-5146: 500 API Error
When Deleting TagDef with a Linked
Tag](https://github.com/apache/ranger/blob/ff36aabe36169b94862c51a5b403f59c9d728b94/dev-support/ranger-docker/docker-compose.ranger-build.yml)
+# with some modification to make build successful in GitHub Action:
+# - Add :Z option to volume mount.
+# - Use docker volume instead of bind mount.
+version: "3"
+services:
+ ranger-build:
+ build:
+ context: .
+ dockerfile: Dockerfile.ranger-build
+ args:
+ - RANGER_BUILD_JAVA_VERSION=${RANGER_BUILD_JAVA_VERSION}
+ image: ranger-build
+ container_name: ranger-build
+ hostname: ranger-build.example.com
+ networks:
+ - ranger
+ volumes:
+ # SELinux
+ # https://github.com/moby/moby/issues/41202#issuecomment-657422171
+ - m2:/home/ranger/.m2:Z
+ - ./scripts:/home/ranger/scripts
+ - ./patches:/home/ranger/patches
+ - dist:/home/ranger/dist:Z
+ # mount from host project root to container's project root
+ - ${RANGER_HOME:-./../../}:/home/ranger/src:Z
+ depends_on:
+ - ranger-base
+ environment:
+ - BRANCH
+ - BUILD_HOST_SRC
+ - BUILD_OPTS
+ - PROFILE
+ - GIT_URL
+ - RANGER_VERSION
+ - SKIPTESTS
+
+networks:
+ ranger:
+ name: rangernw
+
+volumes:
+ m2:
+ dist:
diff --git a/dev/docker/ranger/ranger-dependency.sh
b/dev/docker/ranger/ranger-dependency.sh
index 6a77dbc4e5..164889f069 100755
--- a/dev/docker/ranger/ranger-dependency.sh
+++ b/dev/docker/ranger/ranger-dependency.sh
@@ -41,30 +41,32 @@ if [ ! -f "${ranger_dir}/packages/${RANGER_PACKAGE_NAME}"
]; then
git clone https://github.com/apache/ranger --branch master --single-branch
${ranger_dir}/packages/apache-ranger
# set the commit to RANGER-5146: 500 API Error When Deleting TagDef with a
Linked Tag
#
https://github.com/apache/ranger/commit/ff36aabe36169b94862c51a5b403f59c9d728b94
+ cd ${ranger_dir}/packages/apache-ranger
git reset --hard ff36aabe36169b94862c51a5b403f59c9d728b94
fi
cp ${ranger_dir}/.env
${ranger_dir}/packages/apache-ranger/dev-support/ranger-docker
+ # overwrite docker-compose.ranger-base.yml in apache-ranger, add :Z option
to volume to solve
+ # SELinux problem
+ cp ${ranger_dir}/docker-compose.ranger-build.yml
${ranger_dir}/packages/apache-ranger/dev-support/ranger-docker
cd ${ranger_dir}/packages/apache-ranger/dev-support/ranger-docker
- # Prevent builder to pull remote image
- # https://github.com/moby/buildkit/issues/2343#issuecomment-1311890308
- docker builder prune -f
- docker builder use default
+ mkdir -p ${ranger_dir}/packages/apache-ranger/dev-support/ranger-docker/dist
- export DOCKER_BUILDKIT=1
- export COMPOSE_DOCKER_CLI_BUILD=1
- export RANGER_DB_TYPE=mysql
+ # run command in subshell to avoid environment variable pollution
+ (
+ export DOCKER_BUILDKIT=1
+ export COMPOSE_DOCKER_CLI_BUILD=1
+ export RANGER_DB_TYPE=mysql
+ # Prevent builder to pull remote image
+ # https://github.com/moby/buildkit/issues/2343#issuecomment-1311890308
+ export BUILDX_BUILDER=default
- # run docker compose command to build packages
- docker compose -f docker-compose.ranger-base.yml -f
docker-compose.ranger-build.yml up --pull=never
- # copy packages from volume to host
- docker compose -f docker-compose.ranger-base.yml -f
docker-compose.ranger-build.yml cp ranger-build:/home/ranger/dist .
+ docker compose -f docker-compose.ranger-base.yml -f
docker-compose.ranger-build.yml up --pull=never
+ docker compose -f docker-compose.ranger-base.yml -f
docker-compose.ranger-build.yml cp ranger-build:/home/ranger/dist .
+ )
- cp ./dist/* ${ranger_dir}/packages
-
- # change back to gravitino-builder
- docker builder use gravitino-builder
+ cp ${ranger_dir}/packages/apache-ranger/dev-support/ranger-docker/dist/*
${ranger_dir}/packages
fi
if [ ! -f "${ranger_dir}/packages/${MYSQL_CONNECTOR_PACKAGE_NAME}" ]; then
diff --git a/docs/docker-image-details.md b/docs/docker-image-details.md
index 1c187b16d9..dbab9ee8c2 100644
--- a/docs/docker-image-details.md
+++ b/docs/docker-image-details.md
@@ -193,6 +193,8 @@ You can use this kind of image to test the catalog of
Apache Hive.
Changelog
+- apache/gravitino-ci:hive-0.1.19
+ - Build ranger packages from source.
- apache/gravitino-ci:hive-0.1.18
- Support UTF-8 encoding for the `hive-site.xml` file and Hive Metastore.
@@ -368,6 +370,9 @@ You can use this image to control Trino's permissions.
Changelog
+- apache/gravitino-ci:ranger-0.1.2
+ - Build ranger packages from source.
+
- apache/gravitino-ci:ranger-0.1.1 (Switch to Apache official DockerHub
repository)
- Use Gravitino release 0.6.0 Dockerfile to build the image.
