This is an automated email from the ASF dual-hosted git repository.
philo pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git
The following commit(s) were added to refs/heads/main by this push:
new 66bb774f33 [GLUTEN-7514][VL] Reorganize Dockerfiles and document how
to build gluten in docker (#7515)
66bb774f33 is described below
commit 66bb774f33e7419996bd052ae2ba9cbafcf9a489
Author: PHILO-HE <[email protected]>
AuthorDate: Wed Oct 16 10:21:44 2024 +0800
[GLUTEN-7514][VL] Reorganize Dockerfiles and document how to build gluten
in docker (#7515)
---
.github/workflows/docker_image.yml | 6 +-
.gitignore | 1 -
dev/.dockerignore | 3 +
dev/{vcpkg => }/docker/Dockerfile | 4 +-
.../Dockerfile.centos7-static-build} | 0
.../docker/Dockerfile.centos8-dynamic-build | 0
dev/{vcpkg => docker}/Makefile | 20 ++++---
dev/{vcpkg => }/docker/entrypoint.sh | 0
dev/vcpkg/.dockerignore | 5 --
docs/developers/ProfileMemoryOfGlutenWithVelox.md | 2 +-
docs/developers/docker_centos7.md | 53 ------------------
docs/developers/docker_centos8.md | 49 ----------------
docs/developers/docker_ubuntu22.04.md | 65 ----------------------
docs/developers/velox-backend-CI.md | 4 +-
docs/developers/velox-backend-build-in-docker.md | 21 +++++++
15 files changed, 46 insertions(+), 187 deletions(-)
diff --git a/.github/workflows/docker_image.yml
b/.github/workflows/docker_image.yml
index 47ebfac5cd..0a6742099b 100644
--- a/.github/workflows/docker_image.yml
+++ b/.github/workflows/docker_image.yml
@@ -21,6 +21,8 @@ on:
- main
paths:
- '.github/workflows/docker_image.yml'
+ - 'dev/docker/Dockerfile.centos7-static-build'
+ - 'dev/docker/Dockerfile.centos8-dynamic-build'
schedule:
- cron: '0 20 * * 0'
@@ -45,7 +47,7 @@ jobs:
uses: docker/build-push-action@v2
with:
context: .
- file: dev/vcpkg/docker/Dockerfile.gha
+ file: dev/docker/Dockerfile.centos7-static-build
push: true
tags: apache/gluten:vcpkg-centos-7
@@ -69,6 +71,6 @@ jobs:
uses: docker/build-push-action@v2
with:
context: .
- file: .github/workflows/docker/Dockerfile.gha
+ file: dev/docker/Dockerfile.centos8-dynamic-build
push: true
tags: apache/gluten:centos-8
diff --git a/.gitignore b/.gitignore
index 7e82d0e53e..d30ea813ac 100644
--- a/.gitignore
+++ b/.gitignore
@@ -52,7 +52,6 @@ thirdparty/*.tar*
CMakeFiles/
CMakeCache.txt
CTestTestfile.cmake
-Makefile
cmake_install.cmake
build/
*-build/
diff --git a/dev/.dockerignore b/dev/.dockerignore
new file mode 100644
index 0000000000..43ca88b253
--- /dev/null
+++ b/dev/.dockerignore
@@ -0,0 +1,3 @@
+*
+!/vcpkg/setup-build-depends.sh
+!/docker/entrypoint.sh
diff --git a/dev/vcpkg/docker/Dockerfile b/dev/docker/Dockerfile
similarity index 87%
rename from dev/vcpkg/docker/Dockerfile
rename to dev/docker/Dockerfile
index 7332bbf45f..5fb5a976f5 100644
--- a/dev/vcpkg/docker/Dockerfile
+++ b/dev/docker/Dockerfile
@@ -3,7 +3,7 @@ FROM centos:7
RUN sed -i "s/enabled=1/enabled=0/" /etc/yum/pluginconf.d/fastestmirror.conf
WORKDIR /tmp/vcpkg
-COPY setup-build-depends.sh setup-build-depends.sh
+COPY vcpkg/setup-build-depends.sh setup-build-depends.sh
RUN ./setup-build-depends.sh
WORKDIR /
@@ -22,4 +22,4 @@ USER ${BUILDER_UID}:${BUILDER_GID}
ENV VCPKG_BINARY_SOURCES=default
COPY docker/entrypoint.sh /entrypoint
ENTRYPOINT ["/entrypoint"]
-CMD ["sleep", "inf"]
\ No newline at end of file
+CMD ["sleep", "inf"]
diff --git a/dev/vcpkg/docker/Dockerfile.gha
b/dev/docker/Dockerfile.centos7-static-build
similarity index 100%
rename from dev/vcpkg/docker/Dockerfile.gha
rename to dev/docker/Dockerfile.centos7-static-build
diff --git a/.github/workflows/docker/Dockerfile.gha
b/dev/docker/Dockerfile.centos8-dynamic-build
similarity index 100%
rename from .github/workflows/docker/Dockerfile.gha
rename to dev/docker/Dockerfile.centos8-dynamic-build
diff --git a/dev/vcpkg/Makefile b/dev/docker/Makefile
similarity index 68%
rename from dev/vcpkg/Makefile
rename to dev/docker/Makefile
index d37f75b546..ffe25955e3 100644
--- a/dev/vcpkg/Makefile
+++ b/dev/docker/Makefile
@@ -1,4 +1,4 @@
-DOCKER_IMAGE=apache/gluten:vcpkg-centos-7
+DOCKER_IMAGE=apache/gluten:centos-7-vcpkg-build-depends
GLUTEN_REPO=$(shell realpath -L ../..)
CCACHE_DIR=$(HOME)/.ccache
@@ -17,26 +17,32 @@ build: $(CCACHE_DIR) $(VCPKG_BINARY_CACHE_DIR)
$(MAVEN_M2_DIR)
-e http_proxy \
-e https_proxy \
--workdir $(GLUTEN_REPO) \
- -ti \
$(DOCKER_IMAGE) \
./dev/package-vcpkg.sh
docker-image:
docker build \
- --file docker/Dockerfile \
+ --file Dockerfile \
--build-arg BUILDER_UID=`id -u` \
--build-arg BUILDER_GID=`id -g` \
--tag "$(DOCKER_IMAGE)" \
- .
+ ..
-docker-image-gha:
+docker-image-static-build:
docker build \
- --file docker/Dockerfile.gha \
- --tag "$(DOCKER_IMAGE)" \
+ --file Dockerfile.centos7-static-build \
+ --tag "apache/gluten:vcpkg-centos-7" \
--build-arg HTTPS_PROXY="" \
--build-arg HTTP_PROXY="" \
.
+docker-image-dynamic-build:
+ docker build \
+ --file Dockerfile.centos8-dynamic-build \
+ --tag "apache/gluten:centos-8" \
+ --build-arg HTTPS_PROXY="" \
+ --build-arg HTTP_PROXY="" \
+ .
$(CCACHE_DIR) $(VCPKG_BINARY_CACHE_DIR) $(MAVEN_M2_DIR): %:
mkdir -p $@
diff --git a/dev/vcpkg/docker/entrypoint.sh b/dev/docker/entrypoint.sh
similarity index 100%
rename from dev/vcpkg/docker/entrypoint.sh
rename to dev/docker/entrypoint.sh
diff --git a/dev/vcpkg/.dockerignore b/dev/vcpkg/.dockerignore
deleted file mode 100644
index 7874389a76..0000000000
--- a/dev/vcpkg/.dockerignore
+++ /dev/null
@@ -1,5 +0,0 @@
-/.dockerignore
-/.vcpkg
-/vcpkg_installed
-/README.md
-/Dockerfile
\ No newline at end of file
diff --git a/docs/developers/ProfileMemoryOfGlutenWithVelox.md
b/docs/developers/ProfileMemoryOfGlutenWithVelox.md
index 270a06b5a8..480755cb99 100644
--- a/docs/developers/ProfileMemoryOfGlutenWithVelox.md
+++ b/docs/developers/ProfileMemoryOfGlutenWithVelox.md
@@ -1,7 +1,7 @@
---
layout: page
title: Profile memory consumption of Gluten
-nav_order: 15
+nav_order: 8
has_children: true
parent: /developer-overview/
---
diff --git a/docs/developers/docker_centos7.md
b/docs/developers/docker_centos7.md
deleted file mode 100644
index 6678cc7ab9..0000000000
--- a/docs/developers/docker_centos7.md
+++ /dev/null
@@ -1,53 +0,0 @@
----
-layout: page
-title: Docker script for CentOS 7
-nav_order: 8
-parent: Developer Overview
----
-Here is a docker script we verified to build Gluten+Velox backend on CentOS 7:
-
-Run on host as root user:
-```
-docker pull centos:7
-docker run -itd --name gluten centos:7 /bin/bash
-docker attach gluten
-```
-
-Run in docker:
-```
-yum -y install epel-release centos-release-scl
-yum -y install \
- git \
- dnf \
- cmake3 \
- devtoolset-9 \
- java-1.8.0-openjdk \
- java-1.8.0-openjdk-devel \
- ninja-build \
- wget \
- ca-certificates \
- sudo
-
-# gluten need maven version >=3.6.3
-wget
https://downloads.apache.org/maven/maven-3/3.8.8/binaries/apache-maven-3.8.8-bin.tar.gz
-tar -xvf apache-maven-3.8.8-bin.tar.gz
-mv apache-maven-3.8.8 /usr/lib/maven
-export MAVEN_HOME=/usr/lib/maven
-export PATH=${PATH}:${MAVEN_HOME}/bin
-
-# cmake 3.x is required
-ln -s /usr/bin/cmake3 /usr/local/bin/cmake
-
-# enable gcc 9
-. /opt/rh/devtoolset-9/enable || exit 1
-
-export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk
-export PATH=$JAVA_HOME/bin:$PATH
-
-git clone https://github.com/apache/incubator-gluten.git
-cd incubator-gluten
-
-# To access HDFS or S3, you need to add the parameters `--enable_hdfs=ON` and
`--enable_s3=ON`
-# If you have the same error with issue-3283, you need to add the parameter
`--compile_arrow_java=ON`
-./dev/buildbundle-veloxbe.sh
-```
diff --git a/docs/developers/docker_centos8.md
b/docs/developers/docker_centos8.md
deleted file mode 100755
index 1b604bf209..0000000000
--- a/docs/developers/docker_centos8.md
+++ /dev/null
@@ -1,49 +0,0 @@
----
-layout: page
-title: Docker script for CentOS 8
-nav_order: 7
-parent: Developer Overview
----
-Here is a docker script we verified to build Gluten+Velox backend on Centos8:
-
-Run on host as root user:
-```
-docker pull centos:8
-docker run -itd --name gluten centos:8 /bin/bash
-docker attach gluten
-```
-
-Run in docker:
-```
-#update mirror
-sed -i -e "s|mirrorlist=|#mirrorlist=|g" /etc/yum.repos.d/CentOS-*
-sed -i -e
"s|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g"
/etc/yum.repos.d/CentOS-*
-
-dnf install -y epel-release sudo
-yum install -y dnf-plugins-core
-yum config-manager --set-enabled powertools
-dnf --enablerepo=powertools install -y ninja-build
-dnf --enablerepo=powertools install -y libdwarf-devel
-dnf install -y --setopt=install_weak_deps=False ccache gcc-toolset-9 git wget
which libevent-devel \
- openssl-devel re2-devel libzstd-devel lz4-devel double-conversion-devel \
- curl-devel cmake libicu-devel
-
-source /opt/rh/gcc-toolset-9/enable || exit 1
-
-yum install -y java-1.8.0-openjdk-devel patch
-export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk
-export PATH=$JAVA_HOME/bin:$PATH
-
-#gluten need maven version >=3.6.3
-wget
https://downloads.apache.org/maven/maven-3/3.8.8/binaries/apache-maven-3.8.8-bin.tar.gz
-tar -xvf apache-maven-3.8.8-bin.tar.gz
-mv apache-maven-3.8.8 /usr/lib/maven
-export MAVEN_HOME=/usr/lib/maven
-export PATH=${PATH}:${MAVEN_HOME}/bin
-
-git clone https://github.com/apache/incubator-gluten.git
-cd incubator-gluten
-
-# To access HDFS or S3, you need to add the parameters `--enable_hdfs=ON` and
`--enable_s3=ON`
-./dev/buildbundle-veloxbe.sh
-```
\ No newline at end of file
diff --git a/docs/developers/docker_ubuntu22.04.md
b/docs/developers/docker_ubuntu22.04.md
deleted file mode 100644
index b139fecdcf..0000000000
--- a/docs/developers/docker_ubuntu22.04.md
+++ /dev/null
@@ -1,65 +0,0 @@
----
-layout: page
-title: Docker script for Ubuntu 22.04/20.04
-nav_order: 6
-parent: Developer Overview
----
-
-To the first build, it's suggested to build Gluten in a clean docker image.
Otherwise it's easy to run into library version conflict issues.
-Here is a docker script we verified to build Gluten+Velox backend on
Ubuntu22.04/20.04:
-
-Run on host as root user:
-```
-docker pull ubuntu:22.04
-docker run -itd --network host --name gluten ubuntu:22.04 /bin/bash
-docker attach gluten
-```
-
-Run in docker:
-```
-apt-get update
-
-#install gcc and libraries to build arrow
-apt install software-properties-common
-apt install maven build-essential libssl-dev libre2-dev libcurl4-openssl-dev
clang lldb lld libz-dev git ninja-build uuid-dev autoconf-archive curl zip
unzip tar pkg-config bison libtool flex vim python3-pip
-pip3 install cmake==3.28.3
-
-#velox script needs sudo to install dependency libraries
-apt install sudo
-
-# make sure jemalloc is uninstalled, jemalloc will be build in vcpkg, which
conflicts with the default jemalloc in system
-apt purge libjemalloc-dev libjemalloc2 librust-jemalloc-sys-dev
-
-#make sure jdk8 is used. New version of jdk is not supported
-apt install -y openjdk-8-jdk
-apt install -y default-jdk
-export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
-export PATH=$JAVA_HOME/bin:$PATH
-
-#manually install tzdata to avoid the interactive timezone config
-ln -fs /usr/share/zoneinfo/America/New_York /etc/localtime
-DEBIAN_FRONTEND=noninteractive apt-get install -y tzdata
-dpkg --configure -a
-
-#setup proxy on necessary
-#export http_proxy=xxxx
-#export https_proxy=xxxx
-
-#clone gluten
-git clone https://github.com/apache/incubator-gluten.git
-cd incubator-gluten/
-
-#config maven proxy
-#mkdir ~/.m2/
-#vim ~/.m2/settings.xml
-
-# the script download velox & arrow and compile all dependency library
automatically
-# To access HDFS or S3, you need to add the parameters `--enable_hdfs=ON` and
`--enable_s3=ON`
-# It's suggested to build using static link, enabled by `--enable_vcpkg=ON`
-# For developer, it's suggested to enable Debug info, by
--build_type=RelWithDebInfo. Note RelWithDebInfo uses -o2, release uses -o3
-./dev/buildbundle-veloxbe.sh --enable_vcpkg=ON --build_type=RelWithDebInfo
-
-# If you meet the error like `epoll_wait: Function not implemented` when
running tests,
-# Please enable epoll_pwait by,
-export EVENT_NOEPOLL=1
-```
diff --git a/docs/developers/velox-backend-CI.md
b/docs/developers/velox-backend-CI.md
index e4dceffdf7..b70887ad56 100644
--- a/docs/developers/velox-backend-CI.md
+++ b/docs/developers/velox-backend-CI.md
@@ -1,7 +1,7 @@
---
layout: page
title: Velox Backend CI
-nav_order: 14
+nav_order: 6
parent: Developer Overview
---
# Velox Backend CI
@@ -9,7 +9,7 @@ parent: Developer Overview
GHA workflows are defined under `.github/workflows/`.
## Docker Build
-We have a weekly job to build a docker based on `Dockerfile.gha` for CI
verification, defined in `docker_image.yml`.
+We have a weekly job defined in `docker_image.yml` to build docker images
based on `Dockerfile.centos7-static-build` and
`Dockerfile.centos8-dynamic-build` for CI verification.
## Vcpkg Caching
Gluten main branch is pulled down during docker build. And vcpkg will cache
binary data of all dependencies defined under dev/vcpkg.
diff --git a/docs/developers/velox-backend-build-in-docker.md
b/docs/developers/velox-backend-build-in-docker.md
new file mode 100755
index 0000000000..4820c7cdc7
--- /dev/null
+++ b/docs/developers/velox-backend-build-in-docker.md
@@ -0,0 +1,21 @@
+---
+layout: page
+title: Build Gluten Velox backend in docker
+nav_order: 7
+parent: Developer Overview
+---
+
+Currently, Centos-7/8/9 and Ubuntu 20.04/22.04 are supported to build Gluten
Velox backend. Please refer to
+`.github/workflows/velox_weekly.yml` to install required tools before the
build.
+
+There are two docker images with almost all dependencies installed, respective
for static build and dynamic build.
+The according Dockerfiles are respectively `Dockerfile.centos7-static-build`
and `Dockerfile.centos8-dynamic-build`
+under `dev/docker/`.
+
+```shell
+# For static build on centos-7.
+docker pull apache/gluten:vcpkg-centos-7
+
+# For dynamic build on centos-8.
+docker pull apache/gluten:centos-8 (dynamic build)
+```
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
