Copilot commented on code in PR #13422:
URL: https://github.com/apache/gluten/pull/13422#discussion_r4364785633
##########
.github/workflows/util/install-spark-deps.sh:
##########
@@ -24,12 +24,33 @@ set -e
# Install Hadoop binary
function install_hadoop() {
+ local HADOOP_VERSION=3.3.6
+ # apache/gluten:ubuntu-22.04-runtime ships Hadoop here, so CI jobs skip the
download.
+ if [ -d "/opt/hadoop-${HADOOP_VERSION}" ]; then
+ echo "Using pre-installed Hadoop in /opt/hadoop-${HADOOP_VERSION}"
+ export HADOOP_HOME="/opt/hadoop-${HADOOP_VERSION}"
+ else
+ download_hadoop "$HADOOP_VERSION"
+ export HADOOP_HOME="$PWD/hadoop-${HADOOP_VERSION}"
+ fi
+ export PATH="$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH"
+ export LD_LIBRARY_PATH="$HADOOP_HOME/lib/native:$LD_LIBRARY_PATH"
Review Comment:
If `LD_LIBRARY_PATH` is unset/empty, this expansion produces a trailing `:`
which introduces an empty path element (often treated as the current directory)
into the dynamic linker search path. Prefer a conditional append (e.g., only
add `:$LD_LIBRARY_PATH` when non-empty) to avoid weakening runtime library
search security.
##########
dev/docker/Dockerfile.centos8-runtime:
##########
@@ -0,0 +1,39 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Runtime image for the CentOS 8 TPC jobs, which check that the static native
library runs on a
+# plain CentOS 8. It adds only JDKs, CI tools and Celeborn releases to the OS
image, no native
+# libraries. All JDKs are installed side by side under
/usr/lib/jvm/java-<version>-openjdk; jobs
+# select one through JAVA_HOME instead of installing it at runtime.
+# Produces the apache/gluten:centos-8-runtime image.
+FROM centos:8
+
+ENV LANG=C.UTF-8
+ENV LC_ALL=C.UTF-8
+
+COPY dev/docker/install-ci-tools.sh /opt/ci/install-ci-tools.sh
+RUN set -ex; \
+ sed -i -e "s|mirrorlist=|#mirrorlist=|g" /etc/yum.repos.d/CentOS-* ||
true; \
+ sed -i -e
"s|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g"
/etc/yum.repos.d/CentOS-* || true; \
Review Comment:
This switches CentOS repository URLs to `http://vault.centos.org`, which is
vulnerable to MITM/tampering. Use `https://vault.centos.org` instead (and keep
GPG checks enabled) to improve transport integrity during image builds.
##########
dev/docker/install-ci-tools.sh:
##########
@@ -0,0 +1,67 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Installs the tools that CI jobs used to download at runtime, so PR jobs do
not depend on
+# package mirrors. Distro packages are installed by each Dockerfile; this
script adds:
+# - gh and jq, needed by the Apache Stash cache actions;
+# - with --pyspark-python=<python>, one virtualenv per PySpark version used
by the Spark
+# unit tests, under /opt/pyspark/<version>. Jobs select one by putting its
bin/ first in
+# PATH, which leaves the system python3 (and dnf) untouched.
+
+set -ex
+
+GH_VERSION=2.63.2
+JQ_VERSION=1.7.1
+PYSPARK_VERSIONS="3.4.4 3.5.5"
+PYSPARK_PYTHON=""
+
+for arg in "$@"; do
+ case $arg in
+ --pyspark-python=*)
+ PYSPARK_PYTHON="${arg#*=}"
+ ;;
+ *)
+ echo "Unknown argument: $arg" >&2
+ exit 1
+ ;;
+ esac
+done
+
+case "$(uname -m)" in
+ x86_64) ARCH=amd64 ;;
+ aarch64) ARCH=arm64 ;;
+ *) echo "unsupported arch: $(uname -m)" >&2; exit 1 ;;
+esac
+
+curl -fsSL
"https://github.com/cli/cli/releases/download/v${GH_VERSION}/gh_${GH_VERSION}_linux_${ARCH}.tar.gz";
\
+ | tar -xz -C /usr/local --strip-components=1
"gh_${GH_VERSION}_linux_${ARCH}/bin/gh"
+curl -fsSL -o /usr/local/bin/jq \
+
"https://github.com/jqlang/jq/releases/download/jq-${JQ_VERSION}/jq-linux-${ARCH}";
+chmod +x /usr/local/bin/jq
Review Comment:
These downloads are executed without any integrity verification. To reduce
supply-chain risk, download and verify a published checksum/signature (e.g.,
`sha256sum -c`) for both `gh` and `jq` before installing, and fail the build if
verification fails.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
