This is an automated email from the ASF dual-hosted git repository.
hello-stephen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new ccfd4378d42 [chore](docker)Support rootless Docker for third-party
environments (#70457)
ccfd4378d42 is described below
commit ccfd4378d424292a363a96142ad795b8c9ace21e
Author: zhangstar333 <[email protected]>
AuthorDate: Fri Nov 13 14:57:12 2026 +0800
[chore](docker)Support rootless Docker for third-party environments (#70457)
Allow non-root users to start third-party test environments with
rootless Docker.
Remove host privilege escalation and adapt file ownership, networking,
initialization, and health checks.
Linux now requires a rootless daemon.
---------
Co-authored-by: Dongyang Li <[email protected]>
---
docker/thirdparties/README.md | 73 ++++++++
docker/thirdparties/docker-command.sh | 92 ++++++++++
.../thirdparties/docker-compose/db2/db2.yaml.tpl | 5 +-
.../scripts/init-db2.sh} | 23 ++-
.../docker-compose/elasticsearch/es.yaml.tpl | 2 +
.../elasticsearch/scripts/es_init.sh | 3 +
docker/thirdparties/docker-compose/fluss/README.md | 7 +
.../docker-compose/fluss/fluss-network.sh | 50 ++++++
.../docker-compose/fluss/fluss.env.tpl | 2 +-
docker/thirdparties/docker-compose/hive/README.md | 4 +-
.../thirdparties/docker-compose/hive/README_ZH.md | 4 +-
.../hive/scripts/snapshot-hive-baseline.sh | 31 +++-
.../thirdparties/docker-compose/hudi/hudi.env.tpl | 2 +-
.../thirdparties/docker-compose/hudi/hudi.yaml.tpl | 24 ++-
.../iceberg-rest/docker-compose.yaml.tpl | 10 +-
.../iceberg-rest/hdfs-rootless.yaml.tpl | 44 +++++
.../docker-compose/kafka/kafka.yaml.tpl | 5 +-
.../docker-compose/kerberos/kerberos.yaml.tpl | 6 +
.../minio/minio-RELEASE.2024-11-07.yaml.tpl | 1 -
.../docker-compose/mysql/mysql-5.7.yaml.tpl | 5 +-
.../docker-compose/oracle/oracle-11.yaml.tpl | 9 +-
.../docker-compose/polaris/docker-compose.yaml.tpl | 5 +
.../docker-compose/polaris/init-catalog.sh | 2 +
.../docker-compose/polaris/polaris_settings.env | 8 +-
.../postgresql/postgresql-14.yaml.tpl | 5 +-
.../docker-compose/ranger/ranger_settings.env | 6 +-
.../docker-compose/sqlserver/sqlserver.yaml.tpl | 20 +--
docker/thirdparties/docker-health.sh | 4 +-
docker/thirdparties/run-thirdparties-docker.sh | 197 +++++++++------------
docker/thirdparties/test/docker-command-test.sh | 138 +++++++++++++++
docker/thirdparties/test/fluss-network-test.sh | 72 ++++++++
docker/thirdparties/test/launcher-stop-test.sh | 81 +++++++++
.../test/snapshot-hive-baseline-test.sh | 99 +++++++++++
33 files changed, 868 insertions(+), 171 deletions(-)
diff --git a/docker/thirdparties/README.md b/docker/thirdparties/README.md
new file mode 100644
index 00000000000..2e2ea8012d8
--- /dev/null
+++ b/docker/thirdparties/README.md
@@ -0,0 +1,73 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Thirdparty Docker environments
+
+Run a component with Bash 4 or newer and Docker Compose v2:
+
+```bash
+bash run-thirdparties-docker.sh -c iceberg
+bash run-thirdparties-docker.sh -c iceberg --stop
+```
+
+## Rootless Docker on Linux
+
+Install and start the current user's rootless Docker daemon first. Its setup
+requires subordinate UID/GID ranges, `newuidmap`/`newgidmap`, and enabled
+unprivileged user namespaces. Once configured, launching these environments
+does not require host `sudo` or membership in the host `docker` group.
+
+On Linux, the launcher requires a rootless daemon and rejects rootful daemons.
+It detects the user's `$XDG_RUNTIME_DIR/docker.sock` socket unless an endpoint
+or context was explicitly selected. Docker commands always run as the current
+user; there is no automatic elevation or `DOCKER_USE_SUDO` switch.
+The endpoint can also be selected explicitly:
+
+```bash
+export DOCKER_HOST="unix://${XDG_RUNTIME_DIR:-/run/user/$(id -u)}/docker.sock"
+docker info --format '{{json .SecurityOptions}}' # must include name=rootless
+bash run-thirdparties-docker.sh -c iceberg
+```
+
+On macOS, the launcher uses the current user's Docker Desktop endpoint.
+
+Keep the working directory writable by the current user. The launcher cleans
+fixture files owned by subordinate container UIDs from inside the same rootless
+user namespace, using `alpine:3.20`. It does not change host ownership to root.
+
+Hive, Kerberos and Iceberg REST's built-in HDFS require Docker Engine 29.5 or
+newer for rootless host networking. Hive defaults to
`HIVE_HOST_ALIAS=localhost`
+and uses loopback for its same-host services. On Linux, when `IP_HOST` is
loopback, Fluss
+detects a host IPv4 address; set `FLUSS_HOST_IP` explicitly when a different
+address is needed for both containers and host clients. Iceberg REST HDFS uses
+host ports 20020, 20870, 20864, 20866 and
+20867, so its DataNode block addresses are reachable by host clients.
+
+Kerberos client configuration and keytabs remain under
+`docker-compose/kerberos/two-kerberos-hives/`; use the printed `KRB5_CONFIG`
+path for host clients. The rootless launcher does not write `/etc/hosts`,
+`/etc/krb5.conf` or `/keytabs`. The DB2 image retains its container-level
+`privileged` setting within the rootless daemon's user namespace.
+
+`--reserve-ports` changes a host sysctl and is unavailable in rootless mode.
+It is optional; an operator can reserve ports separately when required.
+Image pulls and fixture downloads still require access to their registries
+and artifact repositories. `--no-load-data` skips Hive fixture refresh;
+combine it with `--hive-mode rebuild` to validate a fresh Hive installation
+without downloading a regression-data baseline.
diff --git a/docker/thirdparties/docker-command.sh
b/docker/thirdparties/docker-command.sh
new file mode 100755
index 00000000000..ffd4533cd3e
--- /dev/null
+++ b/docker/thirdparties/docker-command.sh
@@ -0,0 +1,92 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+configure_docker_access() {
+ local rootless_runtime_dir
+ local rootless_socket
+ local security_options
+
+ rootless_runtime_dir="${XDG_RUNTIME_DIR:-/run/user/$(id -u)}"
+ rootless_socket="${rootless_runtime_dir}/docker.sock"
+
+ # An explicit endpoint or context takes precedence over socket discovery.
+ if [[ "${HOST_OS}" != "Darwin" && -z "${DOCKER_HOST+x}" \
+ && -z "${DOCKER_CONTEXT:-}" && -S "${rootless_socket}" ]]; then
+ export DOCKER_HOST="unix://${rootless_socket}"
+ fi
+
+ if ! security_options="$(docker info --format '{{json
.SecurityOptions}}')"; then
+ echo "Unable to access Docker as the current user; start the user's
rootless daemon or select its endpoint" >&2
+ return 1
+ fi
+ # Docker Desktop on macOS runs its daemon in a VM. Linux requires the
+ # user's rootless daemon; an accessible system daemon is not accepted.
+ if [[ "${HOST_OS}" != "Darwin" && "${security_options}" !=
*'"name=rootless"'* ]]; then
+ echo "This launcher requires rootless Docker on Linux; the selected
daemon is rootful" >&2
+ return 1
+ fi
+}
+
+require_rootless_host_network() {
+ local server_version
+ local major
+ local minor
+
+ if [[ "${HOST_OS}" == "Darwin" ]]; then
+ return 0
+ fi
+ server_version="$(docker_cli info --format '{{.ServerVersion}}')" || return
+ major="${server_version%%.*}"
+ minor="${server_version#*.}"
+ minor="${minor%%.*}"
+ # Older rootless host networks cannot reach ports published on the real
+ # host, which these fixtures use for PostgreSQL and HDFS RPC.
+ if ((major < 29 || (major == 29 && minor < 5))); then
+ echo "Rootless host networking requires Docker Engine 29.5 or newer
(found ${server_version})" >&2
+ return 1
+ fi
+}
+
+rootless_fixture_cmd() {
+ local fixture_dir="$1"
+ shift
+
+ # Stay in the daemon's user namespace when images have chowned bind mounts
+ # to subordinate UIDs. Mount only the fixture directory being operated on.
+ docker_cli run --rm --network none --user 0:0 \
+ --security-opt no-new-privileges \
+ --mount "type=bind,source=${fixture_dir},target=/fixture" \
+ "${THIRDPARTY_FILESYSTEM_IMAGE:-alpine:3.20}" "$@"
+}
+
+reset_rootless_data_dir() {
+ local data_dir="$1"
+
+ mkdir -p "${data_dir}"
+ rootless_fixture_cmd "${data_dir}" sh -ec '
+ find /fixture -mindepth 1 -maxdepth 1 -exec rm -rf -- {} +
+ chown 0:0 /fixture
+ chmod 755 /fixture
+ '
+}
+
+chmod_fixture_dir() {
+ local mode="$1"
+ local fixture_dir="$2"
+ rootless_fixture_cmd "${fixture_dir}" chmod -R "${mode}" /fixture
+}
diff --git a/docker/thirdparties/docker-compose/db2/db2.yaml.tpl
b/docker/thirdparties/docker-compose/db2/db2.yaml.tpl
index 49a09656549..c8e944d52cf 100644
--- a/docker/thirdparties/docker-compose/db2/db2.yaml.tpl
+++ b/docker/thirdparties/docker-compose/db2/db2.yaml.tpl
@@ -24,13 +24,14 @@ services:
- ${DOCKER_DB2_EXTERNAL_PORT}:50000
privileged: true
healthcheck:
- test: ["CMD-SHELL", "su - db2inst1 -c \"source ~/.bash_profile; db2
connect to doris && db2 'select 1 from sysibm.sysdummy1'\""]
+ test: ["CMD-SHELL", "test -f /tmp/doris_db2_initialized && su - db2inst1
-c \"source ~/.bash_profile; db2 connect to doris && db2 'select count(*) from
doris_test.sample_table'\""]
interval: 20s
timeout: 60s
retries: 30
start_period: 300s
volumes:
- - ./init:/docker-entrypoint-initdb.d
+ - ./init:/docker-entrypoint-initdb.d:ro
+ - ./scripts/init-db2.sh:/var/custom/init-db2.sh:ro
environment:
- LICENSE=accept
- DBNAME=doris
diff --git a/docker/thirdparties/docker-compose/ranger/ranger_settings.env
b/docker/thirdparties/docker-compose/db2/scripts/init-db2.sh
old mode 100644
new mode 100755
similarity index 52%
copy from docker/thirdparties/docker-compose/ranger/ranger_settings.env
copy to docker/thirdparties/docker-compose/db2/scripts/init-db2.sh
index 2b2fe8b5a18..e98334d1e97
--- a/docker/thirdparties/docker-compose/ranger/ranger_settings.env
+++ b/docker/thirdparties/docker-compose/db2/scripts/init-db2.sh
@@ -16,6 +16,23 @@
# specific language governing permissions and limitations
# under the License.
-export RANGER_SOLR_PORT=8983
-export RANGER_PORT=6081
-export RANGER_MYSQL_PORT=33061
+set -euo pipefail
+rm -f /tmp/doris_db2_initialized
+
+# The IBM image executes /var/custom scripts after creating DBNAME. It does
+# not execute SQL files mounted at /docker-entrypoint-initdb.d by itself.
+su - "${DB2INSTANCE}" -c 'bash -se' <<'EOF'
+. "${HOME}/sqllib/db2profile"
+db2 connect to doris
+schema_count=$(db2 -x "SELECT COUNT(*) FROM SYSCAT.SCHEMATA WHERE
SCHEMANAME='DORIS_TEST'")
+if ((schema_count == 0)); then
+ db2 -stvf /docker-entrypoint-initdb.d/02-create-schema.sql
+ db2 -stvf /docker-entrypoint-initdb.d/03-create-table.sql
+ db2 -stvf /docker-entrypoint-initdb.d/04-insert.sql
+fi
+rows=$(db2 -x 'SELECT COUNT(*) FROM doris_test.sample_table')
+[[ "${rows}" -eq 3 ]]
+db2 connect reset
+EOF
+
+touch /tmp/doris_db2_initialized
diff --git a/docker/thirdparties/docker-compose/elasticsearch/es.yaml.tpl
b/docker/thirdparties/docker-compose/elasticsearch/es.yaml.tpl
index 83303c250a0..f838af7e829 100644
--- a/docker/thirdparties/docker-compose/elasticsearch/es.yaml.tpl
+++ b/docker/thirdparties/docker-compose/elasticsearch/es.yaml.tpl
@@ -127,6 +127,8 @@ services:
ES_8_HOST: "doris--es_8"
command: [ "sh", "-c", "/mnt/scripts/es_init.sh" ]
depends_on:
+ doris--es_5:
+ condition: service_healthy
doris--es_6:
condition: service_healthy
doris--es_7:
diff --git
a/docker/thirdparties/docker-compose/elasticsearch/scripts/es_init.sh
b/docker/thirdparties/docker-compose/elasticsearch/scripts/es_init.sh
index 51f81ced938..d8e31323b00 100755
--- a/docker/thirdparties/docker-compose/elasticsearch/scripts/es_init.sh
+++ b/docker/thirdparties/docker-compose/elasticsearch/scripts/es_init.sh
@@ -16,6 +16,8 @@
# specific language governing permissions and limitations
# under the License.
+set -eo pipefail
+
generate_bulk_request() {
local index_name=$1
local type_value=$2
@@ -234,3 +236,4 @@ generate_bulk_request "test_keyword_array" "" "item_"
"$keyword_array_data_file"
curl -X POST "http://${ES_8_HOST}:9200/_bulk"; --data-binary
"@$keyword_array_bulk_file" -H "Content-Type: application/json"
touch /tmp/SUCCESS
+exec tail -f /dev/null
diff --git a/docker/thirdparties/docker-compose/fluss/README.md
b/docker/thirdparties/docker-compose/fluss/README.md
index ae9ffbed067..95753e06f48 100644
--- a/docker/thirdparties/docker-compose/fluss/README.md
+++ b/docker/thirdparties/docker-compose/fluss/README.md
@@ -87,6 +87,13 @@ enableFlussTest=true
The servers advertise `<host ip>:<published port>`, because Doris FE/BE run on
the host rather than inside the compose network.
+On Linux, when `IP_HOST` is loopback, the launcher detects a host IPv4 address
for Fluss
+so the bridge containers can reach its published ports. Override it with
+`FLUSS_HOST_IP` when the host has multiple networks. This must be an address
+reachable from both Doris and the bridge containers; loopback and wildcard
+addresses are rejected. Other components can continue using
`IP_HOST=127.0.0.1`.
+On Docker Desktop, set `FLUSS_HOST_IP` explicitly to a mutually reachable
address.
+
The remote-data directory is bind mounted at the same absolute path inside the
containers and on the host because Doris reads those files directly and the
path
string is recorded rather than translated:
diff --git a/docker/thirdparties/docker-compose/fluss/fluss-network.sh
b/docker/thirdparties/docker-compose/fluss/fluss-network.sh
new file mode 100644
index 00000000000..31c6df44b61
--- /dev/null
+++ b/docker/thirdparties/docker-compose/fluss/fluss-network.sh
@@ -0,0 +1,50 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+resolve_fluss_host_ip() {
+ local address="${FLUSS_HOST_IP:-${IP_HOST:-}}"
+
+ # Unlike host-network Hive, the Fluss servers and Flink clients live in
+ # separate bridge containers. Their advertised endpoint cannot be loopback.
+ case "${address}" in
+ ""|localhost|localhost.|127.*|::1|"[::1]"|::|"[::]"|0.0.0.0)
+ if [[ -n "${FLUSS_HOST_IP:-}" ]]; then
+ echo "FLUSS_HOST_IP must be reachable from both the host and
bridge containers; loopback and wildcard addresses cannot be advertised" >&2
+ return 1
+ fi
+ address=""
+ if [[ "${HOST_OS}" != "Darwin" ]]; then
+ if command -v ip >/dev/null 2>&1; then
+ # Reading the route does not send traffic to this address.
+ address=$(ip -4 route get 1.1.1.1 2>/dev/null | awk '{for
(i=1; i<NF; i++) if ($i == "src") {print $(i+1); exit}}') || address=""
+ fi
+ if [[ -z "${address}" ]]; then
+ address=$(hostname -I 2>/dev/null | awk '{for (i=1; i<=NF;
i++) if ($i ~ /^[0-9]+\./ && $i !~ /^127\./) {print $i; exit}}') || address=""
+ fi
+ fi
+ ;;
+ esac
+
+ case "${address}" in
+ ""|localhost|localhost.|127.*|::1|"[::1]"|::|"[::]"|0.0.0.0)
+ echo "Cannot determine a Fluss host address; set FLUSS_HOST_IP to an
address reachable from both the host and bridge containers" >&2
+ return 1
+ ;;
+ esac
+ printf '%s\n' "${address}"
+}
diff --git a/docker/thirdparties/docker-compose/fluss/fluss.env.tpl
b/docker/thirdparties/docker-compose/fluss/fluss.env.tpl
index 85820859844..bca9dc85138 100644
--- a/docker/thirdparties/docker-compose/fluss/fluss.env.tpl
+++ b/docker/thirdparties/docker-compose/fluss/fluss.env.tpl
@@ -42,7 +42,7 @@ FLUSS_FLINK_IMAGE=apache/fluss-quickstart-flink:1.20-1.0.0
# Address the fluss servers advertise to clients. Doris FE/BE run on the host,
# so the servers must hand out the host address plus the published ports, not
# their in-container hostnames.
-FLUSS_HOST_IP=${IP_HOST}
+FLUSS_HOST_IP="${FLUSS_HOST_IP:-${IP_HOST}}"
# remote.data.dir holds remote log segments and kv snapshots. Doris BE reads
# those files directly (primary-key table reads), so the directory is bind
diff --git a/docker/thirdparties/docker-compose/hive/README.md
b/docker/thirdparties/docker-compose/hive/README.md
index 57c6d414b1d..65b8266217b 100644
--- a/docker/thirdparties/docker-compose/hive/README.md
+++ b/docker/thirdparties/docker-compose/hive/README.md
@@ -116,11 +116,11 @@ Relevant env vars:
After bootstrapping a clean Hive stack, stop the containers and run:
```bash
-sudo docker compose -p "${CONTAINER_UID}hive3" \
+docker compose -p "${CONTAINER_UID}hive3" \
-f docker/thirdparties/docker-compose/hive/hive-3x.yaml down
bash docker/thirdparties/docker-compose/hive/scripts/snapshot-hive-baseline.sh
\
- "${CONTAINER_UID}hive3" /tmp/hive3-baseline.tar.gz
+ "doris-shared-hive3" /tmp/hive3-baseline.tar.gz
```
Then upload the resulting tarball to OSS at
`oss://<s3BucketName>/regression/datalake/pipeline_data/hive_baseline/hive3-baseline-<version>.tar.gz`
(same convention for `hive2`).
diff --git a/docker/thirdparties/docker-compose/hive/README_ZH.md
b/docker/thirdparties/docker-compose/hive/README_ZH.md
index 8cc32162dc6..546eaf3b8e5 100644
--- a/docker/thirdparties/docker-compose/hive/README_ZH.md
+++ b/docker/thirdparties/docker-compose/hive/README_ZH.md
@@ -116,11 +116,11 @@ Hive 运行态（HDFS 数据、Postgres Metastore、模块 SHA 记录）存放
在一次完整 bootstrap 成功后，停止容器并运行：
```bash
-sudo docker compose -p "${CONTAINER_UID}hive3" \
+docker compose -p "${CONTAINER_UID}hive3" \
-f docker/thirdparties/docker-compose/hive/hive-3x.yaml down
bash docker/thirdparties/docker-compose/hive/scripts/snapshot-hive-baseline.sh
\
- "${CONTAINER_UID}hive3" /tmp/hive3-baseline.tar.gz
+ "doris-shared-hive3" /tmp/hive3-baseline.tar.gz
```
然后把得到的 tarball 上传到
`oss://<s3BucketName>/regression/datalake/pipeline_data/hive_baseline/hive3-baseline-<version>.tar.gz`（`hive2`
同理）。
diff --git
a/docker/thirdparties/docker-compose/hive/scripts/snapshot-hive-baseline.sh
b/docker/thirdparties/docker-compose/hive/scripts/snapshot-hive-baseline.sh
index edbe2788dcc..e8024398a2b 100755
--- a/docker/thirdparties/docker-compose/hive/scripts/snapshot-hive-baseline.sh
+++ b/docker/thirdparties/docker-compose/hive/scripts/snapshot-hive-baseline.sh
@@ -24,7 +24,7 @@ set -eo pipefail
# snapshot-hive-baseline.sh <volume_prefix> <output_path>
#
# Example:
-# bash snapshot-hive-baseline.sh doris--syt--hive3 /tmp/hive3-baseline.tar.gz
+# bash snapshot-hive-baseline.sh doris-shared-hive3
/tmp/hive3-baseline.tar.gz
#
# Prerequisites:
# - All Hive containers must be STOPPED before running this script.
@@ -33,35 +33,48 @@ set -eo pipefail
VOLUME_PREFIX="${1:?Usage: $0 <volume_prefix> <output_path>}"
OUTPUT_PATH="${2:?Usage: $0 <volume_prefix> <output_path>}"
+THIRDPARTY_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." &>/dev/null &&
pwd)"
+HOST_OS="$(uname -s)"
+# shellcheck source=../../../docker-command.sh
+. "${THIRDPARTY_ROOT}/docker-command.sh"
+configure_docker_access
+
+docker_cli() {
+ docker "$@"
+}
+
echo "[snapshot] volume prefix: ${VOLUME_PREFIX}"
echo "[snapshot] output: ${OUTPUT_PATH}"
# Verify all 4 volumes exist
for suffix in namenode datanode pgdata state; do
vol="${VOLUME_PREFIX}-${suffix}"
- if ! sudo docker volume inspect "${vol}" >/dev/null 2>&1; then
+ if ! docker_cli volume inspect "${vol}" >/dev/null 2>&1; then
echo "ERROR: volume ${vol} does not exist" >&2
exit 1
fi
done
_t0=$(date +%s)
+tmp_output="$(mktemp "${OUTPUT_PATH}.XXXXXX")"
+trap 'rm -f "${tmp_output}"' EXIT
# Mount all 4 volumes read-only into a single alpine container and tar them
# in one pass. Copy only the small state volume to ephemeral container storage
# so legacy baseline.version files can be dropped from newly exported tarballs.
-sudo docker run --rm \
- -v "${VOLUME_PREFIX}-namenode:/snapshot/namenode:ro" \
- -v "${VOLUME_PREFIX}-datanode:/snapshot/datanode:ro" \
- -v "${VOLUME_PREFIX}-pgdata:/snapshot/pgdata:ro" \
+docker_cli run --rm --network none --security-opt no-new-privileges \
+ -v "${VOLUME_PREFIX}-namenode:/work/namenode:ro" \
+ -v "${VOLUME_PREFIX}-datanode:/work/datanode:ro" \
+ -v "${VOLUME_PREFIX}-pgdata:/work/pgdata:ro" \
-v "${VOLUME_PREFIX}-state:/snapshot/state:ro" \
- alpine sh -c '
+ "${THIRDPARTY_FILESYSTEM_IMAGE:-alpine:3.20}" sh -ec '
mkdir -p /work/state
cp -a /snapshot/state/. /work/state/
rm -f /work/state/baseline.version
- tar czf - -C /snapshot namenode datanode pgdata -C /work state
+ tar czf - -C /work namenode datanode pgdata state
' \
- > "${OUTPUT_PATH}"
+ > "${tmp_output}"
+mv "${tmp_output}" "${OUTPUT_PATH}"
size=$(du -h "${OUTPUT_PATH}" | cut -f1)
echo "[snapshot] done took=$(( $(date +%s) - _t0 ))s size=${size}"
diff --git a/docker/thirdparties/docker-compose/hudi/hudi.env.tpl
b/docker/thirdparties/docker-compose/hudi/hudi.env.tpl
index e04012010e9..69eefee256c 100644
--- a/docker/thirdparties/docker-compose/hudi/hudi.env.tpl
+++ b/docker/thirdparties/docker-compose/hudi/hudi.env.tpl
@@ -15,7 +15,7 @@
# limitations under the License.
#
-CONTAINER_UID=doris--
+CONTAINER_UID=${CONTAINER_UID}
HUDI_NETWORK=${CONTAINER_UID}hudi-network
# Ports exposed to host
diff --git a/docker/thirdparties/docker-compose/hudi/hudi.yaml.tpl
b/docker/thirdparties/docker-compose/hudi/hudi.yaml.tpl
index 94cd3adb3e0..554e77f033d 100644
--- a/docker/thirdparties/docker-compose/hudi/hudi.yaml.tpl
+++ b/docker/thirdparties/docker-compose/hudi/hudi.yaml.tpl
@@ -27,13 +27,18 @@ services:
${CONTAINER_UID}hudi-minio:
image: doristhirdpartydocker/minio:RELEASE.2025-01-20T14-49-07Z
container_name: ${CONTAINER_UID}hudi-minio
- command: server /data --console-address ":${MINIO_CONSOLE_PORT}"
+ command: server /data --console-address ":9001"
environment:
MINIO_ROOT_USER: ${MINIO_ROOT_USER}
MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD}
ports:
- "${MINIO_API_PORT}:9000"
- "${MINIO_CONSOLE_PORT}:9001"
+ healthcheck:
+ test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/ready";]
+ interval: 5s
+ timeout: 10s
+ retries: 60
networks:
- ${HUDI_NETWORK}
@@ -43,13 +48,13 @@ services:
entrypoint: |
/bin/bash -c "
set -euo pipefail
- sleep 5
mc alias set myminio http://${CONTAINER_UID}hudi-minio:9000
${MINIO_ROOT_USER} ${MINIO_ROOT_PASSWORD}
- mc mb --quiet myminio/${HUDI_BUCKET} || true
- mc mb --quiet myminio/${HUDI_BUCKET}-tmp || true
+ mc mb --quiet --ignore-existing myminio/${HUDI_BUCKET}
+ mc mb --quiet --ignore-existing myminio/${HUDI_BUCKET}-tmp
"
depends_on:
- - ${CONTAINER_UID}hudi-minio
+ ${CONTAINER_UID}hudi-minio:
+ condition: service_healthy
networks:
- ${HUDI_NETWORK}
@@ -123,9 +128,12 @@ services:
- ./scripts:/opt/hudi-scripts
- ./cache:/opt/hudi-cache
depends_on:
- - ${CONTAINER_UID}hudi-minio
- - ${CONTAINER_UID}hudi-minio-mc
- - ${CONTAINER_UID}hudi-metastore
+ ${CONTAINER_UID}hudi-minio:
+ condition: service_started
+ ${CONTAINER_UID}hudi-minio-mc:
+ condition: service_completed_successfully
+ ${CONTAINER_UID}hudi-metastore:
+ condition: service_started
command: ["/opt/hudi-scripts/init.sh"]
ports:
- "${SPARK_UI_PORT}:8080"
diff --git
a/docker/thirdparties/docker-compose/iceberg-rest/docker-compose.yaml.tpl
b/docker/thirdparties/docker-compose/iceberg-rest/docker-compose.yaml.tpl
index 3f99dd90dc2..5ca7664f1ed 100644
--- a/docker/thirdparties/docker-compose/iceberg-rest/docker-compose.yaml.tpl
+++ b/docker/thirdparties/docker-compose/iceberg-rest/docker-compose.yaml.tpl
@@ -171,8 +171,8 @@ services:
-
CORE_CONF_io_compression_codecs=org.apache.hadoop.io.compress.SnappyCodec
- HDFS_CONF_dfs_webhdfs_enabled=true
- HDFS_CONF_dfs_permissions_enabled=false
- - HDFS_CONF_dfs_namenode_rpc_address=hdfs-namenode:8020
- - HDFS_CONF_dfs_namenode_http_address=0.0.0.0:9870
+ - HDFS_CONF_dfs_namenode_rpc___address=hdfs-namenode:8020
+ - HDFS_CONF_dfs_namenode_http___address=0.0.0.0:9870
- HDFS_CONF_dfs_replication=1
- HDFS_CONF_dfs_namenode_name_dir=/hadoop/dfs/name
-
HDFS_CONF_dfs_namenode_datanode_registration_ip___hostname___check=false
@@ -228,6 +228,12 @@ services:
depends_on:
- hdfs-namenode
- hdfs-datanode
+ healthcheck:
+ # This image has bash but does not include curl or wget.
+ test: ["CMD", "bash", "-ec", "exec 3<>/dev/tcp/127.0.0.1/8181; printf
'GET /v1/config HTTP/1.0\\r\\nHost: localhost\\r\\n\\r\\n' >&3; head -n 1 <&3 |
grep -q '^HTTP/1.[01] 200 '"]
+ interval: 10s
+ timeout: 5s
+ retries: 5
networks:
${CONTAINER_UID}iceberg-rest:
diff --git
a/docker/thirdparties/docker-compose/iceberg-rest/hdfs-rootless.yaml.tpl
b/docker/thirdparties/docker-compose/iceberg-rest/hdfs-rootless.yaml.tpl
new file mode 100644
index 00000000000..2c85f826c35
--- /dev/null
+++ b/docker/thirdparties/docker-compose/iceberg-rest/hdfs-rootless.yaml.tpl
@@ -0,0 +1,44 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Use distinct real host ports so HDFS block locations work for host clients
+# as well as the REST catalog containers. Requires rootless Docker >= 29.5.
+services:
+ hdfs-namenode:
+ network_mode: host
+ networks: !reset []
+ ports: !reset []
+ environment:
+ - CORE_CONF_fs_defaultFS=hdfs://${LOCAL_IP}:${HDFS_NAMENODE_RPC_PORT}
+ -
HDFS_CONF_dfs_namenode_rpc___address=${LOCAL_IP}:${HDFS_NAMENODE_RPC_PORT}
+ -
HDFS_CONF_dfs_namenode_http___address=0.0.0.0:${HDFS_NAMENODE_WEBUI_PORT}
+ healthcheck:
+ test: ["CMD", "curl", "-f",
"http://127.0.0.1:${HDFS_NAMENODE_WEBUI_PORT}/";]
+
+ hdfs-datanode:
+ network_mode: host
+ networks: !reset []
+ ports: !reset []
+ environment:
+ - CORE_CONF_fs_defaultFS=hdfs://${LOCAL_IP}:${HDFS_NAMENODE_RPC_PORT}
+ - HDFS_CONF_dfs_datanode_hostname=${LOCAL_IP}
+ - HDFS_CONF_dfs_datanode_address=0.0.0.0:20866
+ - HDFS_CONF_dfs_datanode_ipc_address=0.0.0.0:20867
+ - HDFS_CONF_dfs_datanode_http_address=0.0.0.0:${HDFS_DATANODE_WEBUI_PORT}
+ - SERVICE_PRECONDITION=${LOCAL_IP}:${HDFS_NAMENODE_WEBUI_PORT}
+ healthcheck:
+ test: ["CMD", "curl", "-f",
"http://127.0.0.1:${HDFS_DATANODE_WEBUI_PORT}/";]
diff --git a/docker/thirdparties/docker-compose/kafka/kafka.yaml.tpl
b/docker/thirdparties/docker-compose/kafka/kafka.yaml.tpl
index e3322ce2ff6..777f744b8b6 100644
--- a/docker/thirdparties/docker-compose/kafka/kafka.yaml.tpl
+++ b/docker/thirdparties/docker-compose/kafka/kafka.yaml.tpl
@@ -31,8 +31,7 @@ services:
ports:
- ${DOCKER_ZOOKEEPER_EXTERNAL_PORT}:2181
healthcheck:
- #
https://github.com/bitnami/charts/blob/62399ed3863e70775d66f9581e28129026a86e5d/bitnami/zookeeper/templates/statefulset.yaml#L404
- test: /bin/bash -ec ZOO_HC_TIMEOUT=10
/opt/bitnami/scripts/zookeeper/healthcheck.sh
+ test: ["CMD", "/opt/bitnami/zookeeper/bin/zkServer.sh", "status"]
start_period: 10s
interval: 5s
timeout: 60s
@@ -63,7 +62,5 @@ services:
- KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:19193
- KAFKA_ZOOKEEPER_CONNECT=doris--zookeeper:2181
- ALLOW_PLAINTEXT_LISTENER=yes
- volumes:
- - /var/run/docker.sock:/var/run/docker.sock
networks:
- doris--kafka--network
diff --git a/docker/thirdparties/docker-compose/kerberos/kerberos.yaml.tpl
b/docker/thirdparties/docker-compose/kerberos/kerberos.yaml.tpl
index cf1dc94867c..2a749642d32 100644
--- a/docker/thirdparties/docker-compose/kerberos/kerberos.yaml.tpl
+++ b/docker/thirdparties/docker-compose/kerberos/kerberos.yaml.tpl
@@ -24,6 +24,12 @@ x-kerberos-image: &kerberos-image
x-kerberos-service: &kerberos-service
<<: *kerberos-image
network_mode: "host"
+ extra_hosts:
+ - "hadoop-master:${IP_HOST}"
+ - "hadoop-master-2:${IP_HOST}"
+ # Hadoop's Kerberos resolver performs absolute (trailing-dot) lookups.
+ - "hadoop-master.:${IP_HOST}"
+ - "hadoop-master-2.:${IP_HOST}"
mem_limit: 1536m
stop_grace_period: 30s
diff --git
a/docker/thirdparties/docker-compose/minio/minio-RELEASE.2024-11-07.yaml.tpl
b/docker/thirdparties/docker-compose/minio/minio-RELEASE.2024-11-07.yaml.tpl
index 81ddd8ba027..6dfad86b97a 100644
--- a/docker/thirdparties/docker-compose/minio/minio-RELEASE.2024-11-07.yaml.tpl
+++ b/docker/thirdparties/docker-compose/minio/minio-RELEASE.2024-11-07.yaml.tpl
@@ -23,7 +23,6 @@ services:
restart: always
ports:
- ${DOCKER_MINIO_EXTERNAL_PORT}:9000
- privileged: true
healthcheck:
test: [ "CMD", "bash", "-c", "mc ping --count 1 local" ]
interval: 20s
diff --git a/docker/thirdparties/docker-compose/mysql/mysql-5.7.yaml.tpl
b/docker/thirdparties/docker-compose/mysql/mysql-5.7.yaml.tpl
index c172bdf28c4..1b689454bad 100644
--- a/docker/thirdparties/docker-compose/mysql/mysql-5.7.yaml.tpl
+++ b/docker/thirdparties/docker-compose/mysql/mysql-5.7.yaml.tpl
@@ -36,6 +36,9 @@ services:
- bash
- -c
- |
+ set -e
+ mkdir -p /etc/mysql/certs
+ cp /etc/mysql/certs-source/* /etc/mysql/certs/
chown mysql:mysql /etc/mysql/certs/*
chmod 600 /etc/mysql/certs/server.key
chmod 644 /etc/mysql/certs/server.crt /etc/mysql/certs/root.crt
@@ -56,7 +59,7 @@ services:
- ./data/:/var/lib/mysql
- ./init:/docker-entrypoint-initdb.d
- ./my.cnf:/etc/mysql/conf.d/my.cnf
- - ./certs:/etc/mysql/certs
+ - ./certs:/etc/mysql/certs-source:ro
networks:
- doris--mysql_57
diff --git a/docker/thirdparties/docker-compose/oracle/oracle-11.yaml.tpl
b/docker/thirdparties/docker-compose/oracle/oracle-11.yaml.tpl
index 09314c9dc23..087833bb649 100644
--- a/docker/thirdparties/docker-compose/oracle/oracle-11.yaml.tpl
+++ b/docker/thirdparties/docker-compose/oracle/oracle-11.yaml.tpl
@@ -23,9 +23,14 @@ services:
restart: always
ports:
- ${DOCKER_ORACLE_EXTERNAL_PORT}:1521
- privileged: true
healthcheck:
- test: [ "CMD", "bash", "-c", "echo 'SELECT 1 FROM doris_test.deadline;'
| ORACLE_HOME=/u01/app/oracle/product/11.2.0/xe
/u01/app/oracle/product/11.2.0/xe/bin/sqlplus -s DORIS_TEST/123456@localhost" ]
+ test:
+ - CMD-SHELL
+ - >-
+ printf '%s\n' 'WHENEVER SQLERROR EXIT FAILURE' 'WHENEVER OSERROR
EXIT FAILURE'
+ 'SELECT COUNT(*) FROM doris_test.deadline;' 'EXIT;'
+ | ORACLE_HOME=/u01/app/oracle/product/11.2.0/xe
+ /u01/app/oracle/product/11.2.0/xe/bin/sqlplus -L -s
DORIS_TEST/123456@localhost
interval: 20s
timeout: 60s
retries: 120
diff --git a/docker/thirdparties/docker-compose/polaris/docker-compose.yaml.tpl
b/docker/thirdparties/docker-compose/polaris/docker-compose.yaml.tpl
index 98124890872..78c941cf03b 100644
--- a/docker/thirdparties/docker-compose/polaris/docker-compose.yaml.tpl
+++ b/docker/thirdparties/docker-compose/polaris/docker-compose.yaml.tpl
@@ -113,6 +113,11 @@ services:
volumes:
- ./init-catalog.sh:/init-catalog.sh:ro
entrypoint: ["/bin/sh","-c","/init-catalog.sh"]
+ healthcheck:
+ test: ["CMD-SHELL", "test -f /tmp/polaris_initialized"]
+ interval: 5s
+ timeout: 3s
+ retries: 60
networks:
- ${CONTAINER_UID}polaris
diff --git a/docker/thirdparties/docker-compose/polaris/init-catalog.sh
b/docker/thirdparties/docker-compose/polaris/init-catalog.sh
index 124e41845be..6e0b5a9fb0b 100755
--- a/docker/thirdparties/docker-compose/polaris/init-catalog.sh
+++ b/docker/thirdparties/docker-compose/polaris/init-catalog.sh
@@ -142,4 +142,6 @@ fi
echo "[polaris-init] Permissions setup completed."
echo "[polaris-init] Done."
+touch /tmp/polaris_initialized
+exec tail -f /dev/null
diff --git a/docker/thirdparties/docker-compose/polaris/polaris_settings.env
b/docker/thirdparties/docker-compose/polaris/polaris_settings.env
index 70ba1a96b93..2242db926d4 100644
--- a/docker/thirdparties/docker-compose/polaris/polaris_settings.env
+++ b/docker/thirdparties/docker-compose/polaris/polaris_settings.env
@@ -17,15 +17,15 @@
# under the License.
# Polaris uses 20181-20182 for S3
-export POLARIS_S3_PORT=20181
-export POLARIS_S3_ADMIN_PORT=20182
+export POLARIS_S3_PORT="${POLARIS_S3_PORT:-20181}"
+export POLARIS_S3_ADMIN_PORT="${POLARIS_S3_ADMIN_PORT:-20182}"
# Polaris authentication
export POLARIS_BOOTSTRAP_PASSWORD="secret123"
# MinIO (S3-compatible) configuration for local development
-export MINIO_API_PORT=20001
-export MINIO_CONSOLE_PORT=20002
+export MINIO_API_PORT="${POLARIS_MINIO_API_PORT:-20001}"
+export MINIO_CONSOLE_PORT="${POLARIS_MINIO_CONSOLE_PORT:-20002}"
export MINIO_ACCESS_KEY="admin"
export MINIO_SECRET_KEY="password"
export MINIO_BUCKET="warehouse"
diff --git
a/docker/thirdparties/docker-compose/postgresql/postgresql-14.yaml.tpl
b/docker/thirdparties/docker-compose/postgresql/postgresql-14.yaml.tpl
index 54758285290..a8d1233c1f1 100644
--- a/docker/thirdparties/docker-compose/postgresql/postgresql-14.yaml.tpl
+++ b/docker/thirdparties/docker-compose/postgresql/postgresql-14.yaml.tpl
@@ -29,6 +29,9 @@ services:
- bash
- -c
- |
+ set -e
+ mkdir -p /var/lib/postgresql/certs
+ cp /var/lib/postgresql/certs-source/* /var/lib/postgresql/certs/
chmod 600 /var/lib/postgresql/certs/server.key
chmod 644 /var/lib/postgresql/certs/server.crt
chmod 644 /var/lib/postgresql/certs/root.crt
@@ -59,7 +62,7 @@ services:
volumes:
- ./data/data/:/var/lib/postgresql/data/
- ./init:/docker-entrypoint-initdb.d
- - ./certs:/var/lib/postgresql/certs
+ - ./certs:/var/lib/postgresql/certs-source:ro
networks:
- doris--postgres
diff --git a/docker/thirdparties/docker-compose/ranger/ranger_settings.env
b/docker/thirdparties/docker-compose/ranger/ranger_settings.env
index 2b2fe8b5a18..08c11de6495 100644
--- a/docker/thirdparties/docker-compose/ranger/ranger_settings.env
+++ b/docker/thirdparties/docker-compose/ranger/ranger_settings.env
@@ -16,6 +16,6 @@
# specific language governing permissions and limitations
# under the License.
-export RANGER_SOLR_PORT=8983
-export RANGER_PORT=6081
-export RANGER_MYSQL_PORT=33061
+export RANGER_SOLR_PORT="${RANGER_SOLR_PORT:-8983}"
+export RANGER_PORT="${RANGER_PORT:-6081}"
+export RANGER_MYSQL_PORT="${RANGER_MYSQL_PORT:-33061}"
diff --git a/docker/thirdparties/docker-compose/sqlserver/sqlserver.yaml.tpl
b/docker/thirdparties/docker-compose/sqlserver/sqlserver.yaml.tpl
index c2195d2f022..b14707155de 100644
--- a/docker/thirdparties/docker-compose/sqlserver/sqlserver.yaml.tpl
+++ b/docker/thirdparties/docker-compose/sqlserver/sqlserver.yaml.tpl
@@ -23,7 +23,7 @@ services:
ports:
- ${DOCKER_SQLSERVER_EXTERNAL_PORT}:1433
healthcheck:
- test: ["CMD", "/opt/mssql-tools/bin/sqlcmd", "-Usa", "-PDoris123456",
"-Q", "select 1"]
+ test: ["CMD-SHELL", "test -f /tmp/doris_sqlserver_initialized &&
/opt/mssql-tools/bin/sqlcmd -Usa -PDoris123456 -b -Q 'SELECT 1 FROM
doris_test.dbo.student'"]
interval: 5s
timeout: 30s
retries: 120
@@ -33,26 +33,26 @@ services:
- /bin/bash
- -c
- |
+ set -e
+ rm -f /tmp/doris_sqlserver_initialized
# Launch MSSQL and send to background
/opt/mssql/bin/sqlservr &
+ sqlserver_pid=$$!
+ trap 'kill -TERM "$$sqlserver_pid"; wait "$$sqlserver_pid"' TERM INT
# Wait for it to be available
echo "Waiting for MS SQL to be available ⏳"
- /opt/mssql-tools/bin/sqlcmd -l 30 -S localhost -h-1 -V1 -U sa -P
Doris123456 -Q "SET NOCOUNT ON SELECT \"YAY WE ARE UP\" , @@servername"
- is_up=$$?
- while [ $$is_up -ne 0 ] ; do
- echo -e $$(date)
- /opt/mssql-tools/bin/sqlcmd -l 30 -S localhost -h-1 -V1 -U sa -P
Doris123456 -Q "SET NOCOUNT ON SELECT \"YAY WE ARE UP\" , @@servername"
- is_up=$$?
+ until /opt/mssql-tools/bin/sqlcmd -l 30 -S localhost -h-1 -b -U sa -P
Doris123456 -Q "SET NOCOUNT ON SELECT \"YAY WE ARE UP\" , @@servername"; do
+ kill -0 "$$sqlserver_pid"
sleep 5
done
# Run every script in /scripts
# TODO set a flag so that this is only done once on creation,
# and not every time the container runs
for foo in /docker-entrypoint-initdb.d/*.sql
- do /opt/mssql-tools/bin/sqlcmd -U sa -P Doris123456 -l 30 -e -i $$foo
+ do /opt/mssql-tools/bin/sqlcmd -U sa -P Doris123456 -b -l 30 -e -i
"$$foo"
done
- # So that the container doesn't shut down, sleep this thread
- sleep infinity
+ touch /tmp/doris_sqlserver_initialized
+ wait "$$sqlserver_pid"
restart: always
environment:
diff --git a/docker/thirdparties/docker-health.sh
b/docker/thirdparties/docker-health.sh
index 9148465a54d..7eca1e6f30a 100644
--- a/docker/thirdparties/docker-health.sh
+++ b/docker/thirdparties/docker-health.sh
@@ -18,12 +18,12 @@
docker_container_state_cmd() {
local container_name="$1"
- sudo docker inspect --format '{{.State.Status}}|{{if
.State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' "${container_name}"
+ docker_cli inspect --format '{{.State.Status}}|{{if
.State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' "${container_name}"
}
docker_container_env_cmd() {
local container_name="$1"
- sudo docker inspect --format '{{range .Config.Env}}{{println .}}{{end}}'
"${container_name}"
+ docker_cli inspect --format '{{range .Config.Env}}{{println .}}{{end}}'
"${container_name}"
}
docker_container_ready() {
diff --git a/docker/thirdparties/run-thirdparties-docker.sh
b/docker/thirdparties/run-thirdparties-docker.sh
index af59bbc54e8..f8227b7b487 100755
--- a/docker/thirdparties/run-thirdparties-docker.sh
+++ b/docker/thirdparties/run-thirdparties-docker.sh
@@ -31,28 +31,10 @@ fi
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" &>/dev/null && pwd)"
HOST_OS="$(uname -s)"
-if [[ -z "${DOCKER_USE_SUDO+x}" ]]; then
- if [[ "${HOST_OS}" == "Darwin" ]]; then
- DOCKER_USE_SUDO=0
- else
- DOCKER_USE_SUDO=1
- fi
-fi
+. "${ROOT}/docker-command.sh"
docker_cli() {
- if [[ "${DOCKER_USE_SUDO}" -eq 1 ]]; then
- sudo docker "$@"
- else
- docker "$@"
- fi
-}
-
-host_admin_cmd() {
- if [[ "${HOST_OS}" == "Darwin" ]]; then
- "$@"
- else
- sudo "$@"
- fi
+ docker "$@"
}
. "${ROOT}/custom_settings.env"
@@ -69,7 +51,7 @@ Usage: $0 <options>
-c mysql start MySQL
-c mysql,hive3 start MySQL and Hive3
--stop stop the specified components
- --reserve-ports reserve host ports by setting
'net.ipv4.ip_local_reserved_ports' to avoid port already bind error
+ --reserve-ports unsupported; configure host port reservations
separately
--no-load-data do not load data into the components
--load-parallel <num> set the parallel number to load data, default is
the 50% of CPU cores
--hive-mode <mode> hive startup mode: fast, refresh, rebuild
@@ -81,7 +63,7 @@ Usage: $0 <options>
exit 1
}
DEFAULT_COMPONENTS="mysql,es,hive2,hive3,pg,oracle,sqlserver,clickhouse,mariadb,iceberg,hudi,db2,oceanbase,kerberos,minio"
-ALL_COMPONENTS="${DEFAULT_COMPONENTS},kafka,ranger,polaris,fluss,mongodb"
+ALL_COMPONENTS="${DEFAULT_COMPONENTS},iceberg-rest,kafka,ranger,polaris,fluss,mongodb"
COMPONENTS=$2
HELP=0
STOP=0
@@ -191,8 +173,10 @@ if [[ "${CONTAINER_UID}"x == "doris--"x ]]; then
exit 1
fi
+configure_docker_access
+
if [[ -z "${HIVE_HOST_ALIAS:-}" ]]; then
- export HIVE_HOST_ALIAS="hadoop-master-${HIVE_SHARED_ID}"
+ export HIVE_HOST_ALIAS=localhost
fi
echo "Components are: ${COMPONENTS}"
@@ -353,10 +337,8 @@ reserve_ports() {
return
fi
- if [[ "${RESERVED_PORTS}"x != ""x ]]; then
- echo "Reserve ports: ${RESERVED_PORTS}"
- sudo sysctl -w net.ipv4.ip_local_reserved_ports="${RESERVED_PORTS}"
- fi
+ echo "--reserve-ports changes host sysctl settings; configure it
separately as an operator" >&2
+ return 1
}
JFS_META_FORMATTED=0
@@ -581,8 +563,8 @@ prepare_juicefs_meta_for_hive() {
fi
local bucket_dir="${JFS_BUCKET_DIR:-/tmp/jfs-bucket}"
- sudo mkdir -p "${bucket_dir}"
- sudo chmod 777 "${bucket_dir}"
+ mkdir -p "${bucket_dir}"
+ chmod_fixture_dir 777 "${bucket_dir}"
# For local docker metadata DSNs (mysql/postgresql), ensure metadata
database exists.
ensure_juicefs_meta_database "${jfs_meta}"
@@ -598,7 +580,8 @@ prepare_juicefs_meta_for_hive() {
# and will cause "juicefs format" to fail with "Storage ... is not empty".
if [[ -d "${bucket_dir}" ]]; then
echo "Cleaning stale JuiceFS bucket directory: ${bucket_dir}"
- sudo rm -rf "${bucket_dir:?}"/*
+ reset_data_dirs "${bucket_dir:?}"
+ chmod_fixture_dir 777 "${bucket_dir}"
fi
if ! run_juicefs_cli \
@@ -653,8 +636,7 @@ reset_data_dirs() {
local data_dir
for data_dir in "$@"; do
- host_admin_cmd mkdir -p "${data_dir}"
- host_admin_cmd rm -rf "${data_dir:?}"/*
+ reset_rootless_data_dir "${data_dir}"
done
}
@@ -942,22 +924,16 @@ start_es() {
register_stack_metadata "es"
"${ROOT}/docker-compose/elasticsearch/es.yaml"
"${ROOT}/docker-compose/elasticsearch/es.env"
compose_down_stack "${ROOT}/docker-compose/elasticsearch/es.yaml"
"${ROOT}/docker-compose/elasticsearch/es.env" --remove-orphans
if [[ "${STOP}" -ne 1 ]]; then
- sudo mkdir -p "${ROOT}"/docker-compose/elasticsearch/data/es6/
- sudo rm -rf "${ROOT}"/docker-compose/elasticsearch/data/es6/*
- sudo mkdir -p "${ROOT}"/docker-compose/elasticsearch/data/es7/
- sudo rm -rf "${ROOT}"/docker-compose/elasticsearch/data/es7/*
- sudo mkdir -p "${ROOT}"/docker-compose/elasticsearch/data/es8/
- sudo rm -rf "${ROOT}"/docker-compose/elasticsearch/data/es8/*
- sudo chmod -R 777 "${ROOT}"/docker-compose/elasticsearch/data
- sudo mkdir -p "${ROOT}"/docker-compose/elasticsearch/logs/es6/
- sudo rm -rf "${ROOT}"/docker-compose/elasticsearch/logs/es6/*
- sudo mkdir -p "${ROOT}"/docker-compose/elasticsearch/logs/es7/
- sudo rm -rf "${ROOT}"/docker-compose/elasticsearch/logs/es7/*
- sudo mkdir -p "${ROOT}"/docker-compose/elasticsearch/logs/es8/
- sudo rm -rf "${ROOT}"/docker-compose/elasticsearch/logs/es8/*
- sudo chmod -R 777 "${ROOT}"/docker-compose/elasticsearch/logs
- sudo chmod -R 777 "${ROOT}"/docker-compose/elasticsearch/config
- compose_cmd "${ROOT}/docker-compose/elasticsearch/es.yaml"
"${ROOT}/docker-compose/elasticsearch/es.env" up -d --remove-orphans
+ local version
+ for version in 5 6 7 8; do
+ reset_data_dirs \
+ "${ROOT}/docker-compose/elasticsearch/data/es${version}" \
+ "${ROOT}/docker-compose/elasticsearch/logs/es${version}"
+ done
+ chmod_fixture_dir 777 "${ROOT}/docker-compose/elasticsearch/data"
+ chmod_fixture_dir 777 "${ROOT}/docker-compose/elasticsearch/logs"
+ chmod_fixture_dir 777 "${ROOT}/docker-compose/elasticsearch/config"
+ compose_cmd "${ROOT}/docker-compose/elasticsearch/es.yaml"
"${ROOT}/docker-compose/elasticsearch/es.env" up -d --wait --remove-orphans
fi
}
@@ -972,7 +948,7 @@ start_mysql() {
}
start_mongodb() {
- # sudo may clear the caller's environment; pass the selected port through
the rendered env file.
+ # Pass the selected port through the rendered env file.
export MONGODB_PORT="${MONGODB_PORT:-27017}"
envsubst '${MONGODB_PORT}'
<"${ROOT}/docker-compose/mongodb/mongodb.env.tpl" \
>"${ROOT}/docker-compose/mongodb/mongodb.env"
@@ -1081,7 +1057,7 @@ start_kafka() {
}
if [[ "${STOP}" -ne 1 ]]; then
- compose_up_stack "${ROOT}/docker-compose/kafka/kafka.yaml"
"${ROOT}/docker-compose/kafka/kafka.env" --build --remove-orphans -d
+ compose_up_stack "${ROOT}/docker-compose/kafka/kafka.yaml"
"${ROOT}/docker-compose/kafka/kafka.env" --build --remove-orphans -d --wait
wait_for_kafka_ready "${KAFKA_CONTAINER_ID}" "${IP_HOST}"
create_kafka_topics "${KAFKA_CONTAINER_ID}" "${IP_HOST}"
fi
@@ -1272,41 +1248,14 @@ hive_metastore_container_for() {
ensure_hosts_alias() {
local alias_name="$1"
local alias_ip="$2"
- local tmp_hosts
- local sudo_cmd=()
-
- tmp_hosts="$(mktemp)"
- awk -v alias_name="${alias_name}" '
- {
- keep = 1
- for (i = 2; i <= NF; ++i) {
- if ($i == alias_name) {
- keep = 0
- break
- }
- }
- if (keep) {
- print
- }
- }
- ' /etc/hosts >"${tmp_hosts}"
- printf "%s %s\n" "${alias_ip}" "${alias_name}" >>"${tmp_hosts}"
-
- # Writing /etc/hosts is the only privileged step in the whole script, so
- # take it only when the alias is actually missing or points somewhere else.
- # On macOS sudo has no passwordless default and no tty to prompt on, which
- # would otherwise abort every re-run of an already-correct environment.
- if cmp -s "${tmp_hosts}" /etc/hosts; then
- rm -f "${tmp_hosts}"
+ if [[ "${alias_name}" == "localhost" && "${alias_ip}" == "127.0.0.1" ]];
then
return 0
fi
-
- if [[ "$(id -u)" -ne 0 ]]; then
- sudo_cmd=(sudo)
+ if getent ahostsv4 "${alias_name}" | awk -v ip="${alias_ip}" '$1 == ip {
found = 1 } END { exit !found }'; then
+ return 0
fi
- "${sudo_cmd[@]}" chmod a+w /etc/hosts
- "${sudo_cmd[@]}" cp "${tmp_hosts}" /etc/hosts
- rm -f "${tmp_hosts}"
+ echo "${alias_name} must already resolve to ${alias_ip}; set
HIVE_HOST_ALIAS=localhost and IP_HOST=127.0.0.1 for local fixtures" >&2
+ return 1
}
render_hive_compose() {
@@ -1385,6 +1334,12 @@ start_hive_stack() {
local volume_prefix
local baseline_restore_result="missing"
+ # localhost metadata and host-network services must use loopback on both
+ # sides, even when other components advertise the host's LAN address.
+ if [[ "${HIVE_HOST_ALIAS}" == "localhost" ]]; then
+ export IP_HOST=127.0.0.1
+ fi
+
export HIVE_BOOTSTRAP_GROUPS="$(hive_bootstrap_groups_for
"${hive_version}")"
echo "${hive_version} selected bootstrap files: ${HIVE_BOOTSTRAP_GROUPS}"
@@ -1393,16 +1348,17 @@ start_hive_stack() {
export HIVE_VOLUME_PREFIX="${volume_prefix}"
log_hive_volumes "${hive_version}" "${volume_prefix}"
- # Keep a stable hostname in metastore/HDFS metadata while allowing the
- # backing host IP to change across restarts.
- ensure_hosts_alias "${HIVE_HOST_ALIAS}" "${IP_HOST}"
-
if [[ "${STOP}" -eq 1 ]]; then
render_hive_compose "${hive_version}"
hive_compose_cmd "${hive_version}" down
return 0
fi
+ # Keep a stable hostname in metastore/HDFS metadata while allowing the
+ # backing host IP to change across restarts. Stopping needs no DNS setup.
+ require_rootless_host_network
+ ensure_hosts_alias "${HIVE_HOST_ALIAS}" "${IP_HOST}"
+
# refresh/rebuild: tear down the stack and clear volumes first.
# fast: keep existing volumes and only restore the baseline when they are
empty.
if [[ "${HIVE_MODE}" == "rebuild" || "${HIVE_MODE}" == "refresh" ]]; then
@@ -1458,7 +1414,8 @@ start_iceberg() {
for jar in $(grep -o '\./data/input/jars/[^:]*\.jar'
"${ICEBERG_DIR}/iceberg.yaml"); do
if [[ ! -f "${ICEBERG_DIR}/${jar#./}" ]]; then
echo "${ICEBERG_DIR}/${jar#./} is missing,
${ICEBERG_DIR}/data is from an older archive"
- host_admin_cmd rm -rf "${ICEBERG_DIR:?}/data"
+ reset_data_dirs "${ICEBERG_DIR:?}/data"
+ rmdir "${ICEBERG_DIR}/data"
break
fi
done
@@ -1475,9 +1432,9 @@ start_iceberg() {
wget -P "${ROOT}/docker-compose/iceberg" \
"https://${s3BucketName}.${s3Endpoint}/regression/datalake/pipeline_data/iceberg_data_spark40_paimon142.zip";
fi
- host_admin_cmd unzip iceberg_data_spark40_paimon142.zip
- host_admin_cmd mv iceberg_data data
- host_admin_cmd rm -rf iceberg_data_spark40_paimon142.zip
+ unzip iceberg_data_spark40_paimon142.zip
+ mv iceberg_data data
+ rm -rf iceberg_data_spark40_paimon142.zip
)
else
echo "${ICEBERG_DIR}/data exist, continue !"
@@ -1500,7 +1457,7 @@ start_hudi() {
. "${HUDI_DIR}"/hudi.env
set +a
envsubst <"${HUDI_DIR}"/hudi.yaml.tpl >"${HUDI_DIR}"/hudi.yaml
- sudo chmod +x "${HUDI_DIR}"/scripts/init.sh
+ chmod +x "${HUDI_DIR}"/scripts/init.sh
register_stack_metadata "hudi" "${HUDI_DIR}/hudi.yaml"
"${HUDI_DIR}/hudi.env"
compose_down_stack "${HUDI_DIR}/hudi.yaml" "${HUDI_DIR}/hudi.env"
--remove-orphans
if [[ "${STOP}" -ne 1 ]]; then
@@ -1520,6 +1477,13 @@ start_mariadb() {
start_fluss() {
local fluss_dir="${ROOT}/docker-compose/fluss"
+ local fluss_host_ip
+
+ if [[ "${STOP}" -ne 1 ]]; then
+ . "${fluss_dir}/fluss-network.sh"
+ fluss_host_ip="$(resolve_fluss_host_ip)" || return
+ export FLUSS_HOST_IP="${fluss_host_ip}"
+ fi
# The compose file bind mounts remote.data.dir and the paimon warehouse at
# the same absolute paths it uses inside the containers, so Doris (running
on
@@ -1548,10 +1512,10 @@ start_fluss() {
bash "${fluss_dir}/fetch-paimon-s3.sh"
reset_data_dirs "${FLUSS_REMOTE_DATA_DIR}" "${FLUSS_PAIMON_WAREHOUSE_DIR}"
- # The fluss and flink images run as uid 9999, the host directories are
- # created by root.
- host_admin_cmd chmod 777 "${FLUSS_REMOTE_DATA_DIR}"
"${FLUSS_PAIMON_WAREHOUSE_DIR}"
- host_admin_cmd chmod +x "${fluss_dir}/scripts/run-init-sql.sh"
+ # The fluss and flink images run as uid 9999, mapped to a subordinate
+ # host UID in rootless Docker. Let both images write the fixture paths.
+ chmod 777 "${FLUSS_REMOTE_DATA_DIR}" "${FLUSS_PAIMON_WAREHOUSE_DIR}"
+ chmod +x "${fluss_dir}/scripts/run-init-sql.sh"
compose_up_stack "${fluss_dir}/fluss.yaml" "${fluss_dir}/fluss.env" -d
--wait
}
@@ -1571,7 +1535,7 @@ cleanup_kerberos_ready_wait() {
local wait_pid="$1"
[[ -n "${wait_pid}" ]] || return 0
- sudo kill -TERM -- "-${wait_pid}" >/dev/null 2>&1 || true
+ kill -TERM -- "-${wait_pid}" >/dev/null 2>&1 || true
wait "${wait_pid}" >/dev/null 2>&1 || true
}
@@ -1581,7 +1545,7 @@ wait_for_kerberos_ready() {
local wait_status=0
echo "Waiting for ${container} readiness event"
- setsid sudo timeout --signal=TERM --kill-after=5s 20m bash -c '
+ setsid timeout --signal=TERM --kill-after=5s 20m bash -c '
log_dir=$(mktemp -d)
mkfifo "${log_dir}/container.log"
docker logs --follow "$1" >"${log_dir}/container.log" 2>&1 &
@@ -1708,21 +1672,12 @@ start_kerberos() {
envsubst <"${KERBEROS_DIR}/conf/${config}.tpl"
>"${KERBEROS_DIR}/conf/kerberos${i}/${config}"
done
done
- sudo chmod a+w /etc/hosts
- if ! awk -v ip="${IP_HOST}" '$1 == ip && $2 == "hadoop-master" { found = 1
} END { exit !found }' /etc/hosts; then
- sudo sed -i "1i${IP_HOST} hadoop-master" /etc/hosts
- fi
- if ! awk -v ip="${IP_HOST}" '$1 == ip && $2 == "hadoop-master-2" { found =
1 } END { exit !found }' /etc/hosts; then
- sudo sed -i "1i${IP_HOST} hadoop-master-2" /etc/hosts
- fi
register_stack_metadata "kerberos" "${KERBEROS_DIR}/kerberos.yaml" ""
compose_cmd "${KERBEROS_DIR}/kerberos.yaml" "" down --remove-orphans
- sudo rm -rf "${KERBEROS_DIR}/data"
if [[ "${STOP}" -ne 1 ]]; then
+ require_rootless_host_network
echo "PREPARE KERBEROS DATA"
- rm -rf "${KERBEROS_DIR}"/two-kerberos-hives/*.keytab
- rm -rf "${KERBEROS_DIR}"/two-kerberos-hives/*.jks
- rm -rf "${KERBEROS_DIR}"/two-kerberos-hives/*.conf
+ reset_data_dirs "${KERBEROS_DIR}/data"
"${KERBEROS_DIR}/two-kerberos-hives"
compose_cmd "${KERBEROS_DIR}/kerberos.yaml" "" up --build
--remove-orphans -d
trap 'cleanup_kerberos_readiness_jobs "${readiness_pids[@]}"' EXIT
trap 'exit 143' TERM
@@ -1744,9 +1699,13 @@ start_kerberos() {
for container in "${containers[@]}"; do
validate_kerberos_container "${container}"
done
- sudo ln -sfn "${KERBEROS_DIR}/two-kerberos-hives" /keytabs
- sudo cp "${KERBEROS_DIR}/common/conf/doris-krb5.conf"
/keytabs/krb5.conf
- sudo cp "${KERBEROS_DIR}/common/conf/doris-krb5.conf" /etc/krb5.conf
+ sed -e "s/kdc = hadoop-master:/kdc = ${IP_HOST}:/" \
+ -e "s/kdc = hadoop-master-2:/kdc = ${IP_HOST}:/" \
+ -e "s#FILE:/var/log/#FILE:${KERBEROS_DIR}/two-kerberos-hives/#g" \
+ "${KERBEROS_DIR}/common/conf/doris-krb5.conf" \
+ >"${KERBEROS_DIR}/two-kerberos-hives/krb5.conf"
+ echo "Kerberos client files: ${KERBEROS_DIR}/two-kerberos-hives"
+ echo "For host clients: export
KRB5_CONFIG=${KERBEROS_DIR}/two-kerberos-hives/krb5.conf"
sleep 2
fi
}
@@ -1857,10 +1816,22 @@ start_iceberg_rest() {
export CONTAINER_UID=${CONTAINER_UID}
. "${ROOT}"/docker-compose/iceberg-rest/iceberg-rest_settings.env
envsubst <"${ICEBERG_REST_DIR}/docker-compose.yaml.tpl"
>"${ICEBERG_REST_DIR}/docker-compose.yaml"
+ if [[ "${HOST_OS}" != "Darwin" ]]; then
+ if [[ "${STOP}" -ne 1 ]]; then
+ require_rootless_host_network
+ fi
+ # A host HDFS client must reach the DataNode address returned by the
+ # NameNode. A rootless bridge address is not routable from the host.
+ envsubst <"${ICEBERG_REST_DIR}/hdfs-rootless.yaml.tpl"
>"${ICEBERG_REST_DIR}/hdfs-rootless.yaml"
+ docker_cli compose -f "${ICEBERG_REST_DIR}/docker-compose.yaml" \
+ -f "${ICEBERG_REST_DIR}/hdfs-rootless.yaml" config \
+ >"${ICEBERG_REST_DIR}/docker-compose.rootless.yaml"
+ mv "${ICEBERG_REST_DIR}/docker-compose.rootless.yaml"
"${ICEBERG_REST_DIR}/docker-compose.yaml"
+ fi
register_stack_metadata "iceberg-rest"
"${ICEBERG_REST_DIR}/docker-compose.yaml" ""
compose_cmd "${ICEBERG_REST_DIR}/docker-compose.yaml" "" down
--remove-orphans
if [[ "${STOP}" -ne 1 ]]; then
- # Start all three REST catalogs (S3, OSS, COS)
+ # Start the REST catalogs and their built-in HDFS.
compose_cmd "${ICEBERG_REST_DIR}/docker-compose.yaml" "" up -d
--remove-orphans --wait
fi
}
@@ -1871,7 +1842,7 @@ reserve_ports
# Ensure hive data is downloaded before starting hive2/hive3, but only once
need_prepare_hive_data=0
-if [[ "$NEED_LOAD_DATA" -eq 1 ]]; then
+if [[ "${STOP}" -ne 1 && "$NEED_LOAD_DATA" -eq 1 ]]; then
if [[ "${RUN_HIVE2}" -eq 1 ]] || [[ "${RUN_HIVE3}" -eq 1 ]]; then
if [[ "${HIVE_MODE}" == "refresh" || "${HIVE_MODE}" == "rebuild" ]];
then
need_prepare_hive_data=1
diff --git a/docker/thirdparties/test/docker-command-test.sh
b/docker/thirdparties/test/docker-command-test.sh
new file mode 100755
index 00000000000..3d7c0eab304
--- /dev/null
+++ b/docker/thirdparties/test/docker-command-test.sh
@@ -0,0 +1,138 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -eo pipefail
+
+ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." &>/dev/null && pwd)"
+TEST_DIR="$(mktemp -d /tmp/doris-docker-test.XXXXXX)"
+TEST_LOG="${TEST_DIR}/commands.log"
+ORIGINAL_PATH="${PATH}"
+trap 'rm -rf "${TEST_DIR}"' EXIT
+
+fail() {
+ echo "FAIL: $*" >&2
+ exit 1
+}
+
+mkdir -p "${TEST_DIR}/bin"
+
+cat >"${TEST_DIR}/bin/docker" <<'EOF'
+#!/usr/bin/env bash
+if [[ "${1:-}" == "info" ]]; then
+ if [[ "${DOCKER_INFO_EXIT:-0}" -ne 0 ]]; then
+ exit "${DOCKER_INFO_EXIT}"
+ fi
+ if [[ "${2:-}" == "--format" ]]; then
+ if [[ "${3}" == '{{.ServerVersion}}' ]]; then
+ printf '%s\n' "${DOCKER_SERVER_VERSION}"
+ else
+ printf '["name=%s"]\n' "${DOCKER_SECURITY_MODE}"
+ fi
+ fi
+ exit 0
+fi
+printf 'docker %s\n' "$*" >>"${TEST_LOG}"
+EOF
+
+cat >"${TEST_DIR}/bin/sudo" <<'EOF'
+#!/usr/bin/env bash
+printf 'sudo %s\n' "$*" >>"${TEST_LOG}"
+exit 125
+EOF
+
+chmod +x "${TEST_DIR}/bin/docker" "${TEST_DIR}/bin/sudo"
+export PATH="${TEST_DIR}/bin:${ORIGINAL_PATH}"
+export TEST_LOG DOCKER_SECURITY_MODE
+HOST_OS=Linux
+
+# shellcheck source=../docker-command.sh
+. "${ROOT}/docker-command.sh"
+
+docker_cli() {
+ docker "$@"
+}
+
+unset DOCKER_HOST DOCKER_CONTEXT
+DOCKER_SECURITY_MODE=rootless
+configure_docker_access
+: >"${TEST_LOG}"
+export DOCKER_USE_SUDO=1 # A legacy environment must not enable elevation.
+docker_cli ps
+unset DOCKER_USE_SUDO
+grep -Fxq 'docker ps' "${TEST_LOG}" || fail "docker_cli did not call Docker
directly"
+if grep -q '^sudo ' "${TEST_LOG}"; then
+ fail "rootless mode invoked sudo"
+fi
+
+export DOCKER_SERVER_VERSION=29.4.0
+if require_rootless_host_network 2>/dev/null; then
+ fail "rootless Docker before 29.5 was accepted for host networking"
+fi
+DOCKER_SERVER_VERSION=29.5.0
+require_rootless_host_network || fail "rootless Docker 29.5 was rejected"
+DOCKER_SERVER_VERSION=30.0.0
+require_rootless_host_network || fail "newer rootless Docker was rejected"
+
+DOCKER_SECURITY_MODE=seccomp
+: >"${TEST_LOG}"
+if configure_docker_access 2>/dev/null; then
+ fail "Linux accepted a rootful daemon"
+fi
+[[ ! -s "${TEST_LOG}" ]] || fail "a rootful daemon triggered elevation"
+
+# macOS keeps direct Docker Desktop access without Linux rootless detection.
+HOST_OS=Darwin
+configure_docker_access || fail "Docker Desktop was rejected"
+require_rootless_host_network || fail "Linux version requirement applied to
macOS"
+HOST_OS=Linux
+DOCKER_SECURITY_MODE=rootless
+
+# Discover the user's socket even if the default system daemon is accessible.
+export XDG_RUNTIME_DIR="${TEST_DIR}/runtime"
+mkdir -p "${XDG_RUNTIME_DIR}"
+python3 - "${XDG_RUNTIME_DIR}/docker.sock" <<'PY'
+import socket
+import sys
+connection = socket.socket(socket.AF_UNIX)
+connection.bind(sys.argv[1])
+connection.close()
+PY
+unset DOCKER_HOST DOCKER_CONTEXT
+configure_docker_access
+[[ "${DOCKER_HOST}" == "unix://${XDG_RUNTIME_DIR}/docker.sock" ]] || fail
"user socket was not discovered"
+
+DOCKER_HOST=unix:///explicit/docker.sock
+configure_docker_access
+[[ "${DOCKER_HOST}" == unix:///explicit/docker.sock ]] || fail "explicit
endpoint was replaced"
+unset DOCKER_HOST
+export DOCKER_CONTEXT=review-rootless
+configure_docker_access
+[[ -z "${DOCKER_HOST+x}" ]] || fail "explicit context was replaced"
+unset DOCKER_CONTEXT
+
+# A failed daemon probe must not run a host operation or silently escalate it.
+export DOCKER_INFO_EXIT=17
+export DOCKER_HOST="unix://${XDG_RUNTIME_DIR}/docker.sock"
+: >"${TEST_LOG}"
+status=0
+(configure_docker_access && touch "${TEST_DIR}/unexpected") 2>/dev/null ||
status=$?
+[[ "${status}" -eq 1 ]] || fail "failed daemon probe did not propagate its
error"
+[[ ! -e "${TEST_DIR}/unexpected" ]] || fail "host command ran after daemon
probe failed"
+[[ ! -s "${TEST_LOG}" ]] || fail "failed daemon probe invoked sudo or another
Docker operation"
+
+echo "PASS"
diff --git a/docker/thirdparties/test/fluss-network-test.sh
b/docker/thirdparties/test/fluss-network-test.sh
new file mode 100644
index 00000000000..bd647f652c3
--- /dev/null
+++ b/docker/thirdparties/test/fluss-network-test.sh
@@ -0,0 +1,72 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -euo pipefail
+ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." &>/dev/null && pwd)"
+. "${ROOT}/docker-compose/fluss/fluss-network.sh"
+test_dir="$(mktemp -d /tmp/doris-fluss-network-test.XXXXXX)"
+trap 'rm -rf "${test_dir}"' EXIT
+mkdir "${test_dir}/bin"
+cat >"${test_dir}/bin/ip" <<'EOF'
+#!/bin/sh
+test "${TEST_ROUTE_FAIL:-0}" = 0 || exit 1
+printf '1.1.1.1 via 192.0.2.1 dev eth0 src %s uid 1000\n' "${TEST_ROUTE_IP}"
+EOF
+cat >"${test_dir}/bin/hostname" <<'EOF'
+#!/bin/sh
+test "${TEST_HOSTNAME_FAIL:-0}" = 0 || exit 1
+printf '%s\n' "${TEST_HOST_IPS}"
+EOF
+chmod +x "${test_dir}/bin/ip" "${test_dir}/bin/hostname"
+export PATH="${test_dir}/bin:${PATH}"
+export TEST_ROUTE_IP=192.0.2.10 TEST_HOST_IPS='127.0.0.1 192.0.2.11 ::1'
+export HOST_OS=Linux
+unset FLUSS_HOST_IP
+for IP_HOST in 127.0.0.1 localhost 0.0.0.0 ::1; do
+ [[ "$(resolve_fluss_host_ip)" == 192.0.2.10 ]]
+done
+[[ "${IP_HOST}" == ::1 ]] # Resolving Fluss must not change other fixtures.
+
+TEST_ROUTE_FAIL=1
+export TEST_ROUTE_FAIL
+IP_HOST=192.0.2.20
+[[ "$(resolve_fluss_host_ip)" == 192.0.2.20 ]]
+FLUSS_HOST_IP=build-host.example
+[[ "$(resolve_fluss_host_ip)" == build-host.example ]]
+for FLUSS_HOST_IP in 127.0.0.1 localhost localhost. 0.0.0.0 ::1 '[::1]' ::
'[::]'; do
+ if resolve_fluss_host_ip >/dev/null 2>&1; then
+ echo "FAIL: advertised an unusable address" >&2
+ exit 1
+ fi
+done
+unset FLUSS_HOST_IP
+IP_HOST=127.0.0.1
+[[ "$(resolve_fluss_host_ip)" == 192.0.2.11 ]]
+export TEST_HOSTNAME_FAIL=1
+if resolve_fluss_host_ip >/dev/null 2>&1; then
+ echo "FAIL: accepted missing host addresses" >&2
+ exit 1
+fi
+HOST_OS=Darwin
+if resolve_fluss_host_ip >/dev/null 2>&1; then
+ echo "FAIL: used loopback for Docker Desktop bridge clients" >&2
+ exit 1
+fi
+FLUSS_HOST_IP=192.0.2.30
+[[ "$(resolve_fluss_host_ip)" == 192.0.2.30 ]]
+echo "Fluss host-address selection tests passed"
diff --git a/docker/thirdparties/test/launcher-stop-test.sh
b/docker/thirdparties/test/launcher-stop-test.sh
new file mode 100644
index 00000000000..2f2574fe64a
--- /dev/null
+++ b/docker/thirdparties/test/launcher-stop-test.sh
@@ -0,0 +1,81 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Run the real CLI against an isolated fixture tree and a Docker stub.
+# Requires Bash 4+, like the launcher; does not access a real daemon.
+set -euo pipefail
+ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." &>/dev/null && pwd)"
+test_dir="$(mktemp -d /tmp/doris-stop-test.XXXXXX)"
+trap 'rm -rf "${test_dir}"' EXIT
+fixture="${test_dir}/thirdparties"
+mkdir -p "${fixture}" "${test_dir}/bin"
+for file in \
+ run-thirdparties-docker.sh docker-command.sh docker-health.sh \
+ juicefs-helpers.sh custom_settings.env \
+ docker-compose/hive/scripts/bootstrap/bootstrap-groups.sh \
+ docker-compose/hive/hive-3x_settings.env \
+ docker-compose/hive/hadoop-hive.env.tpl \
+ docker-compose/hive/hadoop-hive-3x.env.tpl \
+ docker-compose/hive/hive-3x.yaml.tpl \
+ docker-compose/fluss/fluss.env.tpl docker-compose/fluss/fluss.yaml.tpl; do
+ mkdir -p "$(dirname "${fixture}/${file}")"
+ cp "${ROOT}/${file}" "${fixture}/${file}"
+done
+
+export STOP_TEST_DOCKER_LOG="${test_dir}/docker.log"
+export STOP_TEST_PREPARE_MARKER="${test_dir}/prepare-called"
+cat >"${fixture}/docker-compose/hive/scripts/prepare-hive-data.sh" <<'EOF'
+touch "${STOP_TEST_PREPARE_MARKER}"
+exit 99
+EOF
+cat >"${test_dir}/bin/docker" <<'EOF'
+#!/usr/bin/env bash
+if [[ "${1:-}" == info ]]; then
+ echo '["name=rootless"]'
+ exit 0
+fi
+printf '%s\n' "$*" >>"${STOP_TEST_DOCKER_LOG}"
+[[ "${1:-}" == compose ]] || exit 1
+for argument in "$@"; do
+ if [[ "${argument}" == down ]]; then exit 0; fi
+done
+exit 1
+EOF
+cat >"${test_dir}/bin/envsubst" <<'EOF'
+#!/bin/sh
+cat
+EOF
+chmod +x "${test_dir}/bin/docker" "${test_dir}/bin/envsubst"
+
+export PATH="${test_dir}/bin:${PATH}"
+export CONTAINER_UID=rootless-stop-test-
+export DOCKER_HOST=unix:///rootless-stop-test.sock
+export IP_HOST=127.0.0.1
+export HIVE_HOST_ALIAS=localhost
+export JFS_CLUSTER_META=redis://127.0.0.1:6379/0
+export HIVE_MODE=refresh
+bash "${fixture}/run-thirdparties-docker.sh" -c hive3 --stop \
+ >"${test_dir}/launcher.log" 2>&1
+[[ ! -e "${STOP_TEST_PREPARE_MARKER}" ]]
+grep -q 'compose -p rootless-stop-test-hive3 ' "${STOP_TEST_DOCKER_LOG}"
+grep -q ' down$' "${STOP_TEST_DOCKER_LOG}"
+bash "${fixture}/run-thirdparties-docker.sh" -c fluss --stop \
+ >"${test_dir}/fluss-stop.log" 2>&1
+grep -q '/fluss/fluss.yaml ' "${STOP_TEST_DOCKER_LOG}"
+echo "Hive stop bypasses data preparation and only tears down compose"
+echo "Fluss stop needs no network-address discovery"
diff --git a/docker/thirdparties/test/snapshot-hive-baseline-test.sh
b/docker/thirdparties/test/snapshot-hive-baseline-test.sh
new file mode 100644
index 00000000000..8c03a0eee43
--- /dev/null
+++ b/docker/thirdparties/test/snapshot-hive-baseline-test.sh
@@ -0,0 +1,99 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Integration test: requires the current user's Docker daemon and Python 3.
+set -euo pipefail
+ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." &>/dev/null && pwd)"
+HOST_OS="$(uname -s)"
+. "${ROOT}/docker-command.sh"
+configure_docker_access
+
+test_dir="$(mktemp -d /tmp/doris-snapshot-test.XXXXXX)"
+prefix="$(basename "${test_dir}" | tr '[:upper:]' '[:lower:]')"
+image="${THIRDPARTY_FILESYSTEM_IMAGE:-alpine:3.20}"
+snapshot="${ROOT}/docker-compose/hive/scripts/snapshot-hive-baseline.sh"
+volumes=()
+
+cleanup() {
+ if ((${#volumes[@]})); then
+ docker volume rm "${volumes[@]}" >/dev/null
+ fi
+ rm -rf "${test_dir}"
+}
+trap cleanup EXIT
+
+# Missing volumes must fail before producing an output file.
+if bash "${snapshot}" "${prefix}" "${test_dir}/missing.tar.gz"
>"${test_dir}/missing.log" 2>&1; then
+ echo "FAIL: snapshot accepted missing volumes" >&2
+ exit 1
+fi
+[[ ! -e "${test_dir}/missing.tar.gz" ]]
+
+for suffix in namenode datanode pgdata state; do
+ volumes+=("${prefix}-${suffix}")
+ docker volume create "${prefix}-${suffix}" >/dev/null
+done
+
+docker run --rm --network none --security-opt no-new-privileges \
+ -v "${prefix}-namenode:/fixture/namenode" \
+ -v "${prefix}-datanode:/fixture/datanode" \
+ -v "${prefix}-pgdata:/fixture/pgdata" \
+ -v "${prefix}-state:/fixture/state" \
+ "${image}" sh -ec '
+ for part in namenode datanode pgdata state; do
+ printf "%s" "$part" >"/fixture/$part/payload"
+ done
+ printf legacy >/fixture/state/baseline.version
+ chown 123:456 /fixture/pgdata /fixture/pgdata/payload
+ chmod 700 /fixture/pgdata
+ chmod 600 /fixture/pgdata/payload
+ '
+
+bash "${snapshot}" "${prefix}" "${test_dir}/baseline.tar.gz"
+python3 - "${test_dir}/baseline.tar.gz" <<'PY'
+import sys
+import tarfile
+
+with tarfile.open(sys.argv[1]) as archive:
+ for part in ['namenode', 'datanode', 'pgdata', 'state']:
+ assert archive.extractfile(part + '/payload').read() == part.encode()
+ assert 'state/baseline.version' not in archive.getnames()
+ entry = archive.getmember('pgdata/payload')
+ assert (entry.uid, entry.gid, entry.mode) == (123, 456, 0o600)
+PY
+docker run --rm --network none -v "${prefix}-state:/fixture:ro" \
+ "${image}" test -f /fixture/baseline.version
+
+# An interrupted exporter must preserve an existing baseline and remove its
+# temporary output. Forward info/volume queries to the real rootless daemon.
+printf previous-baseline >"${test_dir}/previous.tar.gz"
+docker() {
+ if [[ "${1:-}" == "run" ]]; then
+ return 42
+ fi
+ command docker "$@"
+}
+export -f docker
+status=0
+bash "${snapshot}" "${prefix}" "${test_dir}/previous.tar.gz" \
+ >"${test_dir}/failure.log" 2>&1 || status=$?
+unset -f docker
+[[ "${status}" -eq 42 ]]
+[[ "$(cat "${test_dir}/previous.tar.gz")" == previous-baseline ]]
+[[ "$(find "${test_dir}" -name 'previous.tar.gz.*' | wc -l | tr -d ' ')" -eq 0
]]
+echo "Snapshot export, UID/GID preservation and failure cleanup passed"
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
