This is an automated email from the ASF dual-hosted git repository.
pan3793 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/celeborn.git
The following commit(s) were added to refs/heads/main by this push:
new 4e20a611c [CELEBORN-2390] Add docker-compose for local cluster
4e20a611c is described below
commit 4e20a611c9953bc817b83712caaee233a7b1db10
Author: yew1eb <[email protected]>
AuthorDate: Mon Aug 24 16:50:48 2026 +0800
[CELEBORN-2390] Add docker-compose for local cluster
### What changes were proposed in this pull request?
Add a `docker-compose.yaml` under `docker/` to run a minimal local Celeborn
cluster (1 master + 3 workers) for development and debugging, with a
`build-image.sh` wrapper and a `conf/`. The existing `docker/Dockerfile` is
unchanged.
### Why are the changes needed?
There is no lightweight local deployment path today — only the K8s Helm
chart or running master/worker manually on the host.
### Does this PR resolve a correctness bug?
- [ ] Yes
### Does this PR introduce _any_ user-facing change?
- [ ] Yes
### How was this patch tested?
Ran the cluster locally with `docker compose up -d`:
> $ docker compose -f docker/docker-compose.yaml up -d
> $ docker compose -f docker/docker-compose.yaml ps
<img width="1354" height="132" alt="image"
src="https://github.com/user-attachments/assets/86b8e10b-9058-498f-b532-15ab2ce5f3fc";
/>
<img width="1510" height="119" alt="image"
src="https://github.com/user-attachments/assets/96cd905d-be6a-4c46-9c62-70f77927a3d1";
/>
Closes #3768 from yew1eb/add-docker-compose.
Authored-by: yew1eb <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
---
.gitignore | 2 +
docker/README.md | 107 +++++++++++++++++++++++++++
docker/build-image.sh | 107 +++++++++++++++++++++++++++
docker/conf/celeborn-defaults.conf | 72 ++++++++++++++++++
docker/conf/log4j2.xml | 129 ++++++++++++++++++++++++++++++++
docker/conf/metrics.properties | 20 +++++
docker/docker-compose.yaml | 134 ++++++++++++++++++++++++++++++++++
docker/spark/Dockerfile | 40 ++++++++++
docker/spark/conf/spark-defaults.conf | 43 +++++++++++
9 files changed, 654 insertions(+)
diff --git a/.gitignore b/.gitignore
index 970fea471..847ea21f3 100644
--- a/.gitignore
+++ b/.gitignore
@@ -76,3 +76,5 @@ spark-warehouse/
logs
pids
tmp/
+
+*.jar
diff --git a/docker/README.md b/docker/README.md
new file mode 100644
index 000000000..3187d0d5d
--- /dev/null
+++ b/docker/README.md
@@ -0,0 +1,107 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Celeborn Local Cluster (docker-compose)
+
+A local Celeborn cluster — 1 master (HA off) + 3 workers — for development and
+debugging, optionally accompanied by an example Spark cluster (1 master + 2
+workers) whose shuffle data is served by Celeborn, so the whole stack can be
+exercised end-to-end.
+
+## Prerequisites
+
+- Docker (daemon running) and the `docker compose` plugin.
+- The repo bundles `build/mvn` used by `make-distribution.sh`. No system Maven
+ required, but a JDK is needed to run the build.
+
+## 1. Build the images
+
+```bash
+# Celeborn only (no Spark example cluster):
+./docker/build-image.sh
+
+# Also build the Spark example image (celeborn-spark:dev):
+./docker/build-image.sh -Pspark-3.5 # Spark 3 client
+./docker/build-image.sh -Pspark-4.0 # or Spark 4 client
+```
+
+This runs `./build/make-distribution.sh` (producing `dist/`) and then:
+
+- `docker build -t celeborn:dev -f dist/docker/Dockerfile dist/` — the
+ Celeborn master/worker image, always built.
+- If a Spark client profile was passed (e.g. `-Pspark-3.5` or `-Pspark-4.0`),
+ the distribution also produces
`dist/spark/celeborn-client-spark-*-shaded_*.jar`;
+ `build-image.sh` copies it into `docker/spark/` and runs
+ `docker build -t celeborn-spark:dev docker/spark/` — a Spark image with the
+ Celeborn client jar baked in. Without a Spark profile, the Spark image is
+ skipped and only the Celeborn cluster is available.
+
+## 2. Start the cluster
+
+```bash
+docker compose -f docker/docker-compose.yaml up -d
+docker compose -f docker/docker-compose.yaml ps
+```
+
+The Celeborn master has a healthcheck; the 3 workers start once the master is
+healthy and register with it. When `celeborn-spark:dev` is present, the Spark
+master and 2 workers start after the Celeborn master is healthy.
+
+## 3. Verify
+
+```bash
+# Master HTTP (expect HTTP 200)
+curl -s -o /dev/null -w '%{http_code}\n' http://localhost:9098/metrics/json
+
+# Tail logs; look for "Registered worker" on the master
+docker compose -f docker/docker-compose.yaml logs -f celeborn-master
+```
+
+## 4. Run a Spark job that shuffles through Celeborn
+
+Spark is pre-wired to use Celeborn via `docker/spark/conf/spark-defaults.conf`
+(`spark.shuffle.manager=...CelebornShuffleManager`, master endpoints
+`celeborn-master:9097`), so no extra `--conf` flags are needed. Submit Spark's
+built-in `GroupByTest`, which forces a shuffle:
+
+```bash
+docker exec celeborn-spark-master /opt/spark/bin/run-example \
+ --master spark://celeborn-spark-master:7077 \
+ GroupByTest 10 100 1000 10
+```
+
+The job should finish with `final status: SUCCEEDED`. While it runs, watch the
+shuffle traffic land on Celeborn workers (push/fetch requests) and confirm the
+shuffle really went through Celeborn rather than the local Spark shuffle:
+
+```bash
+docker compose -f docker/docker-compose.yaml logs -f celeborn-worker
+# Master HTTP metrics: the worker push/fetch counters should increase.
+curl -s http://localhost:9098/metrics/json | grep -i shuffle
+```
+
+The Spark master Web UI is at http://localhost:8080/.
+
+## 5. Stop
+
+```bash
+docker compose -f docker/docker-compose.yaml down
+```
+
+`down` removes the containers (Celeborn and Spark) and the `shared` volume, so
+worker shuffle data (in the containers' writable layer) is discarded.
+`restart` keeps it.
diff --git a/docker/build-image.sh b/docker/build-image.sh
new file mode 100755
index 000000000..ea09bd93e
--- /dev/null
+++ b/docker/build-image.sh
@@ -0,0 +1,107 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Build the local Celeborn docker image (celeborn:dev) for docker-compose.
+#
+# Two steps:
+# 1. ./build/make-distribution.sh -> produces dist/ (bin/sbin/conf/jars/...
+# plus dist/docker/Dockerfile copied by the distribution script itself)
+# 2. docker build -> builds the image with context = dist/
+# (the Dockerfile COPYs bin/sbin/... relative to the context root)
+#
+# The image is NOT built via the compose `build:` key, because
make-distribution
+# must run before docker build and dist/ may not exist yet when invoking
+# `docker compose up`.
+#
+# When a Spark client profile is passed (e.g. -Pspark-3.5 or -Pspark-4.0),
+# make-distribution also drops the client shaded jar at dist/spark/. Step 3
+# then builds an additional Spark image (celeborn-spark:dev) on top of
+# apache/spark with that jar baked in, so docker-compose can run an end-to-end
+# Spark cluster whose shuffle is served by Celeborn.
+#
+# Usage:
+# ./docker/build-image.sh # core master/worker/cli
+# ./docker/build-image.sh -Pspark-3.5 # also build Spark 3 client +
image
+# ./docker/build-image.sh -Pspark-4.0 # also build Spark 4 client +
image
+# CELEBORN_IMAGE_TAG=celeborn:dev ./docker/build-image.sh
+# CELEBORN_SPARK_IMAGE_TAG=celeborn-spark:dev ./docker/build-image.sh
-Pspark-3.5
+# CELEBORN_SPARK_BASE_TAG=3.5.9 ./docker/build-image.sh -Pspark-3.5
+#
+# Requirements: docker on PATH; build/mvn is bundled in the repo.
+
+set -euo pipefail
+
+# Resolve repo root (parent of this script's directory).
+cd "$(dirname "$0")/.."
+
+IMAGE_TAG="${CELEBORN_IMAGE_TAG:-celeborn:dev}"
+SPARK_IMAGE_TAG="${CELEBORN_SPARK_IMAGE_TAG:-celeborn-spark:dev}"
+# Base apache/spark image tag. Renamed away from SPARK_IMAGE_TAG because that
+# env var is commonly exported by Spark's own tooling and would otherwise leak
+# in here as the *output* image name (e.g. celeborn-spark:dev), producing an
+# invalid "apache/spark:celeborn-spark:dev" FROM reference. When unset, the
+# default is derived from the built client jar's Spark major version below.
+BASE_SPARK_IMAGE_TAG="${CELEBORN_SPARK_BASE_TAG:-}"
+
+echo "==> Building distribution (./build/make-distribution.sh --sbt-enabled
$*) ..."
+./build/make-distribution.sh --sbt-enabled "$@"
+
+echo "==> Building docker image ${IMAGE_TAG} from dist/ ..."
+if [ ! -f dist/docker/Dockerfile ]; then
+ echo "ERROR: dist/docker/Dockerfile not found. make-distribution may have
failed." >&2
+ exit 1
+fi
+docker build -t "${IMAGE_TAG}" -f dist/docker/Dockerfile dist/
+
+# --- Spark image (optional)
---------------------------------------------------
+# Only build it when make-distribution produced a Spark client shaded jar,
which
+# happens when a -Pspark-3.x/-Pspark-4.x profile was passed. If absent, skip
+# gracefully so the Celeborn-only workflow keeps working.
+SPARK_CLIENT_JAR="$(ls dist/spark/celeborn-client-spark-*-shaded_*.jar
2>/dev/null || true)"
+
+if [ -n "${SPARK_CLIENT_JAR}" ]; then
+ echo "==> Found Spark client jar: $(basename "${SPARK_CLIENT_JAR}")"
+ # Derive the default base image tag from the jar's Spark major version, e.g.
+ # celeborn-client-spark-4-shaded_2.13-<version>.jar -> 4 ->
apache/spark:4.0.4.
+ SPARK_MAJOR_VERSION="$(basename "${SPARK_CLIENT_JAR}" | sed -E
's/celeborn-client-spark-([0-9]+)-shaded_.*/\1/')"
+ if [ -z "${BASE_SPARK_IMAGE_TAG}" ]; then
+ case "${SPARK_MAJOR_VERSION}" in
+ 3) BASE_SPARK_IMAGE_TAG="3.5.9" ;;
+ 4) BASE_SPARK_IMAGE_TAG="4.0.4" ;;
+ *)
+ echo "ERROR: unsupported Spark major version
'${SPARK_MAJOR_VERSION}'," >&2
+ echo " set CELEBORN_SPARK_BASE_TAG explicitly." >&2
+ exit 1
+ ;;
+ esac
+ fi
+ # Copy into the Spark image build context (excluded from git via .gitignore).
+ cp "${SPARK_CLIENT_JAR}" docker/spark/
+ echo "==> Building Spark docker image ${SPARK_IMAGE_TAG} (base
apache/spark:${BASE_SPARK_IMAGE_TAG}) ..."
+ docker build -t "${SPARK_IMAGE_TAG}" \
+ --build-arg spark_image_tag="${BASE_SPARK_IMAGE_TAG}" \
+ docker/spark/
+ echo "==> Done. Images: ${IMAGE_TAG}, ${SPARK_IMAGE_TAG}"
+ echo "Start the cluster (Celeborn + Spark):"
+ echo " docker compose -f docker/docker-compose.yaml up -d"
+else
+ echo "==> No Spark client jar under dist/spark/ (pass e.g. -Pspark-3.5 or
-Pspark-4.0 to build one)."
+ echo "==> Skipped Spark image. Done. Image: ${IMAGE_TAG}"
+ echo "Start the Celeborn-only cluster:"
+ echo " docker compose -f docker/docker-compose.yaml up -d"
+fi
diff --git a/docker/conf/celeborn-defaults.conf
b/docker/conf/celeborn-defaults.conf
new file mode 100644
index 000000000..a55234202
--- /dev/null
+++ b/docker/conf/celeborn-defaults.conf
@@ -0,0 +1,72 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# This is a docker-compose local cluster configuration for Apache Celeborn.
+# Topology: 1 master (HA disabled) + 3 workers.
+# Config values are aligned with charts/celeborn/values.yaml, with local
+# downsizing (single master, 1 disk x 10G, smaller memory via compose env).
+#
+# Mounted (read-only) into /opt/celeborn/conf/ by docker-compose.yaml, which
+# overrides the image's conf (the image only ships *.template files that are
+# NOT loaded by default -- see Utils.getDefaultPropertiesFile).
+
+celeborn.metrics.enabled true
+
+# ----------------------------------------------------------------------------
+# Master (single node, HA disabled)
+# ----------------------------------------------------------------------------
+# Bind/advertise host = compose service name, so workers/clients can reach
+# master via celeborn-master:9097 on the compose network.
+celeborn.master.host celeborn-master
+celeborn.master.port 9097
+celeborn.master.http.port 9098
+celeborn.master.endpoints celeborn-master:9097
+celeborn.master.ha.enabled false
+celeborn.master.heartbeat.worker.timeout 120s
+celeborn.master.heartbeat.application.timeout 300s
+
+# ----------------------------------------------------------------------------
+# Worker
+# ----------------------------------------------------------------------------
+celeborn.worker.http.port 9096
+# Store shuffle data in the image-writable /opt/celeborn/data (owned by the
+# celeborn user). No host bind-mount; data lives in the container's writable
+# layer, so it survives restarts but is discarded when the container is removed
+# (e.g. `docker compose down`).
+celeborn.worker.storage.dirs
/opt/celeborn/data:disktype=SSD:capacity=10G
+celeborn.worker.storage.disk.reserve.size 1G
+# Containers have no real /sys/block disk info; disable device monitor to
+# avoid workers being wrongly excluded as unhealthy.
+celeborn.worker.monitor.disk.enabled false
+celeborn.worker.fetch.io.threads 32
+celeborn.worker.push.io.threads 32
+celeborn.worker.heartbeat.timeout 120s
+
+# ----------------------------------------------------------------------------
+# Network
+# ----------------------------------------------------------------------------
+celeborn.rpc.io.serverThreads 64
+celeborn.rpc.io.numConnectionsPerPeer 2
+celeborn.rpc.io.clientThreads 64
+celeborn.rpc.dispatcher.numThreads 4
+
+# ----------------------------------------------------------------------------
+# Shuffle / Client
+# ----------------------------------------------------------------------------
+celeborn.shuffle.chunk.size 8m
+celeborn.client.push.stageEnd.timeout 120s
+
diff --git a/docker/conf/log4j2.xml b/docker/conf/log4j2.xml
new file mode 100644
index 000000000..68a304c3e
--- /dev/null
+++ b/docker/conf/log4j2.xml
@@ -0,0 +1,129 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+<!--
+ ~ Extra logging related to initialization of Log4j.
+ ~ Set to debug or trace if log4j initialization is failing.
+ -->
+<Configuration status="INFO">
+ <Appenders>
+ <Console name="stdout" target="SYSTEM_OUT">
+ <!--
+ ~ In the pattern layout configuration below, we specify an
explicit `%ex` conversion
+ ~ pattern for logging Throwables. If this was omitted, then (by
default) Log4J would
+ ~ implicitly add an `%xEx` conversion pattern which logs
stacktraces with additional
+ ~ class packaging information. That extra information can
sometimes add a substantial
+ ~ performance overhead, so we disable it in our default logging
config.
+ -->
+ <PatternLayout pattern="%d{yy/MM/dd HH:mm:ss,SSS} %p [%t] %c{1}:
%m%n%ex"/>
+ </Console>
+ <RollingRandomAccessFile name="file"
fileName="${env:CELEBORN_LOG_DIR}/celeborn.log"
+
filePattern="${env:CELEBORN_LOG_DIR}/celeborn.log.%d-%i">
+ <PatternLayout pattern="%d{yy/MM/dd HH:mm:ss,SSS} %p [%t] %c{1}:
%m%n%ex"/>
+ <Policies>
+ <SizeBasedTriggeringPolicy size="200 MB"/>
+ </Policies>
+ <DefaultRolloverStrategy max="7">
+ <Delete basePath="${env:CELEBORN_LOG_DIR}" maxDepth="1">
+ <IfFileName glob="celeborn.log*">
+ <IfAny>
+ <IfAccumulatedFileSize exceeds="1 GB"/>
+ <IfAccumulatedFileCount exceeds="10"/>
+ </IfAny>
+ </IfFileName>
+ </Delete>
+ </DefaultRolloverStrategy>
+ </RollingRandomAccessFile>
+ <RollingRandomAccessFile name="restAuditFile"
fileName="${env:CELEBORN_LOG_DIR}/audit/rest-audit.log"
+
filePattern="${env:CELEBORN_LOG_DIR}/audit/rest-audit.log.%d-%i">
+ <PatternLayout pattern="%d{yy/MM/dd HH:mm:ss,SSS} %p [%t] %c{1}:
%m%n%ex"/>
+ <Policies>
+ <SizeBasedTriggeringPolicy size="200 MB"/>
+ </Policies>
+ <DefaultRolloverStrategy max="7">
+ <Delete basePath="${env:CELEBORN_LOG_DIR}/audit" maxDepth="1">
+ <IfFileName glob="rest-audit.log*">
+ <IfAny>
+ <IfAccumulatedFileSize exceeds="1 GB"/>
+ <IfAccumulatedFileCount exceeds="10"/>
+ </IfAny>
+ </IfFileName>
+ </Delete>
+ </DefaultRolloverStrategy>
+ </RollingRandomAccessFile>
+ <RollingRandomAccessFile name="shuffleAuditFile"
fileName="${env:CELEBORN_LOG_DIR}/audit/shuffle-audit.log"
+
filePattern="${env:CELEBORN_LOG_DIR}/audit/shuffle-audit.log.%d-%i">
+ <PatternLayout pattern="%d{yy/MM/dd HH:mm:ss,SSS} %p [%t] %c{1}:
%m%n%ex"/>
+ <Policies>
+ <SizeBasedTriggeringPolicy size="200 MB"/>
+ </Policies>
+ <DefaultRolloverStrategy max="7">
+ <Delete basePath="${env:CELEBORN_LOG_DIR}/audit" maxDepth="1">
+ <IfFileName glob="shuffle-audit.log*">
+ <IfAny>
+ <IfAccumulatedFileSize exceeds="1 GB"/>
+ <IfAccumulatedFileCount exceeds="10"/>
+ </IfAny>
+ </IfFileName>
+ </Delete>
+ </DefaultRolloverStrategy>
+ </RollingRandomAccessFile>
+ <RollingRandomAccessFile name="metricsAuditFile"
fileName="${env:CELEBORN_LOG_DIR}/audit/metrics-audit.log"
+
filePattern="${env:CELEBORN_LOG_DIR}/audit/metrics-audit.log.%d-%i">
+ <PatternLayout pattern="%d{yy/MM/dd HH:mm:ss,SSS} %p [%t] %c{1}:
%m%n%ex"/>
+ <Policies>
+ <SizeBasedTriggeringPolicy size="200 MB"/>
+ </Policies>
+ <DefaultRolloverStrategy max="7">
+ <Delete basePath="${env:CELEBORN_LOG_DIR}/audit" maxDepth="1">
+ <IfFileName glob="metrics-audit.log*">
+ <IfAny>
+ <IfAccumulatedFileSize exceeds="1 GB"/>
+ <IfAccumulatedFileCount exceeds="10"/>
+ </IfAny>
+ </IfFileName>
+ </Delete>
+ </DefaultRolloverStrategy>
+ </RollingRandomAccessFile>
+ </Appenders>
+
+ <Loggers>
+ <Root level="INFO">
+ <AppenderRef ref="stdout"/>
+ <AppenderRef ref="file"/>
+ </Root>
+ <Logger name="org.apache.hadoop.hdfs" level="WARN" additivity="false">
+ <Appender-ref ref="stdout" level="WARN"/>
+ <Appender-ref ref="file" level="WARN"/>
+ </Logger>
+ <Logger name="org.apache.ratis.server.RaftServerConfigKeys"
level="WARN" additivity="false">
+ <Appender-ref ref="stdout" level="WARN"/>
+ <Appender-ref ref="file" level="WARN"/>
+ </Logger>
+ <Logger name="org.apache.celeborn.server.common.http.RestAuditLogger"
level="INFO" additivity="false">
+ <Appender-ref ref="restAuditFile" level="INFO"/>
+ </Logger>
+ <Logger
name="org.apache.celeborn.service.deploy.master.audit.ShuffleAuditLogger"
level="INFO"
+ additivity="false">
+ <Appender-ref ref="shuffleAuditFile" level="INFO"/>
+ </Logger>
+ <Logger name="org.apache.celeborn.common.metrics.sink.LoggerSink"
level="INFO" additivity="false">
+ <Appender-ref ref="metricsAuditFile" level="INFO"/>
+ </Logger>
+ </Loggers>
+</Configuration>
diff --git a/docker/conf/metrics.properties b/docker/conf/metrics.properties
new file mode 100644
index 000000000..e3b521369
--- /dev/null
+++ b/docker/conf/metrics.properties
@@ -0,0 +1,20 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+*.sink.prometheusServlet.class=org.apache.celeborn.common.metrics.sink.PrometheusServlet
+*.sink.jsonServlet.class=org.apache.celeborn.common.metrics.sink.JsonServlet
+*.sink.loggerSink.class=org.apache.celeborn.common.metrics.sink.LoggerSink
diff --git a/docker/docker-compose.yaml b/docker/docker-compose.yaml
new file mode 100644
index 000000000..a6f310c5e
--- /dev/null
+++ b/docker/docker-compose.yaml
@@ -0,0 +1,134 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Local docker-compose cluster for Apache Celeborn.
+# Topology: 1 master (HA disabled) + 3 workers (via replicas)
+# + an example Spark cluster (1 master + 2 workers) whose shuffle
+# data is served by Celeborn, so the whole stack can be exercised
+# end-to-end without exposing any extra ports.
+#
+# Prerequisites: build the images first ->
+# ./docker/build-image.sh -Pspark-3.5
+# (produces celeborn:dev and celeborn-spark:dev)
+#
+# Usage:
+# docker compose -f docker/docker-compose.yaml up -d
+# docker compose -f docker/docker-compose.yaml logs -f
+# docker compose -f docker/docker-compose.yaml down
+
+name: celeborn
+
+services:
+ celeborn-master:
+ image: celeborn:dev
+ container_name: celeborn-master
+ hostname: celeborn-master
+ command: /opt/celeborn/sbin/start-master.sh
+ environment:
+ # Foreground mode so the JVM stays as PID 1 (else nohup & container
exits).
+ CELEBORN_NO_DAEMONIZE: "true"
+ CELEBORN_MASTER_MEMORY: 1g
+ ports:
+ - "9097:9097" # RPC
+ - "9098:9098" # HTTP REST / metrics
+ volumes:
+ - ./conf:/opt/celeborn/conf:ro
+ networks:
+ - celeborn-net
+ healthcheck:
+ # Master binds to its container IP (via celeborn-master host), not
127.0.0.1.
+ test: ["CMD-SHELL", "bash -c 'echo > /dev/tcp/celeborn-master/9098' ||
exit 1"]
+ interval: 10s
+ timeout: 3s
+ retries: 6
+ start_period: 40s
+ restart: unless-stopped
+
+ celeborn-worker:
+ image: celeborn:dev
+ command: /opt/celeborn/sbin/start-worker.sh
+ deploy:
+ replicas: 3
+ environment:
+ CELEBORN_NO_DAEMONIZE: "true"
+ CELEBORN_WORKER_MEMORY: 1g
+ CELEBORN_WORKER_OFFHEAP_MEMORY: 1g
+ volumes:
+ - ./conf:/opt/celeborn/conf:ro
+ networks:
+ - celeborn-net
+ depends_on:
+ celeborn-master:
+ condition: service_healthy
+ restart: unless-stopped
+
+ spark-master:
+ image: celeborn-spark:dev
+ container_name: celeborn-spark-master
+ hostname: celeborn-spark-master
+ command: /opt/spark/bin/spark-class org.apache.spark.deploy.master.Master
-h celeborn-spark-master
+ environment:
+ SPARK_MASTER_HOST: celeborn-spark-master
+ SPARK_MASTER_PORT: 7077
+ SPARK_MASTER_WEBUI_PORT: 8080
+ SPARK_PUBLIC_DNS: localhost
+ volumes:
+ # Pre-wire Spark to use Celeborn shuffle (see spark-defaults.conf).
+ - ./spark/conf/spark-defaults.conf:/opt/spark/conf/spark-defaults.conf:ro
+ - shared:/shared:rw
+ ports:
+ - "4040:4040" # Spark application UI (when a job runs on the master)
+ - "7077:7077" # Spark master RPC
+ - "8080:8080" # Spark master Web UI
+ networks:
+ - celeborn-net
+ depends_on:
+ celeborn-master:
+ condition: service_healthy
+ healthcheck:
+ test: ["CMD-SHELL", "bash -c 'echo >
/dev/tcp/celeborn-spark-master/8080' || exit 1"]
+ interval: 10s
+ timeout: 3s
+ retries: 6
+ start_period: 20s
+ restart: unless-stopped
+
+ spark-worker:
+ image: celeborn-spark:dev
+ command: /opt/spark/bin/spark-class org.apache.spark.deploy.worker.Worker
spark://celeborn-spark-master:7077
+ deploy:
+ replicas: 2
+ environment:
+ SPARK_WORKER_CORES: 1
+ SPARK_WORKER_MEMORY: 1g
+ SPARK_WORKER_WEBUI_PORT: 8081
+ SPARK_PUBLIC_DNS: localhost
+ volumes:
+ - ./spark/conf/spark-defaults.conf:/opt/spark/conf/spark-defaults.conf:ro
+ - shared:/shared:rw
+ networks:
+ - celeborn-net
+ depends_on:
+ spark-master:
+ condition: service_healthy
+ restart: unless-stopped
+
+networks:
+ celeborn-net:
+ driver: bridge
+
+volumes:
+ shared:
diff --git a/docker/spark/Dockerfile b/docker/spark/Dockerfile
new file mode 100644
index 000000000..355475eb4
--- /dev/null
+++ b/docker/spark/Dockerfile
@@ -0,0 +1,40 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Spark image pre-wired to use Celeborn as its shuffle service.
+#
+# Built by docker/build-image.sh, which copies the Celeborn client shaded jar
+# produced by make-distribution.sh (under dist/spark/) into this directory so
+# it can be COPYed into /opt/spark/jars. The jar is not committed to the repo.
+#
+# Usage (built by build-image.sh, not directly):
+# docker build -t celeborn-spark:dev docker/spark/
+
+ARG spark_image_tag=3.5.9
+FROM apache/spark:${spark_image_tag}
+
+ENV PATH="${PATH}:/opt/spark/bin"
+
+USER root
+RUN mkdir -p /home/spark /shared && chown spark:spark /home/spark /shared
+
+# Celeborn client shaded jar enables
spark.shuffle.manager=...CelebornShuffleManager
+COPY celeborn-client-spark-*-shaded_*.jar /opt/spark/jars/
+
+VOLUME /shared
+
+USER spark
diff --git a/docker/spark/conf/spark-defaults.conf
b/docker/spark/conf/spark-defaults.conf
new file mode 100644
index 000000000..583f7b1b5
--- /dev/null
+++ b/docker/spark/conf/spark-defaults.conf
@@ -0,0 +1,43 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Spark defaults that route shuffle data through the Celeborn cluster running
+# alongside this Spark container in docker-compose. Mounted into
+# /opt/spark/conf/spark-defaults.conf so spark-submit/spark-shell pick it up
+# automatically (no need to repeat --conf flags on every invocation).
+#
+# See README.md "Configure Spark" of the Celeborn repo and
+# tests/spark-it/.../SparkTestBase.scala for the authoritative option set.
+
+# Route shuffle through Celeborn. The shuffle manager class name changed in
+# Celeborn 0.3.0 (was RssShuffleManager); this repo is >= 0.3.0.
+spark.shuffle.manager
org.apache.spark.shuffle.celeborn.SparkShuffleManager
+# Kryo is required: the shaded client relocates packages, which the Java
+# serializer cannot handle.
+spark.serializer
org.apache.spark.serializer.KryoSerializer
+
+# Connect to the Celeborn master started by docker-compose (RPC port 9097).
+# Spark only talks to the master; worker push/fetch ports are random and
+# handed out by the master, so workers need no exposed ports.
+spark.celeborn.master.endpoints celeborn-master:9097
+spark.shuffle.service.enabled false
+spark.celeborn.client.spark.shuffle.writer hash
+spark.celeborn.client.push.replicate.enabled true
+
+# AQE local shuffle reader is incompatible with remote (Celeborn) shuffle.
+spark.sql.adaptive.enabled true
+spark.sql.adaptive.localShuffleReader.enabled false
