yew1eb commented on code in PR #3768: URL: https://github.com/apache/celeborn/pull/3768#discussion_r3823579985
########## docker/build-image.sh: ########## @@ -0,0 +1,91 @@ +#!/usr/bin/env bash +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# Build the local Celeborn docker image (celeborn:dev) for docker-compose. +# +# Two steps: +# 1. ./build/make-distribution.sh -> produces dist/ (bin/sbin/conf/jars/... +# plus dist/docker/Dockerfile copied by the distribution script itself) +# 2. docker build -> builds the image with context = dist/ +# (the Dockerfile COPYs bin/sbin/... relative to the context root) +# +# The image is NOT built via the compose `build:` key, because make-distribution +# must run before docker build and dist/ may not exist yet when invoking +# `docker compose up`. +# +# When a Spark client profile is passed (e.g. -Pspark-3.5), make-distribution +# also drops the client shaded jar at dist/spark/. Step 3 then builds an +# additional Spark image (celeborn-spark:dev) on top of apache/spark with that +# jar baked in, so docker-compose can run an end-to-end Spark cluster whose +# shuffle is served by Celeborn. +# +# Usage: +# ./docker/build-image.sh # core master/worker/cli +# ./docker/build-image.sh -Pspark-3.5 # also build Spark client + image +# CELEBORN_IMAGE_TAG=celeborn:dev ./docker/build-image.sh +# CELEBORN_SPARK_IMAGE_TAG=celeborn-spark:dev ./docker/build-image.sh -Pspark-3.5 +# CELEBORN_SPARK_BASE_TAG=3.5.0 ./docker/build-image.sh -Pspark-3.5 +# +# Requirements: docker on PATH; build/mvn is bundled in the repo. + +set -euo pipefail + +# Resolve repo root (parent of this script's directory). +cd "$(dirname "$0")/.." + +IMAGE_TAG="${CELEBORN_IMAGE_TAG:-celeborn:dev}" +SPARK_IMAGE_TAG="${CELEBORN_SPARK_IMAGE_TAG:-celeborn-spark:dev}" +# Base apache/spark image tag. Renamed away from SPARK_IMAGE_TAG because that +# env var is commonly exported by Spark's own tooling and would otherwise leak +# in here as the *output* image name (e.g. celeborn-spark:dev), producing an +# invalid "apache/spark:celeborn-spark:dev" FROM reference. +BASE_SPARK_IMAGE_TAG="${CELEBORN_SPARK_BASE_TAG:-3.5.0}" + +echo "==> Building distribution (./build/make-distribution.sh $*) ..." +./build/make-distribution.sh "$@" + +echo "==> Building docker image ${IMAGE_TAG} from dist/ ..." +if [ ! -f dist/docker/Dockerfile ]; then + echo "ERROR: dist/docker/Dockerfile not found. make-distribution may have failed." >&2 + exit 1 +fi +docker build -t "${IMAGE_TAG}" -f dist/docker/Dockerfile dist/ + +# --- Spark image (optional) --------------------------------------------------- +# Only build it when make-distribution produced a Spark client shaded jar, which +# happens when a -Pspark-3.x profile was passed. If absent, skip gracefully so +# the Celeborn-only workflow keeps working. +SPARK_CLIENT_JAR="$(ls dist/spark/celeborn-client-spark-3-shaded_*.jar 2>/dev/null || true)" Review Comment: Supported ########## docker/README.md: ########## @@ -0,0 +1,107 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# Celeborn Local Cluster (docker-compose) + +A local Celeborn cluster — 1 master (HA off) + 3 workers — for development and +debugging, optionally accompanied by an example Spark cluster (1 master + 2 +workers) whose shuffle data is served by Celeborn, so the whole stack can be +exercised end-to-end. + +## Prerequisites + +- Docker (daemon running) and the `docker compose` plugin. +- The repo bundles `build/mvn` used by `make-distribution.sh`. No system Maven + required, but a JDK is needed to run the build. + +## 1. Build the images + +```bash +# Celeborn only (no Spark example cluster): +./docker/build-image.sh + +# Also build the Spark example image (celeborn-spark:dev): +./docker/build-image.sh -Pspark-3.5 +``` + +This runs `./build/make-distribution.sh` (producing `dist/`) and then: + +- `docker build -t celeborn:dev -f dist/docker/Dockerfile dist/` — the + Celeborn master/worker image, always built. +- If a Spark client profile was passed (e.g. `-Pspark-3.5`), the distribution + also produces `dist/spark/celeborn-client-spark-3-shaded_*.jar`; + `build-image.sh` copies it into `docker/spark/` and runs + `docker build -t celeborn-spark:dev docker/spark/` — a Spark image with the + Celeborn client jar baked in. Without a Spark profile, the Spark image is + skipped and only the Celeborn cluster is available. + +## 2. Start the cluster + +```bash +docker compose -f docker/docker-compose.yaml up -d +docker compose -f docker/docker-compose.yaml ps +``` + +The Celeborn master has a healthcheck; the 3 workers start once the master is +healthy and register with it. When `celeborn-spark:dev` is present, the Spark +master and 2 workers start after the Celeborn master is healthy. + +## 3. Verify + +```bash +# Master HTTP (expect HTTP 200) +curl -s -o /dev/null -w '%{http_code}\n' http://localhost:9098/metrics/json + +# Tail logs; look for "Registered worker" on the master +docker compose -f docker/docker-compose.yaml logs -f celeborn-master +``` + +## 4. Run a Spark job that shuffles through Celeborn + +Spark is pre-wired to use Celeborn via `docker/spark/conf/spark-defaults.conf` +(`spark.shuffle.manager=...CelebornShuffleManager`, master endpoints +`celeborn-master:9097`), so no extra `--conf` flags are needed. Submit Spark's +built-in `GroupByTest`, which forces a shuffle: + +```bash +docker exec celeborn-spark-master /opt/spark/bin/spark-submit \ + --master spark://celeborn-spark-master:7077 \ + --class org.apache.spark.examples.GroupByTest \ + /opt/spark/examples/jars/spark-examples_2.12-3.5.0.jar 10 100 1000 10 Review Comment: Done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
