yew1eb commented on code in PR #3768: URL: https://github.com/apache/celeborn/pull/3768#discussion_r3823576954
########## .gitignore: ########## @@ -76,3 +76,6 @@ spark-warehouse/ logs pids tmp/ + +# Spark client shaded jar copied into the docker build context by build-image.sh +docker/spark/*.jar Review Comment: Done ########## docker/build-image.sh: ########## @@ -0,0 +1,91 @@ +#!/usr/bin/env bash +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# Build the local Celeborn docker image (celeborn:dev) for docker-compose. +# +# Two steps: +# 1. ./build/make-distribution.sh -> produces dist/ (bin/sbin/conf/jars/... +# plus dist/docker/Dockerfile copied by the distribution script itself) +# 2. docker build -> builds the image with context = dist/ +# (the Dockerfile COPYs bin/sbin/... relative to the context root) +# +# The image is NOT built via the compose `build:` key, because make-distribution +# must run before docker build and dist/ may not exist yet when invoking +# `docker compose up`. +# +# When a Spark client profile is passed (e.g. -Pspark-3.5), make-distribution +# also drops the client shaded jar at dist/spark/. Step 3 then builds an +# additional Spark image (celeborn-spark:dev) on top of apache/spark with that +# jar baked in, so docker-compose can run an end-to-end Spark cluster whose +# shuffle is served by Celeborn. +# +# Usage: +# ./docker/build-image.sh # core master/worker/cli +# ./docker/build-image.sh -Pspark-3.5 # also build Spark client + image +# CELEBORN_IMAGE_TAG=celeborn:dev ./docker/build-image.sh +# CELEBORN_SPARK_IMAGE_TAG=celeborn-spark:dev ./docker/build-image.sh -Pspark-3.5 +# CELEBORN_SPARK_BASE_TAG=3.5.0 ./docker/build-image.sh -Pspark-3.5 +# +# Requirements: docker on PATH; build/mvn is bundled in the repo. + +set -euo pipefail + +# Resolve repo root (parent of this script's directory). +cd "$(dirname "$0")/.." + +IMAGE_TAG="${CELEBORN_IMAGE_TAG:-celeborn:dev}" +SPARK_IMAGE_TAG="${CELEBORN_SPARK_IMAGE_TAG:-celeborn-spark:dev}" +# Base apache/spark image tag. Renamed away from SPARK_IMAGE_TAG because that +# env var is commonly exported by Spark's own tooling and would otherwise leak +# in here as the *output* image name (e.g. celeborn-spark:dev), producing an +# invalid "apache/spark:celeborn-spark:dev" FROM reference. +BASE_SPARK_IMAGE_TAG="${CELEBORN_SPARK_BASE_TAG:-3.5.0}" + +echo "==> Building distribution (./build/make-distribution.sh $*) ..." +./build/make-distribution.sh "$@" Review Comment: Done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
