This is an automated email from the ASF dual-hosted git repository.
voonhous pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/master by this push:
new bcaf2d48e6b7 chore(docker): address PR #18520 review comments for
Spark 4.0.1 stack (#18524)
bcaf2d48e6b7 is described below
commit bcaf2d48e6b77c5b4bb56346adefdad0df25e550
Author: voonhous <[email protected]>
AuthorDate: Wed Aug 26 18:18:42 2026 +0800
chore(docker): address PR #18520 review comments for Spark 4.0.1 stack
(#18524)
* chore(docker): address PR #18520 review comments for Spark 4.0.1 stack
* fix(docker): match hadoop-aws and the AWS SDK bundle to the Hadoop that
Spark bundles
The hadoop-aws jar lands on Spark's classpath next to the hadoop-client the
Spark distribution bundles (3.3.4 for Spark 3.5.x, 3.4.1 for 4.0.x, 3.4.2
for
4.1.x), so key the selection off SPARK_VERSION rather than the cluster
HADOOP_VERSION. hadoop-aws 3.4.x is built against
software.amazon.awssdk:bundle
(SDK v2), not com.amazonaws:aws-java-sdk-bundle, so spark_base now picks the
bundle artifact from the SDK major version and Spark 4 builds pull the v2
version each hadoop-aws release declares.
* fix(docker): add the Spark 4.2 hadoop-aws pairing and fix the spark401
compose files
- build_docker_images.sh: explicit 4.1 and 4.2 arms (Spark 4.2.x bundles
Hadoop 3.5.0,
SDK bundle 2.35.4); unmapped 4.x lines now warn before falling back to
the newest pairing
- drop the 19888 mapping: run_history.sh only starts the YARN timeline
server on 8188
- arm64: same apache/kafka:3.7.2 KRaft broker as amd64 so kafkabroker:29092
has a listener
* fix(docker): key the hadoop-aws fallback off the Spark major and move the
mapred-site guard to base_java11
- a 5.x Spark line fell past 4.* into the Spark 3 pairing while the base
image had already
switched to java17; unmapped 4+ lines now warn and take the newest pairing
- Hadoop 3.x ships mapred-site.xml directly (MAPREDUCE-6875), so the guard
was dead in
base_java17 and missing in base_java11, where --hadoop-version 3.3.4
failed on the bare cp
* fix(docker): ship analyticsaccelerator-s3 alongside hadoop-aws 3.4.2+
hadoop-aws 3.5.0 made the analytics input stream the default
(StreamIntegration.DEFAULT_STREAM_TYPE), so the Spark 4.2 arm and the 4+
fallback failed S3A init without the jar. spark_base now takes
ANALYTICS_ACCELERATOR_VERSION and fetches the jar when it is set; the script
pins 1.2.1 for 3.4.2 (still opt-in there) and 1.3.1 for 3.5.0, the versions
each hadoop-project pom declares. Spark 4.0.x (hadoop-aws 3.4.1) and 3.x
leave
it empty since the stream type does not exist there. The jar declares no
dependencies of its own.
---
docker/README.md | 7 ++
docker/build_docker_images.sh | 93 ++++++++++++++++------
...er-compose_hadoop340_hive313_spark401_amd64.yml | 31 +++++++-
...er-compose_hadoop340_hive313_spark401_arm64.yml | 17 +++-
docker/hoodie/hadoop/base_java11/Dockerfile | 3 +-
docker/hoodie/hadoop/spark_base/Dockerfile | 14 +++-
6 files changed, 133 insertions(+), 32 deletions(-)
diff --git a/docker/README.md b/docker/README.md
index 560d800063e6..5b19e85c4334 100644
--- a/docker/README.md
+++ b/docker/README.md
@@ -43,6 +43,13 @@ Downstream Dockerfiles (`datanode`, `historyserver`,
`hive_base`, `namenode`, `p
`BASE_IMAGE_TAG` build arg (default `java11`). `build_docker_images.sh` sets
it automatically; bare `docker build`
invocations targeting the Java 17 base must pass `--build-arg
BASE_IMAGE_TAG=java17`.
+`spark_base` additionally takes `HADOOP_AWS_VERSION`, `AWS_SDK_VERSION` and
`ANALYTICS_ACCELERATOR_VERSION` for the
+S3A jars it adds to the Spark classpath. `build_docker_images.sh` derives all
three from `--spark-version`, matching
+the Hadoop line each Spark distribution bundles: Spark 4.0.x gets `hadoop-aws`
3.4.1, Spark 4.1.x gets 3.4.2 and
+Spark 4.2.x gets 3.5.0, all with the AWS SDK v2 bundle; Spark 3.x gets 3.3.4
with the SDK v1 bundle. Spark 4.1.x and
+4.2.x also get `analyticsaccelerator-s3` (1.2.1 and 1.3.1), which backs the
S3A analytics input stream that
+hadoop-aws 3.5.0 makes the default; an empty `ANALYTICS_ACCELERATOR_VERSION`
skips that jar.
+
### Docker compose config for the Demo - `/compose`
The `/compose` folder contains the yaml file to compose the Docker environment
for running Hudi Demo.
diff --git a/docker/build_docker_images.sh b/docker/build_docker_images.sh
index bb613a2a5155..e68ce44080c9 100755
--- a/docker/build_docker_images.sh
+++ b/docker/build_docker_images.sh
@@ -54,6 +54,7 @@ done
if [ "$MULTI_ARCH" = true ]; then
DOCKER_PLATFORM='linux/amd64,linux/arm64'
echo "Building multi-arch images (amd64 + arm64)"
+ export BUILDX_EXPERIMENTAL=1
else
ARCHITECTURE=$(uname -m)
case "$ARCHITECTURE" in
@@ -70,9 +71,8 @@ else
esac
export DOCKER_DEFAULT_PLATFORM="$DOCKER_PLATFORM"
fi
-export BUILDX_EXPERIMENTAL=1
# Get the directory of this script for relative paths
-SCRIPT_DIR=$(cd $(dirname "$0") && pwd)
+SCRIPT_DIR=$(cd "$(dirname "$0")" && pwd)
# Determine VERSION_TAG (command line arg or Maven project version)
if [ -n "$VERSION_TAG_ARG" ]; then
@@ -93,7 +93,11 @@ DOCKER_CONTEXT_DIR="hoodie/hadoop"
# Select Java base image based on Spark version (Spark 4.0+ requires Java 17)
SPARK_MAJOR=$(echo "$SPARK_VERSION" | cut -d. -f1)
-if [ "$SPARK_MAJOR" -ge 4 ] 2>/dev/null; then
+if ! [[ "$SPARK_MAJOR" =~ ^[0-9]+$ ]]; then
+ echo "Error: invalid SPARK_VERSION='$SPARK_VERSION'" >&2
+ exit 1
+fi
+if [ "$SPARK_MAJOR" -ge 4 ]; then
BASE_IMAGE_DIR="base_java17"
BASE_JAVA_TAG="java17"
echo "Using Java 17 base image for Spark ${SPARK_VERSION}"
@@ -103,8 +107,51 @@ else
echo "Using Java 11 base image for Spark ${SPARK_VERSION}"
fi
+# Select hadoop-aws/aws-sdk versions from the Hadoop line each Spark
distribution bundles:
+# the jars land on Spark's classpath next to its own hadoop-client, not the
cluster Hadoop.
+# hadoop-aws 3.4+ is built against AWS SDK v2 (software.amazon.awssdk:bundle);
3.3.x uses
+# SDK v1 (com.amazonaws:aws-java-sdk-bundle). spark_base picks the artifact
from the SDK major.
+# hadoop-aws 3.4.2+ also compiles against analyticsaccelerator-s3 for its
analytics input
+# stream: opt-in via fs.s3a.input.stream.type in 3.4.2, the default from 3.5.0
on, so S3A
+# init fails there without the jar. Each arm pins the version its
hadoop-project pom declares.
+SPARK_MAJOR_MINOR=$(echo "$SPARK_VERSION" | cut -d. -f1,2)
+case "$SPARK_MAJOR_MINOR" in
+ 4.0)
+ # Spark 4.0.x bundles Hadoop 3.4.1
+ HADOOP_AWS_VERSION="3.4.1"
+ AWS_SDK_VERSION="2.24.6"
+ ANALYTICS_ACCELERATOR_VERSION="" # no analytics stream type before
hadoop-aws 3.4.2
+ ;;
+ 4.1)
+ # Spark 4.1.x bundles Hadoop 3.4.2
+ HADOOP_AWS_VERSION="3.4.2"
+ AWS_SDK_VERSION="2.29.52"
+ ANALYTICS_ACCELERATOR_VERSION="1.2.1"
+ ;;
+ 4.2)
+ # Spark 4.2.x bundles Hadoop 3.5.0
+ HADOOP_AWS_VERSION="3.5.0"
+ AWS_SDK_VERSION="2.35.4"
+ ANALYTICS_ACCELERATOR_VERSION="1.3.1"
+ ;;
+ *)
+ if [ "$SPARK_MAJOR" -ge 4 ]; then
+ # Unmapped 4+ line (4.3, 5.x, ...): fall back to the newest mapped
pairing and say so,
+ # rather than silently shipping hadoop-aws from an older Hadoop line
than Spark bundles.
+ echo "Warning: no hadoop-aws mapping for Spark ${SPARK_VERSION}; using
the Spark 4.2 pairing" >&2
+ HADOOP_AWS_VERSION="3.5.0"
+ AWS_SDK_VERSION="2.35.4"
+ ANALYTICS_ACCELERATOR_VERSION="1.3.1"
+ else
+ # Spark 3.x bundles Hadoop 3.3.x
+ HADOOP_AWS_VERSION="3.3.4"
+ AWS_SDK_VERSION="1.12.734"
+ ANALYTICS_ACCELERATOR_VERSION=""
+ fi
+ ;;
+esac
+
# List of images to build: "subdir|image_base_name"
-# Each entry: <subdir>|<image_base_name>
DOCKER_IMAGES=(
"${BASE_IMAGE_DIR}|apachehudi/hudi-hadoop_${HADOOP_VERSION}-base-${BASE_JAVA_TAG}"
"datanode|apachehudi/hudi-hadoop_${HADOOP_VERSION}-datanode"
@@ -116,6 +163,13 @@ DOCKER_IMAGES=(
"sparkmaster|apachehudi/hudi-hadoop_${HADOOP_VERSION}-hive_${HIVE_VERSION}-sparkmaster_${SPARK_VERSION}"
"sparkworker|apachehudi/hudi-hadoop_${HADOOP_VERSION}-hive_${HIVE_VERSION}-sparkworker_${SPARK_VERSION}"
)
+# Select docker build command once (MULTI_ARCH doesn't change per image).
+if [ "$MULTI_ARCH" = true ]; then
+ DOCKER_BUILD_CMD=(docker buildx build --platform "$DOCKER_PLATFORM" --push)
+else
+ DOCKER_BUILD_CMD=(docker build)
+fi
+
# Build each Docker image in the list
for IMAGE_CONFIG in "${DOCKER_IMAGES[@]}"; do
# Split config into subdir and image base name
@@ -125,26 +179,17 @@ for IMAGE_CONFIG in "${DOCKER_IMAGES[@]}"; do
TAG_VERSIONED="$IMAGE_BASE:$VERSION_TAG"
echo "Building $IMAGE_CONTEXT as $TAG_LATEST and $TAG_VERSIONED"
# Build the Docker image with both latest and versioned tags
- if [ "$MULTI_ARCH" = true ]; then
- if ! docker buildx build --platform "$DOCKER_PLATFORM" --push \
- --build-arg HADOOP_VERSION=${HADOOP_VERSION} \
- --build-arg SPARK_VERSION=${SPARK_VERSION} \
- --build-arg HIVE_VERSION=${HIVE_VERSION} \
- --build-arg BASE_IMAGE_TAG=${BASE_JAVA_TAG} \
- "$IMAGE_CONTEXT" -t "$TAG_LATEST" -t "$TAG_VERSIONED"; then
- echo "Error: Failed to build docker image for $IMAGE_CONTEXT"
- exit 1
- fi
- else
- if ! docker build \
- --build-arg HADOOP_VERSION=${HADOOP_VERSION} \
- --build-arg SPARK_VERSION=${SPARK_VERSION} \
- --build-arg HIVE_VERSION=${HIVE_VERSION} \
- --build-arg BASE_IMAGE_TAG=${BASE_JAVA_TAG} \
- "$IMAGE_CONTEXT" -t "$TAG_LATEST" -t "$TAG_VERSIONED"; then
- echo "Error: Failed to build docker image for $IMAGE_CONTEXT"
- exit 1
- fi
+ if ! "${DOCKER_BUILD_CMD[@]}" \
+ --build-arg HADOOP_VERSION=${HADOOP_VERSION} \
+ --build-arg SPARK_VERSION=${SPARK_VERSION} \
+ --build-arg HIVE_VERSION=${HIVE_VERSION} \
+ --build-arg BASE_IMAGE_TAG=${BASE_JAVA_TAG} \
+ --build-arg HADOOP_AWS_VERSION=${HADOOP_AWS_VERSION} \
+ --build-arg AWS_SDK_VERSION=${AWS_SDK_VERSION} \
+ --build-arg ANALYTICS_ACCELERATOR_VERSION=${ANALYTICS_ACCELERATOR_VERSION}
\
+ "$IMAGE_CONTEXT" -t "$TAG_LATEST" -t "$TAG_VERSIONED"; then
+ echo "Error: Failed to build docker image for $IMAGE_CONTEXT"
+ exit 1
fi
done
diff --git a/docker/compose/docker-compose_hadoop340_hive313_spark401_amd64.yml
b/docker/compose/docker-compose_hadoop340_hive313_spark401_amd64.yml
index 483ce92e0c03..638f3eee2663 100644
--- a/docker/compose/docker-compose_hadoop340_hive313_spark401_amd64.yml
+++ b/docker/compose/docker-compose_hadoop340_hive313_spark401_amd64.yml
@@ -17,6 +17,7 @@ services:
namenode:
image: apachehudi/hudi-hadoop_3.4.0-namenode:latest
+ platform: linux/amd64
hostname: namenode
container_name: namenode
environment:
@@ -35,6 +36,7 @@ services:
datanode1:
image: apachehudi/hudi-hadoop_3.4.0-datanode:latest
+ platform: linux/amd64
container_name: datanode1
hostname: datanode1
environment:
@@ -58,6 +60,7 @@ services:
historyserver:
image: apachehudi/hudi-hadoop_3.4.0-history:latest
+ platform: linux/amd64
hostname: historyserver
container_name: historyserver
environment:
@@ -80,6 +83,7 @@ services:
hive-metastore-postgresql:
image: bde2020/hive-metastore-postgresql:3.1.0
+ platform: linux/amd64
volumes:
- hive-metastore-postgresql:/var/lib/postgresql
hostname: hive-metastore-postgresql
@@ -87,6 +91,7 @@ services:
hivemetastore:
image: apachehudi/hudi-hadoop_3.4.0-hive_3.1.3:latest
+ platform: linux/amd64
hostname: hivemetastore
container_name: hivemetastore
links:
@@ -110,6 +115,7 @@ services:
hiveserver:
image: apachehudi/hudi-hadoop_3.4.0-hive_3.1.3:latest
+ platform: linux/amd64
hostname: hiveserver
container_name: hiveserver
env_file:
@@ -130,6 +136,7 @@ services:
zookeeper:
image: 'bitnamilegacy/zookeeper:3.6.4'
+ platform: linux/amd64
hostname: zookeeper
container_name: zookeeper
ports:
@@ -138,17 +145,30 @@ services:
- ALLOW_ANONYMOUS_LOGIN=yes
kafka:
- image: 'bitnamilegacy/kafka:3.4.1'
+ image: 'apache/kafka:3.7.2'
+ platform: linux/amd64
hostname: kafkabroker
container_name: kafkabroker
ports:
- "9092:9092"
environment:
- - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
- - ALLOW_PLAINTEXT_LISTENER=yes
+ - KAFKA_NODE_ID=1
+ - KAFKA_PROCESS_ROLES=broker,controller
+ -
KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:29092,CONTROLLER://0.0.0.0:9093,PLAINTEXT_HOST://0.0.0.0:9092
+ -
KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafkabroker:29092,PLAINTEXT_HOST://localhost:9092
+ - KAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLER
+ -
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
+ - KAFKA_INTER_BROKER_LISTENER_NAME=PLAINTEXT
+ - KAFKA_CONTROLLER_QUORUM_VOTERS=1@kafkabroker:9093
+ - KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
+ - KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR=1
+ - KAFKA_TRANSACTION_STATE_LOG_MIN_ISR=1
+ - KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS=0
+ - KAFKA_NUM_PARTITIONS=3
sparkmaster:
image: apachehudi/hudi-hadoop_3.4.0-hive_3.1.3-sparkmaster_4.0.1:latest
+ platform: linux/amd64
hostname: sparkmaster
container_name: sparkmaster
env_file:
@@ -170,6 +190,7 @@ services:
spark-worker-1:
image: apachehudi/hudi-hadoop_3.4.0-hive_3.1.3-sparkworker_4.0.1:latest
+ platform: linux/amd64
hostname: spark-worker-1
container_name: spark-worker-1
env_file:
@@ -188,6 +209,7 @@ services:
adhoc-1:
image: apachehudi/hudi-hadoop_3.4.0-hive_3.1.3-sparkadhoc_4.0.1:latest
+ platform: linux/amd64
hostname: adhoc-1
container_name: adhoc-1
env_file:
@@ -208,6 +230,7 @@ services:
adhoc-2:
image: apachehudi/hudi-hadoop_3.4.0-hive_3.1.3-sparkadhoc_4.0.1:latest
+ platform: linux/amd64
hostname: adhoc-2
container_name: adhoc-2
env_file:
@@ -226,6 +249,7 @@ services:
minio:
image: 'minio/minio:latest'
+ platform: linux/amd64
hostname: minio
container_name: minio
ports:
@@ -241,6 +265,7 @@ services:
mc:
image: minio/mc
+ platform: linux/amd64
container_name: mc
entrypoint: >
/bin/sh -c "
diff --git a/docker/compose/docker-compose_hadoop340_hive313_spark401_arm64.yml
b/docker/compose/docker-compose_hadoop340_hive313_spark401_arm64.yml
index 483ce92e0c03..1ee2bb90de86 100644
--- a/docker/compose/docker-compose_hadoop340_hive313_spark401_arm64.yml
+++ b/docker/compose/docker-compose_hadoop340_hive313_spark401_arm64.yml
@@ -138,14 +138,25 @@ services:
- ALLOW_ANONYMOUS_LOGIN=yes
kafka:
- image: 'bitnamilegacy/kafka:3.4.1'
+ image: 'apache/kafka:3.7.2'
hostname: kafkabroker
container_name: kafkabroker
ports:
- "9092:9092"
environment:
- - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
- - ALLOW_PLAINTEXT_LISTENER=yes
+ - KAFKA_NODE_ID=1
+ - KAFKA_PROCESS_ROLES=broker,controller
+ -
KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:29092,CONTROLLER://0.0.0.0:9093,PLAINTEXT_HOST://0.0.0.0:9092
+ -
KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafkabroker:29092,PLAINTEXT_HOST://localhost:9092
+ - KAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLER
+ -
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
+ - KAFKA_INTER_BROKER_LISTENER_NAME=PLAINTEXT
+ - KAFKA_CONTROLLER_QUORUM_VOTERS=1@kafkabroker:9093
+ - KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
+ - KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR=1
+ - KAFKA_TRANSACTION_STATE_LOG_MIN_ISR=1
+ - KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS=0
+ - KAFKA_NUM_PARTITIONS=3
sparkmaster:
image: apachehudi/hudi-hadoop_3.4.0-hive_3.1.3-sparkmaster_4.0.1:latest
diff --git a/docker/hoodie/hadoop/base_java11/Dockerfile
b/docker/hoodie/hadoop/base_java11/Dockerfile
index 121f94cd09ea..7ec8356846d9 100644
--- a/docker/hoodie/hadoop/base_java11/Dockerfile
+++ b/docker/hoodie/hadoop/base_java11/Dockerfile
@@ -36,7 +36,8 @@ RUN set -x \
&& tar -xvf /tmp/hadoop.tar.gz -C /opt/ \
&& rm /tmp/hadoop.tar.gz* \
&& ln -s /opt/hadoop-$HADOOP_VERSION/etc/hadoop /etc/hadoop \
- && cp /etc/hadoop/mapred-site.xml.template /etc/hadoop/mapred-site.xml \
+ # Hadoop 2.x ships only mapred-site.xml.template; 3.x ships
mapred-site.xml directly
+ && if [ -f /etc/hadoop/mapred-site.xml.template ]; then cp
/etc/hadoop/mapred-site.xml.template /etc/hadoop/mapred-site.xml; fi \
&& mkdir /hadoop-data
ENV HADOOP_PREFIX=/opt/hadoop-$HADOOP_VERSION
diff --git a/docker/hoodie/hadoop/spark_base/Dockerfile
b/docker/hoodie/hadoop/spark_base/Dockerfile
index 4a064971536e..cfbf606db95c 100644
--- a/docker/hoodie/hadoop/spark_base/Dockerfile
+++ b/docker/hoodie/hadoop/spark_base/Dockerfile
@@ -66,9 +66,21 @@ ENV SPARK_BLOCKMGR_PORT=5003
EXPOSE $SPARK_DRIVER_PORT $SPARK_UI_PORT $SPARK_BLOCKMGR_PORT
# Without this spark-shell fails - Download if it is not already there in
$SPARK_INSTALL
+# hadoop-aws 3.4+ is built against AWS SDK v2 (software.amazon.awssdk:bundle);
older releases
+# use SDK v1 (com.amazonaws:aws-java-sdk-bundle). The artifact is picked from
the SDK major version.
+# hadoop-aws 3.4.2+ also needs analyticsaccelerator-s3 for its analytics input
stream, which 3.5.0
+# made the default; pass the version hadoop-project declares, or leave it
empty to skip the jar.
ARG HADOOP_AWS_VERSION=3.3.4
ARG AWS_SDK_VERSION=1.12.734
+ARG ANALYTICS_ACCELERATOR_VERSION=""
RUN wget -nc -q -O "${SPARK_INSTALL}/jars/jersey-bundle-1.19.4.jar"
"https://repo1.maven.org/maven2/com/sun/jersey/jersey-bundle/1.19.4/jersey-bundle-1.19.4.jar";
&& \
wget -O "${SPARK_INSTALL}/jars/hadoop-aws-${HADOOP_AWS_VERSION}.jar"
https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/${HADOOP_AWS_VERSION}/hadoop-aws-${HADOOP_AWS_VERSION}.jar
&& \
- wget -O "${SPARK_INSTALL}/jars/aws-java-sdk-bundle-${AWS_SDK_VERSION}.jar"
https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/${AWS_SDK_VERSION}/aws-java-sdk-bundle-${AWS_SDK_VERSION}.jar
+ if [ "${AWS_SDK_VERSION%%.*}" -ge 2 ]; then \
+ wget -O "${SPARK_INSTALL}/jars/bundle-${AWS_SDK_VERSION}.jar"
https://repo1.maven.org/maven2/software/amazon/awssdk/bundle/${AWS_SDK_VERSION}/bundle-${AWS_SDK_VERSION}.jar;
\
+ else \
+ wget -O
"${SPARK_INSTALL}/jars/aws-java-sdk-bundle-${AWS_SDK_VERSION}.jar"
https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/${AWS_SDK_VERSION}/aws-java-sdk-bundle-${AWS_SDK_VERSION}.jar;
\
+ fi && \
+ if [ -n "${ANALYTICS_ACCELERATOR_VERSION}" ]; then \
+ wget -O
"${SPARK_INSTALL}/jars/analyticsaccelerator-s3-${ANALYTICS_ACCELERATOR_VERSION}.jar"
https://repo1.maven.org/maven2/software/amazon/s3/analyticsaccelerator/analyticsaccelerator-s3/${ANALYTICS_ACCELERATOR_VERSION}/analyticsaccelerator-s3-${ANALYTICS_ACCELERATOR_VERSION}.jar;
\
+ fi
