wombatu-kun commented on code in PR #18524:
URL: https://github.com/apache/hudi/pull/18524#discussion_r3860767022
##########
docker/hoodie/hadoop/base_java17/Dockerfile:
##########
@@ -47,7 +47,9 @@ RUN DEBIAN_FRONTEND=noninteractive apt-get -yq update \
&& mkdir /hadoop-data
COPY --from=hadoop-builder /opt/hadoop-${HADOOP_VERSION}
/opt/hadoop-${HADOOP_VERSION}
-RUN ln -s /opt/hadoop-${HADOOP_VERSION}/etc/hadoop /etc/hadoop
+# Hadoop 3.4.x ships mapred-site.xml directly; older releases only ship the
.template.
Review Comment:
Hadoop has shipped `mapred-site.xml` directly since 3.0 (MAPREDUCE-6875
renamed the template), so this branch never fires on a 3.x tarball, while
`base_java11` still does the same copy unguarded and fails the
`--hadoop-version 3.3.4` build the README documents. Should the guard go to
`base_java11` instead, or is that out of scope here?
##########
docker/build_docker_images.sh:
##########
@@ -103,8 +107,44 @@ else
echo "Using Java 11 base image for Spark ${SPARK_VERSION}"
fi
+# Select hadoop-aws/aws-sdk versions from the Hadoop line each Spark
distribution bundles:
+# the jars land on Spark's classpath next to its own hadoop-client, not the
cluster Hadoop.
+# hadoop-aws 3.4+ is built against AWS SDK v2 (software.amazon.awssdk:bundle);
3.3.x uses
+# SDK v1 (com.amazonaws:aws-java-sdk-bundle). spark_base picks the artifact
from the SDK major.
+# The opt-in analytics stream type of hadoop-aws 3.4.2+ also needs
analyticsaccelerator-s3,
+# not shipped here.
+SPARK_MAJOR_MINOR=$(echo "$SPARK_VERSION" | cut -d. -f1,2)
+case "$SPARK_MAJOR_MINOR" in
+ 4.0)
+ # Spark 4.0.x bundles Hadoop 3.4.1
+ HADOOP_AWS_VERSION="3.4.1"
+ AWS_SDK_VERSION="2.24.6"
+ ;;
+ 4.1)
+ # Spark 4.1.x bundles Hadoop 3.4.2
+ HADOOP_AWS_VERSION="3.4.2"
+ AWS_SDK_VERSION="2.29.52"
+ ;;
+ 4.2)
Review Comment:
The summary still says both compose files expose 19888 and stops the
hadoop-aws mapping at Spark 4.1, neither of which matches the latest commit.
Worth refreshing that section and the image-growth estimate under Impact.
##########
docker/build_docker_images.sh:
##########
@@ -103,8 +107,44 @@ else
echo "Using Java 11 base image for Spark ${SPARK_VERSION}"
fi
+# Select hadoop-aws/aws-sdk versions from the Hadoop line each Spark
distribution bundles:
+# the jars land on Spark's classpath next to its own hadoop-client, not the
cluster Hadoop.
+# hadoop-aws 3.4+ is built against AWS SDK v2 (software.amazon.awssdk:bundle);
3.3.x uses
+# SDK v1 (com.amazonaws:aws-java-sdk-bundle). spark_base picks the artifact
from the SDK major.
+# The opt-in analytics stream type of hadoop-aws 3.4.2+ also needs
analyticsaccelerator-s3,
+# not shipped here.
+SPARK_MAJOR_MINOR=$(echo "$SPARK_VERSION" | cut -d. -f1,2)
+case "$SPARK_MAJOR_MINOR" in
+ 4.0)
+ # Spark 4.0.x bundles Hadoop 3.4.1
+ HADOOP_AWS_VERSION="3.4.1"
+ AWS_SDK_VERSION="2.24.6"
+ ;;
+ 4.1)
+ # Spark 4.1.x bundles Hadoop 3.4.2
+ HADOOP_AWS_VERSION="3.4.2"
+ AWS_SDK_VERSION="2.29.52"
+ ;;
+ 4.2)
+ # Spark 4.2.x bundles Hadoop 3.5.0
+ HADOOP_AWS_VERSION="3.5.0"
+ AWS_SDK_VERSION="2.35.4"
+ ;;
+ 4.*)
+ # Unmapped 4.x line: fall back to the newest mapped pairing and say so,
rather than
+ # silently shipping hadoop-aws from an older Hadoop line than the one
Spark bundles.
+ echo "Warning: no hadoop-aws mapping for Spark ${SPARK_VERSION}; using the
Spark 4.2 pairing" >&2
+ HADOOP_AWS_VERSION="3.5.0"
+ AWS_SDK_VERSION="2.35.4"
+ ;;
+ *)
Review Comment:
`SPARK_MAJOR_MINOR` for a 5.x line falls past `4.*` into the `*)` arm and
silently gets hadoop-aws 3.3.4 with the SDK v1 bundle, while the base image
still switches to java17 on major >= 4. Worth keying the warn arm off
`SPARK_MAJOR` >= 4 rather than the literal `4.*` glob?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
