voonhous commented on code in PR #18524:
URL: https://github.com/apache/hudi/pull/18524#discussion_r3860615372
##########
docker/build_docker_images.sh:
##########
@@ -103,8 +107,31 @@ else
echo "Using Java 11 base image for Spark ${SPARK_VERSION}"
fi
+# Select hadoop-aws/aws-sdk versions from the Hadoop line each Spark
distribution bundles:
+# the jars land on Spark's classpath next to its own hadoop-client, not the
cluster Hadoop.
+# hadoop-aws 3.4.x is built against AWS SDK v2
(software.amazon.awssdk:bundle); 3.3.x uses
+# SDK v1 (com.amazonaws:aws-java-sdk-bundle). spark_base picks the artifact
from the SDK major.
+SPARK_MAJOR_MINOR=$(echo "$SPARK_VERSION" | cut -d. -f1,2)
+case "$SPARK_MAJOR_MINOR" in
+ 4.0)
+ # Spark 4.0.x bundles Hadoop 3.4.1
+ HADOOP_AWS_VERSION="3.4.1"
+ AWS_SDK_VERSION="2.24.6"
+ ;;
+ 4.*)
Review Comment:
Not deliberate -- Hudi already ships a hudi-spark4.2.x module, so 4.2 needs
its own arm. Added `4.2)` with hadoop-aws 3.5.0 / SDK bundle 2.35.4 in 9852795:
Spark v4.2.0's pom pins hadoop.version 3.5.0, and Hadoop 3.5.0 pins
aws-java-sdk-v2 2.35.4. 4.1 got its own arm too, and the `4.*` catch-all now
warns on stderr before falling back to the newest pairing, so the next unmapped
line is loud rather than silent.
##########
docker/compose/docker-compose_hadoop340_hive313_spark401_amd64.yml:
##########
@@ -68,6 +71,7 @@ services:
- "namenode"
ports:
- "8188:8188"
+ - "19888:19888"
Review Comment:
Dropped from both compose files in 9852795. run_history.sh only runs `yarn
historyserver` (timeline server on 8188); nothing in the image starts the MR
JobHistoryServer and the demo does not need one. The line was copied from the
hadoop334 amd64 compose, which carries the same dead mapping -- left that for a
separate cleanup since this PR does not touch it.
##########
docker/compose/docker-compose_hadoop340_hive313_spark401_amd64.yml:
##########
@@ -138,17 +146,30 @@ services:
- ALLOW_ANONYMOUS_LOGIN=yes
kafka:
- image: 'bitnamilegacy/kafka:3.4.1'
+ image: 'apache/kafka:3.7.2'
Review Comment:
Swapped arm64 to the same apache/kafka:3.7.2 KRaft block in 9852795, minus
the platform pin since the image publishes a native arm64 build. Without it
kafkabroker:29092 from kafka-source.properties had no listener on arm64
(bitnami only listens on 9092).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
