wombatu-kun commented on code in PR #18524:
URL: https://github.com/apache/hudi/pull/18524#discussion_r3859110033
##########
docker/build_docker_images.sh:
##########
@@ -103,8 +107,31 @@ else
echo "Using Java 11 base image for Spark ${SPARK_VERSION}"
fi
+# Select hadoop-aws/aws-sdk versions from the Hadoop line each Spark
distribution bundles:
+# the jars land on Spark's classpath next to its own hadoop-client, not the
cluster Hadoop.
+# hadoop-aws 3.4.x is built against AWS SDK v2
(software.amazon.awssdk:bundle); 3.3.x uses
+# SDK v1 (com.amazonaws:aws-java-sdk-bundle). spark_base picks the artifact
from the SDK major.
+SPARK_MAJOR_MINOR=$(echo "$SPARK_VERSION" | cut -d. -f1,2)
+case "$SPARK_MAJOR_MINOR" in
+ 4.0)
+ # Spark 4.0.x bundles Hadoop 3.4.1
+ HADOOP_AWS_VERSION="3.4.1"
+ AWS_SDK_VERSION="2.24.6"
+ ;;
+ 4.*)
Review Comment:
Spark 4.2.x lands in this arm but bundles Hadoop 3.5.0, so a 4.2 build would
ship hadoop-aws 3.4.2 next to a 3.5.0 client. Worth a `4.2)` arm with
hadoop-aws 3.5.0 and AWS SDK 2.35.4, or is 4.2 deliberately out of scope here?
##########
docker/compose/docker-compose_hadoop340_hive313_spark401_amd64.yml:
##########
@@ -68,6 +71,7 @@ services:
- "namenode"
ports:
- "8188:8188"
+ - "19888:19888"
Review Comment:
run_history.sh starts the YARN ApplicationHistoryServer, which serves its UI
on 8188; nothing in the history image binds 19888, so both compose files now
publish a closed port. Should the MapReduce JobHistoryServer be started here as
well, or is the mapping better dropped?
##########
docker/compose/docker-compose_hadoop340_hive313_spark401_amd64.yml:
##########
@@ -138,17 +146,30 @@ services:
- ALLOW_ANONYMOUS_LOGIN=yes
kafka:
- image: 'bitnamilegacy/kafka:3.4.1'
+ image: 'apache/kafka:3.7.2'
Review Comment:
The arm64 file keeps the ZooKeeper-backed bitnami broker, so
kafka-source.properties' bootstrap of kafkabroker:29092 has no matching
listener there once amd64 moves to KRaft. Should arm64 get the same swap, or is
it intentionally left for the 4.1.x move (follow-up, not a blocker)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
