Csaba Ringhofer has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/25043 )
Change subject: IMPALA-15506: Add utility to test Spark with impala minicluster ...................................................................... IMPALA-15506: Add utility to test Spark with impala minicluster Add convenience scripts to test Spark (4.1.3 by default) with the Impala minicluster. Based on similar scripts for Trino: start/stop/run sql with run-spark.sh/kill-spark.sh/spark-sql.sh The official Spark docker image is used unmodified, so there is no image to build. run-spark.sh bind mounts the Iceberg runtime jar and a conf directory assembled from the minicluster configuration, and passes everything the image would otherwise bake in as environment variables. The jar is downloaded from Maven Central on the first run. Both mount sources live under testdata/bin/minicluster_spark and are gitignored. No tests added at this point, the goal is only to help with testing Spark<->Impala interop. The main motivation was GEOMETRY testing, so newest (1.12.0) Iceberg is used in Spark instead of Impala's 1.11.0. Versions used can be overridden with env vars read by run-spark.sh. Change-Id: I7c87024e898f49e5c4afb737463f728210d018c6 Assisted-by: Claude Opus 5 Reviewed-on: http://gerrit.cloudera.org:8080/25043 Reviewed-by: Peter Rozsa <[email protected]> Reviewed-by: Csaba Ringhofer <[email protected]> Tested-by: Csaba Ringhofer <[email protected]> --- A testdata/bin/SPARK-README.md A testdata/bin/kill-spark.sh A testdata/bin/minicluster_spark/.gitignore A testdata/bin/minicluster_spark/spark-defaults-iceberg.conf A testdata/bin/minicluster_spark/spark-defaults.conf A testdata/bin/run-spark.sh A testdata/bin/spark-sql.sh 7 files changed, 383 insertions(+), 0 deletions(-) Approvals: Peter Rozsa: Looks good to me, approved Csaba Ringhofer: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/25043 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I7c87024e898f49e5c4afb737463f728210d018c6 Gerrit-Change-Number: 25043 Gerrit-PatchSet: 5 Gerrit-Owner: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Peter Rozsa <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
