Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/24544 )
Change subject: IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite ...................................................................... IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite Add a custom cluster test suite that exercises Impala <-> Trino interoperability over Apache Iceberg V3 tables, plus the test-infra support to drive and verify Trino from .test files. Test infra: - New TRINO_QUERY / RESULTS .test sections, analogous to HIVE_QUERY / RESULTS. TRINO_QUERY statements run against the Trino container (iceberg catalog, test database as schema); RESULTS verifies the returned rows (with an optional VERIFY_* modifier). - tests/common/trino_cluster.py: TrinoCluster lifecycle helper (start/stop/is_container_running) that drives the Trino CLI inside the 'impala-minicluster-trino' container via 'docker exec', so no new Python dependency is required. Results are formatted into the Impala RESULTS textual convention and compared as opaque strings. start() builds the image (build-trino-docker-image.sh) lazily if it is missing and starts the container, so the Docker cost is paid only by runs that actually execute the interop tests (mirrors the MySQL JDBC tests) and never by data loading, cluster setup or run-all-tests.sh. - run_test_case dispatches TRINO_QUERY to Trino and verifies against RESULTS; run_stmt_in_trino mirrors run_stmt_in_hive. - CustomClusterTestSuite.with_args(run_trino=True): the framework starts the Trino container before the Impala cluster and stops it on teardown (only if it started it), so interop suites no longer hand-roll setup_class/teardown_class. If the Impala setup fails after Trino started, Trino is stopped so a failed setup does not leak the container. - Disable Trino's DELTA_LENGTH_BYTE_ARRAY Parquet encoding for the Iceberg catalog, which Impala cannot read for string/binary columns yet (Trino 482 defaults it on). This is applied server-side via the image's file session property manager (session-property-config.*), setting parquet_writer_delta_length_byte_array_encoding_enabled=false. - Set hive.non-managed-table-writes-enabled=true in the image's hive.properties. Our HMS runs the default metadata transformer, which converts the MANAGED tables Trino requests into EXTERNAL ones (Trino does not advertise Hive ACID write capabilities), and Trino refuses to write into existing external tables by default; enabling this lets the hive-catalog interop test use plain CREATE TABLE + INSERT. Tests (run in run-tests.py's serial phase like other custom cluster tests, skipped only on non-HDFS filesystems, and failing rather than skipping if the Trino container/Docker is unavailable so problems are visible): - tests/custom_cluster/test_iceberg_trino_interop.py (iceberg catalog): - tests/custom_cluster/test_trino_interop.py (hive catalog): a minimal suite over legacy (non-Iceberg) Hive tables Pin the Trino Docker image to trinodb/trino:482 instead of 'latest' so the interop tests are reproducible and not made flaky by future Trino releases (482 is the first release with full Iceberg V3 support). Docs: testdata/bin/TRINO-README.md explains how to run the suites and author TRINO_QUERY / RESULTS sections. Change-Id: I7706244f909f32dff4d390537e71877b0635c661 Assisted-by: Claude Opus 4.8 (1M context) <[email protected]> Reviewed-on: http://gerrit.cloudera.org:8080/24544 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M testdata/bin/TRINO-README.md M testdata/bin/minicluster_trino/Dockerfile M testdata/bin/minicluster_trino/hive.properties A testdata/bin/minicluster_trino/session-property-config.json A testdata/bin/minicluster_trino/session-property-config.properties A testdata/workloads/functional-query/queries/QueryTest/iceberg-trino-interop-default-values.test A testdata/workloads/functional-query/queries/QueryTest/iceberg-trino-interop-delete-dv.test A testdata/workloads/functional-query/queries/QueryTest/iceberg-trino-interop-insert.test M tests/common/custom_cluster_test_suite.py M tests/common/impala_test_suite.py M tests/common/test_result_verifier.py A tests/common/trino_cluster.py A tests/custom_cluster/test_iceberg_trino_interop.py A tests/custom_cluster/test_trino_interop.py M tests/util/test_file_parser.py 15 files changed, 892 insertions(+), 11 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/24544 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I7706244f909f32dff4d390537e71877b0635c661 Gerrit-Change-Number: 24544 Gerrit-PatchSet: 6 Gerrit-Owner: Zoltan Borok-Nagy <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Noemi Pap-Takacs <[email protected]> Gerrit-Reviewer: Peter Rozsa <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
