Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24544
Change subject: IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite ...................................................................... IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite Add a custom cluster test suite that exercises Impala <-> Trino interoperability over Apache Iceberg V3 tables, plus the test-infra support to drive and verify Trino from .test files. Test infra: - New TRINO_QUERY / TRINO_RESULTS .test sections, analogous to HIVE_QUERY / RESULTS. TRINO_QUERY statements run against the Trino container (iceberg catalog, test database as schema); TRINO_RESULTS verifies the returned rows (with an optional VERIFY_* modifier). - tests/common/trino_cluster.py: TrinoCluster lifecycle helper (start/stop/is_container_running) that drives the Trino CLI inside the 'impala-minicluster-trino' container via 'docker exec', so no new Python dependency is required. Results are formatted into the Impala RESULTS textual convention and compared as opaque strings. start() builds the image (build-trino-docker-image.sh) lazily if it is missing and starts the container, so the Docker cost is paid only by runs that actually execute the interop tests (mirrors the MySQL JDBC tests) and never by data loading, cluster setup or run-all-tests.sh. start() is guarded by an inter-process file lock and the container is not stopped from an individual pytest-xdist worker, so the suite is safe to run concurrently (e.g. -n3). - run_test_case dispatches TRINO_QUERY to Trino and verifies against TRINO_RESULTS; run_stmt_in_trino mirrors run_stmt_in_hive. - Disable Trino's DELTA_LENGTH_BYTE_ARRAY Parquet encoding for the Iceberg catalog, which Impala cannot read for string/binary columns yet (Trino 482 defaults it on). This is applied server-side via the image's file session property manager (session-property-config.*), setting parquet_writer_delta_length_byte_array_encoding_enabled=false. Tests (tests/custom_cluster/test_iceberg_trino_interop.py; run in the core exploration strategy, skipped only on non-HDFS filesystems, and failing rather than skipping if the Trino container/Docker is unavailable so problems are visible): - INSERT / SELECT round-trips (both directions), format-version check. - DELETE / UPDATE / MERGE materialized as Puffin deletion vectors (both directions), asserting the delete files are Puffin DVs via Iceberg "$files" metadata. - Column default values written by one engine and read by the other. Pin the Trino Docker image to trinodb/trino:482 instead of 'latest' so the interop tests are reproducible and not made flaky by future Trino releases (482 is the first release with full Iceberg V3 support). Docs: testdata/bin/TRINO-README.md explains how to run the suite and author TRINO_QUERY / TRINO_RESULTS sections. Change-Id: I7706244f909f32dff4d390537e71877b0635c661 Assisted-by: Claude Opus 4.8 (1M context) <[email protected]> --- M testdata/bin/TRINO-README.md M testdata/bin/minicluster_trino/Dockerfile A testdata/bin/minicluster_trino/session-property-config.json A testdata/bin/minicluster_trino/session-property-config.properties A testdata/workloads/functional-query/queries/QueryTest/iceberg-trino-interop-default-values.test A testdata/workloads/functional-query/queries/QueryTest/iceberg-trino-interop-delete-dv.test A testdata/workloads/functional-query/queries/QueryTest/iceberg-trino-interop-insert.test M tests/common/impala_test_suite.py M tests/common/test_result_verifier.py A tests/common/trino_cluster.py A tests/custom_cluster/test_iceberg_trino_interop.py M tests/util/test_file_parser.py 12 files changed, 715 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/44/24544/1 -- To view, visit http://gerrit.cloudera.org:8080/24544 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I7706244f909f32dff4d390537e71877b0635c661 Gerrit-Change-Number: 24544 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy <[email protected]>
