Hello Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24544
to look at the new patch set (#2).
Change subject: IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom
cluster test suite
......................................................................
IMPALA-12422: Add Impala/Trino Iceberg V3 interop custom cluster test suite
Add a custom cluster test suite that exercises Impala <-> Trino
interoperability over Apache Iceberg V3 tables, plus the test-infra
support to drive and verify Trino from .test files.
Test infra:
- New TRINO_QUERY / RESULTS .test sections, analogous to
HIVE_QUERY / RESULTS. TRINO_QUERY statements run against the Trino
container (iceberg catalog, test database as schema); RESULTS
verifies the returned rows (with an optional VERIFY_* modifier).
- tests/common/trino_cluster.py: TrinoCluster lifecycle helper
(start/stop/is_container_running) that drives the Trino CLI inside the
'impala-minicluster-trino' container via 'docker exec', so no new
Python dependency is required. Results are formatted into the Impala
RESULTS textual convention and compared as opaque strings. start()
builds the image (build-trino-docker-image.sh) lazily if it is missing
and starts the container, so the Docker cost is paid only by runs that
actually execute the interop tests (mirrors the MySQL JDBC tests) and
never by data loading, cluster setup or run-all-tests.sh. start() is
guarded by an inter-process file lock and the container is not stopped
from an individual pytest-xdist worker, so the suite is safe to run
concurrently (e.g. -n3).
- run_test_case dispatches TRINO_QUERY to Trino and verifies against
RESULTS; run_stmt_in_trino mirrors run_stmt_in_hive.
- CustomClusterTestSuite.with_args(run_trino=True): the framework starts
the Trino container before the Impala cluster and stops it on teardown
(only if it started it, and never mid-run under pytest-xdist), so
interop suites no longer hand-roll setup_class/teardown_class. If the
Impala setup fails after Trino started, Trino is stopped so a failed
setup does not leak the container.
- Disable Trino's DELTA_LENGTH_BYTE_ARRAY Parquet encoding for the
Iceberg catalog, which Impala cannot read for string/binary columns
yet (Trino 482 defaults it on). This is applied server-side via the
image's file session property manager (session-property-config.*),
setting parquet_writer_delta_length_byte_array_encoding_enabled=false.
- Set hive.non-managed-table-writes-enabled=true in the image's
hive.properties. Our HMS runs the default metadata transformer, which
converts the MANAGED tables Trino requests into EXTERNAL ones (Trino
does not advertise Hive ACID write capabilities), and Trino refuses to
write into existing external tables by default; enabling this lets the
hive-catalog interop test use plain CREATE TABLE + INSERT.
Tests (run in the core exploration strategy, skipped only on non-HDFS
filesystems, and failing rather than skipping if the Trino
container/Docker is unavailable so problems are visible):
- tests/custom_cluster/test_iceberg_trino_interop.py (iceberg catalog):
- tests/custom_cluster/test_trino_interop.py (hive catalog): a minimal
suite over legacy (non-Iceberg) Hive tables
Pin the Trino Docker image to trinodb/trino:482 instead of 'latest' so
the interop tests are reproducible and not made flaky by future Trino
releases (482 is the first release with full Iceberg V3 support).
Docs: testdata/bin/TRINO-README.md explains how to run the suites and
author TRINO_QUERY / RESULTS sections.
Change-Id: I7706244f909f32dff4d390537e71877b0635c661
Assisted-by: Claude Opus 4.8 (1M context) <[email protected]>
---
M testdata/bin/TRINO-README.md
M testdata/bin/minicluster_trino/Dockerfile
M testdata/bin/minicluster_trino/hive.properties
A testdata/bin/minicluster_trino/session-property-config.json
A testdata/bin/minicluster_trino/session-property-config.properties
A
testdata/workloads/functional-query/queries/QueryTest/iceberg-trino-interop-default-values.test
A
testdata/workloads/functional-query/queries/QueryTest/iceberg-trino-interop-delete-dv.test
A
testdata/workloads/functional-query/queries/QueryTest/iceberg-trino-interop-insert.test
M tests/common/custom_cluster_test_suite.py
M tests/common/impala_test_suite.py
M tests/common/test_result_verifier.py
A tests/common/trino_cluster.py
A tests/custom_cluster/test_iceberg_trino_interop.py
A tests/custom_cluster/test_trino_interop.py
M tests/util/test_file_parser.py
15 files changed, 880 insertions(+), 11 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/44/24544/2
--
To view, visit http://gerrit.cloudera.org:8080/24544
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7706244f909f32dff4d390537e71877b0635c661
Gerrit-Change-Number: 24544
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <[email protected]>
Gerrit-Reviewer: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
