This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new a7ee5ab2d68 [test](paimon) Cover Variant reads of a deduplicate
primary-key Paimon table (#69227)
a7ee5ab2d68 is described below
commit a7ee5ab2d68bad37f2de6d4888be494ab07fafca
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Mon Oct 19 11:02:36 2026 +0100
[test](paimon) Cover Variant reads of a deduplicate primary-key Paimon
table (#69227)
### What problem does this PR solve?
Issue Number: None
Related PR: #66466, #68242
Problem Summary:
**In short.** Ports the primary-key Variant coverage of branch-4.1
#66466 to master: a Paimon deduplicate table whose second snapshot
updates two keys, and three blocks of `test_paimon_catalog_variant` that
read it through JNI. The rest of #66466 and the WAL fix #68242 are not
ported, because master handles those cases another way; see below.
**Background.** #66466 did three things on branch-4.x:
1. In BE, a generic gate in `TableReader::validate_file_mapping` rejects
a physical Variant read from any file format but Parquet. #68242 then
exempted WAL, which stores Doris blocks and supports Variant, because
the gate broke group commit WAL replay.
2. In FE, `PaimonScanNode` rejects a query that projects a Variant
column while Variant V2 is disabled.
3. In the tests, `run13.sql` gets `variant_primary_key_smoke`, and
`test_paimon_catalog_variant` reads it.
**The problem, and what it cost.** Every Variant table master's
`test_paimon_catalog_variant` reads is append-only, so the regression
does not check that a Paimon merge read keeps only the latest row of
each key while it materializes Variant values.
**How this PR fixes it.**
- `run13.sql` creates `variant_primary_key_smoke` as on branch-4.x: a
deduplicate primary-key table in Parquet, written in two snapshots. The
second updates ids 1 and 2 and inserts id 4; id 3 holds a root array.
- The JNI section of `test_paimon_catalog_variant` gets the three
branch-4.x blocks: the whole Variant value, Variant paths cast to
scalars, and a predicate on a Variant path. All three expect the latest
value of ids 1 and 2.
The other parts of #66466 and #68242 do not apply to master:
| Branch-4.x change | Master |
|---|---|
| FE rejects a Variant projection while Variant V2 is disabled | #66858
made Variant V2 the default and removed the switch, so there is nothing
to check |
| BE rejects a physical Variant read from any format but Parquet | Would
reject the native Variant reads of the ADBC reader (#68883). Master
keeps the Iceberg-specific gate in `IcebergTableReader`, admits Arrow
Variant reads in `FileScannerV2` only for ADBC, and plans a Paimon split
that projects a Variant natively only when every file is Parquet or an
ORC file whose schema predates the Variant column. Paimon 1.4.2 cannot
create an ORC table with a VARIANT column (`Unsupported type: VARIANT`)
|
| WAL is exempted from that gate | WAL is not gated on master |
**Results.** On a local cluster against the Iceberg docker environment,
Spark reads `variant_primary_key_smoke` as four rows, with the
second-snapshot values for ids 1 and 2. Doris returns the same rows in
the three new blocks, and every other result block of the suite is
unchanged.
Co-authored-by: zhangstar333 <[email protected]>
---
.../create_preinstalled_scripts/paimon/run13.sql | 26 ++++++++++++++++++++++
.../paimon/test_paimon_catalog_variant.out | 16 +++++++++++++
.../paimon/test_paimon_catalog_variant.groovy | 26 +++++++++++++++++++++-
3 files changed, 67 insertions(+), 1 deletion(-)
diff --git
a/docker/thirdparties/docker-compose/iceberg/scripts/create_preinstalled_scripts/paimon/run13.sql
b/docker/thirdparties/docker-compose/iceberg/scripts/create_preinstalled_scripts/paimon/run13.sql
index ec071966382..49ff2d2369d 100644
---
a/docker/thirdparties/docker-compose/iceberg/scripts/create_preinstalled_scripts/paimon/run13.sql
+++
b/docker/thirdparties/docker-compose/iceberg/scripts/create_preinstalled_scripts/paimon/run13.sql
@@ -70,3 +70,29 @@ alter table variant_mixed_su unset tblproperties (
);
insert into variant_mixed_su values
(2, date '2026-07-01',
parse_json('{"name":"bob","age":30,"layout":"unshredded"}'));
+
+-- Keep a primary-key table separate from variant_smoke so the regression
covers both Paimon
+-- append-only reads and deduplicate merge reads with Variant payloads.
+drop table if exists variant_primary_key_smoke;
+create table variant_primary_key_smoke (
+ id BIGINT,
+ payload VARIANT
+) using paimon
+tblproperties (
+ 'primary-key' = 'id',
+ 'bucket' = '1',
+ 'merge-engine' = 'deduplicate',
+ 'file.format' = 'parquet'
+);
+
+insert into variant_primary_key_smoke values
+ (1,
parse_json('{"name":"alice","version":1,"active":false,"profile":{"city":"beijing"}}')),
+ (2, parse_json('{"name":"bob","version":1,"tags":["old"]}')),
+ (3, parse_json('[10,"original"]'));
+
+-- A second snapshot updates two existing keys and inserts a new key. Doris
should expose only the
+-- latest logical row for ids 1 and 2.
+insert into variant_primary_key_smoke values
+ (1,
parse_json('{"name":"alice-updated","version":2,"active":true,"profile":{"city":"hangzhou"}}')),
+ (2, parse_json('{"name":"bob","version":2,"tags":["new","primary-key"]}')),
+ (4, parse_json('{"name":"carol","version":1,"active":true}'));
diff --git
a/regression-test/data/external_table_p0/paimon/test_paimon_catalog_variant.out
b/regression-test/data/external_table_p0/paimon/test_paimon_catalog_variant.out
index 256a115c8d2..d1e70d1b963 100644
---
a/regression-test/data/external_table_p0/paimon/test_paimon_catalog_variant.out
+++
b/regression-test/data/external_table_p0/paimon/test_paimon_catalog_variant.out
@@ -24,6 +24,22 @@ payload variant<PROPERTIES
("variant_max_subcolumns_count" = "0","variant_enable
-- !subpath_predicate --
2 bob
+-- !primary_key_full_variant --
+1
{"active":true,"name":"alice-updated","profile":{"city":"hangzhou"},"version":2}
+2 {"name":"bob","tags":["new","primary-key"],"version":2}
+3 [10,"original"]
+4 {"active":true,"name":"carol","version":1}
+
+-- !primary_key_deduplicate --
+1 alice-updated 2 true
+2 bob 2 \N
+3 \N \N \N
+4 carol 1 true
+
+-- !primary_key_subpath_predicate --
+1 alice-updated
+2 bob
+
-- !jni_shredded_projection --
2 bob 30
diff --git
a/regression-test/suites/external_table_p0/paimon/test_paimon_catalog_variant.groovy
b/regression-test/suites/external_table_p0/paimon/test_paimon_catalog_variant.groovy
index f8364971e8f..8aa1933acc7 100644
---
a/regression-test/suites/external_table_p0/paimon/test_paimon_catalog_variant.groovy
+++
b/regression-test/suites/external_table_p0/paimon/test_paimon_catalog_variant.groovy
@@ -106,8 +106,32 @@ suite("test_paimon_catalog_variant",
"p0,external,doris,external_docker,external
order by id
"""
+ // variant_smoke is append-only. This table has duplicate primary-key
writes, so the scan
+ // must preserve Paimon's deduplicate semantics while materializing
Variant values.
+ order_qt_primary_key_full_variant """
+ select id, payload
+ from variant_primary_key_smoke
+ order by id
+ """
+
+ order_qt_primary_key_deduplicate """
+ select id,
+ cast(payload['name'] as string),
+ cast(payload['version'] as int),
+ cast(payload['active'] as boolean)
+ from variant_primary_key_smoke
+ order by id
+ """
+
+ order_qt_primary_key_subpath_predicate """
+ select id, cast(payload['name'] as string)
+ from variant_primary_key_smoke
+ where cast(payload['version'] as int) = 2
+ order by id
+ """
+
// Exercise Paimon's metadata-marked read type against a physically
shredded file. The
- // variant_smoke cases above read unshredded files.
+ // unshredded and primary-key cases above cover Paimon's raw-value and
merge fallbacks.
order_qt_jni_shredded_projection """
select id,
cast(payload['name'] as string),
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
