This is an automated email from the ASF dual-hosted git repository.
Gabriel39 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 60cb5518438 [fix](regression) Correct Iceberg nullability and Paimon
GCS flag (#69416)
60cb5518438 is described below
commit 60cb55184380a80863ef3baf91e630f41b46f314
Author: Gabriel <[email protected]>
AuthorDate: Thu Oct 22 22:26:41 2026 +0800
[fix](regression) Correct Iceberg nullability and Paimon GCS flag (#69416)
### What problem does this PR solve?
This PR corrects three external-regression test issues:
- `test_iceberg_modify_binary_column`: the initial golden treats the
required `digest` field as nullable. Regenerate that expectation as
non-nullable; retain the nullable expectation after `ALTER ... NULL`.
- `test_iceberg_show_nullable` and `test_iceberg_modify_binary_column`:
their six negative-write assertions still expect generic file-writer
errors. After #69327, Iceberg validates required NULLs before creating
writers/files and reports `Iceberg required field '<path>' contains
NULL`. Update the expected diagnostics for `id`, `value`, `digest`,
`arr.element`, and `m.value`. The first two errors stop their respective
suites; the remaining four assertions have the same stale expectation.
The writes correctly fail, so this fixes test expectations without
changing product behavior or weakening the required-field checks.
- `test_paimon_hms_catalog`: use the recognized `fs.gcs.support`
property instead of `fs.gs.support` for explicit GCS-provider selection,
retaining both credential-property variants.
The Iceberg coverage still checks metadata nullability, rejected NULL
writes, successful writes after constraints are relaxed, nested fields,
and Spark read-back. Both TVF suites retain their original fixture
handling.
Related PRs: #68640, #69285, #69327.
### Validation
Current failure fix, using the same CI binary package (FE `436941a8bb`,
BE `0a77e2dd98`; package master revision `e779124b5f60`):
- Ran both original suites locally with `run-regression-test.sh`: 2
failed, reproducing the exact stale exception assertions for `digest`
and `id`.
- Updated all six assertions and reran both complete suites in normal
comparison mode: **2 passed, 0 failed, 0 fatal, 0 skipped**. All six
expected NULL rejections, post-ALTER writes, metadata/snapshot checks,
and Spark read-back passed. No golden output was regenerated for this
assertion-only update.
- One intermediate run encountered `SQLITE_BUSY` while the local REST
fixture cleaned up an existing table. Restarted that fixture, then reran
both complete suites successfully; the environment failure is not
counted as a passing validation.
- Reviewed the full diff and passed `git diff --check`. Only regression
scripts change in this update; no FE/BE source changes.
Earlier validation:
- Reproduced the initial Iceberg `digest` nullability mismatch with
runtime `6ca5ef36d8`. Generated the golden with `run-regression-test.sh
-forceGenOut`, then passed the complete binary-column suite in normal
comparison mode, including Spark read-back.
- Loaded the actual GCS directory plugin and checked both `gs.*` and
`s3.*` credential variants: the old flag fails `gs` filesystem
resolution; the corrected flag selects GCS and resolves `S3AFileSystem`.
This checks plugin binding and filesystem resolution; the full remote
Paimon HMS/GCS suite was not run locally.
- Regression framework build and its 50 tests passed.
### Release note
None
### Check List (For Author)
- Test:
- [x] Regression test
- [x] Manual test
- Behavior changed:
- [x] No product behavior change.
- Does this need documentation?
- [x] No.
---
.../iceberg/test_iceberg_modify_binary_column.out | 2 +-
.../iceberg/test_iceberg_modify_binary_column.groovy | 10 ++++++----
.../iceberg/test_iceberg_show_nullable.groovy | 10 +++++-----
.../external_table_p2/paimon/test_paimon_hms_catalog.groovy | 3 ++-
4 files changed, 14 insertions(+), 11 deletions(-)
diff --git
a/regression-test/data/external_table_p0/iceberg/test_iceberg_modify_binary_column.out
b/regression-test/data/external_table_p0/iceberg/test_iceberg_modify_binary_column.out
index 15b3bacfe57..a9f38ea985b 100644
---
a/regression-test/data/external_table_p0/iceberg/test_iceberg_modify_binary_column.out
+++
b/regression-test/data/external_table_p0/iceberg/test_iceberg_modify_binary_column.out
@@ -2,7 +2,7 @@
-- !desc_before --
id int Yes true \N
payload varbinary(2147483647) Yes true \N
-digest varbinary(16) Yes true \N sha
+digest varbinary(16) No true \N sha
s struct<a:varbinary(2147483647) comment "raw",b:int> Yes true
\N
arr array<varbinary(4)> Yes true \N
m map<text,varbinary(2147483647)> Yes true \N
diff --git
a/regression-test/suites/external_table_p0/iceberg/test_iceberg_modify_binary_column.groovy
b/regression-test/suites/external_table_p0/iceberg/test_iceberg_modify_binary_column.groovy
index e9487f4c04d..8473962f01c 100644
---
a/regression-test/suites/external_table_p0/iceberg/test_iceberg_modify_binary_column.groovy
+++
b/regression-test/suites/external_table_p0/iceberg/test_iceberg_modify_binary_column.groovy
@@ -84,20 +84,22 @@ suite("test_iceberg_modify_binary_column",
"p0,external,iceberg,external_docker,
named_struct('a', X'AB', 'b', 1), [X'01020304'], map('k',
X'02'))"""
// DESC shows column and STRUCT field comments too.
sql """set show_column_comment_in_describe = true"""
+ // DESC preserves Iceberg requiredness: digest is non-nullable until the
ALTER below.
qt_desc_before """DESC ${tableName}"""
- // The required column, ARRAY element and MAP value reject NULL.
+ // The Iceberg validator rejects required NULLs before format-specific
writers run and
+ // reports full field paths for ARRAY elements and MAP values.
test {
sql """INSERT INTO ${tableName} VALUES (2, X'03', NULL, NULL, NULL,
NULL)"""
- exception "Column 'digest' is declared non-nullable"
+ exception "Iceberg required field 'digest' contains NULL"
}
test {
sql """INSERT INTO ${tableName} VALUES (2, X'03',
X'00112233445566778899AABBCCDDEEFF', NULL, [NULL], NULL)"""
- exception "Column 'element' is declared non-nullable"
+ exception "Iceberg required field 'arr.element' contains NULL"
}
test {
sql """INSERT INTO ${tableName} VALUES (2, X'03',
X'00112233445566778899AABBCCDDEEFF', NULL, NULL,
map('k', NULL))"""
- exception "Column 'value' is declared non-nullable"
+ exception "Iceberg required field 'm.value' contains NULL"
}
// Each statement restates the type DESC shows, to change the position,
nullability or comment.
diff --git
a/regression-test/suites/external_table_p0/iceberg/test_iceberg_show_nullable.groovy
b/regression-test/suites/external_table_p0/iceberg/test_iceberg_show_nullable.groovy
index 5f1b9eb94e0..c2fb2ce4ecf 100644
---
a/regression-test/suites/external_table_p0/iceberg/test_iceberg_show_nullable.groovy
+++
b/regression-test/suites/external_table_p0/iceberg/test_iceberg_show_nullable.groovy
@@ -76,15 +76,15 @@ suite("test_iceberg_show_nullable", "p0,external") {
assertTrue(ddl.contains("`event_time` datetimev2(6) NULL"), ddl)
order_qt_select """select * from iceberg_show_nullable"""
- // Only the metadata view is NOT NULL; the columns are read as nullable,
and a write still enforces
- // the declared constraint.
+ // Columns are read as nullable, but the Iceberg validator rejects
required NULLs before
+ // format-specific writers run.
test {
sql """insert into iceberg_show_nullable values (null, 'bad',
'2026-01-02 00:00:00')"""
- exception "Column 'id' is declared non-nullable but contains nulls"
+ exception "Iceberg required field 'id' contains NULL"
}
test {
sql """insert into iceberg_show_nullable values (3, null, '2026-01-02
00:00:00')"""
- exception "Column 'value' is declared non-nullable but contains nulls"
+ exception "Iceberg required field 'value' contains NULL"
}
// Relaxing the constraint is reflected without a new data snapshot.
@@ -99,6 +99,6 @@ suite("test_iceberg_show_nullable", "p0,external") {
order_qt_select_after_alter """select * from iceberg_show_nullable"""
test {
sql """insert into iceberg_show_nullable values (null, 'still
required', null)"""
- exception "Column 'id' is declared non-nullable but contains nulls"
+ exception "Iceberg required field 'id' contains NULL"
}
}
diff --git
a/regression-test/suites/external_table_p2/paimon/test_paimon_hms_catalog.groovy
b/regression-test/suites/external_table_p2/paimon/test_paimon_hms_catalog.groovy
index 1ef30d8a94e..9a359cd4f9b 100644
---
a/regression-test/suites/external_table_p2/paimon/test_paimon_hms_catalog.groovy
+++
b/regression-test/suites/external_table_p2/paimon/test_paimon_hms_catalog.groovy
@@ -224,8 +224,9 @@ suite("test_paimon_hms_catalog", "p2,external") {
String paimon_fs_s3_support = """
'fs.s3.support' = 'true',
"""
+ // An explicit filesystem flag disables endpoint inference, so it must use
the provider name.
String paimon_fs_gcs_support = """
- 'fs.gs.support' = 'true',
+ 'fs.gcs.support' = 'true',
"""
testQuery(paimon_hms_catalog_properties + paimon_fs_hdfs_support +
hdfs_warehouse_properties + hdfs_storage_properties, "support_hdfs", "hdfs_db")
testQuery(paimon_hms_type_prop + paimon_fs_hdfs_support +
hdfs_warehouse_properties + hms_kerberos_new_prop + hdfs_kerberos_properties,
"support_hdfs_kerberos", "hdfs_db")
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
