This is an automated email from the ASF dual-hosted git repository.
Gabriel39 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 13947a50583 [fix](iceberg) Allow nested decimal precision widening
(#68378)
13947a50583 is described below
commit 13947a505830cd1031dc962d61d82b7a07538e05
Author: Oliveira <[email protected]>
AuthorDate: Thu Sep 24 10:23:48 2026 +0800
[fix](iceberg) Allow nested decimal precision widening (#68378)
[fix](iceberg) Allow nested decimal precision widening
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: Iceberg allows increasing DECIMAL precision when the
scale is unchanged, but Doris rejected this change when the decimal
field was nested in a STRUCT, ARRAY, or MAP. The rejection caused the
nested decimal schema-evolution cases in the Trino Iceberg compatibility
suite to fail. This change delegates nested primitive promotion
validation to Iceberg's `TypeUtil`, so the connector accepts only
promotions supported by Iceberg, including same-scale decimal precision
widening, while continuing to reject narrowing and incompatible changes.
The regression coverage creates an Iceberg table with
`STRUCT<amount:DECIMAL(5,3)>`, widens it to
`STRUCT<amount:DECIMAL(10,3)>`, verifies the resulting schema, and
verifies that the pre-existing value `12.345` remains readable.
### Release note
Iceberg complex column changes now support safe nested decimal precision
widening.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
Tests:
- `IcebergNestedColumnEvolutionTest`: 40 passed
- `CatalogBackedIcebergCatalogOpsColumnEvolutionTest`: 44 passed
-
`external_table_p0/iceberg/test_iceberg_nested_schema_evolution_ddl.groovy`:
1 suite passed locally
- Added nested decimal precision-widening schema and old-row-read
assertions to the regression suite.
- The six original cross-engine Trino product-test cases are left to
TeamCity CI validation.
- Behavior changed:
- [ ] No.
- [x] Yes. Nested Iceberg decimal precision widening with an unchanged
scale is now accepted.
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
---
.../connector/iceberg/IcebergComplexTypeDiff.java | 21 +++++----------
...BackedIcebergCatalogOpsColumnEvolutionTest.java | 17 ++++++------
.../iceberg/IcebergNestedColumnEvolutionTest.java | 20 +++++++++++++++
.../test_iceberg_nested_schema_evolution_ddl.out | 6 +++++
...test_iceberg_nested_schema_evolution_ddl.groovy | 30 ++++++++++++++++++++++
5 files changed, 72 insertions(+), 22 deletions(-)
diff --git
a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergComplexTypeDiff.java
b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergComplexTypeDiff.java
index ad8ab08a917..a7fae7b7968 100644
---
a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergComplexTypeDiff.java
+++
b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergComplexTypeDiff.java
@@ -22,6 +22,7 @@ import org.apache.doris.connector.spi.DorisConnectorException;
import org.apache.iceberg.UpdateSchema;
import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
import org.apache.iceberg.types.Types;
import java.util.HashSet;
@@ -50,7 +51,7 @@ import java.util.Set;
* {@code UpdateSchema.commit()}, so any guard throwing aborts the whole
change atomically.</p>
*
* <p><b>Supported shape changes (legacy parity):</b> widen an existing nested
field's primitive type (only the
- * iceberg-representable safe promotions int→long, float→double, or
an exact match), change a nested
+ * iceberg-representable safe promotions, including same-scale decimal
precision widening), change a nested
* field's comment, widen a NOT NULL nested field to nullable, and append new
(nullable) STRUCT fields. The
* category of every nested level must stay the same (struct/array/map);
struct fields may not be renamed,
* reordered, dropped, or narrowed to NOT NULL; a MAP key type may not
change.</p>
@@ -310,21 +311,13 @@ public final class IcebergComplexTypeDiff {
/**
* Whether changing a nested primitive {@code oldType} to {@code newType}
is a legal promotion, mirroring
- * legacy {@code ColumnType.checkSupportSchemaChangeForNestedPrimitive}
restricted to the iceberg-representable
- * cases: an exact match (covers VARCHAR length growth, which both map to
iceberg STRING), INT→BIGINT
- * (iceberg INTEGER→LONG), and FLOAT→DOUBLE. Everything else
(e.g. a nested DECIMAL precision change,
- * any narrowing, a category change) is rejected — matching legacy's
restrictive nested rule.
+ * Iceberg's primitive-promotion rules. Besides INTEGER→LONG and
FLOAT→DOUBLE, Iceberg allows a
+ * DECIMAL precision increase when scale is unchanged. Delegating to
Iceberg keeps the connector's validation
+ * aligned with the UpdateSchema operation it is about to commit.
*/
private static boolean isLegalNestedPrimitivePromotion(Type oldType, Type
newType) {
- if (oldType.equals(newType)) {
- return true;
- }
- Type.TypeID oldId = oldType.typeId();
- Type.TypeID newId = newType.typeId();
- if (oldId == Type.TypeID.INTEGER && newId == Type.TypeID.LONG) {
- return true;
- }
- return oldId == Type.TypeID.FLOAT && newId == Type.TypeID.DOUBLE;
+ return newType.isPrimitiveType()
+ && TypeUtil.isPromotionAllowed(oldType,
newType.asPrimitiveType());
}
/** The iceberg type category (struct/list/map) of {@code newType} must
equal {@code oldType}'s. */
diff --git
a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/CatalogBackedIcebergCatalogOpsColumnEvolutionTest.java
b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/CatalogBackedIcebergCatalogOpsColumnEvolutionTest.java
index 450ae232695..e23b0cfb309 100644
---
a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/CatalogBackedIcebergCatalogOpsColumnEvolutionTest.java
+++
b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/CatalogBackedIcebergCatalogOpsColumnEvolutionTest.java
@@ -606,17 +606,18 @@ public class
CatalogBackedIcebergCatalogOpsColumnEvolutionTest {
}
@Test
- public void testModifyNestedDecimalPrecisionFailsLoud() {
- // Legacy parity: a nested primitive change is restricted to int->long
/ float->double / exact; a
- // DECIMAL precision change inside a struct is rejected
(checkSupportSchemaChangeForNestedPrimitive).
+ public void testModifyNestedDecimalPrecisionWidens() {
+ // Iceberg permits a nested DECIMAL precision increase when the scale
stays fixed.
createTable("s_dec", new ConnectorColumn("st",
structType(Arrays.asList("a"),
Arrays.asList(ConnectorType.of("DECIMALV3", 10, 2)),
Arrays.asList(true), Arrays.asList((String) null)),
"", true, null, false));
- DorisConnectorException ex =
Assertions.assertThrows(DorisConnectorException.class,
- () -> modifyComplex("s_dec", "st",
- structType(Arrays.asList("a"),
Arrays.asList(ConnectorType.of("DECIMALV3", 20, 2)),
- Arrays.asList(true), Arrays.asList((String)
null)), true));
- Assertions.assertTrue(ex.getMessage().contains("nested"));
+ modifyComplex("s_dec", "st",
+ structType(Arrays.asList("a"),
Arrays.asList(ConnectorType.of("DECIMALV3", 20, 2)),
+ Arrays.asList(true), Arrays.asList((String) null)),
true);
+ Types.DecimalType amount = (Types.DecimalType)
reload("s_dec").findField("st")
+ .type().asStructType().field("a").type();
+ Assertions.assertEquals(20, amount.precision());
+ Assertions.assertEquals(2, amount.scale());
}
@Test
diff --git
a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergNestedColumnEvolutionTest.java
b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergNestedColumnEvolutionTest.java
index 0f74d82f735..c2029243c3a 100644
---
a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergNestedColumnEvolutionTest.java
+++
b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergNestedColumnEvolutionTest.java
@@ -713,6 +713,26 @@ public class IcebergNestedColumnEvolutionTest {
Assertions.assertEquals(Type.TypeID.LONG, metric.type().typeId());
}
+ @Test
+ public void testModifyNestedStructDecimalPrecisionPromotionAllowed() {
+ // Iceberg permits DECIMAL precision widening when the scale stays
fixed. This is the same complex MODIFY
+ // issued by the Trino product test for STRUCT<field:DECIMAL(5,3)> ->
STRUCT<field:DECIMAL(10,3)>.
+ Schema schema = new Schema(Types.NestedField.optional(1, "info",
Types.StructType.of(
+ Types.NestedField.optional(2, "amount",
Types.DecimalType.of(5, 3)))));
+ createTable("m_decimal_promote", schema);
+
+ ops.modifyColumn("db1", "m_decimal_promote",
+ structModify("info", Collections.singletonList("amount"),
+
Collections.singletonList(ConnectorType.of("DECIMALV3", 10, 3)),
+ Collections.singletonList(""),
Collections.singletonList(false)),
+ false, null);
+
+ Types.DecimalType amount = (Types.DecimalType)
reload("m_decimal_promote").findField("info")
+ .type().asStructType().field("amount").type();
+ Assertions.assertEquals(10, amount.precision());
+ Assertions.assertEquals(3, amount.scale());
+ }
+
@Test
public void testModifyNestedPrimitivePromotionDisallowedFailsLoud() {
// BIGINT -> INT is not an iceberg-representable promotion.
diff --git
a/regression-test/data/external_table_p0/iceberg/test_iceberg_nested_schema_evolution_ddl.out
b/regression-test/data/external_table_p0/iceberg/test_iceberg_nested_schema_evolution_ddl.out
index 93826ea80b7..485f9d0b540 100644
---
a/regression-test/data/external_table_p0/iceberg/test_iceberg_nested_schema_evolution_ddl.out
+++
b/regression-test/data/external_table_p0/iceberg/test_iceberg_nested_schema_evolution_ddl.out
@@ -10,3 +10,9 @@ m_scalar map<string,bigint(20)>
-- !query_rows --
1 \N 10 \N \N 100 \N \N 1000 \N
\N 7 70
2 first 20 after_a c2 200 202 201 2000 2002
2001 8 80
+
+-- !nested_decimal_schema --
+struct<decimalv3(10, 3)>
+
+-- !nested_decimal_old_row --
+1 12.345
diff --git
a/regression-test/suites/external_table_p0/iceberg/test_iceberg_nested_schema_evolution_ddl.groovy
b/regression-test/suites/external_table_p0/iceberg/test_iceberg_nested_schema_evolution_ddl.groovy
index 012bc88e3d9..7ca5b4b875c 100644
---
a/regression-test/suites/external_table_p0/iceberg/test_iceberg_nested_schema_evolution_ddl.groovy
+++
b/regression-test/suites/external_table_p0/iceberg/test_iceberg_nested_schema_evolution_ddl.groovy
@@ -172,4 +172,34 @@ suite("test_iceberg_nested_schema_evolution_ddl",
"p0,external,doris,external_do
FROM ${tableName}
ORDER BY id
"""
+
+ // Trino compatibility: an existing nested decimal may widen its precision
without changing its scale.
+ sql """DROP TABLE IF EXISTS iceberg_nested_decimal_evolution"""
+ sql """
+ CREATE TABLE iceberg_nested_decimal_evolution (
+ id INT,
+ info STRUCT<amount:DECIMAL(5,3)>
+ )
+ """
+ sql """
+ INSERT INTO iceberg_nested_decimal_evolution VALUES
+ (1, STRUCT(CAST(12.345 AS DECIMAL(5,3))))
+ """
+ sql """
+ ALTER TABLE iceberg_nested_decimal_evolution
+ MODIFY COLUMN info STRUCT<amount:DECIMAL(10,3)>
+ """
+
+ qt_nested_decimal_schema """
+ SELECT COLUMN_TYPE
+ FROM ${catalogName}.information_schema.columns
+ WHERE TABLE_SCHEMA = '${dbName}' AND TABLE_NAME =
'iceberg_nested_decimal_evolution'
+ AND COLUMN_NAME = 'info'
+ """
+
+ order_qt_nested_decimal_old_row """
+ SELECT id, element_at(info, 'amount')
+ FROM iceberg_nested_decimal_evolution
+ ORDER BY id
+ """
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
