This is an automated email from the ASF dual-hosted git repository.
Gabriel39 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 219c6193f24 [fix](iceberg) Preserve nested field case in created
schemas (#67166)
219c6193f24 is described below
commit 219c6193f24ed6c028c5f5469f18681be7c0292c
Author: Gabriel <[email protected]>
AuthorDate: Thu Sep 3 20:02:55 2026 +0800
[fix](iceberg) Preserve nested field case in created schemas (#67166)
### What problem does this PR solve?
Issue Number: N/A
Problem Summary:
Doris used the lowercase runtime name of nested STRUCT fields when
building connector schemas. Creating an Iceberg table through Doris
therefore changed persisted mixed-case field names such as
`CaseSensitive` to `casesensitive`, causing schema compatibility checks
from engines such as Trino and Spark to fail.
The same identity mismatch also affected schema evolution, query
execution, and pruning:
- complex `MODIFY COLUMN` could interpret a case-only spelling
difference as a nested rename;
- flat Iceberg `MODIFY COLUMN` and its `AFTER` reference used caller
spelling even though Iceberg update paths are case-sensitive;
- nested `MODIFY COLUMN` error upgrading resolved only the leaf name and
could inspect an unrelated same-named top-level column;
- runtime struct lookup depended on the JVM default locale, so locales
such as Turkish could map distinct field names to the same key;
- a Unicode field spelling displayed by `DESCRIBE`, such as `Σ` or `ẞ`,
could fail at execution because FE passed that external spelling to BE
while the thrift struct descriptor contained the normalized runtime name
(`σ` or `ß`);
- cast-aware nested pruning used `equalsIgnoreCase`, which could map
distinct ROOT-normalized siblings such as `i` and `ı` to the wrong
source field.
This change keeps two explicit field identities: a locale-independent
lowercase key for Doris runtime lookup and the original spelling for
connector/external metadata. Connector create-table paths, including
FILE TVF CTAS, use the original spelling. Iceberg schema evolution
resolves existing fields case-insensitively and then stages type,
comment, nullability, and position changes with the persisted canonical
path. Only an explicit rename operation changes field spelling. Nested
modify error upgrading resolves the complete `ConnectorColumnPath` and
remains best-effort so it cannot replace the original build error for a
missing target.
After Nereids successfully resolves a STRUCT selector, it replaces the
external spelling with the resolved field's normalized runtime name for
thrift/BE execution. This canonicalization is applied to directly
analyzed `ElementAt` nodes, `ElementAt` produced while binding SQL
function syntax from `UnboundFunction`, and `ElementAt` created for
dotted access on a computed base such as `CAST(... AS
STRUCT<...>).field`. The latter two paths construct and return a new
node without revisiting `visitElementAt`. Consequently,
`element_at(struct, 'field')` and dotted dereference both work while
`DESCRIBE` and connector metadata continue to expose the original
external spelling. Cast-aware pruning compares exact ROOT-normalized
keys so that distinct siblings remain distinct.
#### Metadata and rolling-upgrade compatibility
`StructField.name`, `StructType.fields`, and the current `fieldMap`
lookup index can be present in FE image metadata. Before this change, an
FE running with a locale such as `tr-TR` could therefore persist `I` as
the runtime key `ı`. A new FE using `Locale.ROOT` produces the lookup
key `i`, so exact ROOT lookup alone cannot read that pre-ROOT image.
For fields replayed from metadata that predates `originalName`, this PR
records a runtime-only legacy marker through Catalog-to-Nereids
conversion. Current fields use the exact ROOT lookup key. Legacy fields
first accept an exact persisted runtime spelling; broader case matching
is used only when it identifies a single legacy field. If multiple
legacy runtime names match, lookup rejects the ambiguous selector
instead of silently returning the wrong sibling. Newly created metadata
always has `originalName`, so valid ROOT-distinct names such as `i` and
`ı` are not merged by the compatibility path.
This covers the supported rolling-upgrade direction where upgraded
Followers/Observers replay metadata written by an older Master, followed
by upgrading the Master. It does not make an old FE understand metadata
first written by a new FE. It also cannot reconstruct original spelling
that an old FE already discarded; it only preserves unambiguous lookup
compatibility for the persisted runtime name. Rebuilding `fieldMap`
during deserialization would not recover the old FE locale or discarded
spelling because Doris replays these objects through Gson and that
information was never persisted.
#### Fix boundary
The fix is limited to nested field identity preservation, Iceberg schema
evolution, STRUCT selector canonicalization in FE, unambiguous legacy
pre-ROOT struct lookup, and consistent cast-pruning identity. Doris
runtime lookup remains case-insensitive for current metadata. BE
continues to receive and compare normalized thrift names; this PR does
not add Unicode case folding to BE. Current metadata continues to
persist the existing `fieldMap`; rebuilding or removing that lookup
index is a separate metadata-format change and is intentionally outside
this PR. Removed code paths such as `StructElement` and
`IcebergScanNode` are not reintroduced; their current replacements
already route through the fixed lookup or preserve partition-column
case.
### Release note
Preserve mixed-case nested field names when creating and evolving
Iceberg schemas, allow displayed Unicode nested field names to be
queried, keep nested pruning correct for locale-sensitive Unicode names,
and retain safe lookup compatibility with pre-ROOT FE metadata.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [ ] Manual test
- [ ] No need to test or manual test.
Added coverage for mixed-case nested fields in create-table and FILE TVF
CTAS paths, STRUCT/ARRAY/MAP evolution, locale-independent runtime
lookup and pruning, case-insensitive flat Iceberg `MODIFY
COLUMN`/`AFTER` resolution, full-path nested modify error handling,
pre-ROOT Turkish metadata replay across Catalog and Nereids, ambiguous
legacy-field rejection, cast pruning with ROOT-distinct sibling names,
and execution of exact displayed Unicode names (`Σ` and `ẞ`). The
`ExpressionAnalyzer` coverage constructs an `UnboundFunction` to
exercise the same SQL function-binding path as `element_at(...)`, and
covers direct `ElementAt`, ordinary dotted dereference, and
computed-base dotted dereference.
Latest local validation passed 23 Iceberg connector column-evolution
tests, all 8 `ExpressionAnalyzer` tests, and all 5
`ColumnGsonSerializationTest` tests. Full FE Checkstyle passed all 74
modules. The external Spark/Iceberg regression requires the CI test
environment and was added for CI execution.
- Behavior changed:
- [ ] No.
- [x] Yes. External Iceberg schemas retain their original nested-field
spelling, displayed Unicode nested field names remain executable,
unambiguous legacy struct metadata remains queryable after upgrade,
ambiguous legacy selectors are rejected instead of reading the wrong
field, and pruning keeps ROOT-distinct field identities separate.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
---
.../doris/connector/iceberg/IcebergCatalogOps.java | 17 ++-
.../connector/iceberg/IcebergComplexTypeDiff.java | 13 +-
.../iceberg/IcebergConnectorMetadata.java | 11 +-
.../iceberg/IcebergNestedColumnEvolution.java | 20 +++
...BackedIcebergCatalogOpsColumnEvolutionTest.java | 144 +++++++++++++++++++++
...cebergConnectorMetadataColumnEvolutionTest.java | 47 +++++++
.../converter/ConnectorColumnConverter.java | 2 +-
.../nereids/rules/analysis/ExpressionAnalyzer.java | 29 ++++-
.../rewrite/AccessPathExpressionCollector.java | 6 +-
.../nereids/rules/rewrite/NestedColumnPruning.java | 14 +-
.../functions/scalar/CreateNamedStruct.java | 3 +-
.../org/apache/doris/nereids/types/DataType.java | 4 +-
.../apache/doris/nereids/types/StructField.java | 45 ++++++-
.../org/apache/doris/nereids/types/StructType.java | 22 +++-
.../ExternalFileTableValuedFunction.java | 10 +-
.../doris/catalog/ColumnGsonSerializationTest.java | 72 +++++++++++
...teTableInfoToConnectorRequestConverterTest.java | 16 +++
.../converter/ConnectorColumnConverterTest.java | 43 ++++++
.../rules/analysis/ExpressionAnalyzerTest.java | 101 +++++++++++++++
.../rewrite/AccessPathExpressionCollectorTest.java | 102 +++++++++++++++
.../expressions/literal/StructLiteralTest.java | 20 +++
.../ExternalFileTableValuedFunctionTest.java | 42 ++++++
.../java/org/apache/doris/catalog/StructField.java | 39 +++++-
.../java/org/apache/doris/catalog/StructType.java | 23 +++-
.../main/java/org/apache/doris/catalog/Type.java | 2 +-
.../test_iceberg_struct_schema_evolution.out | 4 +-
.../test_iceberg_struct_schema_evolution.groovy | 6 +-
.../test_iceberg_write_ctas_format_boundary.groovy | 84 +++++++++++-
28 files changed, 888 insertions(+), 53 deletions(-)
diff --git
a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogOps.java
b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogOps.java
index 91d6c9b1c93..3b55a98be7c 100644
---
a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogOps.java
+++
b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogOps.java
@@ -531,10 +531,14 @@ public interface IcebergCatalogOps {
public void modifyColumn(String dbName, String tableName,
IcebergColumnChange column,
boolean commentSpecified, ConnectorColumnPosition position) {
withTable(dbName, tableName, table -> {
- Types.NestedField current =
table.schema().findField(column.getName());
+ Schema schema = table.schema();
+ Types.NestedField current =
IcebergNestedColumnEvolution.findTopLevelField(
+ schema, column.getName());
if (current == null) {
throw new DorisConnectorException("Column " +
column.getName() + " does not exist");
}
+ // Iceberg update paths are case-sensitive, so stage every
change with the persisted spelling.
+ String currentName = current.name();
// Iceberg can widen required -> optional but never optional
-> required (existing data may hold
// nulls), so a NOT NULL request on an already-nullable column
fails loud — legacy parity
// (IcebergMetadataOps.validateForModifyColumn /
validateForModifyComplexColumn).
@@ -555,7 +559,7 @@ public interface IcebergCatalogOps {
throw new DorisConnectorException("Modify column type
from complex to primitive is not"
+ " supported: " + column.getName());
}
- updateSchema.updateColumn(column.getName(),
newType.asPrimitiveType(), targetComment);
+ updateSchema.updateColumn(currentName,
newType.asPrimitiveType(), targetComment);
} else {
// A complex (STRUCT/ARRAY/MAP) modify diffs the new type
against the current one field-by-field
// (IcebergComplexTypeDiff); the top-level column doc is
updated separately, as in legacy.
@@ -563,16 +567,17 @@ public interface IcebergCatalogOps {
throw new DorisConnectorException("Modify column type
from non-complex to complex is not"
+ " supported: " + column.getName());
}
- IcebergComplexTypeDiff.apply(updateSchema,
column.getName(), current.type(), newType,
+ IcebergComplexTypeDiff.apply(updateSchema, currentName,
current.type(), newType,
column.getSourceType());
if (!Objects.equals(current.doc(), targetComment)) {
- updateSchema.updateColumnDoc(column.getName(),
targetComment);
+ updateSchema.updateColumnDoc(currentName,
targetComment);
}
}
if (column.isNullable()) {
- updateSchema.makeColumnOptional(column.getName());
+ updateSchema.makeColumnOptional(currentName);
}
- applyPosition(updateSchema, position, column.getName());
+ IcebergNestedColumnEvolution.applyTopLevelPosition(
+ updateSchema, position, currentName, schema, "modify");
updateSchema.commit();
return null;
});
diff --git
a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergComplexTypeDiff.java
b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergComplexTypeDiff.java
index 36aa7cef28f..ad8ab08a917 100644
---
a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergComplexTypeDiff.java
+++
b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergComplexTypeDiff.java
@@ -172,10 +172,11 @@ public final class IcebergComplexTypeDiff {
Types.NestedField oldField = oldFields.get(i);
Types.NestedField newField = newFields.get(i);
String fieldPath = path + "." + oldField.name();
- existingNames.add(oldField.name());
+ existingNames.add(lowercaseName(oldField.name()));
- // Legacy ColumnType rule: existing fields are matched by position
and may not be renamed.
- if (!oldField.name().equals(newField.name())) {
+ // Iceberg defines case-insensitive identity with ROOT-lowercase
keys. Java equalsIgnoreCase is
+ // broader for some Unicode characters and could otherwise route
an update to the wrong field.
+ if
(!lowercaseName(oldField.name()).equals(lowercaseName(newField.name()))) {
throw new DorisConnectorException("Cannot rename struct field
from '" + oldField.name()
+ "' to '" + newField.name() + "'");
}
@@ -216,7 +217,7 @@ public final class IcebergComplexTypeDiff {
// Append the new fields (legacy parity: must be nullable and not
clash with an existing name).
for (int i = oldFields.size(); i < newFields.size(); i++) {
Types.NestedField newField = newFields.get(i);
- if (existingNames.contains(newField.name())) {
+ if (!existingNames.add(lowercaseName(newField.name()))) {
throw new DorisConnectorException("Added struct field '" +
newField.name()
+ "' conflicts with existing field");
}
@@ -227,6 +228,10 @@ public final class IcebergComplexTypeDiff {
}
}
+ private static String lowercaseName(String name) {
+ return name.toLowerCase(Locale.ROOT);
+ }
+
private static void applyListChange(UpdateSchema updateSchema, String path,
Types.ListType oldList, Types.ListType newList, ConnectorType
newConn) {
String elementPath = path + "." +
oldList.field(oldList.elementId()).name();
diff --git
a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java
b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java
index bbb83894026..10318ee9ee1 100644
---
a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java
+++
b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java
@@ -1303,7 +1303,7 @@ public class IcebergConnectorMetadata implements
ConnectorMetadata {
// generic "Unsupported type for Iceberg: SMALLINT" here. Restore
the legacy parity message ("Cannot
// change int to smallint in nested types") by validating the
requested nested type against the
// CURRENT type — legacy validated in Doris type space, where the
narrow target still exists.
- throw upgradeNestedModifyError(iceHandle, column, buildError);
+ throw upgradeNestedModifyError(iceHandle,
ConnectorColumnPath.of(column.getName()), column, buildError);
}
// Carry the neutral source type so a complex-type diff can read each
STRUCT field's commentSpecified.
IcebergColumnChange change = new IcebergColumnChange(column.getName(),
icebergType,
@@ -1328,15 +1328,15 @@ public class IcebergConnectorMetadata implements
ConnectorMetadata {
* against the CURRENT column type. Best-effort: a scalar modify, a load
failure, or no offending nested leaf
* keeps the original build error — so no other modify path changes.
*/
- private DorisConnectorException
upgradeNestedModifyError(IcebergTableHandle handle, ConnectorColumn column,
- DorisConnectorException buildError) {
+ private DorisConnectorException
upgradeNestedModifyError(IcebergTableHandle handle, ConnectorColumnPath path,
+ ConnectorColumn column, DorisConnectorException buildError) {
if (!isComplexType(column.getType())) {
return buildError;
}
try {
Types.NestedField current = executeAuthenticated(() ->
catalogOps.withTable(handle.getDbName(),
handle.getTableName(),
- table ->
table.schema().findField(column.getName())));
+ table ->
IcebergNestedColumnEvolution.findFieldForErrorUpgrade(table.schema(), path)));
if (current != null && !current.type().isPrimitiveType()) {
IcebergComplexTypeDiff.validateNestedModifyRepresentable(current.type(),
column.getType());
}
@@ -1487,7 +1487,8 @@ public class IcebergConnectorMetadata implements
ConnectorMetadata {
try {
icebergType =
IcebergSchemaBuilder.buildColumnType(column.getType());
} catch (DorisConnectorException buildError) {
- throw upgradeNestedModifyError(iceHandle, column, buildError);
+ // Preserve the complete target identity so error parity cannot
bind a same-named top-level field.
+ throw upgradeNestedModifyError(iceHandle, path, column,
buildError);
}
// Carry the neutral source type so the nested complex-type diff can
read each STRUCT field's
// commentSpecified (an omitted COMMENT on a sub-field must keep its
current doc, not clear it).
diff --git
a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergNestedColumnEvolution.java
b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergNestedColumnEvolution.java
index b4ca1c02c7d..7bc5a0e52b8 100644
---
a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergNestedColumnEvolution.java
+++
b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergNestedColumnEvolution.java
@@ -492,6 +492,19 @@ public final class IcebergNestedColumnEvolution {
return new ResolvedColumnPath(ConnectorColumnPath.of(canonicalParts),
currentType, currentField);
}
+ static NestedField findTopLevelField(Schema schema, String columnName) {
+ return schema.asStruct().caseInsensitiveField(columnName);
+ }
+
+ static NestedField findFieldForErrorUpgrade(Schema schema,
ConnectorColumnPath columnPath) {
+ try {
+ return resolveColumnPath(schema, columnPath, "modify").getField();
+ } catch (DorisConnectorException ignored) {
+ // Error-message upgrading is best-effort and must not replace the
original type-build failure.
+ return null;
+ }
+ }
+
/**
* Resolves {@code columnPath}'s parent (which must be a struct) and its
leaf within that struct
* (case-insensitive). Used by nested DROP / RENAME, which target an
existing struct field.
@@ -584,6 +597,13 @@ public final class IcebergNestedColumnEvolution {
}
}
+ static void applyTopLevelPosition(UpdateSchema updateSchema,
ConnectorColumnPosition position,
+ String columnName, Schema schema, String operation) {
+ if (position != null) {
+ applyPosition(updateSchema, position,
ConnectorColumnPath.of(columnName), schema, operation);
+ }
+ }
+
private static String getPositionReferencePath(Schema schema,
ConnectorColumnPath columnPath,
ConnectorColumnPosition position, String operation) {
if (position == null || position.isFirst()) {
diff --git
a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/CatalogBackedIcebergCatalogOpsColumnEvolutionTest.java
b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/CatalogBackedIcebergCatalogOpsColumnEvolutionTest.java
index c0b9089b6fd..450ae232695 100644
---
a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/CatalogBackedIcebergCatalogOpsColumnEvolutionTest.java
+++
b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/CatalogBackedIcebergCatalogOpsColumnEvolutionTest.java
@@ -367,6 +367,32 @@ public class
CatalogBackedIcebergCatalogOpsColumnEvolutionTest {
return ConnectorType.structOf(names, types, nullable, comments);
}
+ @Test
+ public void testModifyColumnsCanonicalizeMixedCaseRootAndPosition() {
+ createMixedCaseTable();
+ ops.modifyColumn("db1", "mixed", change("id", Types.LongType.get(),
"identifier", true), true, null);
+
+ ConnectorType requestedType = structType(
+ Arrays.asList("metric"),
Arrays.asList(ConnectorType.of("BIGINT")),
+ Arrays.asList(true), Arrays.asList((String) null));
+
+ ops.modifyColumn("db1", "mixed",
+ new IcebergColumnChange("info",
IcebergSchemaBuilder.buildColumnType(requestedType),
+ "updated", null, true, requestedType),
+ true, ConnectorColumnPosition.after("id"));
+
+ Schema schema = reload("mixed");
+ Assertions.assertEquals(Arrays.asList("Id", "Info", "Label"),
schema.columns().stream()
+ .map(Types.NestedField::name).collect(Collectors.toList()));
+ Assertions.assertEquals(Type.TypeID.LONG,
schema.findField("Id").type().typeId());
+ Assertions.assertEquals("identifier", schema.findField("Id").doc());
+ Types.NestedField info = schema.findField("Info");
+ Assertions.assertEquals("updated", info.doc());
+ Types.NestedField metric = info.type().asStructType().fields().get(0);
+ Assertions.assertEquals("Metric", metric.name());
+ Assertions.assertEquals(Type.TypeID.LONG, metric.type().typeId());
+ }
+
@Test
public void testModifyStructAddsNullableField() {
createTable("s_add", new ConnectorColumn("st",
@@ -397,6 +423,124 @@ public class
CatalogBackedIcebergCatalogOpsColumnEvolutionTest {
Assertions.assertEquals("new", a.doc());
}
+ @Test
+ public void testModifyStructMatchesExistingFieldCaseInsensitively() {
+ createTable("s_case", new ConnectorColumn("st",
+ structType(Arrays.asList("CaseSensitive"),
Arrays.asList(ConnectorType.of("INT")),
+ Arrays.asList(true), Arrays.asList((String) null)),
"", true, null, false));
+
+ modifyComplex("s_case", "st",
+ structType(Arrays.asList("casesensitive"),
Arrays.asList(ConnectorType.of("BIGINT")),
+ Arrays.asList(true), Arrays.asList((String) null)),
true);
+
+ Types.NestedField field =
reload("s_case").findField("st").type().asStructType().fields().get(0);
+ Assertions.assertEquals("CaseSensitive", field.name());
+ Assertions.assertEquals(Type.TypeID.LONG, field.type().typeId());
+ }
+
+ @Test
+ public void testModifyStructCommentMatchesExistingFieldCaseInsensitively()
{
+ createTable("s_case_doc", new ConnectorColumn("st",
+ structType(Arrays.asList("CaseSensitive"),
Arrays.asList(ConnectorType.of("INT")),
+ Arrays.asList(true), Arrays.asList("old")), "", true,
null, false));
+
+ modifyComplex("s_case_doc", "st",
+ structType(Arrays.asList("casesensitive"),
Arrays.asList(ConnectorType.of("INT")),
+ Arrays.asList(true), Arrays.asList("new")), true);
+
+ Types.NestedField field =
reload("s_case_doc").findField("st").type().asStructType().fields().get(0);
+ Assertions.assertEquals("CaseSensitive", field.name());
+ Assertions.assertEquals("new", field.doc());
+ }
+
+ @Test
+ public void
testModifyStructNullabilityMatchesExistingFieldCaseInsensitively() {
+ createTable("s_case_null", new ConnectorColumn("st",
+ structType(Arrays.asList("CaseSensitive"),
Arrays.asList(ConnectorType.of("INT")),
+ Arrays.asList(false), Arrays.asList((String) null)),
"", true, null, false));
+
+ modifyComplex("s_case_null", "st",
+ structType(Arrays.asList("casesensitive"),
Arrays.asList(ConnectorType.of("INT")),
+ Arrays.asList(true), Arrays.asList((String) null)),
true);
+
+ Types.NestedField field =
reload("s_case_null").findField("st").type().asStructType().fields().get(0);
+ Assertions.assertEquals("CaseSensitive", field.name());
+ Assertions.assertTrue(field.isOptional());
+ }
+
+ @Test
+ public void
testModifyStructUnderArrayMatchesExistingFieldCaseInsensitively() {
+ ConnectorType oldStruct = structType(
+ Arrays.asList("CaseSensitive"),
Arrays.asList(ConnectorType.of("INT")),
+ Arrays.asList(true), Arrays.asList((String) null));
+ createTable("a_case", new ConnectorColumn(
+ "arr", ConnectorType.arrayOf(oldStruct), "", true, null,
false));
+ ConnectorType newStruct = structType(
+ Arrays.asList("casesensitive"),
Arrays.asList(ConnectorType.of("BIGINT")),
+ Arrays.asList(true), Arrays.asList((String) null));
+
+ modifyComplex("a_case", "arr", ConnectorType.arrayOf(newStruct), true);
+
+ Types.NestedField field =
reload("a_case").findField("arr").type().asListType()
+ .elementType().asStructType().fields().get(0);
+ Assertions.assertEquals("CaseSensitive", field.name());
+ Assertions.assertEquals(Type.TypeID.LONG, field.type().typeId());
+ }
+
+ @Test
+ public void
testModifyStructUnderMapMatchesExistingFieldCaseInsensitively() {
+ ConnectorType oldStruct = structType(
+ Arrays.asList("CaseSensitive"),
Arrays.asList(ConnectorType.of("INT")),
+ Arrays.asList(true), Arrays.asList((String) null));
+ createTable("m_case", new ConnectorColumn("m",
+ ConnectorType.mapOf(ConnectorType.of("STRING"), oldStruct),
"", true, null, false));
+ ConnectorType newStruct = structType(
+ Arrays.asList("casesensitive"),
Arrays.asList(ConnectorType.of("BIGINT")),
+ Arrays.asList(true), Arrays.asList((String) null));
+
+ modifyComplex("m_case", "m",
+ ConnectorType.mapOf(ConnectorType.of("STRING"), newStruct),
true);
+
+ Types.NestedField field =
reload("m_case").findField("m").type().asMapType()
+ .valueType().asStructType().fields().get(0);
+ Assertions.assertEquals("CaseSensitive", field.name());
+ Assertions.assertEquals(Type.TypeID.LONG, field.type().typeId());
+ }
+
+ @Test
+ public void testModifyStructRejectsCaseInsensitiveAppendedFieldCollision()
{
+ createTable("s_case_collision", new ConnectorColumn("st",
+ structType(Arrays.asList("CaseSensitive"),
Arrays.asList(ConnectorType.of("INT")),
+ Arrays.asList(true), Arrays.asList((String) null)),
"", true, null, false));
+
+ DorisConnectorException ex =
Assertions.assertThrows(DorisConnectorException.class,
+ () -> modifyComplex("s_case_collision", "st",
+ structType(Arrays.asList("casesensitive",
"CASESENSITIVE"),
+ Arrays.asList(ConnectorType.of("INT"),
ConnectorType.of("STRING")),
+ Arrays.asList(true, true), Arrays.asList(null,
null)), true));
+
+ Assertions.assertTrue(ex.getMessage().contains("conflicts with
existing field"), ex.getMessage());
+ }
+
+ @Test
+ public void testModifyStructUsesIcebergLowercaseIdentity() {
+ createTable("s_unicode_identity", new ConnectorColumn("st",
+ structType(Arrays.asList("Σ", "ς"),
+ Arrays.asList(ConnectorType.of("INT"),
ConnectorType.of("INT")),
+ Arrays.asList(true, true), Arrays.asList(null, null)),
"", true, null, false));
+
+ DorisConnectorException ex =
Assertions.assertThrows(DorisConnectorException.class,
+ () -> modifyComplex("s_unicode_identity", "st",
+ structType(Arrays.asList("ς", "Σ"),
+ Arrays.asList(ConnectorType.of("BIGINT"),
ConnectorType.of("INT")),
+ Arrays.asList(true, true), Arrays.asList(null,
null)), true));
+
+ Assertions.assertTrue(ex.getMessage().contains("Cannot rename struct
field"), ex.getMessage());
+ Types.StructType fields =
reload("s_unicode_identity").findField("st").type().asStructType();
+ Assertions.assertEquals(Type.TypeID.INTEGER,
fields.field("Σ").type().typeId());
+ Assertions.assertEquals(Type.TypeID.INTEGER,
fields.field("ς").type().typeId());
+ }
+
@Test
public void testModifyStructFieldWidensNotNullToNullable() {
createTable("s_null", new ConnectorColumn("st",
diff --git
a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadataColumnEvolutionTest.java
b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadataColumnEvolutionTest.java
index da1a8e3557b..c82513e421c 100644
---
a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadataColumnEvolutionTest.java
+++
b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadataColumnEvolutionTest.java
@@ -20,6 +20,7 @@ package org.apache.doris.connector.iceberg;
import org.apache.doris.connector.spi.ConnectorColumn;
import org.apache.doris.connector.spi.ConnectorType;
import org.apache.doris.connector.spi.DorisConnectorException;
+import org.apache.doris.connector.spi.ddl.ConnectorColumnPath;
import org.apache.doris.connector.spi.ddl.ConnectorColumnPosition;
import org.apache.iceberg.Schema;
@@ -84,6 +85,21 @@ public class IcebergConnectorMetadataColumnEvolutionTest {
return catalog.createTable(TableIdentifier.of("db1", "t1"), schema);
}
+ /** A real iceberg table with a nested {@code Root.Leaf ARRAY<INT>} target
and an unrelated
+ * top-level {@code LEAF ARRAY<FLOAT>} field sharing the target's leaf
name. */
+ private static Table tableWithNestedArrayAndTopLevelDecoy() {
+ InMemoryCatalog catalog = new InMemoryCatalog();
+ catalog.initialize("test", Collections.emptyMap());
+ catalog.createNamespace(Namespace.of("db1"));
+ Schema schema = new Schema(
+ Types.NestedField.optional(1, "Root", Types.StructType.of(
+ Types.NestedField.optional(2, "Leaf",
+ Types.ListType.ofOptional(3,
Types.IntegerType.get())))),
+ Types.NestedField.optional(4, "LEAF",
+ Types.ListType.ofOptional(5, Types.FloatType.get())));
+ return catalog.createTable(TableIdentifier.of("db1", "t1"), schema);
+ }
+
// ---------- addColumn ----------
@Test
@@ -290,6 +306,37 @@ public class IcebergConnectorMetadataColumnEvolutionTest {
Assertions.assertEquals("Cannot change int to smallint in nested
types", ex.getMessage());
}
+ @Test
+ public void testModifyNestedColumnBuildErrorUsesFullTargetPath() {
+ RecordingIcebergCatalogOps ops = new RecordingIcebergCatalogOps();
+ ops.table = tableWithNestedArrayAndTopLevelDecoy();
+ RecordingConnectorContext ctx = new RecordingConnectorContext();
+ ConnectorColumn leaf = new ConnectorColumn("leaf",
+ ConnectorType.arrayOf(ConnectorType.of("SMALLINT")), "", true,
null, false);
+
+ DorisConnectorException ex =
Assertions.assertThrows(DorisConnectorException.class,
+ () -> metadata(ops, ctx).modifyNestedColumn(null, HANDLE,
+ ConnectorColumnPath.of(Arrays.asList("root", "leaf")),
leaf, null));
+
+ // Error parity must use the resolved nested target, never a
same-named top-level field.
+ Assertions.assertEquals("Cannot change int to smallint in nested
types", ex.getMessage());
+ }
+
+ @Test
+ public void testModifyMissingComplexColumnKeepsBuildError() {
+ RecordingIcebergCatalogOps ops = new RecordingIcebergCatalogOps();
+ ops.table = tableWithArrayIntColumn();
+ RecordingConnectorContext ctx = new RecordingConnectorContext();
+ ConnectorColumn missing = new ConnectorColumn("missing",
+ ConnectorType.arrayOf(ConnectorType.of("SMALLINT")), "", true,
null, false);
+
+ DorisConnectorException ex =
Assertions.assertThrows(DorisConnectorException.class,
+ () -> metadata(ops, ctx).modifyColumn(null, HANDLE, missing,
null));
+
+ // Error upgrading is best-effort; an unresolved target must retain
the original build failure.
+ Assertions.assertEquals("Unsupported type for Iceberg: SMALLINT",
ex.getMessage());
+ }
+
@Test
public void testModifyScalarColumnToUnrepresentableKeepsBuildError() {
// A TOP-LEVEL (non-nested) modify to an iceberg-unrepresentable type
keeps the generic build error: the
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/connector/converter/ConnectorColumnConverter.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/connector/converter/ConnectorColumnConverter.java
index 8a2d4dd38c6..24986554c64 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/connector/converter/ConnectorColumnConverter.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/connector/converter/ConnectorColumnConverter.java
@@ -238,7 +238,7 @@ public final class ConnectorColumnConverter {
// isCommentSpecified() so the diff can tell an omitted COMMENT
(preserve the current doc) from
// COMMENT '' (clear it) — the comment string is "" for both
(#65329 omit-preserves-metadata).
for (StructField f : struct.getFields()) {
- names.add(f.getName());
+ names.add(f.getOriginalName());
types.add(toConnectorType(f.getType()));
nullables.add(f.getContainsNull());
comments.add(f.getComment());
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/ExpressionAnalyzer.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/ExpressionAnalyzer.java
index fd7db58c0b0..bf7e872de20 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/ExpressionAnalyzer.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/ExpressionAnalyzer.java
@@ -273,7 +273,9 @@ public class ExpressionAnalyzer extends
SubExprAnalyzer<ExpressionRewriteContext
StructType structType = (StructType) dataType;
StructField field =
structType.getField(dereferenceExpression.fieldName);
if (field != null) {
- return new ElementAt(expression,
dereferenceExpression.child(1));
+ // This newly constructed node returns directly and will not
be revisited by visitElementAt.
+ return canonicalizeStructSelector(
+ new ElementAt(expression,
dereferenceExpression.child(1)));
}
} else if (dataType.isMapType()) {
return new ElementAt(expression, dereferenceExpression.child(1));
@@ -299,6 +301,7 @@ public class ExpressionAnalyzer extends
SubExprAnalyzer<ExpressionRewriteContext
}
Expression right = elementAt.right().accept(this, context);
elementAt = (ElementAt) elementAt.withChildren(left, right);
+ elementAt = canonicalizeStructSelector(elementAt);
Expression coerced = TypeCoercionUtils.processBoundFunction(elementAt);
if (isEnableVariantSchemaAutoCast(context)) {
return wrapVariantElementAtWithCast(coerced);
@@ -617,7 +620,12 @@ public class ExpressionAnalyzer extends
SubExprAnalyzer<ExpressionRewriteContext
// we do type coercion in build function in alias function, so
it's ok to return directly.
return buildResult.first;
} else {
- Expression castFunction =
TypeCoercionUtils.processBoundFunction((BoundFunction) buildResult.first);
+ BoundFunction boundFunction = (BoundFunction) buildResult.first;
+ if (boundFunction instanceof ElementAt) {
+ // SQL function syntax binds here directly and therefore does
not visit visitElementAt above.
+ boundFunction = canonicalizeStructSelector((ElementAt)
boundFunction);
+ }
+ Expression castFunction =
TypeCoercionUtils.processBoundFunction(boundFunction);
if (castFunction instanceof RewriteWhenAnalyze) {
castFunction = ((RewriteWhenAnalyze)
castFunction).rewriteWhenAnalyze();
}
@@ -631,6 +639,20 @@ public class ExpressionAnalyzer extends
SubExprAnalyzer<ExpressionRewriteContext
return TypeCoercionUtils.processBoundFunction(boundFunction);
}
+ private ElementAt canonicalizeStructSelector(ElementAt elementAt) {
+ Expression left = elementAt.left();
+ Expression right = elementAt.right();
+ if (left.getDataType() instanceof StructType && right instanceof
StringLikeLiteral) {
+ String selector = ((StringLikeLiteral) right).getStringValue();
+ StructField field = ((StructType)
left.getDataType()).getField(selector);
+ if (field != null && !field.getName().equals(selector)) {
+ // BE struct names use the normalized thrift identity and
cannot Unicode-fold external spelling.
+ return (ElementAt) elementAt.withChildren(left, new
StringLiteral(field.getName()));
+ }
+ }
+ return elementAt;
+ }
+
@Override
public Expression visitWindow(WindowExpression windowExpression,
ExpressionRewriteContext context) {
windowExpression = (WindowExpression)
super.visitWindow(windowExpression, context);
@@ -1286,7 +1308,8 @@ public class ExpressionAnalyzer extends
SubExprAnalyzer<ExpressionRewriteContext
throw new AnalysisException("No such struct field '" +
fieldName + "' in '" + lastFieldName + "'");
}
lastFieldName = fieldName;
- expression = new ElementAt(expression, new
StringLiteral(fieldName));
+ // Dereference-created selectors also cross the thrift
boundary and must use runtime identity.
+ expression = new ElementAt(expression, new
StringLiteral(field.getName()));
continue;
} else if (dataType.isMapType()) {
expression = new ElementAt(expression, new
StringLiteral(fieldName));
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AccessPathExpressionCollector.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AccessPathExpressionCollector.java
index 316a0f76833..f03047e6348 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AccessPathExpressionCollector.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AccessPathExpressionCollector.java
@@ -74,6 +74,7 @@ import com.google.common.collect.Multimap;
import java.util.ArrayList;
import java.util.LinkedList;
import java.util.List;
+import java.util.Locale;
import java.util.Map;
import java.util.Map.Entry;
import java.util.Objects;
@@ -148,7 +149,7 @@ public class AccessPathExpressionCollector extends
DefaultExpressionVisitor<Void
return null;
}
if (dataType instanceof NestedColumnPrunable) {
-
context.accessPathBuilder.addPrefix(slotReference.getName().toLowerCase());
+
context.accessPathBuilder.addPrefix(slotReference.getName().toLowerCase(Locale.ROOT));
ImmutableList<String> path =
Utils.fastToImmutableList(context.accessPathBuilder.accessPath);
int slotId = slotReference.getExprId().asInt();
slotToAccessPaths.put(slotId, new CollectAccessPathResult(path,
context.bottomFilter, context.type));
@@ -359,7 +360,8 @@ public class AccessPathExpressionCollector extends
DefaultExpressionVisitor<Void
return continueCollectAccessPath(first, context);
}
}
- context.accessPathBuilder.addPrefix(((Literal)
fieldName).getStringValue().toLowerCase());
+ context.accessPathBuilder.addPrefix(
+ ((Literal)
fieldName).getStringValue().toLowerCase(Locale.ROOT));
return continueCollectAccessPath(first, context);
}
return visit(elementAt, context);
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/NestedColumnPruning.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/NestedColumnPruning.java
index 3faf0d581a5..64153355fc3 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/NestedColumnPruning.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/NestedColumnPruning.java
@@ -56,6 +56,7 @@ import java.util.Collection;
import java.util.Comparator;
import java.util.LinkedHashMap;
import java.util.List;
+import java.util.Locale;
import java.util.Map;
import java.util.Map.Entry;
import java.util.Optional;
@@ -620,7 +621,8 @@ public class NestedColumnPruning implements CustomRewriter {
List<StructField> fields = ((StructType)
cast.type).getFields();
for (int i = 0; i < fields.size(); i++) {
String castFieldName = path.get(index);
- if
(fields.get(i).getName().equalsIgnoreCase(castFieldName)) {
+ // Struct runtime keys are ROOT-normalized; broad folding
can merge distinct siblings.
+ if (fields.get(i).getName().equals(castFieldName)) {
String originFieldName = ((StructType)
type).getFields().get(i).getName();
path.set(index, originFieldName);
return
children.get(originFieldName).replacePathByAnotherTree(
@@ -662,7 +664,7 @@ public class NestedColumnPruning implements CustomRewriter {
accessPartialChild = true;
if (this.type.isStructType()) {
- String fieldName = path.get(accessIndex).toLowerCase();
+ String fieldName =
path.get(accessIndex).toLowerCase(Locale.ROOT);
DataTypeAccessTree child = children.get(fieldName);
if (child != null) {
child.setAccessByPath(path, accessIndex + 1, pathType);
@@ -730,7 +732,8 @@ public class NestedColumnPruning implements CustomRewriter {
accessAll = true;
return;
} else if (isRoot) {
-
children.get(path.get(accessIndex).toLowerCase()).setAccessByPath(path,
accessIndex + 1, pathType);
+ children.get(path.get(accessIndex).toLowerCase(Locale.ROOT))
+ .setAccessByPath(path, accessIndex + 1, pathType);
return;
}
throw new AnalysisException("unsupported data type: " + this.type);
@@ -739,7 +742,7 @@ public class NestedColumnPruning implements CustomRewriter {
public static DataTypeAccessTree ofRoot(Slot slot,
ColumnAccessPathType pathType) {
DataTypeAccessTree child = of(slot.getDataType(), pathType);
DataTypeAccessTree root = new DataTypeAccessTree(true,
NullType.INSTANCE, pathType);
- root.children.put(slot.getName().toLowerCase(), child);
+ root.children.put(slot.getName().toLowerCase(Locale.ROOT), child);
return root;
}
@@ -749,7 +752,8 @@ public class NestedColumnPruning implements CustomRewriter {
if (type instanceof StructType) {
StructType structType = (StructType) type;
for (Entry<String, StructField> kv :
structType.getNameToFields().entrySet()) {
- root.children.put(kv.getKey().toLowerCase(),
of(kv.getValue().getDataType(), pathType));
+ root.children.put(kv.getKey().toLowerCase(Locale.ROOT),
+ of(kv.getValue().getDataType(), pathType));
}
} else if (type instanceof ArrayType) {
root.children.put(AccessPathInfo.ACCESS_ALL, of(((ArrayType)
type).getItemType(), pathType));
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/CreateNamedStruct.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/CreateNamedStruct.java
index c27b5cc94c9..49d089bd4a6 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/CreateNamedStruct.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/CreateNamedStruct.java
@@ -35,6 +35,7 @@ import com.google.common.collect.ImmutableList;
import com.google.common.collect.Sets;
import java.util.List;
+import java.util.Locale;
import java.util.Set;
/**
@@ -70,7 +71,7 @@ public class CreateNamedStruct extends ScalarFunction
implements CustomSignature
throw new AnalysisException("named_struct only allows"
+ " constant string parameter in odd position: " +
this);
} else {
- String name = ((StringLikeLiteral)
child(i)).getStringValue().toLowerCase();
+ String name = ((StringLikeLiteral)
child(i)).getStringValue().toLowerCase(Locale.ROOT);
if (names.contains(name)) {
throw new AnalysisException("The name of the struct field
cannot be repeated."
+ " same name fields are " + name);
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/types/DataType.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/types/DataType.java
index d70079555c7..6c62dbc2bd0 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/types/DataType.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/types/DataType.java
@@ -467,9 +467,9 @@ public abstract class DataType {
if (type.isStructType()) {
List<StructField> structFields =
((org.apache.doris.catalog.StructType) (type)).getFields().stream()
- .map(cf -> new StructField(cf.getName(),
fromCatalogType(cf.getType()),
+ .map(cf -> new StructField(cf.getName(),
cf.getOriginalName(), fromCatalogType(cf.getType()),
cf.getContainsNull(), cf.getComment() == null ? ""
: cf.getComment(),
- cf.isCommentSpecified()))
+ cf.isCommentSpecified(), !cf.hasOriginalName()))
.collect(ImmutableList.toImmutableList());
return new StructType(structFields);
} else if (type.isMapType()) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/types/StructField.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/types/StructField.java
index aefcf20f227..7e8699293ae 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/types/StructField.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/types/StructField.java
@@ -22,6 +22,7 @@ import org.apache.doris.nereids.parser.NereidsParser;
import org.apache.doris.nereids.util.SqlLiteralUtils;
import org.apache.doris.nereids.util.Utils;
+import java.util.Locale;
import java.util.Objects;
/**
@@ -32,10 +33,12 @@ public class StructField {
public static final String DEFAULT_FIELD_NAME = "col";
private final String name;
+ private final String originalName;
private final DataType dataType;
private final boolean nullable;
private final String comment;
private final boolean commentSpecified;
+ private final boolean legacyLocaleDependentName;
/**
* StructField Constructor
@@ -49,17 +52,44 @@ public class StructField {
public StructField(String name, DataType dataType, boolean nullable,
String comment,
boolean commentSpecified) {
- this.name = Objects.requireNonNull(name, "name should not be
null").toLowerCase();
+ this(name, name, dataType, nullable, comment, commentSpecified);
+ }
+
+ /**
+ * Creates a field with separate names for case-insensitive runtime lookup
and external schema spelling.
+ *
+ * @param name field name normalized internally for runtime lookup
+ * @param originalName field spelling preserved for external schema
metadata
+ * @param dataType field data type
+ * @param nullable whether the field accepts null values
+ * @param comment field comment
+ * @param commentSpecified whether the comment was explicitly specified
+ */
+ public StructField(String name, String originalName, DataType dataType,
boolean nullable, String comment,
+ boolean commentSpecified) {
+ this(name, originalName, dataType, nullable, comment,
commentSpecified, false);
+ }
+
+ StructField(String name, String originalName, DataType dataType, boolean
nullable, String comment,
+ boolean commentSpecified, boolean legacyLocaleDependentName) {
+ // Runtime field identity must stay stable across FE locales and match
external schema lookup keys.
+ this.name = Objects.requireNonNull(name, "name should not be
null").toLowerCase(Locale.ROOT);
+ this.originalName = Objects.requireNonNull(originalName, "originalName
should not be null");
this.dataType = Objects.requireNonNull(dataType, "dataType should not
be null");
this.nullable = nullable;
this.comment = Objects.requireNonNull(comment, "comment should not be
null");
this.commentSpecified = commentSpecified;
+ this.legacyLocaleDependentName = legacyLocaleDependentName;
}
public String getName() {
return name;
}
+ public String getOriginalName() {
+ return originalName;
+ }
+
public DataType getDataType() {
return dataType;
}
@@ -76,6 +106,10 @@ public class StructField {
return commentSpecified;
}
+ boolean isLegacyLocaleDependentName() {
+ return legacyLocaleDependentName;
+ }
+
public StructField conversion() {
if (this.dataType.equals(dataType.conversion())) {
return this;
@@ -84,16 +118,19 @@ public class StructField {
}
public StructField withDataType(DataType dataType) {
- return new StructField(name, dataType, nullable, comment,
commentSpecified);
+ return new StructField(name, originalName, dataType, nullable,
comment, commentSpecified,
+ legacyLocaleDependentName);
}
public StructField withDataTypeAndNullable(DataType dataType, boolean
nullable) {
- return new StructField(name, dataType, nullable, comment,
commentSpecified);
+ return new StructField(name, originalName, dataType, nullable,
comment, commentSpecified,
+ legacyLocaleDependentName);
}
public org.apache.doris.catalog.StructField toCatalogDataType() {
return new org.apache.doris.catalog.StructField(
- name, dataType.toCatalogDataType(), comment, nullable,
commentSpecified);
+ name, legacyLocaleDependentName ? null : originalName,
+ dataType.toCatalogDataType(), comment, nullable,
commentSpecified);
}
public String toSql() {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/types/StructType.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/types/StructType.java
index 13f28c2e06e..8f2893ec55f 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/types/StructType.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/types/StructType.java
@@ -29,6 +29,7 @@ import java.util.ArrayList;
import java.util.HashMap;
import java.util.LinkedHashMap;
import java.util.List;
+import java.util.Locale;
import java.util.Map;
import java.util.Objects;
import java.util.stream.Collectors;
@@ -60,7 +61,7 @@ public class StructType extends DataType implements
ComplexDataType, NestedColum
// ATTN: should use LinkedHashMap to keep order
this.nameToFields = new LinkedHashMap<>();
for (StructField field : this.fields) {
- String fieldName = field.getName().toLowerCase();
+ String fieldName = field.getName().toLowerCase(Locale.ROOT);
StructField existingField = this.nameToFields.put(fieldName,
field);
if (existingField != null) {
throw new AnalysisException("Duplicate field name found: " +
fieldName);
@@ -76,8 +77,25 @@ public class StructType extends DataType implements
ComplexDataType, NestedColum
return nameToFields;
}
+ /** Get a field by its case-insensitive runtime name. */
public StructField getField(String name) {
- return nameToFields.get(name.toLowerCase());
+ StructField field = nameToFields.get(name.toLowerCase(Locale.ROOT));
+ if (field != null && (!field.isLegacyLocaleDependentName() ||
field.getName().equals(name))) {
+ return field;
+ }
+ StructField legacyMatch = null;
+ for (int i = fields.size() - 1; i >= 0; i--) {
+ StructField legacyField = fields.get(i);
+ // Limit broad case folding to old replayed fields so new
ROOT-distinct names remain distinct.
+ if (legacyField.isLegacyLocaleDependentName() &&
legacyField.getName().equalsIgnoreCase(name)) {
+ if (legacyMatch != null) {
+ // The old locale was not persisted, so choosing either
folded sibling could return wrong data.
+ return null;
+ }
+ legacyMatch = legacyField;
+ }
+ }
+ return legacyMatch != null ? legacyMatch : field;
}
@Override
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/ExternalFileTableValuedFunction.java
b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/ExternalFileTableValuedFunction.java
index 01abd8b4821..eb21be855a8 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/ExternalFileTableValuedFunction.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/ExternalFileTableValuedFunction.java
@@ -98,6 +98,7 @@ import java.util.Arrays;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
+import java.util.Locale;
import java.util.Map;
import java.util.Optional;
import java.util.Set;
@@ -421,13 +422,16 @@ public abstract class ExternalFileTableValuedFunction
extends TableValuedFunctio
for (int i = 0; i < typeNodes.get(start).getStructFieldsCount();
++i) {
Pair<Type, Integer> fieldType = getColumnType(typeNodes, start
+ parsedNodes);
PStructField structField =
typeNodes.get(start).getStructFields(i);
- String fieldName = structField.getName().toLowerCase();
+ String originalFieldName = structField.getName();
+ String fieldName = originalFieldName.toLowerCase(Locale.ROOT);
if (fieldLowerNames.contains(fieldName)) {
throw new NotSupportedException("Repeated lowercase field
names: " + fieldName);
} else {
fieldLowerNames.add(fieldName);
- fields.add(new StructField(fieldName, fieldType.key(),
structField.getComment(),
- structField.getContainsNull()));
+ // File readers return the external schema spelling, which
must survive CTAS metadata writes;
+ // only the runtime lookup key and duplicate detection are
normalized.
+ fields.add(new StructField(fieldName, originalFieldName,
fieldType.key(), structField.getComment(),
+ structField.getContainsNull(),
!structField.getComment().isEmpty()));
}
parsedNodes += fieldType.value();
}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/catalog/ColumnGsonSerializationTest.java
b/fe/fe-core/src/test/java/org/apache/doris/catalog/ColumnGsonSerializationTest.java
index 6e81121ca04..dddc1a7ecf1 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/catalog/ColumnGsonSerializationTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/catalog/ColumnGsonSerializationTest.java
@@ -23,6 +23,10 @@ import org.apache.doris.common.io.Writable;
import org.apache.doris.persist.gson.GsonUtils;
import com.google.common.collect.Lists;
+import com.google.gson.JsonArray;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParser;
import com.google.gson.annotations.SerializedName;
import org.junit.After;
import org.junit.Assert;
@@ -121,4 +125,72 @@ public class ColumnGsonSerializationTest {
in.close();
}
+ @Test
+ public void testReplayPreRootTurkishStructFieldName() {
+ StructType structType = new StructType(new StructField("I", Type.INT));
+ JsonObject legacyJson = JsonParser.parseString(
+ GsonUtils.GSON.toJson(structType,
Type.class)).getAsJsonObject();
+
+ JsonObject fieldMap = legacyJson.getAsJsonObject("fieldMap");
+ JsonElement legacyMapField = fieldMap.remove("i");
+ setLegacyTurkishFieldName(legacyMapField.getAsJsonObject());
+ fieldMap.add("ı", legacyMapField);
+ JsonArray fields = legacyJson.getAsJsonArray("fields");
+ setLegacyTurkishFieldName(fields.get(0).getAsJsonObject());
+
+ StructType replayed = (StructType) GsonUtils.GSON.fromJson(legacyJson,
Type.class);
+ Assert.assertEquals("ı", replayed.getField("I").getName());
+
+ org.apache.doris.nereids.types.StructType nereidsType =
+ (org.apache.doris.nereids.types.StructType)
+
org.apache.doris.nereids.types.DataType.fromCatalogType(replayed);
+ Assert.assertEquals("ı", nereidsType.getField("I").getName());
+
+ StructType roundTrip = (StructType) nereidsType.toCatalogDataType();
+ Assert.assertEquals("ı", roundTrip.getField("I").getName());
+ }
+
+ @Test
+ public void testCurrentDotlessStructFieldDoesNotMatchAsciiI() {
+ StructType structType = new StructType(new StructField("ı", Type.INT));
+ Assert.assertNull(structType.getField("I"));
+
+ org.apache.doris.nereids.types.StructType nereidsType =
+ (org.apache.doris.nereids.types.StructType)
+
org.apache.doris.nereids.types.DataType.fromCatalogType(structType);
+ Assert.assertNull(nereidsType.getField("I"));
+ }
+
+ @Test
+ public void testReplayPreRootTurkishStructFieldCollisionIsAmbiguous() {
+ StructType structType = new StructType(
+ new StructField("ı", Type.INT), new StructField("i",
Type.BIGINT));
+ JsonObject legacyJson = JsonParser.parseString(
+ GsonUtils.GSON.toJson(structType,
Type.class)).getAsJsonObject();
+ legacyJson.getAsJsonObject("fieldMap").entrySet().forEach(
+ entry ->
entry.getValue().getAsJsonObject().remove("originalName"));
+ legacyJson.getAsJsonArray("fields").forEach(
+ field -> field.getAsJsonObject().remove("originalName"));
+
+ StructType replayed = (StructType) GsonUtils.GSON.fromJson(legacyJson,
Type.class);
+ Assert.assertNull(replayed.getField("I"));
+ Assert.assertEquals("ı", replayed.getField("ı").getName());
+ Assert.assertEquals("i", replayed.getField("i").getName());
+
+ org.apache.doris.nereids.types.StructType nereidsType =
+ (org.apache.doris.nereids.types.StructType)
+
org.apache.doris.nereids.types.DataType.fromCatalogType(replayed);
+ Assert.assertNull(nereidsType.getField("I"));
+ Assert.assertEquals("ı", nereidsType.getField("ı").getName());
+ Assert.assertEquals("i", nereidsType.getField("i").getName());
+
+ StructType roundTrip = (StructType) nereidsType.toCatalogDataType();
+ Assert.assertNull(roundTrip.getField("I"));
+ }
+
+ private static void setLegacyTurkishFieldName(JsonObject field) {
+ field.addProperty("name", "ı");
+ field.remove("originalName");
+ }
+
}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/connector/ddl/CreateTableInfoToConnectorRequestConverterTest.java
b/fe/fe-core/src/test/java/org/apache/doris/connector/ddl/CreateTableInfoToConnectorRequestConverterTest.java
index 6abe426c049..471acd6861d 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/connector/ddl/CreateTableInfoToConnectorRequestConverterTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/connector/ddl/CreateTableInfoToConnectorRequestConverterTest.java
@@ -37,6 +37,8 @@ import
org.apache.doris.nereids.trees.plans.commands.info.PartitionTableInfo;
import org.apache.doris.nereids.trees.plans.commands.info.SortFieldInfo;
import org.apache.doris.nereids.types.IntegerType;
import org.apache.doris.nereids.types.StringType;
+import org.apache.doris.nereids.types.StructField;
+import org.apache.doris.nereids.types.StructType;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableMap;
@@ -98,6 +100,20 @@ public class CreateTableInfoToConnectorRequestConverterTest
{
Assertions.assertNull(req.getBucketSpec());
}
+ @Test
+ public void nestedFieldSpellingIsPreservedForConnectorSchemas() {
+ StructType payloadType = new StructType(ImmutableList.of(
+ new StructField("CaseSensitive", IntegerType.INSTANCE, true,
"")));
+ ColumnDefinition payload = new ColumnDefinition("payload",
payloadType, true);
+ CreateTableInfo info = stubInfo("t",
Collections.singletonList(payload),
+ null, null, "", Collections.emptyMap(), false);
+
+ ConnectorCreateTableRequest request =
CreateTableInfoToConnectorRequestConverter.convert(info, "db");
+
+ Assertions.assertEquals(Collections.singletonList("CaseSensitive"),
+ request.getColumns().get(0).getType().getFieldNames());
+ }
+
@Test
public void autoIncInitValueIsPropagatedAsIsAutoInc() {
// ColumnDefinition is mocked (its auto-inc ctor pulls in
ColumnNullableType machinery);
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/datasource/connector/converter/ConnectorColumnConverterTest.java
b/fe/fe-core/src/test/java/org/apache/doris/datasource/connector/converter/ConnectorColumnConverterTest.java
index 71bb443e186..85a5c65f3e5 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/datasource/connector/converter/ConnectorColumnConverterTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/datasource/connector/converter/ConnectorColumnConverterTest.java
@@ -35,6 +35,7 @@ import org.junit.jupiter.api.Test;
import java.util.ArrayList;
import java.util.Arrays;
+import java.util.Locale;
class ConnectorColumnConverterTest {
@@ -117,6 +118,48 @@ class ConnectorColumnConverterTest {
Assertions.assertEquals(ScalarType.INT,
backStruct.getFields().get(0).getType());
}
+ @Test
+ void mixedCaseStructFieldKeepsSchemaSpellingAndNormalizedRuntimeName() {
+ ConnectorType connectorType = ConnectorType.structOf(
+ Arrays.asList("CaseSensitive"),
Arrays.asList(ConnectorType.of("INT")));
+
+ StructType converted = (StructType)
ConnectorColumnConverter.convertType(connectorType);
+ StructField field = converted.getFields().get(0);
+
+ Assertions.assertEquals("casesensitive", field.getName());
+ Assertions.assertEquals("CaseSensitive", field.getOriginalName());
+ Assertions.assertEquals("struct<CaseSensitive:int>",
converted.toSql());
+ Assertions.assertEquals("casesensitive",
+
converted.toThrift().getTypes().get(0).getStructFields().get(0).getName());
+ }
+
+ @Test
+ void structRuntimeNamesUseRootLocale() {
+ Locale originalLocale = Locale.getDefault();
+ try {
+ Locale.setDefault(Locale.forLanguageTag("tr-TR"));
+ ConnectorType connectorType = ConnectorType.structOf(
+ Arrays.asList("I", "ı"),
+ Arrays.asList(ConnectorType.of("INT"),
ConnectorType.of("STRING")));
+
+ StructType catalogType = (StructType)
ConnectorColumnConverter.convertType(connectorType);
+ Assertions.assertEquals("i",
catalogType.getFields().get(0).getName());
+ Assertions.assertEquals("ı",
catalogType.getFields().get(1).getName());
+ Assertions.assertSame(catalogType.getFields().get(0),
catalogType.getField("i"));
+ Assertions.assertSame(catalogType.getFields().get(1),
catalogType.getField("ı"));
+
+ org.apache.doris.nereids.types.StructType nereidsType =
+ (org.apache.doris.nereids.types.StructType)
+
org.apache.doris.nereids.types.DataType.fromCatalogType(catalogType);
+ Assertions.assertEquals("i",
nereidsType.getFields().get(0).getName());
+ Assertions.assertEquals("ı",
nereidsType.getFields().get(1).getName());
+ Assertions.assertSame(nereidsType.getFields().get(0),
nereidsType.getField("i"));
+ Assertions.assertSame(nereidsType.getFields().get(1),
nereidsType.getField("ı"));
+ } finally {
+ Locale.setDefault(originalLocale);
+ }
+ }
+
@Test
void testNestedComplexType() {
// ARRAY<MAP<STRING, INT>>
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/analysis/ExpressionAnalyzerTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/analysis/ExpressionAnalyzerTest.java
index 6c70a37aa0c..4f15f00ab56 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/analysis/ExpressionAnalyzerTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/analysis/ExpressionAnalyzerTest.java
@@ -17,13 +17,16 @@
package org.apache.doris.nereids.rules.analysis;
+import org.apache.doris.nereids.CascadesContext;
import org.apache.doris.nereids.analyzer.Scope;
import org.apache.doris.nereids.analyzer.UnboundFunction;
import org.apache.doris.nereids.analyzer.UnboundSlot;
import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.trees.expressions.Alias;
import org.apache.doris.nereids.trees.expressions.And;
import org.apache.doris.nereids.trees.expressions.BoundStar;
import org.apache.doris.nereids.trees.expressions.Cast;
+import org.apache.doris.nereids.trees.expressions.DereferenceExpression;
import org.apache.doris.nereids.trees.expressions.ExprId;
import org.apache.doris.nereids.trees.expressions.Expression;
import org.apache.doris.nereids.trees.expressions.IsFalse;
@@ -31,11 +34,17 @@ import org.apache.doris.nereids.trees.expressions.IsNull;
import org.apache.doris.nereids.trees.expressions.IsTrue;
import org.apache.doris.nereids.trees.expressions.Not;
import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.ElementAt;
import org.apache.doris.nereids.trees.expressions.literal.DateTimeV2Literal;
+import org.apache.doris.nereids.trees.expressions.literal.StringLikeLiteral;
import org.apache.doris.nereids.trees.expressions.literal.StringLiteral;
import org.apache.doris.nereids.trees.expressions.literal.TinyIntLiteral;
import org.apache.doris.nereids.types.BigIntType;
import org.apache.doris.nereids.types.BooleanType;
+import org.apache.doris.nereids.types.IntegerType;
+import org.apache.doris.nereids.types.StructField;
+import org.apache.doris.nereids.types.StructType;
+import org.apache.doris.qe.ConnectContext;
import com.google.common.collect.ImmutableList;
import org.junit.jupiter.api.Assertions;
@@ -133,4 +142,96 @@ public class ExpressionAnalyzerTest {
Assertions.assertInstanceOf(Not.class, isNotFalse);
Assertions.assertInstanceOf(And.class, isNotFalse.child(0));
}
+
+ @Test
+ public void testStructElementAtCanonicalizesUnicodeSelector() {
+ StructType structType = new StructType(ImmutableList.of(
+ new StructField("Σ", "Σ", IntegerType.INSTANCE, true, "",
false),
+ new StructField("ẞ", "ẞ", IntegerType.INSTANCE, true, "",
false)));
+ SlotReference payload = new SlotReference(
+ new ExprId(1), "payload", structType, true,
ImmutableList.of());
+ ExpressionAnalyzer analyzer = new ExpressionAnalyzer(null, new
Scope(ImmutableList.of()),
+ null, true, true);
+
+ Expression analyzedSigma = analyzer.analyze(new ElementAt(payload, new
StringLiteral("Σ")));
+ Expression analyzedSharpS = analyzer.analyze(new ElementAt(payload,
new StringLiteral("ẞ")));
+
+ Assertions.assertInstanceOf(ElementAt.class, analyzedSigma);
+ Assertions.assertInstanceOf(ElementAt.class, analyzedSharpS);
+ // The BE receives the ROOT-normalized thrift name and cannot
Unicode-fold the displayed spelling.
+ Assertions.assertEquals("σ", ((StringLikeLiteral)
analyzedSigma.child(1)).getStringValue());
+ Assertions.assertEquals("ß", ((StringLikeLiteral)
analyzedSharpS.child(1)).getStringValue());
+ }
+
+ @Test
+ public void testStructElementAtFunctionCanonicalizesUnicodeSelector() {
+ StructType structType = new StructType(ImmutableList.of(
+ new StructField("Σ", "Σ", IntegerType.INSTANCE, true, "",
false)));
+ SlotReference payload = new SlotReference(
+ new ExprId(1), "payload", structType, true,
ImmutableList.of());
+ ExpressionAnalyzer analyzer = new ExpressionAnalyzer(null, new
Scope(ImmutableList.of()),
+ null, true, true);
+
+ Expression analyzed = analyzer.analyze(new
UnboundFunction("element_at",
+ ImmutableList.of(payload, new StringLiteral("Σ"))));
+
+ Assertions.assertInstanceOf(ElementAt.class, analyzed);
+ Assertions.assertEquals("σ", ((StringLikeLiteral)
analyzed.child(1)).getStringValue());
+ }
+
+ @Test
+ public void testStructDereferenceCanonicalizesUnicodeSelector() {
+ StructType structType = new StructType(ImmutableList.of(
+ new StructField("Σ", "Σ", IntegerType.INSTANCE, true, "",
false)));
+ SlotReference payload = new SlotReference(
+ new ExprId(1), "payload", structType, true,
ImmutableList.of());
+ ConnectContext connectContext = new ConnectContext();
+ connectContext.setThreadLocalInfo();
+ try {
+ CascadesContext cascadesContext =
CascadesContext.initTempContext();
+ ExpressionAnalyzer analyzer = new ExpressionAnalyzer(null, new
Scope(ImmutableList.of(payload)),
+ cascadesContext, true, true);
+
+ Expression analyzed = analyzer.analyze(new UnboundSlot("payload",
"Σ"));
+
+ Assertions.assertInstanceOf(Alias.class, analyzed);
+ Assertions.assertInstanceOf(ElementAt.class, analyzed.child(0));
+ Assertions.assertEquals("σ", ((StringLikeLiteral)
analyzed.child(0).child(1)).getStringValue());
+ } finally {
+ ConnectContext.remove();
+ }
+ }
+
+ @Test
+ public void testComputedStructDereferenceCanonicalizesUnicodeSelector() {
+ StructType structType = new StructType(ImmutableList.of(
+ new StructField("Σ", "Σ", IntegerType.INSTANCE, true, "",
false)));
+ SlotReference payload = new SlotReference(
+ new ExprId(1), "payload", structType, true,
ImmutableList.of());
+ ExpressionAnalyzer analyzer = new ExpressionAnalyzer(null, new
Scope(ImmutableList.of()),
+ null, true, true);
+
+ Expression analyzed = analyzer.analyze(new DereferenceExpression(
+ new Cast(payload, structType), new StringLiteral("Σ")));
+
+ Assertions.assertInstanceOf(ElementAt.class, analyzed);
+ Assertions.assertEquals("σ", ((StringLikeLiteral)
analyzed.child(1)).getStringValue());
+ }
+
+ @Test
+ public void testLegacyStructFieldCollisionRejectsAmbiguousSelector() {
+ org.apache.doris.catalog.StructType catalogType = new
org.apache.doris.catalog.StructType(
+ new org.apache.doris.catalog.StructField(
+ "ı", null, org.apache.doris.catalog.Type.INT, "",
true, false),
+ new org.apache.doris.catalog.StructField(
+ "i", null, org.apache.doris.catalog.Type.BIGINT, "",
true, false));
+ StructType structType = (StructType)
org.apache.doris.nereids.types.DataType.fromCatalogType(catalogType);
+ SlotReference payload = new SlotReference(
+ new ExprId(1), "payload", structType, true,
ImmutableList.of());
+ ExpressionAnalyzer analyzer = new ExpressionAnalyzer(null, new
Scope(ImmutableList.of()),
+ null, true, true);
+
+ Assertions.assertThrows(AnalysisException.class,
+ () -> analyzer.analyze(new ElementAt(payload, new
StringLiteral("I"))));
+ }
}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/AccessPathExpressionCollectorTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/AccessPathExpressionCollectorTest.java
new file mode 100644
index 00000000000..d06e6905109
--- /dev/null
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/AccessPathExpressionCollectorTest.java
@@ -0,0 +1,102 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.analysis.ColumnAccessPathType;
+import
org.apache.doris.nereids.rules.rewrite.AccessPathExpressionCollector.CollectAccessPathResult;
+import
org.apache.doris.nereids.rules.rewrite.NestedColumnPruning.DataTypeAccessTree;
+import org.apache.doris.nereids.trees.expressions.Cast;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.ElementAt;
+import org.apache.doris.nereids.trees.expressions.literal.StringLiteral;
+import org.apache.doris.nereids.types.IntegerType;
+import org.apache.doris.nereids.types.StringType;
+import org.apache.doris.nereids.types.StructField;
+import org.apache.doris.nereids.types.StructType;
+
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Multimap;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Locale;
+
+public class AccessPathExpressionCollectorTest {
+
+ @Test
+ public void testStructAccessPathUsesRootLocale() {
+ Locale originalLocale = Locale.getDefault();
+ try {
+ Locale.setDefault(Locale.forLanguageTag("tr-TR"));
+ StructType structType = new StructType(ImmutableList.of(
+ new StructField("I", IntegerType.INSTANCE, true, ""),
+ new StructField("ı", StringType.INSTANCE, true, "")));
+ SlotReference slot = new SlotReference("I", structType);
+ Multimap<Integer, CollectAccessPathResult> accessPaths =
ArrayListMultimap.create();
+ AccessPathExpressionCollector collector =
+ new AccessPathExpressionCollector(null, accessPaths,
false, false);
+
+ collector.collect(new ElementAt(slot, new StringLiteral("I")));
+
+ List<CollectAccessPathResult> results = new ArrayList<>(
+ accessPaths.get(slot.getExprId().asInt()));
+ Assertions.assertEquals(1, results.size());
+ Assertions.assertEquals(ImmutableList.of("i", "i"),
results.get(0).getPath());
+
+ DataTypeAccessTree tree = DataTypeAccessTree.ofRoot(slot,
ColumnAccessPathType.DATA);
+ tree.setAccessByPath(results.get(0).getPath(), 0,
ColumnAccessPathType.DATA);
+ StructType prunedType = (StructType)
tree.pruneDataType().orElseThrow();
+ Assertions.assertEquals(1, prunedType.getFields().size());
+ Assertions.assertEquals("i",
prunedType.getFields().get(0).getName());
+ Assertions.assertEquals(IntegerType.INSTANCE,
prunedType.getFields().get(0).getDataType());
+ } finally {
+ Locale.setDefault(originalLocale);
+ }
+ }
+
+ @Test
+ public void testCastStructAccessPathKeepsRootKeyIdentity() {
+ StructType originType = new StructType(ImmutableList.of(
+ new StructField("first", IntegerType.INSTANCE, true, ""),
+ new StructField("second", StringType.INSTANCE, true, "")));
+ StructType castType = new StructType(ImmutableList.of(
+ new StructField("I", IntegerType.INSTANCE, true, ""),
+ new StructField("ı", StringType.INSTANCE, true, "")));
+ SlotReference slot = new SlotReference("s", originType);
+ Multimap<Integer, CollectAccessPathResult> accessPaths =
ArrayListMultimap.create();
+ AccessPathExpressionCollector collector =
+ new AccessPathExpressionCollector(null, accessPaths, false,
false);
+
+ collector.collect(new ElementAt(new Cast(slot, castType), new
StringLiteral("ı")));
+
+ List<CollectAccessPathResult> results = new ArrayList<>(
+ accessPaths.get(slot.getExprId().asInt()));
+ Assertions.assertEquals(1, results.size());
+ Assertions.assertEquals(ImmutableList.of("s", "second"),
results.get(0).getPath());
+
+ DataTypeAccessTree tree = DataTypeAccessTree.ofRoot(slot,
ColumnAccessPathType.DATA);
+ tree.setAccessByPath(results.get(0).getPath(), 0,
ColumnAccessPathType.DATA);
+ StructType prunedType = (StructType)
tree.pruneDataType().orElseThrow();
+ Assertions.assertEquals(1, prunedType.getFields().size());
+ Assertions.assertEquals("second",
prunedType.getFields().get(0).getName());
+ Assertions.assertEquals(StringType.INSTANCE,
prunedType.getFields().get(0).getDataType());
+ }
+}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/expressions/literal/StructLiteralTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/expressions/literal/StructLiteralTest.java
index 1982a6ae67f..835f7ba344d 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/expressions/literal/StructLiteralTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/expressions/literal/StructLiteralTest.java
@@ -36,6 +36,8 @@ import org.junit.jupiter.api.Test;
import org.mockito.MockedStatic;
import org.mockito.Mockito;
+import java.util.Locale;
+
public class StructLiteralTest {
@Test
@@ -74,6 +76,24 @@ public class StructLiteralTest {
Assertions.assertTrue(nullableType.getFields().get(0).isNullable());
}
+ @Test
+ public void testNamedStructFieldIdentityUsesRootLocale() {
+ Locale originalLocale = Locale.getDefault();
+ try {
+ Locale.setDefault(Locale.forLanguageTag("tr-TR"));
+ CreateNamedStruct namedStruct = new CreateNamedStruct(
+ new StringLiteral("I"), new IntegerLiteral(1),
+ new StringLiteral("ı"), new IntegerLiteral(2));
+
+
Assertions.assertDoesNotThrow(namedStruct::checkLegalityBeforeTypeCoercion);
+ StructType type = (StructType)
namedStruct.customSignature().returnType;
+ Assertions.assertEquals("i", type.getFields().get(0).getName());
+ Assertions.assertEquals("ı", type.getFields().get(1).getName());
+ } finally {
+ Locale.setDefault(originalLocale);
+ }
+ }
+
@Test
public void testStructFunctionsKeepPhysicalCastNullabilityInStrictMode() {
SlotReference requiredString = new SlotReference("metric",
StringType.INSTANCE, false);
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/tablefunction/ExternalFileTableValuedFunctionTest.java
b/fe/fe-core/src/test/java/org/apache/doris/tablefunction/ExternalFileTableValuedFunctionTest.java
index 34df4964395..a76d9c39b0c 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/tablefunction/ExternalFileTableValuedFunctionTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/tablefunction/ExternalFileTableValuedFunctionTest.java
@@ -19,10 +19,19 @@ package org.apache.doris.tablefunction;
import org.apache.doris.catalog.Column;
import org.apache.doris.catalog.PrimitiveType;
+import org.apache.doris.catalog.StructField;
+import org.apache.doris.catalog.StructType;
+import org.apache.doris.catalog.Type;
import org.apache.doris.common.AnalysisException;
import org.apache.doris.common.Config;
+import org.apache.doris.common.Pair;
import org.apache.doris.common.util.FileFormatConstants;
import org.apache.doris.common.util.FileFormatUtils;
+import org.apache.doris.proto.Types.PScalarType;
+import org.apache.doris.proto.Types.PStructField;
+import org.apache.doris.proto.Types.PTypeNode;
+import org.apache.doris.thrift.TPrimitiveType;
+import org.apache.doris.thrift.TTypeNodeType;
import com.google.common.collect.Lists;
import com.google.common.collect.Maps;
@@ -30,10 +39,43 @@ import org.junit.Assert;
import org.junit.Test;
import org.mockito.Mockito;
+import java.lang.reflect.Method;
+import java.util.Arrays;
import java.util.List;
import java.util.Map;
public class ExternalFileTableValuedFunctionTest {
+ @Test
+ public void testFileSchemaPreservesNestedFieldSpelling() throws Exception {
+ ExternalFileTableValuedFunction tvf = Mockito.mock(
+ ExternalFileTableValuedFunction.class,
Mockito.CALLS_REAL_METHODS);
+ PTypeNode structNode = PTypeNode.newBuilder()
+ .setType(TTypeNodeType.STRUCT.getValue())
+
.setScalarType(PScalarType.newBuilder().setType(TPrimitiveType.STRUCT.getValue()))
+ .addStructFields(PStructField.newBuilder()
+ .setName("CaseSensitive")
+ .setComment("mixed-case child")
+ .setContainsNull(true))
+ .build();
+ PTypeNode intNode = PTypeNode.newBuilder()
+ .setType(TTypeNodeType.SCALAR.getValue())
+
.setScalarType(PScalarType.newBuilder().setType(TPrimitiveType.INT.getValue()))
+ .build();
+
+ Method getColumnType = ExternalFileTableValuedFunction.class
+ .getDeclaredMethod("getColumnType", List.class, int.class);
+ getColumnType.setAccessible(true);
+ @SuppressWarnings("unchecked")
+ Pair<Type, Integer> parsed = (Pair<Type, Integer>)
getColumnType.invoke(
+ tvf, Arrays.asList(structNode, intNode), 0);
+
+ StructField field = ((StructType) parsed.key()).getFields().get(0);
+ Assert.assertEquals("casesensitive", field.getName());
+ Assert.assertEquals("CaseSensitive", field.getOriginalName());
+ Assert.assertEquals("mixed-case child", field.getComment());
+ Assert.assertTrue(field.getContainsNull());
+ }
+
@Test
public void
testHiveParquetTimeZoneIsCanonicalizedAndRemovedFromStorageProperties()
throws AnalysisException {
diff --git a/fe/fe-type/src/main/java/org/apache/doris/catalog/StructField.java
b/fe/fe-type/src/main/java/org/apache/doris/catalog/StructField.java
index e9432c1efad..a4339d239d6 100644
--- a/fe/fe-type/src/main/java/org/apache/doris/catalog/StructField.java
+++ b/fe/fe-type/src/main/java/org/apache/doris/catalog/StructField.java
@@ -24,10 +24,15 @@ import org.apache.doris.thrift.TTypeNode;
import com.google.common.base.Strings;
import com.google.gson.annotations.SerializedName;
+import java.util.Locale;
+
public class StructField {
@SerializedName(value = "name")
protected final String name;
+ @SerializedName(value = "originalName")
+ protected final String originalName;
+
@SerializedName(value = "type")
protected final Type type;
@@ -51,7 +56,24 @@ public class StructField {
public StructField(String name, Type type, String comment, boolean
containsNull,
boolean commentSpecified) {
- this.name = name.toLowerCase();
+ this(name, name, type, comment, containsNull, commentSpecified);
+ }
+
+ /**
+ * Creates a field with separate names for case-insensitive runtime lookup
and external schema spelling.
+ *
+ * @param name field name normalized internally for runtime lookup
+ * @param originalName field spelling preserved for external schema
metadata
+ * @param type field type
+ * @param comment field comment
+ * @param containsNull whether the field accepts null values
+ * @param commentSpecified whether the comment was explicitly specified
+ */
+ public StructField(String name, String originalName, Type type, String
comment, boolean containsNull,
+ boolean commentSpecified) {
+ // Keep runtime identity locale-independent while preserving external
schema spelling separately.
+ this.name = name.toLowerCase(Locale.ROOT);
+ this.originalName = originalName;
this.type = type;
this.comment = comment;
this.containsNull = containsNull;
@@ -82,6 +104,15 @@ public class StructField {
return name;
}
+ public String getOriginalName() {
+ return originalName == null ? name : originalName;
+ }
+
+ /** Whether this field was persisted with its external schema spelling. */
+ public boolean hasOriginalName() {
+ return originalName != null;
+ }
+
public Type getType() {
return type;
}
@@ -105,7 +136,7 @@ public class StructField {
} else {
typeSql = "...";
}
- StringBuilder sb = new StringBuilder(name);
+ StringBuilder sb = new StringBuilder(getOriginalName());
if (type != null) {
sb.append(":").append(typeSql);
}
@@ -121,7 +152,7 @@ public class StructField {
*/
public String prettyPrint(int lpad) {
String leftPadding = Strings.repeat(" ", lpad);
- StringBuilder sb = new StringBuilder(leftPadding + name);
+ StringBuilder sb = new StringBuilder(leftPadding + getOriginalName());
if (type != null) {
// Pass in the padding to make sure nested fields are aligned
properly,
// even if we then strip the top-level padding.
@@ -162,7 +193,7 @@ public class StructField {
@Override
public String toString() {
- StringBuilder sb = new StringBuilder(name);
+ StringBuilder sb = new StringBuilder(getOriginalName());
if (type != null) {
sb.append(":").append(type);
}
diff --git a/fe/fe-type/src/main/java/org/apache/doris/catalog/StructType.java
b/fe/fe-type/src/main/java/org/apache/doris/catalog/StructType.java
index df06da74313..28e3a074382 100644
--- a/fe/fe-type/src/main/java/org/apache/doris/catalog/StructType.java
+++ b/fe/fe-type/src/main/java/org/apache/doris/catalog/StructType.java
@@ -33,6 +33,7 @@ import java.util.ArrayList;
import java.util.Arrays;
import java.util.HashMap;
import java.util.List;
+import java.util.Locale;
import java.util.Objects;
/**
@@ -51,7 +52,7 @@ public class StructType extends Type {
this.fields = fields;
for (int i = 0; i < this.fields.size(); ++i) {
this.fields.get(i).setPosition(i);
- fieldMap.put(this.fields.get(i).getName().toLowerCase(),
this.fields.get(i));
+
fieldMap.put(this.fields.get(i).getName().toLowerCase(Locale.ROOT),
this.fields.get(i));
}
}
@@ -122,7 +123,7 @@ public class StructType extends Type {
public void addField(StructField field) {
field.setPosition(fields.size());
fields.add(field);
- fieldMap.put(field.getName().toLowerCase(), field);
+ fieldMap.put(field.getName().toLowerCase(Locale.ROOT), field);
}
public ArrayList<StructField> getFields() {
@@ -130,7 +131,23 @@ public class StructType extends Type {
}
public StructField getField(String fieldName) {
- return fieldMap.get(fieldName.toLowerCase());
+ StructField field = fieldMap.get(fieldName.toLowerCase(Locale.ROOT));
+ if (field != null && (field.hasOriginalName() ||
field.getName().equals(fieldName))) {
+ return field;
+ }
+ StructField legacyMatch = null;
+ for (int i = fields.size() - 1; i >= 0; i--) {
+ StructField legacyField = fields.get(i);
+ // Old images lack originalName and may contain keys normalized
with the FE's default locale.
+ if (!legacyField.hasOriginalName() &&
legacyField.getName().equalsIgnoreCase(fieldName)) {
+ if (legacyMatch != null) {
+ // The old locale was not persisted, so choosing either
folded sibling could return wrong data.
+ return null;
+ }
+ legacyMatch = legacyField;
+ }
+ }
+ return legacyMatch != null ? legacyMatch : field;
}
@Override
diff --git a/fe/fe-type/src/main/java/org/apache/doris/catalog/Type.java
b/fe/fe-type/src/main/java/org/apache/doris/catalog/Type.java
index 38dcd8f359c..fb203cedf9a 100644
--- a/fe/fe-type/src/main/java/org/apache/doris/catalog/Type.java
+++ b/fe/fe-type/src/main/java/org/apache/doris/catalog/Type.java
@@ -526,7 +526,7 @@ public abstract class Type {
StructType structType = (StructType) this;
for (int i = 0; i < structType.getFields().size(); i++) {
StructField field = structType.getFields().get(i);
- StringBuilder desc = new
StringBuilder(field.getName()).append(":")
+ StringBuilder desc = new
StringBuilder(field.getOriginalName()).append(":")
.append(field.getType().hideVersionForVersionColumn(
isToSql, showNestedComment,
noBackslashEscapes));
// Requiredness is schema semantics and must survive
independently of whether
diff --git
a/regression-test/data/external_table_p0/iceberg/test_iceberg_struct_schema_evolution.out
b/regression-test/data/external_table_p0/iceberg/test_iceberg_struct_schema_evolution.out
index a364316df42..8cfc10b8204 100644
---
a/regression-test/data/external_table_p0/iceberg/test_iceberg_struct_schema_evolution.out
+++
b/regression-test/data/external_table_p0/iceberg/test_iceberg_struct_schema_evolution.out
@@ -81,7 +81,7 @@ a_struct
struct<renamed:bigint,keep:bigint,drop_and_add:bigint,added:bigint> Yes
-- !case_desc --
id bigint Yes true \N
-a_struct
struct<renamed:bigint,keep:bigint,drop_and_add:bigint,added:bigint> Yes
true \N
+a_struct
struct<renamed:bigint,keep:bigint,DROP_AND_ADD:bigint,added:bigint> Yes
true \N
-- !case_select_all --
1 {"renamed":11, "keep":12, "drop_and_add":null, "added":null}
@@ -129,7 +129,7 @@ a_struct
struct<renamed:bigint,keep:bigint,drop_and_add:bigint,added:bigint> Yes
-- !case_orc_desc --
id bigint Yes true \N
-a_struct
struct<renamed:bigint,keep:bigint,drop_and_add:bigint,added:bigint> Yes
true \N
+a_struct
struct<renamed:bigint,keep:bigint,DROP_AND_ADD:bigint,added:bigint> Yes
true \N
-- !case_orc_select_all --
1 {"renamed":11, "keep":12, "drop_and_add":null, "added":null}
diff --git
a/regression-test/suites/external_table_p0/iceberg/test_iceberg_struct_schema_evolution.groovy
b/regression-test/suites/external_table_p0/iceberg/test_iceberg_struct_schema_evolution.groovy
index be03b238c92..d8a6a427d82 100644
---
a/regression-test/suites/external_table_p0/iceberg/test_iceberg_struct_schema_evolution.groovy
+++
b/regression-test/suites/external_table_p0/iceberg/test_iceberg_struct_schema_evolution.groovy
@@ -160,8 +160,7 @@ suite("test_iceberg_struct_schema_evolution",
"p0,external") {
qt_case_struct_renamed """SELECT element_at(a_struct, 'renamed') FROM
${case_table_name} ORDER BY id"""
// Test 3: Query struct field that was dropped and re-added with case
change
- // Note: Even though we use DROP_AND_ADD (uppercase) in SQL, the system
normalizes
- // field names to lowercase, so we query with 'drop_and_add' (lowercase)
+ // Iceberg metadata retains the external spelling, while runtime lookup
still uses the normalized name.
qt_case_struct_drop_and_add """SELECT element_at(a_struct, 'drop_and_add')
FROM ${case_table_name} ORDER BY id"""
// Test 4: Query struct field that was newly added
@@ -198,8 +197,7 @@ suite("test_iceberg_struct_schema_evolution",
"p0,external") {
qt_case_orc_struct_renamed """SELECT element_at(a_struct, 'renamed') FROM
${case_orc_table_name} ORDER BY id"""
// Test 3: Query struct field that was dropped and re-added with case
change
- // Note: Even though we use DROP_AND_ADD (uppercase) in SQL, the system
normalizes
- // field names to lowercase, so we query with 'drop_and_add' (lowercase)
+ // Iceberg metadata retains the external spelling, while runtime lookup
still uses the normalized name.
qt_case_orc_struct_drop_and_add """SELECT element_at(a_struct,
'drop_and_add') FROM ${case_orc_table_name} ORDER BY id"""
// Test 4: Query struct field that was newly added
diff --git
a/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_ctas_format_boundary.groovy
b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_ctas_format_boundary.groovy
index 9be6f3ba421..a6242cc9886 100644
---
a/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_ctas_format_boundary.groovy
+++
b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_ctas_format_boundary.groovy
@@ -156,7 +156,89 @@ suite("test_iceberg_write_ctas_format_boundary",
}
assertEquals(0, (sql """show tables like
'ctas_failed_atomicity'""").size())
- // WC01-S03: Iceberg allows Avro, but the current Doris writer supports
+ // WC01-S03: FILE TVF must keep the Parquet schema spelling until Iceberg
CTAS persists it.
+ // The normalized names remain available for Doris runtime lookup.
+ spark_iceberg_multi """
+ DROP TABLE IF EXISTS demo.${dbName}.file_tvf_case_source;
+ CREATE TABLE demo.${dbName}.file_tvf_case_source (
+ id INT,
+ payload STRUCT<CaseSensitive:BIGINT,
+ NestedArray:ARRAY<STRUCT<ArrayChild:BIGINT>>,
+ NestedMap:MAP<STRING,STRUCT<MapChild:BIGINT>>>
+ ) USING iceberg
+ TBLPROPERTIES ('write.format.default' = 'parquet');
+ INSERT INTO demo.${dbName}.file_tvf_case_source VALUES (
+ 1,
+ NAMED_STRUCT(
+ 'CaseSensitive', CAST(7 AS BIGINT),
+ 'NestedArray', ARRAY(NAMED_STRUCT('ArrayChild', CAST(8 AS
BIGINT))),
+ 'NestedMap', MAP('k', NAMED_STRUCT('MapChild', CAST(9 AS
BIGINT)))
+ )
+ );
+ """
+ sql """refresh catalog ${catalogName}"""
+ String sourceFile = (sql """
+ select file_path from file_tvf_case_source\$files order by file_path
limit 1
+ """)[0][0].toString()
+
+ sql """drop table if exists ctas_file_tvf_case"""
+ sql """
+ create table ctas_file_tvf_case as
+ select payload from file (
+ "uri" = "${sourceFile}",
+ "format" = "parquet",
+ "s3.endpoint" = "http://${externalEnvIp}:${minioPort}";,
+ "s3.region" = "us-east-1",
+ "s3.access_key" = "admin",
+ "s3.secret_key" = "password",
+ "use_path_style" = "true"
+ )
+ """
+
+ def ctasSchema = spark_iceberg """describe
demo.${dbName}.ctas_file_tvf_case"""
+ def payloadRow = ctasSchema.find { row -> row[0].toString() == "payload" }
+ assertNotNull(payloadRow, "payload column should exist in the Iceberg CTAS
schema")
+ String payloadType = payloadRow[1].toString()
+ assertTrue(payloadType.contains("CaseSensitive"), payloadType)
+ assertTrue(payloadType.contains("NestedArray"), payloadType)
+ assertTrue(payloadType.contains("ArrayChild"), payloadType)
+ assertTrue(payloadType.contains("NestedMap"), payloadType)
+ assertTrue(payloadType.contains("MapChild"), payloadType)
+
+ def nestedValues = sql """
+ select element_at(payload, 'casesensitive'),
+ element_at(element_at(payload, 'nestedarray')[1], 'arraychild'),
+ element_at(element_at(payload, 'nestedmap')['k'], 'mapchild')
+ from ctas_file_tvf_case
+ """
+ assertEquals([[7L, 8L, 9L]], nestedValues)
+
+ // WC01-S04: Names displayed from external metadata must remain executable
even when Java ROOT lowercasing
+ // changes their Unicode bytes or UTF-8 length before thrift reaches the
BE.
+ spark_iceberg_multi """
+ DROP TABLE IF EXISTS demo.${dbName}.unicode_struct_fields;
+ CREATE TABLE demo.${dbName}.unicode_struct_fields (
+ id INT,
+ payload STRUCT<`Σ`:BIGINT, `ẞ`:BIGINT>
+ ) USING iceberg;
+ INSERT INTO demo.${dbName}.unicode_struct_fields VALUES (
+ 1, NAMED_STRUCT('Σ', CAST(10 AS BIGINT), 'ẞ', CAST(11 AS BIGINT))
+ );
+ """
+ sql """refresh catalog ${catalogName}"""
+ def unicodeSchema = sql """describe unicode_struct_fields"""
+ def unicodePayloadRow = unicodeSchema.find { row -> row[0].toString() ==
"payload" }
+ assertNotNull(unicodePayloadRow, "payload column should exist in the
Unicode Iceberg schema")
+ String unicodePayloadType = unicodePayloadRow[1].toString()
+ assertTrue(unicodePayloadType.contains("Σ"), unicodePayloadType)
+ assertTrue(unicodePayloadType.contains("ẞ"), unicodePayloadType)
+ def unicodeValues = sql """
+ select element_at(payload, 'Σ'), element_at(payload, 'ẞ')
+ from unicode_struct_fields
+ """
+ assertEquals([[10L, 11L]], unicodeValues)
+
+ // WC01-S05: Iceberg allows Avro, but the current Doris writer supports
// Parquet and ORC only. Reject Avro explicitly instead of silently
falling back.
sql """drop table if exists avro_write_boundary"""
sql """
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
