This is an automated email from the ASF dual-hosted git repository.
Gabriel39 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new d45e369846d [fix](fe) Preserve external table column name case (#65094)
d45e369846d is described below
commit d45e369846d0d04a2a3f2584eb36ad19e7ef5063
Author: Gabriel <[email protected]>
AuthorDate: Thu Jul 9 08:55:18 2026 +0800
[fix](fe) Preserve external table column name case (#65094)
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: Creating Iceberg or Paimon external tables with
mixed-case partition columns could fail because Doris converted
top-level external column names to lower case while building external
schemas and partition specs. Reading external table schemas and
partition metadata also normalized some Paimon and Iceberg column names
to lower case, so SHOW CREATE and partition helpers could lose the
original external column spelling. This change preserves the original
top-level external field names when converting Doris columns to
Iceberg/Paimon schemas, resolves partition and primary key names
case-insensitively back to the external canonical names, and stops
schema/partition parsing paths from lowercasing external column names.
### Release note
Fix Iceberg and Paimon external table column name casing for mixed-case
partition columns.
### Check List (For Author)
- Test: Unit Test
- Maven focused FE test: MAVEN_ARGS=-o
JDK_17=/usr/local/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home
JAVA_HOME=/usr/local/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home
mvn test -pl fe-core -am -Dcheckstyle.skip=true -DfailIfNoTests=false
-Dmaven.build.cache.enabled=false
-Dtest=CreateIcebergTableTest,PaimonMetadataOpsTest,IcebergUtilsTest#testParseSchemaPreservesNonLowercaseColumnNames,PaimonUtilTest#testParseSchemaPreservesNonLowercaseColumnNames
- git diff --check
- A broader focused run including two existing Mockito-based
IcebergUtilsTest methods compiled successfully but those two methods
failed locally because Mockito inline Byte Buddy could not self-attach
to the Homebrew JDK 17 VM.
- Behavior changed: Yes. Iceberg and Paimon external schemas, partition
specs, and partition metadata now preserve external column name casing.
- Does this need documentation: No
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
---
.../org/apache/doris/paimon/PaimonJniScanner.java | 13 ++++++--
.../apache/doris/paimon/PaimonJniScannerTest.java | 11 ++++++
.../datasource/iceberg/DorisTypeToIcebergType.java | 13 ++++++--
.../datasource/iceberg/IcebergMetadataOps.java | 11 ++++--
.../doris/datasource/iceberg/IcebergUtils.java | 33 +++++++++++-------
.../datasource/paimon/PaimonExternalTable.java | 2 +-
.../doris/datasource/paimon/PaimonMetadataOps.java | 24 ++++++++++---
.../datasource/paimon/PaimonSysExternalTable.java | 2 +-
.../apache/doris/datasource/paimon/PaimonUtil.java | 5 ++-
.../paimon/source/PaimonPredicateConverter.java | 15 +++++++--
.../datasource/paimon/source/PaimonScanNode.java | 23 +++++++------
.../datasource/iceberg/CreateIcebergTableTest.java | 39 ++++++++++++++++++++++
.../iceberg/IcebergExternalTableTest.java | 7 ++++
.../doris/datasource/iceberg/IcebergUtilsTest.java | 24 +++++++++----
.../datasource/paimon/PaimonMetadataOpsTest.java | 25 ++++++++++++++
.../doris/datasource/paimon/PaimonUtilTest.java | 21 ++++++++++--
.../paimon/source/PaimonScanNodeTest.java | 17 +++++++---
.../iceberg/test_iceberg_invaild_avro_name.out | 5 ++-
.../paimon/test_paimon_catalog.groovy | 3 +-
.../paimon/test_paimon_jdbc_catalog.groovy | 2 +-
20 files changed, 235 insertions(+), 60 deletions(-)
diff --git
a/fe/be-java-extensions/paimon-scanner/src/main/java/org/apache/doris/paimon/PaimonJniScanner.java
b/fe/be-java-extensions/paimon-scanner/src/main/java/org/apache/doris/paimon/PaimonJniScanner.java
index f6dd3334e76..cd31a0815e8 100644
---
a/fe/be-java-extensions/paimon-scanner/src/main/java/org/apache/doris/paimon/PaimonJniScanner.java
+++
b/fe/be-java-extensions/paimon-scanner/src/main/java/org/apache/doris/paimon/PaimonJniScanner.java
@@ -254,12 +254,21 @@ public class PaimonJniScanner extends JniScanner {
private int[] getProjected() {
return Arrays.stream(fields).mapToInt(fieldName -> {
- int index = paimonAllFieldNames.indexOf(fieldName);
+ int index = getFieldIndex(paimonAllFieldNames, fieldName);
Preconditions.checkArgument(index >= 0, "RequiredField %s not
found in schema", fieldName);
return index;
}).toArray();
}
+ static int getFieldIndex(List<String> fieldNames, String fieldName) {
+ for (int i = 0; i < fieldNames.size(); i++) {
+ if (fieldNames.get(i).equalsIgnoreCase(fieldName)) {
+ return i;
+ }
+ }
+ return -1;
+ }
+
private List<Predicate> getPredicates() {
List<Predicate> predicates = PaimonUtils.deserialize(paimonPredicate);
if (LOG.isDebugEnabled()) {
@@ -281,7 +290,7 @@ public class PaimonJniScanner extends JniScanner {
if (types[i].isDateTimeV2()) {
// paimon support precision > 6, but it has been reset as 6 in
FE
// try to get the right precision for datetimev2
- int index = paimonAllFieldNames.indexOf(fields[i]);
+ int index = getFieldIndex(paimonAllFieldNames, fields[i]);
if (index != -1) {
DataType dataType = table.rowType().getTypeAt(index);
if (dataType instanceof TimestampType) {
diff --git
a/fe/be-java-extensions/paimon-scanner/src/test/java/org/apache/doris/paimon/PaimonJniScannerTest.java
b/fe/be-java-extensions/paimon-scanner/src/test/java/org/apache/doris/paimon/PaimonJniScannerTest.java
index 4344b89ec40..c6c90389fe8 100644
---
a/fe/be-java-extensions/paimon-scanner/src/test/java/org/apache/doris/paimon/PaimonJniScannerTest.java
+++
b/fe/be-java-extensions/paimon-scanner/src/test/java/org/apache/doris/paimon/PaimonJniScannerTest.java
@@ -31,6 +31,7 @@ import org.junit.rules.TemporaryFolder;
import java.io.File;
import java.lang.reflect.Field;
import java.lang.reflect.Proxy;
+import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.Map;
@@ -225,4 +226,14 @@ public class PaimonJniScannerTest {
public void close() {
}
}
+
+ @Test
+ public void testGetFieldIndexMatchesMixedCaseColumns() {
+ Assert.assertEquals(1,
PaimonJniScanner.getFieldIndex(Arrays.asList("data", "mIxEd_COL", "PART"),
+ "mixed_col"));
+ Assert.assertEquals(2,
PaimonJniScanner.getFieldIndex(Arrays.asList("data", "mIxEd_COL", "PART"),
+ "part"));
+ Assert.assertEquals(-1,
PaimonJniScanner.getFieldIndex(Arrays.asList("data", "mIxEd_COL", "PART"),
+ "missing_col"));
+ }
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/DorisTypeToIcebergType.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/DorisTypeToIcebergType.java
index 56fa03120d9..de19d90728d 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/DorisTypeToIcebergType.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/DorisTypeToIcebergType.java
@@ -29,6 +29,7 @@ import com.google.common.collect.Lists;
import org.apache.iceberg.types.Type;
import org.apache.iceberg.types.Types;
+import java.util.Collections;
import java.util.List;
@@ -37,14 +38,21 @@ import java.util.List;
*/
public class DorisTypeToIcebergType extends DorisTypeVisitor<Type> {
private final StructType root;
+ private final List<String> rootFieldNames;
private int nextId = 0;
public DorisTypeToIcebergType() {
this.root = null;
+ this.rootFieldNames = Collections.emptyList();
}
public DorisTypeToIcebergType(StructType root) {
+ this(root, Collections.emptyList());
+ }
+
+ public DorisTypeToIcebergType(StructType root, List<String>
rootFieldNames) {
this.root = root;
+ this.rootFieldNames = rootFieldNames;
// the root struct's fields use the first ids
this.nextId = root.getFields().size();
}
@@ -65,10 +73,11 @@ public class DorisTypeToIcebergType extends
DorisTypeVisitor<Type> {
Type type = types.get(i);
int id = isRoot ? i : getNextId();
+ String fieldName = isRoot && !rootFieldNames.isEmpty() ?
rootFieldNames.get(i) : field.getName();
if (field.getContainsNull()) {
- newFields.add(Types.NestedField.optional(id, field.getName(),
type, field.getComment()));
+ newFields.add(Types.NestedField.optional(id, fieldName, type,
field.getComment()));
} else {
- newFields.add(Types.NestedField.required(id, field.getName(),
type, field.getComment()));
+ newFields.add(Types.NestedField.required(id, fieldName, type,
field.getComment()));
}
}
return Types.StructType.of(newFields);
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergMetadataOps.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergMetadataOps.java
index 321c1627535..b3cfc172731 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergMetadataOps.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergMetadataOps.java
@@ -416,8 +416,8 @@ public class IcebergMetadataOps implements
ExternalMetadataOps {
.map(col -> new StructField(col.getName(), col.getType(),
col.getComment(), col.isAllowNull()))
.collect(Collectors.toList());
StructType structType = new StructType(new ArrayList<>(collect));
- Type visit =
- DorisTypeVisitor.visit(structType, new
DorisTypeToIcebergType(structType));
+ List<String> rootFieldNames =
columns.stream().map(Column::getName).collect(Collectors.toList());
+ Type visit = DorisTypeVisitor.visit(structType, new
DorisTypeToIcebergType(structType, rootFieldNames));
Schema schema = new
Schema(visit.asNestedType().asStructType().fields());
Map<String, String> properties = createTableInfo.getProperties();
properties.put(ExternalCatalog.DORIS_VERSION,
ExternalCatalog.DORIS_VERSION_VALUE);
@@ -1455,7 +1455,7 @@ public class IcebergMetadataOps implements
ExternalMetadataOps {
org.apache.iceberg.SortOrder.Builder builder =
org.apache.iceberg.SortOrder.builderFor(schema);
for (SortFieldInfo sortField : sortFields) {
- String columnName = sortField.getColumnName();
+ String columnName = getIcebergColumnName(schema,
sortField.getColumnName());
if (sortField.isAscending()) {
if (sortField.isNullFirst()) {
builder.asc(columnName,
org.apache.iceberg.NullOrder.NULLS_FIRST);
@@ -1472,4 +1472,9 @@ public class IcebergMetadataOps implements
ExternalMetadataOps {
}
return builder.build();
}
+
+ private static String getIcebergColumnName(Schema schema, String
columnName) {
+ NestedField field = schema.caseInsensitiveFindField(columnName);
+ return field == null ? columnName : field.name();
+ }
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergUtils.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergUtils.java
index b801bc84322..2cd4f2bcf2f 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergUtils.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergUtils.java
@@ -139,7 +139,6 @@ import java.util.Comparator;
import java.util.HashMap;
import java.util.LinkedHashSet;
import java.util.List;
-import java.util.Locale;
import java.util.Map;
import java.util.Optional;
import java.util.Set;
@@ -587,37 +586,43 @@ public class IcebergUtils {
PartitionSpec.Builder builder = PartitionSpec.builderFor(schema);
for (Expr expr : partitionExprs) {
if (expr instanceof SlotRef) {
- builder.identity(((SlotRef) expr).getColumnName());
+ builder.identity(getIcebergColumnName(schema, ((SlotRef)
expr).getColumnName()));
} else if (expr instanceof FunctionCallExpr) {
String exprName = expr.accept(ExprToExprNameVisitor.INSTANCE,
null);
List<Expr> params = ((FunctionCallExpr)
expr).getParams().exprs();
switch (exprName.toLowerCase()) {
case "bucket":
builder.bucket(
-
params.get(1).accept(ExprToExprNameVisitor.INSTANCE, null),
+ getIcebergColumnName(schema,
+
params.get(1).accept(ExprToExprNameVisitor.INSTANCE, null)),
Integer.parseInt(params.get(0).getStringValue()));
break;
case "year":
case "years":
-
builder.year(params.get(0).accept(ExprToExprNameVisitor.INSTANCE, null));
+ builder.year(getIcebergColumnName(schema,
+
params.get(0).accept(ExprToExprNameVisitor.INSTANCE, null)));
break;
case "month":
case "months":
-
builder.month(params.get(0).accept(ExprToExprNameVisitor.INSTANCE, null));
+ builder.month(getIcebergColumnName(schema,
+
params.get(0).accept(ExprToExprNameVisitor.INSTANCE, null)));
break;
case "date":
case "day":
case "days":
-
builder.day(params.get(0).accept(ExprToExprNameVisitor.INSTANCE, null));
+ builder.day(getIcebergColumnName(schema,
+
params.get(0).accept(ExprToExprNameVisitor.INSTANCE, null)));
break;
case "date_hour":
case "hour":
case "hours":
-
builder.hour(params.get(0).accept(ExprToExprNameVisitor.INSTANCE, null));
+ builder.hour(getIcebergColumnName(schema,
+
params.get(0).accept(ExprToExprNameVisitor.INSTANCE, null)));
break;
case "truncate":
builder.truncate(
-
params.get(1).accept(ExprToExprNameVisitor.INSTANCE, null),
+ getIcebergColumnName(schema,
+
params.get(1).accept(ExprToExprNameVisitor.INSTANCE, null)),
Integer.parseInt(params.get(0).getStringValue()));
break;
default:
@@ -628,6 +633,11 @@ public class IcebergUtils {
return builder.build();
}
+ private static String getIcebergColumnName(Schema schema, String
columnName) {
+ Types.NestedField field = schema.caseInsensitiveFindField(columnName);
+ return field == null ? columnName : field.name();
+ }
+
private static Type
icebergPrimitiveTypeToDorisType(org.apache.iceberg.types.Type.PrimitiveType
primitive,
boolean enableMappingVarbinary, boolean enableMappingTimestampTz) {
switch (primitive.typeId()) {
@@ -773,7 +783,7 @@ public class IcebergUtils {
}
String columnName =
table.schema().findColumnName(partitionField.sourceId());
if (columnName != null) {
- partitionColumns.add(columnName.toLowerCase(Locale.ROOT));
+ partitionColumns.add(columnName);
}
}
}
@@ -805,8 +815,7 @@ public class IcebergUtils {
}
Object value = partitionData.get(i);
try {
- partitionInfoMap.put(columnName.toLowerCase(Locale.ROOT),
- serializePartitionValue(field.type(), value,
timeZone));
+ partitionInfoMap.put(columnName,
serializePartitionValue(field.type(), value, timeZone));
} catch (UnsupportedOperationException e) {
LOG.warn("Failed to serialize Iceberg table partition value
for field {}: {}", field.name(),
e.getMessage());
@@ -1173,7 +1182,7 @@ public class IcebergUtils {
List<Types.NestedField> columns = schema.columns();
List<Column> resSchema =
Lists.newArrayListWithCapacity(columns.size());
for (Types.NestedField field : columns) {
- Column column = new Column(field.name().toLowerCase(Locale.ROOT),
+ Column column = new Column(field.name(),
IcebergUtils.icebergTypeToDorisType(field.type(),
enableMappingVarbinary, enableMappingTimestampTz),
true, null,
true, field.doc(), true, -1);
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonExternalTable.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonExternalTable.java
index 1775a984f2c..6a744f765e8 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonExternalTable.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonExternalTable.java
@@ -346,7 +346,7 @@ public class PaimonExternalTable extends ExternalTable
implements MTMVRelatedTab
Set<String> partitionColumnNames =
Sets.newHashSet(tableSchema.partitionKeys());
List<Column> partitionColumns = Lists.newArrayList();
for (DataField field : columns) {
- Column column = new Column(field.name().toLowerCase(),
+ Column column = new Column(field.name(),
PaimonUtil.paimonTypeToDorisType(field.type(),
getCatalog().getEnableMappingVarbinary(),
getCatalog().getEnableMappingTimestampTz()),
true,
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonMetadataOps.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonMetadataOps.java
index b8263250c3d..2f0be3bc305 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonMetadataOps.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonMetadataOps.java
@@ -57,6 +57,7 @@ import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
+import java.util.Locale;
import java.util.Map;
import java.util.Optional;
import java.util.stream.Collectors;
@@ -218,7 +219,9 @@ public class PaimonMetadataOps implements
ExternalMetadataOps {
col.getComment(), col.isNullable()))
.collect(Collectors.toList());
StructType structType = new StructType(new ArrayList<>(collect));
- Schema schema = toPaimonSchema(structType,
createTableInfo.getPartitionDesc(), createTableInfo.getProperties());
+ List<String> rootFieldNames =
columns.stream().map(ColumnDefinition::getName).collect(Collectors.toList());
+ Schema schema = toPaimonSchema(structType, rootFieldNames,
createTableInfo.getPartitionDesc(),
+ createTableInfo.getProperties());
try {
catalog.createTable(new Identifier(createTableInfo.getDbName(),
createTableInfo.getTableName()),
schema, createTableInfo.isIfNotExists());
@@ -228,7 +231,8 @@ public class PaimonMetadataOps implements
ExternalMetadataOps {
return false;
}
- private Schema toPaimonSchema(StructType structType, PartitionDesc
partitionDesc, Map<String, String> properties) {
+ private Schema toPaimonSchema(StructType structType, List<String>
rootFieldNames, PartitionDesc partitionDesc,
+ Map<String, String> properties) {
Map<String, String> normalizedProperties = new HashMap<>(properties);
normalizedProperties.remove(PRIMARY_KEY_IDENTIFIER);
normalizedProperties.remove(PROP_COMMENT);
@@ -242,19 +246,31 @@ public class PaimonMetadataOps implements
ExternalMetadataOps {
.map(String::trim)
.collect(Collectors.toList());
List<String> partitionKeys = partitionDesc == null ? new ArrayList<>()
: partitionDesc.getPartitionColNames();
+ primaryKeys = getPaimonColumnNames(rootFieldNames, primaryKeys);
+ partitionKeys = getPaimonColumnNames(rootFieldNames, partitionKeys);
Schema.Builder schemaBuilder = Schema.newBuilder()
.options(normalizedProperties)
.primaryKey(primaryKeys)
.partitionKeys(partitionKeys)
.comment(properties.getOrDefault(PROP_COMMENT, null));
- for (StructField field : structType.getFields()) {
- schemaBuilder.column(field.getName(),
+ List<StructField> fields = structType.getFields();
+ for (int i = 0; i < fields.size(); i++) {
+ StructField field = fields.get(i);
+ schemaBuilder.column(rootFieldNames.get(i),
toPaimontype(field.getType()).copy(field.getContainsNull()),
field.getComment());
}
return schemaBuilder.build();
}
+ private List<String> getPaimonColumnNames(List<String> paimonColumnNames,
List<String> dorisColumnNames) {
+ Map<String, String> paimonColumnNameMap = paimonColumnNames.stream()
+ .collect(Collectors.toMap(name ->
name.toLowerCase(Locale.ROOT), name -> name));
+ return dorisColumnNames.stream()
+ .map(name ->
paimonColumnNameMap.getOrDefault(name.toLowerCase(Locale.ROOT), name))
+ .collect(Collectors.toList());
+ }
+
private DataType toPaimontype(Type type) {
return DorisTypeVisitor.visit(type, new DorisToPaimonTypeVisitor());
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonSysExternalTable.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonSysExternalTable.java
index b6999b7c50c..744562d0b71 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonSysExternalTable.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonSysExternalTable.java
@@ -255,7 +255,7 @@ public class PaimonSysExternalTable extends ExternalTable {
for (DataField field : fields) {
Column column = new Column(
- field.name().toLowerCase(),
+ field.name(),
PaimonUtil.paimonTypeToDorisType(
field.type(),
getCatalog().getEnableMappingVarbinary(),
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonUtil.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonUtil.java
index 6c988adad30..c96ffa6146c 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonUtil.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonUtil.java
@@ -92,7 +92,6 @@ import java.util.ArrayList;
import java.util.Base64;
import java.util.HashMap;
import java.util.List;
-import java.util.Locale;
import java.util.Map;
import java.util.Optional;
import java.util.regex.Pattern;
@@ -504,7 +503,7 @@ public class PaimonUtil {
boolean enableTimestampTzMapping) {
List<Column> resSchema =
Lists.newArrayListWithCapacity(rowType.getFields().size());
rowType.getFields().forEach(field -> {
- resSchema.add(new Column(field.name().toLowerCase(),
+ resSchema.add(new Column(field.name(),
PaimonUtil.paimonTypeToDorisType(field.type(),
enableVarbinaryMapping, enableTimestampTzMapping),
primaryKeys.contains(field.name()),
null,
@@ -553,7 +552,7 @@ public class PaimonUtil {
try {
String partitionValue =
serializePartitionValue(partitionType.getFields().get(i).type(),
partitionValuesArray[i], timeZone);
-
partitionInfoMap.put(partitionKeys.get(i).toLowerCase(Locale.ROOT),
partitionValue);
+ partitionInfoMap.put(partitionKeys.get(i), partitionValue);
} catch (UnsupportedOperationException e) {
LOG.warn("Failed to serialize table {} partition value for key
{}: {}", table.name(),
partitionKeys.get(i), e.getMessage());
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonPredicateConverter.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonPredicateConverter.java
index 963904a4ff4..867225fdf8b 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonPredicateConverter.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonPredicateConverter.java
@@ -47,7 +47,7 @@ public class PaimonPredicateConverter {
public PaimonPredicateConverter(RowType rowType) {
this.builder = new PredicateBuilder(rowType);
- this.fieldNames = rowType.getFields().stream().map(f ->
f.name().toLowerCase()).collect(Collectors.toList());
+ this.fieldNames =
rowType.getFields().stream().map(DataField::name).collect(Collectors.toList());
this.paimonFieldTypes =
rowType.getFields().stream().map(DataField::type).collect(Collectors.toList());
}
@@ -100,7 +100,7 @@ public class PaimonPredicateConverter {
return null;
}
String colName = slotRef.getColumnName();
- int idx = fieldNames.indexOf(colName);
+ int idx = getFieldIndex(colName);
DataType dataType = paimonFieldTypes.get(idx);
List<Object> valueList = new ArrayList<>();
for (int i = 1; i < predicate.getChildren().size(); i++) {
@@ -132,7 +132,7 @@ public class PaimonPredicateConverter {
return null;
}
String colName = slotRef.getColumnName();
- int idx = fieldNames.indexOf(colName);
+ int idx = getFieldIndex(colName);
DataType dataType = paimonFieldTypes.get(idx);
Object value = dataType.accept(new PaimonValueConverter(literalExpr));
if (value == null) {
@@ -174,6 +174,15 @@ public class PaimonPredicateConverter {
return null;
}
+ private int getFieldIndex(String colName) {
+ for (int i = 0; i < fieldNames.size(); i++) {
+ if (fieldNames.get(i).equalsIgnoreCase(colName)) {
+ return i;
+ }
+ }
+ return fieldNames.indexOf(colName);
+ }
+
public static SlotRef convertDorisExprToSlotRef(Expr expr) {
SlotRef slotRef = null;
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonScanNode.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonScanNode.java
index ad142f08427..f8549855ea6 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonScanNode.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonScanNode.java
@@ -71,7 +71,6 @@ import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
-import java.util.Locale;
import java.util.Map;
import java.util.Optional;
import java.util.concurrent.ConcurrentHashMap;
@@ -238,9 +237,7 @@ public class PaimonScanNode extends FileQueryScanNode {
&& !((PaimonSysExternalTable) externalTable).isDataTable()) {
return Collections.emptyList();
}
- return source.getPaimonTable().partitionKeys().stream()
- .map(key -> key.toLowerCase(Locale.ROOT))
- .collect(Collectors.toList());
+ return source.getPaimonTable().partitionKeys();
}
private void putHistorySchemaInfo(Long schemaId) {
@@ -607,13 +604,9 @@ public class PaimonScanNode extends FileQueryScanNode {
long startTime = System.currentTimeMillis();
try {
Table paimonTable = getProcessedTable();
+ List<String> fieldNames = paimonTable.rowType().getFieldNames();
int[] projected = desc.getSlots().stream().mapToInt(
- slot -> paimonTable.rowType()
- .getFieldNames()
- .stream()
- .map(String::toLowerCase)
- .collect(Collectors.toList())
- .indexOf(slot.getColumn().getName()))
+ slot -> getFieldIndex(fieldNames,
slot.getColumn().getName()))
.filter(i -> i >= 0)
.toArray();
ReadBuilder readBuilder = paimonTable.newReadBuilder();
@@ -637,6 +630,16 @@ public class PaimonScanNode extends FileQueryScanNode {
}
}
+ @VisibleForTesting
+ static int getFieldIndex(List<String> fieldNames, String columnName) {
+ for (int i = 0; i < fieldNames.size(); i++) {
+ if (fieldNames.get(i).equalsIgnoreCase(columnName)) {
+ return i;
+ }
+ }
+ return -1;
+ }
+
private String getFileFormat(String path) {
return
FileFormatUtils.getFileFormatBySuffix(path).orElse(source.getFileFormatFromTableProperties());
}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/CreateIcebergTableTest.java
b/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/CreateIcebergTableTest.java
index 909359146eb..6962f911538 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/CreateIcebergTableTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/CreateIcebergTableTest.java
@@ -214,6 +214,45 @@ public class CreateIcebergTableTest {
Assert.assertEquals("b", table.properties().get("a"));
}
+ @Test
+ public void testPartitionPreservesNonLowercaseColumnNames() throws
UserException {
+ TableIdentifier tb = TableIdentifier.of(dbName, getTableName());
+ String sql = "create table " + tb + " ("
+ + "data int, "
+ + "`PART` int, "
+ + "`mIxEd_COL` int"
+ + ") engine = iceberg "
+ + "partition by (`PART`, bucket(2, `mIxEd_COL`)) ()";
+ createTable(sql);
+ Table table = ops.getCatalog().loadTable(tb);
+ Schema schema = table.schema();
+
+ Assert.assertEquals("PART", schema.columns().get(1).name());
+ Assert.assertEquals("mIxEd_COL", schema.columns().get(2).name());
+ PartitionSpec spec = PartitionSpec.builderFor(schema)
+ .identity("PART")
+ .bucket("mIxEd_COL", 2)
+ .build();
+ Assert.assertEquals(spec, table.spec());
+ }
+
+ @Test
+ public void
testSortOrderResolvesNonLowercaseColumnNamesCaseInsensitively() throws
UserException {
+ TableIdentifier tb = TableIdentifier.of(dbName, getTableName());
+ String sql = "create table " + tb + " ("
+ + "data int, "
+ + "`mIxEd_COL` int"
+ + ") engine = iceberg "
+ + "order by (`mixed_col` asc)";
+ createTable(sql);
+ Table table = ops.getCatalog().loadTable(tb);
+ Schema schema = table.schema();
+
+ Assert.assertEquals("mIxEd_COL", schema.columns().get(1).name());
+ Assert.assertEquals(1, table.sortOrder().fields().size());
+ Assert.assertEquals(schema.findField("mIxEd_COL").fieldId(),
table.sortOrder().fields().get(0).sourceId());
+ }
+
public void createTable(String sql) throws UserException {
LogicalPlan plan = new NereidsParser().parseSingle(sql);
Assertions.assertTrue(plan instanceof CreateTableCommand);
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergExternalTableTest.java
b/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergExternalTableTest.java
index 6bb96bf0d43..0a5a4ab11d4 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergExternalTableTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergExternalTableTest.java
@@ -321,6 +321,13 @@ public class IcebergExternalTableTest {
Assertions.assertEquals("PARTITION BY LIST (`d_year`) ()",
spy.getPartitionSpecSql());
}
+ @Test
+ public void testGetPartitionSpecSqlPreservesNonLowercaseColumnName() {
+ IcebergExternalTable spy = createSpyTable();
+ setupSingleField(mockTransform("identity"), "mIxEd_COL");
+ Assertions.assertEquals("PARTITION BY LIST (`mIxEd_COL`) ()",
spy.getPartitionSpecSql());
+ }
+
@Test
public void testGetPartitionSpecSqlBucket() {
IcebergExternalTable spy = createSpyTable();
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergUtilsTest.java
b/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergUtilsTest.java
index 2c5e2a47e9f..8625b36f283 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergUtilsTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergUtilsTest.java
@@ -287,6 +287,18 @@ public class IcebergUtilsTest {
Assert.assertNotNull(schemaWithRowLineage.findField(MetadataColumns.LAST_UPDATED_SEQUENCE_NUMBER.fieldId()));
}
+ @Test
+ public void testParseSchemaPreservesNonLowercaseColumnNames() {
+ Schema schema = new Schema(
+ Types.NestedField.required(1, "mIxEd_COL",
Types.IntegerType.get()),
+ Types.NestedField.required(2, "PART", Types.StringType.get()));
+
+ List<Column> columns = IcebergUtils.parseSchema(schema, false, false);
+
+ Assert.assertEquals("mIxEd_COL", columns.get(0).getName());
+ Assert.assertEquals("PART", columns.get(1).getName());
+ }
+
@Test
public void testGetPartitionInfoMapSkipBinaryIdentityPartition() {
Schema schema = new Schema(
@@ -305,11 +317,11 @@ public class IcebergUtilsTest {
public void testGetIdentityPartitionColumnsIgnoresTransformPartitions() {
Schema schema = new Schema(
Types.NestedField.required(1, "id", Types.IntegerType.get()),
- Types.NestedField.required(2, "dt", Types.StringType.get()),
+ Types.NestedField.required(2, "Dt", Types.StringType.get()),
Types.NestedField.required(3, "ts",
Types.TimestampType.withoutZone()));
PartitionSpec specWithTransform = PartitionSpec.builderFor(schema)
.withSpecId(1)
- .identity("dt")
+ .identity("Dt")
.day("ts")
.build();
PartitionSpec identityOnlySpec = PartitionSpec.builderFor(schema)
@@ -324,16 +336,16 @@ public class IcebergUtilsTest {
Mockito.when(table.schema()).thenReturn(schema);
Mockito.when(table.specs()).thenReturn(specs);
- Assert.assertEquals(Arrays.asList("dt", "id"),
IcebergUtils.getIdentityPartitionColumns(table));
+ Assert.assertEquals(Arrays.asList("Dt", "id"),
IcebergUtils.getIdentityPartitionColumns(table));
}
@Test
public void testGetIdentityPartitionInfoMapReturnsIdentityColumnsOnly() {
Schema schema = new Schema(
- Types.NestedField.required(1, "dt", Types.StringType.get()),
+ Types.NestedField.required(1, "Dt", Types.StringType.get()),
Types.NestedField.required(2, "ts",
Types.TimestampType.withoutZone()));
PartitionSpec partitionSpec = PartitionSpec.builderFor(schema)
- .identity("dt")
+ .identity("Dt")
.day("ts")
.build();
PartitionData partitionData = new
PartitionData(partitionSpec.partitionType());
@@ -345,7 +357,7 @@ public class IcebergUtilsTest {
Map<String, String> partitionInfoMap =
IcebergUtils.getIdentityPartitionInfoMap(
partitionData, partitionSpec, table, "UTC");
- Assert.assertEquals(Collections.singletonMap("dt", "2025-01-01"),
partitionInfoMap);
+ Assert.assertEquals(Collections.singletonMap("Dt", "2025-01-01"),
partitionInfoMap);
}
@Test
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/PaimonMetadataOpsTest.java
b/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/PaimonMetadataOpsTest.java
index e4146faa690..dda2c3d2344 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/PaimonMetadataOpsTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/PaimonMetadataOpsTest.java
@@ -53,6 +53,7 @@ import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.UUID;
+import java.util.stream.Collectors;
public class PaimonMetadataOpsTest {
public static String warehouse;
@@ -196,6 +197,30 @@ public class PaimonMetadataOpsTest {
Assert.assertEquals(1, table.primaryKeys().size());
}
+ @Test
+ public void testPartitionPreservesNonLowercaseColumnNames() throws
Exception {
+ String tableName = getTableName();
+ Identifier identifier = new Identifier(dbName, tableName);
+ String sql = "create table " + dbName + "." + tableName + " ("
+ + "data int, "
+ + "`PART` int, "
+ + "`mIxEd_COL` int"
+ + ") engine = paimon "
+ + "partition by (`PART`) ()";
+ createTable(sql);
+ Catalog catalog = ops.getCatalog();
+ Table table = catalog.getTable(identifier);
+
+ List<String> columnNames = table.rowType().getFields().stream()
+ .map(DataField::name)
+ .collect(Collectors.toList());
+
+ Assert.assertEquals("PART", columnNames.get(1));
+ Assert.assertEquals("mIxEd_COL", columnNames.get(2));
+ Assert.assertEquals(1, table.partitionKeys().size());
+ Assert.assertEquals("PART", table.partitionKeys().get(0));
+ }
+
@Test
public void testBucket() throws Exception {
String tableName = getTableName();
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/PaimonUtilTest.java
b/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/PaimonUtilTest.java
index 4a2b609023b..050e547816d 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/PaimonUtilTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/PaimonUtilTest.java
@@ -17,6 +17,7 @@
package org.apache.doris.datasource.paimon;
+import org.apache.doris.catalog.Column;
import org.apache.doris.catalog.Type;
import org.apache.doris.thrift.TPrimitiveType;
import org.apache.doris.thrift.schema.external.TFieldPtr;
@@ -30,6 +31,7 @@ import org.apache.paimon.table.Table;
import org.apache.paimon.types.CharType;
import org.apache.paimon.types.DataField;
import org.apache.paimon.types.DataTypes;
+import org.apache.paimon.types.RowType;
import org.apache.paimon.types.VarCharType;
import org.junit.Assert;
import org.junit.Test;
@@ -84,7 +86,20 @@ public class PaimonUtilTest {
}
@Test
- public void testGetPartitionInfoMapUsesLowerCaseKeys() {
+ public void testParseSchemaPreservesNonLowercaseColumnNames() {
+ RowType rowType = DataTypes.ROW(
+ DataTypes.FIELD(0, "mIxEd_COL", DataTypes.INT()),
+ DataTypes.FIELD(1, "PART", DataTypes.STRING()));
+
+ List<Column> columns = PaimonUtil.parseSchema(rowType,
Collections.singletonList("PART"), false, false);
+
+ Assert.assertEquals("mIxEd_COL", columns.get(0).getName());
+ Assert.assertEquals("PART", columns.get(1).getName());
+ Assert.assertTrue(columns.get(1).isKey());
+ }
+
+ @Test
+ public void testGetPartitionInfoMapPreservesNonLowercaseKeys() {
DataField mixedCasePartition = DataTypes.FIELD(0, "Dt",
DataTypes.STRING());
Table table = Mockito.mock(Table.class);
Mockito.when(table.name()).thenReturn("mock_table");
@@ -95,8 +110,8 @@ public class PaimonUtilTest {
Map<String, String> partitionInfoMap =
PaimonUtil.getPartitionInfoMap(table, partitionValues, "UTC");
- Assert.assertFalse(partitionInfoMap.containsKey("Dt"));
- Assert.assertEquals("2026-05-26", partitionInfoMap.get("dt"));
+ Assert.assertFalse(partitionInfoMap.containsKey("dt"));
+ Assert.assertEquals("2026-05-26", partitionInfoMap.get("Dt"));
}
@Test
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/source/PaimonScanNodeTest.java
b/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/source/PaimonScanNodeTest.java
index d745ee451da..034be2185e2 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/source/PaimonScanNodeTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/source/PaimonScanNodeTest.java
@@ -578,7 +578,7 @@ public class PaimonScanNodeTest {
Mockito.when(sysTable.isDataTable()).thenReturn(true);
node.setSource(source);
- Assert.assertEquals(Arrays.asList("dt", "region"),
node.getPathPartitionKeys());
+ Assert.assertEquals(Arrays.asList("Dt", "Region"),
node.getPathPartitionKeys());
}
@Test
@@ -611,19 +611,28 @@ public class PaimonScanNodeTest {
rangeDesc.setColumnsFromPath(Collections.singletonList("old"));
rangeDesc.setColumnsFromPathIsNull(Collections.singletonList(false));
Map<String, String> partitionValues = new HashMap<>();
- partitionValues.put("dt", "2025-01-01");
- partitionValues.put("pt", "p1");
+ partitionValues.put("Dt", "2025-01-01");
+ partitionValues.put("Pt", "p1");
PaimonSplit split = new
PaimonSplit(createDataSplit("ordered.parquet"));
split.setPaimonPartitionValues(partitionValues);
invokePrivateMethod(node, "setPaimonParams",
new Class<?>[] {TFileRangeDesc.class, PaimonSplit.class},
rangeDesc, split);
- Assert.assertEquals(Arrays.asList("pt", "dt"),
rangeDesc.getColumnsFromPathKeys());
+ Assert.assertEquals(Arrays.asList("Pt", "Dt"),
rangeDesc.getColumnsFromPathKeys());
Assert.assertEquals(Arrays.asList("p1", "2025-01-01"),
rangeDesc.getColumnsFromPath());
Assert.assertEquals(Arrays.asList(false, false),
rangeDesc.getColumnsFromPathIsNull());
}
+ @Test
+ public void testGetFieldIndexMatchesMixedCaseColumns() {
+ List<String> fieldNames = Arrays.asList("data", "mIxEd_COL", "PART");
+
+ Assert.assertEquals(1, PaimonScanNode.getFieldIndex(fieldNames,
"mixed_col"));
+ Assert.assertEquals(2, PaimonScanNode.getFieldIndex(fieldNames,
"part"));
+ Assert.assertEquals(-1, PaimonScanNode.getFieldIndex(fieldNames,
"missing_col"));
+ }
+
private void mockJniReader(PaimonScanNode spyNode) {
Mockito.doReturn(false).when(spyNode).supportNativeReader(ArgumentMatchers.any(Optional.class));
}
diff --git
a/regression-test/data/external_table_p0/iceberg/test_iceberg_invaild_avro_name.out
b/regression-test/data/external_table_p0/iceberg/test_iceberg_invaild_avro_name.out
index 1b8af743c6a..be0c0e22364 100644
---
a/regression-test/data/external_table_p0/iceberg/test_iceberg_invaild_avro_name.out
+++
b/regression-test/data/external_table_p0/iceberg/test_iceberg_invaild_avro_name.out
@@ -1,7 +1,7 @@
-- This file is automatically generated. You should know what you did if you
want to edit this
-- !desc --
id int Yes true \N
-test:a1b2.raw.abc-gg-1-a text Yes true \N
+TEST:A1B2.RAW.ABC-GG-1-A text Yes true \N
-- !q_1 --
1 row1
@@ -29,7 +29,7 @@ test:a1b2.raw.abc-gg-1-a text Yes true \N
-- !desc --
id int Yes true \N
-test:a1b2.raw.abc-gg-1-a text Yes true \N
+TEST:A1B2.RAW.ABC-GG-1-A text Yes true \N
-- !q_1 --
1 row1
@@ -54,4 +54,3 @@ test:a1b2.raw.abc-gg-1-a text Yes true \N
3 row3
2 row2
1 row1
-
diff --git
a/regression-test/suites/external_table_p0/paimon/test_paimon_catalog.groovy
b/regression-test/suites/external_table_p0/paimon/test_paimon_catalog.groovy
index 720c4e21911..a011a9f64a7 100644
--- a/regression-test/suites/external_table_p0/paimon/test_paimon_catalog.groovy
+++ b/regression-test/suites/external_table_p0/paimon/test_paimon_catalog.groovy
@@ -307,7 +307,7 @@ suite("test_paimon_catalog", "p0,external") {
test {
sql """select * from dup_columns_table;"""
- exception "Duplicate column name found: id"
+ exception "Duplicate column name found: ID"
}
sql """ set force_jni_scanner=false; """
@@ -332,4 +332,3 @@ suite("test_paimon_catalog", "p0,external") {
}
}
-
diff --git
a/regression-test/suites/external_table_p0/paimon/test_paimon_jdbc_catalog.groovy
b/regression-test/suites/external_table_p0/paimon/test_paimon_jdbc_catalog.groovy
index 3974051f9f6..b5f233aac67 100644
---
a/regression-test/suites/external_table_p0/paimon/test_paimon_jdbc_catalog.groovy
+++
b/regression-test/suites/external_table_p0/paimon/test_paimon_jdbc_catalog.groovy
@@ -267,7 +267,7 @@ suite("test_paimon_jdbc_catalog", "p0,external") {
assertSystemTableReadable(
"paimon_jdbc_row_tracking_tbl\$row_tracking",
- ["_row_id", "_sequence_number"],
+ ["_ROW_ID", "_SEQUENCE_NUMBER"],
1
)
} finally {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
