This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new e771daa9b01 [fix](paimon) Keep CHAR/VARCHAR lengths and DATETIME
precision when creating Paimon tables (#69045)
e771daa9b01 is described below
commit e771daa9b01e53ef51e73e96de14a231b68d6115
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Wed Oct 14 22:30:58 2026 +0800
[fix](paimon) Keep CHAR/VARCHAR lengths and DATETIME precision when
creating Paimon tables (#69045)
### What problem does this PR solve?
Issue Number: None
Related PR: #66227, #68825
Problem Summary:
**In short.** When Doris creates or alters a Paimon table, master drops
the declared length of CHAR(n) and VARCHAR(n) columns (they all become
VARCHAR(2147483647)) and gives DATETIME a precision of 6. Paimon relies
on these bounds when it reads files written under an older schema, so a
table created or altered through Doris does not behave like the schema
the user declared. This PR keeps the bounds, as branch-4.1 #66227 does.
**Background.** `PaimonTypeMapping.toPaimonType` (fe-connector-paimon)
turns the column types of `CREATE TABLE`, `ADD COLUMN` and `MODIFY
COLUMN` into the Paimon schema. Paimon stores that schema with every
data file. When it reads a file written under an older schema, it casts
each column from the file's type to the current type, so the current
type decides what a reader sees: a VARCHAR(3) column truncates older
longer values, and a TIMESTAMP(3) column reads a numeric string such as
`42` as milliseconds. The mapping was ported from fe-core's
`DorisToPaimonTypeVisitor` when Paimon moved into the connector plugin;
branch-4.1 #66227 later changed that visitor to keep the bounds, and the
change never reached master. #68825 (the master port of the rest of
#66227) updated this method's Javadoc to describe the new mapping, but
not the code.
**The problem, and what it cost.**
| Doris column type | Paimon type on master | Paimon type with this PR |
What changes for users |
|---|---|---|---|
| CHAR(n) | VARCHAR(2147483647) | CHAR(n) | The column reads back as
CHAR(n) instead of STRING. |
| VARCHAR(n) | VARCHAR(2147483647) | VARCHAR(n) | The column reads back
as VARCHAR(n) instead of STRING. `MODIFY COLUMN c VARCHAR(3)` narrows
the column, and Paimon truncates older values when it reads them; on
master it widens the column to the maximum length instead. |
| STRING | VARCHAR(2147483647) | unchanged | |
| DATETIME (v1, only when date conversion is disabled) | TIMESTAMP(6) |
TIMESTAMP(0) | |
| DATETIMEV2(p) | TIMESTAMP(p) | unchanged | |
The same applies inside ARRAY, MAP and STRUCT, and to `CREATE TABLE ...
AS SELECT`: query columns that Doris' CTAS rules keep as VARCHAR(n)
(columns of internal tables, and expressions as VARCHAR(65533) by
default) keep that length instead of becoming STRING. Only Doris DDL
uses this mapping: `ADD COLUMN` and `MODIFY COLUMN` on a table created
by Spark are affected, reading a table is not.
**How this PR fixes it.** `toPaimonType` maps CHAR(n) to `CharType(n)`,
VARCHAR(n) to `VarCharType(n)` (CREATE TABLE already carries the length
in `ConnectorType.precision`), STRING to `VarCharType(MAX_LENGTH)`,
DATETIME to `TimestampType(0)` and DATETIMEV2(p) to `TimestampType(p)`.
A column declared as plain `varchar` or `char` is normalized to
VARCHAR(65533) or CHAR(1) by `ColumnDefinition` before it reaches the
connector, so the length is always set. The Javadoc is the same text as
in #68825.
Two existing behaviors now also apply to these tables, as they already
do on master to CHAR(n) and VARCHAR(n) columns of Paimon tables created
by Spark or Flink. Doris's native Parquet/ORC readers apply the declared
length to older values only when `truncate_char_or_varchar_columns` is
on (the same as for Hive and Iceberg tables); JNI reads always apply
Paimon's cast. Doris's Paimon writers do not pad CHAR(n) values to their
length, while Paimon's own cast does. Both are left to separate changes.
**Results.** `PaimonTypeMappingToPaimonTest` now checks the bounds and
precisions, the nested types, and Paimon's own casts on the mapped
types: from VARCHAR(10) to the mapped VARCHAR(3), `abcdef` becomes
`abc`; from STRING to the mapped DATETIMEV2(3), `42` becomes
`1970-01-01T00:00:00.042`. With master's mapping all four of these cases
fail. All fe-connector-paimon unit tests (726) pass. No existing
regression suite creates Paimon tables with CHAR(n), VARCHAR(n) or
DATETIME v1 columns through Doris; the
`test_paimon_rust_reader_eq_for_null` suite in #68825 narrows a column
to VARCHAR(3) and needs this change.
Co-authored-by: Fangyuan Deng <[email protected]>
---
.../doris/connector/paimon/PaimonTypeMapping.java | 20 ++++----
.../paimon/PaimonTypeMappingToPaimonTest.java | 58 +++++++++++++++++-----
2 files changed, 55 insertions(+), 23 deletions(-)
diff --git
a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonTypeMapping.java
b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonTypeMapping.java
index 65cc58ab1db..a26d4468e87 100644
---
a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonTypeMapping.java
+++
b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonTypeMapping.java
@@ -209,13 +209,10 @@ public final class PaimonTypeMapping {
* Convert a Doris {@link ConnectorType} (as produced by the CREATE TABLE
request path)
* to a Paimon {@link DataType}.
*
- * <p>This is the faithful reverse of the legacy fe-core
- * {@code DorisToPaimonTypeVisitor}: the scalar set is intentionally
narrow (it mirrors
- * the visitor's {@code atomic} branches and NOT MaxCompute's richer set),
CHAR/VARCHAR/STRING
- * all collapse to {@code VarChar(MAX)} (declared length dropped),
DATETIME/DATETIMEV2 map to a
- * plain {@code TimestampType()} (scale dropped), and the MAP key is
forced non-null. Types the
- * legacy visitor did not handle (TINYINT, SMALLINT, LARGEINT, TIME,
IPV4/6, JSON, ...) throw,
- * preserving the legacy gap.</p>
+ * <p>This is the connector-SPI equivalent of the former fe-core
+ * {@code DorisToPaimonTypeVisitor}. It preserves declared CHAR/VARCHAR
bounds and timestamp
+ * precision because Paimon uses them when applying casts and interpreting
numeric timestamp
+ * values. The MAP key remains non-null, matching Paimon's type
contract.</p>
*
* <p>The returned type carries Paimon's default (nullable) flag;
column-level nullability is
* applied by the caller via {@code .copy(nullable)} (mirroring legacy
@@ -243,10 +240,11 @@ public final class PaimonTypeMapping {
case "DOUBLE":
return new DoubleType();
case "CHAR":
+ // DDL must retain bounds: widening to STRING changes
historical narrowing casts.
+ return new CharType(type.getPrecision());
case "VARCHAR":
+ return new VarCharType(type.getPrecision());
case "STRING":
- // Legacy parity: all char-family types collapse to
VarChar(MAX); declared
- // length is intentionally dropped
(DorisToPaimonTypeVisitor.atomic isCharFamily).
return new VarCharType(VarCharType.MAX_LENGTH);
case "DATE":
case "DATEV2":
@@ -259,9 +257,9 @@ public final class PaimonTypeMapping {
case "DECIMAL256":
return new DecimalType(type.getPrecision(), type.getScale());
case "DATETIME":
- return new TimestampType();
case "DATETIMEV2":
- return new TimestampType(type.getPrecision());
+ // Paimon interprets numeric timestamp strings in units
derived from this precision.
+ return new TimestampType("DATETIMEV2".equals(name) ?
type.getPrecision() : 0);
case "TIMESTAMPTZ":
// Preserve the instant contract when deriving an external
schema.
return new LocalZonedTimestampType(type.getPrecision());
diff --git
a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonTypeMappingToPaimonTest.java
b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonTypeMappingToPaimonTest.java
index af0aad30474..055709ac090 100644
---
a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonTypeMappingToPaimonTest.java
+++
b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonTypeMappingToPaimonTest.java
@@ -20,9 +20,13 @@ package org.apache.doris.connector.paimon;
import org.apache.doris.connector.spi.ConnectorType;
import org.apache.doris.connector.spi.DorisConnectorException;
+import org.apache.paimon.casting.CastExecutor;
+import org.apache.paimon.casting.CastExecutors;
+import org.apache.paimon.data.BinaryString;
import org.apache.paimon.types.ArrayType;
import org.apache.paimon.types.BigIntType;
import org.apache.paimon.types.BooleanType;
+import org.apache.paimon.types.CharType;
import org.apache.paimon.types.DataField;
import org.apache.paimon.types.DataType;
import org.apache.paimon.types.DateType;
@@ -46,7 +50,7 @@ import java.util.Collections;
/**
* P5-T11 — pins the Doris->Paimon reverse type mapping in
- * {@link PaimonTypeMapping#toPaimonType} to byte-parity with the legacy
fe-core
+ * {@link PaimonTypeMapping#toPaimonType}, the connector equivalent of the
former fe-core
* {@code DorisToPaimonTypeVisitor}.
*
* <p>The CREATE TABLE path produces these {@link ConnectorType} descriptors;
this mapping is
@@ -74,26 +78,56 @@ public class PaimonTypeMappingToPaimonTest {
}
@Test
- public void charFamilyCollapsesToVarcharMaxDroppingLength() {
- // WHY: legacy isCharFamily -> VarCharType(MAX_LENGTH)
unconditionally; the declared length
- // is intentionally dropped and CHAR is NOT mapped to paimon CharType.
MUTATION: honoring
- // the declared length (e.g. new VarCharType(10)) or mapping CHAR ->
CharType makes these red.
- DataType expected = new VarCharType(VarCharType.MAX_LENGTH);
- Assertions.assertEquals(expected,
PaimonTypeMapping.toPaimonType(ConnectorType.of("CHAR", 10, 0)));
- Assertions.assertEquals(expected,
PaimonTypeMapping.toPaimonType(ConnectorType.of("VARCHAR", 20, 0)));
- Assertions.assertEquals(expected,
PaimonTypeMapping.toPaimonType(ConnectorType.of("STRING")));
+ public void characterTypesKeepDeclaredBounds() {
+ // WHY: the declared length is part of the Paimon schema; Paimon
truncates to it when it casts
+ // a historical data file to the current column type. MUTATION:
collapsing CHAR/VARCHAR to
+ // VarChar(MAX), or mapping CHAR to VarCharType, makes these red.
+ Assertions.assertEquals(new CharType(3),
PaimonTypeMapping.toPaimonType(ConnectorType.of("CHAR", 3, 0)));
+ Assertions.assertEquals(new VarCharType(3),
+ PaimonTypeMapping.toPaimonType(ConnectorType.of("VARCHAR", 3,
0)));
+ Assertions.assertEquals(new VarCharType(VarCharType.MAX_LENGTH),
+ PaimonTypeMapping.toPaimonType(ConnectorType.of("STRING")));
}
@Test
public void datetimePreservesPrecisionAndTimestampKind() {
- Assertions.assertEquals(new TimestampType(3),
- PaimonTypeMapping.toPaimonType(ConnectorType.of("DATETIMEV2",
3, 0)));
- Assertions.assertEquals(new TimestampType(),
+ for (int precision : new int[] {0, 3, 6}) {
+ Assertions.assertEquals(new TimestampType(precision),
+
PaimonTypeMapping.toPaimonType(ConnectorType.of("DATETIMEV2", precision, 0)));
+ }
+ // DATETIME (v1) has no fractional seconds.
+ Assertions.assertEquals(new TimestampType(0),
PaimonTypeMapping.toPaimonType(ConnectorType.of("DATETIME")));
Assertions.assertEquals(new
org.apache.paimon.types.LocalZonedTimestampType(6),
PaimonTypeMapping.toPaimonType(ConnectorType.of("TIMESTAMPTZ",
6, 0)));
}
+ @Test
+ @SuppressWarnings("unchecked")
+ public void mappedTypesDriveHistoricalCasts() {
+ // WHY: Paimon reads a data file written under an older schema by
casting the file type to the
+ // current column type, so the mapped bounds decide the values a
reader sees.
+ CastExecutor<BinaryString, ?> varcharCast =
(CastExecutor<BinaryString, ?>) CastExecutors.resolve(
+ new VarCharType(10),
PaimonTypeMapping.toPaimonType(ConnectorType.of("VARCHAR", 3, 0)));
+ Assertions.assertNotNull(varcharCast);
+ Assertions.assertEquals("abc",
varcharCast.cast(BinaryString.fromString("abcdef")).toString());
+ CastExecutor<BinaryString, ?> timestampCast =
(CastExecutor<BinaryString, ?>) CastExecutors.resolve(
+ new VarCharType(VarCharType.MAX_LENGTH),
+ PaimonTypeMapping.toPaimonType(ConnectorType.of("DATETIMEV2",
3, 0)));
+ Assertions.assertNotNull(timestampCast);
+ Assertions.assertEquals("1970-01-01T00:00:00.042",
+ timestampCast.cast(BinaryString.fromString("42")).toString());
+ }
+
+ @Test
+ public void nestedTypesKeepBoundsAndPrecision() {
+ Assertions.assertEquals(new ArrayType(new VarCharType(3)),
+
PaimonTypeMapping.toPaimonType(ConnectorType.arrayOf(ConnectorType.of("VARCHAR",
3, 0))));
+ Assertions.assertEquals(new MapType(new CharType(false, 2), new
TimestampType(3)),
+ PaimonTypeMapping.toPaimonType(ConnectorType.mapOf(
+ ConnectorType.of("CHAR", 2, 0),
ConnectorType.of("DATETIMEV2", 3, 0))));
+ }
+
@Test
public void decimalCarriesPrecisionAndScale() {
// WHY: every decimal family member carries precision/scale through
verbatim. MUTATION:
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
