This is an automated email from the ASF dual-hosted git repository.
yuqi1129 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/main by this push:
new 4091d5c56b [#12272] feat(clickhouse): map ClickHouse Date32 to
Gravitino DateType and verify Enum round-trip (#12275)
4091d5c56b is described below
commit 4091d5c56b217cc03516e69eee19780584631d64
Author: StormSpirit <[email protected]>
AuthorDate: Fri Aug 21 15:38:08 2026 +0800
[#12272] feat(clickhouse): map ClickHouse Date32 to Gravitino DateType and
verify Enum round-trip (#12275)
### What changes were proposed in this pull request?
1. **Date32 → DateType**: Add `case DATE32` in
`ClickHouseTypeConverter.toGravitino()`, mapping ClickHouse Date32 to
Gravitino `Types.DateType`. Previously Date32 fell through to the
default branch and was mapped to `ExternalType`, despite being a
calendar date type semantically identical to Date (just with a wider
year range: 1900–2299 vs 1970–2149).
2. **Enum IT test**: Add `testEnumRoundTrip()` integration test
verifying that Enum8/Enum16 round-trip correctly through the Gravitino
type system via ExternalType. The enum definitions (including spaces and
equals signs) are fully preserved across create → load → recreate
cycles.
### Why are the changes needed?
Date32 was the last user-visible ClickHouse type that has a Gravitino
native counterpart (`Types.DateType`) but was not mapped to it. The
`DateTime64(3)` → `TimestampType.withoutTimeZone(3)` → `DateTime(0)`
pattern already establishes the precedent of one-way precision/range
narrowing in the ClickHouse converter — Date32 follows the same pattern.
Mapping Date32 to `DateType` (instead of `ExternalType`) also keeps the
column usable in downstream systems: the Gravitino Spark connector
currently throws `UnsupportedOperationException` for `ExternalType`
columns, making the entire table inaccessible (see #12104), and the
proposed fix there maps them to `StringType`, losing the date semantics
entirely. A native `DateType` mapping preserves the date semantics
across Spark/Flink/Trino integrations.
Enum8/Enum16 already work correctly via ExternalType pass-through, but
had no integration test coverage, leaving a gap in verifying their
round-trip correctness. The test confirms no code change is needed for
Enum support.
Fix: #12272
### Does this PR introduce _any_ user-facing change?
Yes, with a caveat. After this change:
- **Read path**: ClickHouse `Date32` columns are now loaded as Gravitino
`DateType` (instead of `ExternalType("Date32")`)
- **Write path**: `DateType` is still mapped to ClickHouse `Date` (not
`Date32`), so round-tripping a Date32 table through Gravitino will
produce `Date` on the ClickHouse side, narrowing the supported range
from 1900–2299 to 1970–2149-06-06
⚠️ **Silent saturation of out-of-range values** (verified against
ClickHouse 25.3.2.39): ClickHouse does not reject dates outside the
`Date` range — it silently saturates them. Writing `1900-01-01` to a
`Date` column stores `1970-01-01`, and writing `2150-01-01` stores
`2149-06-06`. This only affects values in 1900–1969 or after 2149-06-06,
which are rare in practice.
Users who need the full 1900–2299 range should use
`ExternalType("Date32")` explicitly when creating tables — the
ExternalType pass-through path remains fully available.
This is consistent with the existing `DateTime64(3)` → `DateTime(0)`
precision narrowing pattern.
### How was this patch tested?
- Unit tests: `TestClickHouseTypeConverter` — verified Date32 now maps
to `DateType.get()` (previously `ExternalType.of("Date32")`)
- Docker integration tests: `CatalogClickHouseIT.testEnumRoundTrip` —
verified Enum8/Enum16 round-trip correctness (create → load → verify →
recreate → SHOW CREATE TABLE) against ClickHouse 25.3.2.39
- Static checks: `spotlessCheck`, `rat` (license headers),
`gravitino-pr-precheck.py` — all passed
---------
Signed-off-by: jiangxt2 <[email protected]>
---
.../converter/ClickHouseTypeConverter.java | 2 +-
.../integration/test/CatalogClickHouseIT.java | 125 +++++++++++++++++++++
2 files changed, 126 insertions(+), 1 deletion(-)
diff --git
a/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/converter/ClickHouseTypeConverter.java
b/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/converter/ClickHouseTypeConverter.java
index 03f184b1f7..cc4d9c8adf 100644
---
a/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/converter/ClickHouseTypeConverter.java
+++
b/catalogs-contrib/catalog-jdbc-clickhouse/src/main/java/org/apache/gravitino/catalog/clickhouse/converter/ClickHouseTypeConverter.java
@@ -162,7 +162,7 @@ public class ClickHouseTypeConverter extends
JdbcTypeConverter {
case DATE:
return Types.DateType.get();
case DATE32:
- // Date32 supports 1900-2299 vs Date's 1970-2149. Use ExternalType to
preserve round-trip.
+ // Date32 has a wider range than Date; preserve its catalog type for
round-trip.
return Types.ExternalType.of(DATE32);
case DATETIME:
// Default is 0 precision
diff --git
a/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/integration/test/CatalogClickHouseIT.java
b/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/integration/test/CatalogClickHouseIT.java
index 5ffa5344c4..71d9bc9203 100644
---
a/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/integration/test/CatalogClickHouseIT.java
+++
b/catalogs-contrib/catalog-jdbc-clickhouse/src/test/java/org/apache/gravitino/catalog/clickhouse/integration/test/CatalogClickHouseIT.java
@@ -78,6 +78,7 @@ import
org.apache.gravitino.rel.expressions.transforms.Transforms;
import org.apache.gravitino.rel.indexes.Index;
import org.apache.gravitino.rel.indexes.Indexes;
import org.apache.gravitino.rel.types.Decimal;
+import org.apache.gravitino.rel.types.Type;
import org.apache.gravitino.rel.types.Types;
import org.apache.gravitino.utils.RandomNameUtils;
import org.junit.jupiter.api.AfterAll;
@@ -271,6 +272,50 @@ public class CatalogClickHouseIT extends BaseIT {
return properties;
}
+ private static String normalizeEnumFormatting(String value) {
+ StringBuilder normalized = new StringBuilder(value.length());
+ boolean inSingleQuote = false;
+ boolean pendingWhitespace = false;
+ for (int i = 0; i < value.length(); i++) {
+ char current = value.charAt(i);
+ boolean escapedQuote =
+ current == '\''
+ && i > 0
+ && value.charAt(i - 1) == '\\'
+ && (i < 2 || value.charAt(i - 2) != '\\');
+ if (current == '\'' && !escapedQuote) {
+ if (pendingWhitespace) {
+ appendWhitespaceUnlessSeparator(normalized);
+ pendingWhitespace = false;
+ }
+ normalized.append(current);
+ inSingleQuote = !inSingleQuote;
+ } else if (!inSingleQuote && Character.isWhitespace(current)) {
+ pendingWhitespace = true;
+ } else if (!inSingleQuote && (current == '=' || current == ',')) {
+ pendingWhitespace = false;
+ normalized.append(current);
+ } else {
+ if (pendingWhitespace) {
+ appendWhitespaceUnlessSeparator(normalized);
+ pendingWhitespace = false;
+ }
+ normalized.append(current);
+ }
+ }
+ return normalized.toString().trim();
+ }
+
+ private static void appendWhitespaceUnlessSeparator(StringBuilder builder) {
+ if (builder.length() == 0) {
+ return;
+ }
+ char previous = builder.charAt(builder.length() - 1);
+ if (previous != '=' && previous != ',') {
+ builder.append(' ');
+ }
+ }
+
@Test
void testOperationClickhouseSchema() {
SupportsSchemas schemas = catalog.asSchemas();
@@ -2836,6 +2881,7 @@ public class CatalogClickHouseIT extends BaseIT {
// may normalize the enum definition format (e.g., spacing around '=' and
',').
Assertions.assertTrue(loadedTable.columns()[4].dataType() instanceof
Types.ExternalType);
Assertions.assertTrue(loadedTable.columns()[5].dataType() instanceof
Types.ExternalType);
+ // Date32 remains ExternalType so its wider range and catalog type survive
round-trip.
Assertions.assertEquals(Types.ExternalType.of("Date32"),
loadedTable.columns()[6].dataType());
}
@@ -2872,4 +2918,83 @@ public class CatalogClickHouseIT extends BaseIT {
getSortOrders("id"),
Indexes.EMPTY_INDEXES));
}
+
+ @Test
+ void testEnumRoundTrip() {
+ // Create a table in ClickHouse with Enum8 and Enum16 columns
+ String tableName = GravitinoITUtils.genRandomName("enum_test");
+ clickhouseService.executeQuery(
+ String.format(
+ "CREATE TABLE `%s`.`%s` ("
+ + "id Int32, "
+ + "status Enum8('active' = 1, 'inactive' = 2), "
+ + "priority Enum16('low' = 100, 'medium' = 200, 'high' = 300)"
+ + ") ENGINE = MergeTree ORDER BY id",
+ schemaName, tableName));
+
+ // Load through Gravitino and verify column types are ExternalType
+ Table loaded =
catalog.asTableCatalog().loadTable(NameIdentifier.of(schemaName, tableName));
+ Column[] columns = loaded.columns();
+ Assertions.assertEquals(3, columns.length);
+
+ // Enum8 type verification
+ Type enum8Type = columns[1].dataType();
+ Assertions.assertTrue(
+ enum8Type instanceof Types.ExternalType,
+ "Enum8 should map to ExternalType, but got: " +
enum8Type.simpleString());
+ Assertions.assertEquals(
+ normalizeEnumFormatting("Enum8('active' = 1, 'inactive' = 2)"),
+ normalizeEnumFormatting(((Types.ExternalType)
enum8Type).catalogString()));
+
+ // Enum16 type verification
+ Type enum16Type = columns[2].dataType();
+ Assertions.assertTrue(
+ enum16Type instanceof Types.ExternalType,
+ "Enum16 should map to ExternalType, but got: " +
enum16Type.simpleString());
+ Assertions.assertEquals(
+ normalizeEnumFormatting("Enum16('low' = 100, 'medium' = 200, 'high' =
300)"),
+ normalizeEnumFormatting(((Types.ExternalType)
enum16Type).catalogString()));
+
+ // Round-trip: recreate table through Gravitino with the loaded schema
+ String rtTableName = GravitinoITUtils.genRandomName("enum_rt");
+ catalog
+ .asTableCatalog()
+ .createTable(
+ NameIdentifier.of(schemaName, rtTableName),
+ columns,
+ "enum round-trip test",
+ createProperties(),
+ Transforms.EMPTY_TRANSFORM,
+ Distributions.NONE,
+ new SortOrder[] {SortOrders.of(NamedReference.field("id"),
SortDirection.ASCENDING)});
+
+ // Verify types survived round-trip
+ Table rtLoaded =
catalog.asTableCatalog().loadTable(NameIdentifier.of(schemaName, rtTableName));
+ Column[] rtColumns = rtLoaded.columns();
+ Type rtEnum8 = rtColumns[1].dataType();
+ Type rtEnum16 = rtColumns[2].dataType();
+ Assertions.assertTrue(
+ rtEnum8 instanceof Types.ExternalType, "Enum8 should survive
round-trip as ExternalType");
+ Assertions.assertTrue(
+ rtEnum16 instanceof Types.ExternalType, "Enum16 should survive
round-trip as ExternalType");
+ Assertions.assertEquals(
+ normalizeEnumFormatting(((Types.ExternalType)
enum8Type).catalogString()),
+ normalizeEnumFormatting(((Types.ExternalType)
rtEnum8).catalogString()));
+ Assertions.assertEquals(
+ normalizeEnumFormatting(((Types.ExternalType)
enum16Type).catalogString()),
+ normalizeEnumFormatting(((Types.ExternalType)
rtEnum16).catalogString()));
+
+ // Verify DDL on ClickHouse side contains full enum definitions
+ String createSql =
+ clickhouseService.executeQueryForResult(
+ String.format("SHOW CREATE TABLE `%s`.`%s`", schemaName,
rtTableName));
+ Assertions.assertNotNull(createSql, "SHOW CREATE TABLE should return a
result");
+ String normalizedCreateSql = normalizeEnumFormatting(createSql);
+ Assertions.assertTrue(
+ normalizedCreateSql.contains("Enum8('active'=1,'inactive'=2)"),
+ "SHOW CREATE TABLE should contain Enum8 definition: " + createSql);
+ Assertions.assertTrue(
+
normalizedCreateSql.contains("Enum16('low'=100,'medium'=200,'high'=300)"),
+ "SHOW CREATE TABLE should contain Enum16 definition: " + createSql);
+ }
}
