jiangxt2 opened a new issue, #11879:
URL: https://github.com/apache/gravitino/issues/11879
### Version
main branch
### Describe what's wrong
The ClickHouse catalog has type conversion issues in
`ClickHouseTypeConverter` that cause
tables to either crash on `loadTable` or lose type semantics on round-trip.
1. **Decimal(precision > 38) crashes loadTable**: ClickHouse supports
`Decimal` up to precision 76,
but `Types.DecimalType.checkPrecisionScale()` enforces precision ≤ 38.
Tables with
`Decimal(50, 10)` etc. throw `IllegalArgumentException` and cannot be
loaded at all.
2. **DateTime64(N) mapped to ExternalType**:
`TypeUtils.extractDateTimePrecision()` regex
`^DateTime\((\d+)\)$` does not match `DateTime64(3)`. The entire type
falls through to
`ExternalType`. In `fromGravitino`, `TimestampType` always returns bare
`DateTime` with
no precision — `DateTime64(N)` cannot be reconstructed.
3. **LowCardinality wrapper not stripped**: `ClickHouseTypeConverter` (line
79) calls
`TypeUtils.stripNullable()` to unwrap the type name, but `TypeUtils` has
no
`stripLowCardinality()` method. For `LowCardinality(Nullable(String))`,
the outermost
wrapper is `LowCardinality`, so `stripNullable` does not match — the
entire string falls
through to `ExternalType`.
4. **IPv4/IPv6 constants defined but not mapped**: Constants `IPV4` and
`IPV6` are declared
at lines 57-58 but have no corresponding `case` branches in the
`toGravitino` switch
statement. Both fall through to the default `ExternalType` case.
### Error message and/or stacktrace
For Decimal(50, 10):
java.lang.IllegalArgumentException: Decimal precision must be in
range[1, 38]: precision: 50
at
com.google.common.base.Preconditions.checkArgument(Preconditions.java:191)
at
org.apache.gravitino.rel.types.Types$DecimalType.checkPrecisionScale(Types.java:282)
at
org.apache.gravitino.rel.types.Types$DecimalType.<init>(Types.java:276)
at
org.apache.gravitino.rel.types.Types$DecimalType.of(Types.java:269)
at
org.apache.gravitino.catalog.clickhouse.converter.ClickHouseTypeConverter.toGravitino(ClickHouseTypeConverter.java:129)
at
org.apache.gravitino.catalog.clickhouse.converter.ClickHouseTypeConverter.toGravitino(ClickHouseTypeConverter.java:26)
### How to reproduce
1. Create tables in ClickHouse:
```sql
CREATE TABLE t1 (id Int64, val Decimal(50, 10)) ENGINE=MergeTree ORDER BY
id;
CREATE TABLE t2 (id Int64, ts DateTime64(3)) ENGINE=MergeTree ORDER BY id;
CREATE TABLE t3 (id Int64, name LowCardinality(Nullable(String)))
ENGINE=MergeTree ORDER BY id;
CREATE TABLE t4 (id Int64, addr IPv4) ENGINE=MergeTree ORDER BY id;
```
2. Load each table via Gravitino REST API: `GET
/api/metalakes/{metalake}/catalogs/{catalog}/schemas/{schema}/tables/{table}`
3. Expected: `t1.val` → `decimal(50,10)`, `t2.ts` → `timestamp`, `t3.name` →
`string`, `t4.addr` → mapped type
4. Actual: `t1` crashes with `IllegalArgumentException`; `t2`/`t3`/`t4`
return `ExternalType`
### Additional context
- ClickHouse supports Decimal precision up to 76, but `Types.DecimalType` in
Gravitino core enforces precision ≤ 38. Using `ExternalType` for precision > 38
is consistent with the catalog's existing pattern for non-standard types (IPv4,
DateTime64).
- `TypeUtils.java`:32 regex needs `64?` to match both `DateTime` and
`DateTime64`
- `TypeUtils.java` needs a `stripLowCardinality` method, applied before
`stripNullable` (LowCardinality is the outermost wrapper; stripping Nullable
first won't match `LowCardinality(Nullable(...))` patterns)
- IPv4/IPv6 switch cases are trivial additions (constants already defined)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
