jiangxt2 opened a new issue, #12104:
URL: https://github.com/apache/gravitino/issues/12104
### Describe what's wrong
When loading a JDBC catalog table through the Spark connector, if any column
is mapped to `Types.ExternalType` in Gravitino metadata, `loadTable()` throws
`UnsupportedOperationException` and the entire table becomes unusable.
Root cause: `SparkTypeConverter.toSparkType()` handles ~20 Gravitino native
types but does not have a branch for `Types.ExternalType`. Any column with this
type falls through to the final `throw` statement.
The problem affects all JDBC catalogs where `toGravitino()` produces
`ExternalType` for database-specific types:
| Catalog | Types producing ExternalType |
|---------|------------------------------|
| ClickHouse | IPv4, IPv6, Date32, DateTime64, Decimal(>38), LowCardinality,
Map, Tuple, Array, JSON, Enum, Variant, Geo, Nested, Dynamic, AggregateFunction
|
| Doris | LARGEINT, BITMAP, HLL, JSON, Variant, IPV4, IPV6 |
| PostgreSQL | interval, custom types, and unknown-type catch-all |
| MySQL | TINYTEXT, MEDIUMTEXT, YEAR, ENUM, SET, JSON, GEOMETRY and other
types |
| OceanBase | Unknown-type catch-all |
| Hologres | Unknown-type catch-all |
| Glue (via Hive connector) | Unknown Hive types |
The Glue catalog uses `SparkHiveTypeConverter` which also delegates unknown
types to the base class `SparkTypeConverter.toSparkType()`, triggering the same
crash.
### Error message and/or stacktrace
```
java.lang.UnsupportedOperationException: Not support ExternalType: IPv4
at
org.apache.gravitino.spark.connector.SparkTypeConverter.toSparkType(SparkTypeConverter.java:196)
at
org.apache.gravitino.spark.connector.jdbc.SparkJdbcTypeConverter.toSparkType(SparkJdbcTypeConverter.java:39)
at
org.apache.gravitino.spark.connector.utils.GravitinoTableInfoHelper.getSchema(GravitinoTableInfoHelper.java:89)
```
### How to reproduce
1. Register a ClickHouse catalog in Gravitino with a table containing an
`IPv4` column:
```sql
CREATE TABLE test.ipv4_table (id Int32, ip IPv4) ENGINE = MergeTree ORDER
BY id;
```
2. Load the table through the Spark connector:
```sql
SELECT * FROM clickhouse_catalog.test.ipv4_table;
```
3. Result: `UnsupportedOperationException` — the entire table is
inaccessible.
### Proposed fix
Add an `ExternalType → DataTypes.StringType` mapping in
`SparkTypeConverter.toSparkType()` before the final `throw`. This follows the
existing pattern used for `UUID` (also mapped to `StringType` because Spark has
no native UUID type). Placing the fix in the base class automatically covers
all subclasses (JDBC, Hive, and Glue connectors across all Spark versions
3.3/3.4/3.5).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
