mchades commented on code in PR #12105:
URL: https://github.com/apache/gravitino/pull/12105#discussion_r3747345497
##########
spark-connector/spark-common/src/main/java/org/apache/gravitino/spark/connector/SparkTypeConverter.java:
##########
@@ -192,6 +192,11 @@ public DataType toSparkType(Type gravitinoType) {
return DataTypes.createStructType(fields);
} else if (gravitinoType instanceof Types.NullType) {
return DataTypes.NullType;
+ } else if (gravitinoType instanceof Types.ExternalType) {
+ // ExternalType represents types with no Gravitino-native mapping (e.g.
ClickHouse
+ // IPv4/IPv6, Doris LARGEINT/BITMAP, Glue unknown types). Map to
StringType so that
+ // tables containing these columns remain loadable in Spark. Users can
CAST as needed.
+ return DataTypes.StringType;
Review Comment:
I understand the availability concern, but keeping queries running should
not be the primary goal. Since `SparkJdbcTable` is built on Spark JDBC, its
default behavior should remain consistent with the native connector.
A query that succeeds with silently changed semantics can be worse than an
explicit failure. For example, mapping `BLOB` to `StringType` changes
`getBytes`/`setBytes` to `getString`/`setString`. Exposing different schemas
for the same table also increases debugging and explanation costs.
Therefore, `jdbcTable.schema()` should be authoritative. If the schemas
cannot be reconciled, we should fail explicitly by default; a `StringType`
fallback could be offered as an opt-in policy with a warning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
