diqiu50 opened a new pull request, #11451: URL: https://github.com/apache/gravitino/pull/11451
## What changes were proposed in this pull request? MySQL `datetime` columns were unusable as filter predicates when accessed via Gravitino's Spark JDBC connector (e.g. `WHERE update_time >= '2025-12-04 09:07:43'` would fail with `SQLSyntaxErrorException: ... near ':07:43)'`). **Root cause:** Gravitino's JDBC type converters mapped MySQL `datetime` (`Types.TimestampType.withoutTimeZone()`) to Spark `TimestampNTZType` in Spark 3.4+, while the MySQL JDBC dialect only knows how to format `TimestampType` literals during filter pushdown. In Spark 3.3, this path threw `UnsupportedOperationException` instead. Without Gravitino, Spark's native JDBC MySQL catalog maps `datetime` → `TimestampType`, so filter pushdown works correctly. **Fix:** - `SparkJdbcTypeConverter`: map `TimestampType.withoutTimeZone()` → `TimestampType` (fixes Spark 3.3) - `SparkJdbcTypeConverter34`: override parent's `TimestampNTZType` mapping → `TimestampType` (fixes Spark 3.4) - `GravitinoJdbcCatalogSpark35`: use `SparkJdbcTypeConverter34` instead of `SparkTypeConverter34` (fixes Spark 3.5, also corrects a pre-existing gap where the Spark 3.5 JDBC catalog was missing `VarCharType` handling) Note: `toGravitinoType(TimestampType)` continues to return `withTimeZone()` (inherited, unchanged). Both MySQL `TIMESTAMP` and `DATETIME` columns round-trip to `withTimeZone()` in Gravitino metadata — this asymmetry is intentional for JDBC compatibility. ## How was this patch tested? - Added unit tests for all three affected converters (`TestSparkJdbcTypeConverter`, `TestSparkJdbcTypeConverter34`, `TestSparkJdbcTypeConverter35`) - Added `TimestampType.withoutTimeZone()` to `notSupportGravitinoTypes` in `TestSparkTypeConverter` to document the base-class behavior - All new tests assert the JDBC override, the delegation path, and the intentional `toGravitinoType` asymmetry -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
