nevzheng opened a new pull request, #11941: URL: https://github.com/apache/gravitino/pull/11941
> **Stacked on #11938** (Iceberg V3 nanosecond timestamps). That PR makes Gravitino surface Iceberg > `timestamp_ns` / `timestamptz_ns` columns as `timestamp(9)` / `timestamp_tz(9)`. This PR makes the > Spark connector handle that precision safely. Please review/merge after #11938; the first commit > here is #11938 and will drop out once it lands (I'll rebase). ### What changes were proposed in this pull request? The Spark connector maps every Gravitino `timestamp[_tz]` to Spark's microsecond timestamp regardless of precision, so a `timestamp(9)` is silently truncated to microseconds. Spark itself has no released nanosecond timestamp type — the nanosecond types in [SPARK-57454](https://issues.apache.org/jira/browse/SPARK-57454) are preview-only and flag-gated (`spark.sql.timestampNanosTypes.enabled`), and both `TimestampType` and `TimestampNTZType` are [microsecond precision](https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/types/TimestampType.html). So there is no lossless target; the connector should reject a precision it cannot represent rather than drop digits silently. This adds a single precision guard (`precision() > 6` is rejected) and routes every Spark type converter through it via a shared `convertTimestamp` seam, so it cannot be bypassed: - `SparkTypeConverter` / `SparkTypeConverter34` (iceberg, paimon). - `SparkJdbcTypeConverter` / `SparkJdbcTypeConverter34` (jdbc). - `SparkHiveTypeConverter` / `SparkHiveTypeConverter34` (hive). Precision `6` and below (and unset) are unchanged. > **Draft status:** the guard currently covers the base converters (`SparkTypeConverter` / > `SparkTypeConverter34`, i.e. the iceberg/paimon paths). The jdbc and hive converters override the > timestamp mapping and are being moved onto the shared `convertTimestamp` seam so the guard applies > there too — that is the remaining work before this leaves draft. ### Why are the changes needed? Once #11938 lands, reading an Iceberg V3 `timestamptz_ns` table through the Spark connector would otherwise present a microsecond column and silently discard the sub-microsecond digits. Failing loudly is the correct behavior until Spark ships a stable nanosecond type (at which point a `spark-4.x` converter can map it natively). ### Does this PR introduce _any_ user-facing change? Yes. The Spark connector now throws `UnsupportedOperationException` when asked to represent a `timestamp[_tz]` with precision greater than microsecond (`> 6`), instead of silently truncating it. Precision `6` and below is unchanged. ### How was this patch tested? - Unit tests asserting `timestamp[_tz]` at precision 7/8/9 throws and precision unset/0/6 maps, across the base, jdbc, and hive converters (zoned and `TIMESTAMP_NTZ` paths), for Spark 3.3/3.4/3.5. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
