anshulbaliga7 commented on PR #60211: URL: https://github.com/apache/spark/pull/60211#issuecomment-6716142493
Thanks for the thorough review. The read regression is real: native ORC stores `TIMESTAMP_NTZ` as a `LONG` column and the Hive reader decodes it directly, so gating reads would have broken tables written by the default implementation. Updated in the latest commit: - The rejection moved out of `supportDataType` into `prepareWrite`, so a `TIMESTAMP_NTZ` write (at any nesting level) still fails on the driver with `UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE`, before job setup, while reads and `CREATE TABLE` behave as on master. - Reading a Hive timestamp column with a `TIMESTAMP_NTZ` schema now fails with `UNSUPPORTED_FEATURE.ORC_TYPE_CAST`, as the native reader does, instead of returning the instant shifted into a wall-clock value. Fields containing `TIMESTAMP_NTZ` go through the data-type-aware unwrapper for that; `LONG`-backed values still decode directly. - Tests cover the write rejection, native-written `TIMESTAMP_NTZ` data (top-level, struct, array, map) read back through the Hive implementation via a path and a catalog table including a projection that skips the column, and the conversion error top-level and nested. - The comment now states the actual reason (no serializer mapping for microsecond `TIMESTAMP_NTZ`, unlike `TIMESTAMP_NTZ(7..9)`), and the PR description lists the behavior changes. On the precision split: it predates this PR (SPARK-57455 added the nanosecond mapping deliberately), and this PR only turns an existing failure into a clear error, following SPARK-60545 rather than deciding whether this format should store microsecond `TIMESTAMP_NTZ`. Supporting it would also need a choice between the Hive-timestamp layout the nanosecond types use and the `LONG` layout the native implementation uses, which affects interoperability between the two implementations, so I'd rather leave that to a separate JIRA if the maintainers want it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
