MaxGekk opened a new pull request, #56557: URL: https://github.com/apache/spark/pull/56557
### What changes were proposed in this pull request? This PR adds end-to-end support for nanosecond-capable timestamp types in ORC for both datasource v1 and v2, including Hive ORC paths. It introduces ORC schema/type mappings for nanos timestamps, implements native ORC serializer/deserializer and vectorized reader conversions for `TimestampNTZNanosType` and `TimestampLTZNanosType`, and removes ORC-specific datasource guardrails that previously rejected nanos timestamp types. For Hive ORC, it updates serializer/deserializer and inspector/table-reader conversion logic so nanos timestamps round-trip correctly while remaining compatible with Hive type parsing. It also extends tests to cover: - ORC support in v1 and v2 for precisions 7/8/9 - Native ORC vectorized vs non-vectorized read parity - Hive ORC nanos support with `CONVERT_METASTORE_ORC` enabled/disabled - Backward compatibility: plain ORC `timestamp` (without Spark nanos metadata) is still inferred as `TimestampType` ### Why are the changes needed? SPARK-57455 tracks enabling ORC to support Spark nanosecond timestamp types. ORC was still rejecting these types in datasource capability checks and lacked complete conversion coverage in native/Hive read-write paths. Without these changes, nanos timestamp columns could not be reliably written/read through ORC. ### Does this PR introduce _any_ user-facing change? Yes. Users can now write and read `TimestampNTZNanosType(p)` and `TimestampLTZNanosType(p)` (`p` in 7..9) with ORC in both datasource v1 and v2, including Hive ORC integration paths. ### How was this patch tested? - `./dev/scalastyle` - `./build/mvn scalafmt:format -Dscalafmt.skip=false -Dscalafmt.validateOnly=false -Dscalafmt.changedOnly=false -pl sql/api -pl sql/connect/common -pl sql/connect/server -pl sql/connect/shims -pl sql/connect/client/jvm` - `build/sbt 'sql/Test/compile'` - `build/sbt 'hive/Test/compile'` - `build/sbt 'sql/testOnly org.apache.spark.sql.FileBasedDataSourceSuite -- -z "SPARK-57166"'` - `build/sbt 'hive/testOnly org.apache.spark.sql.hive.orc.HiveOrcSourceSuite -- -z "SPARK-57166"'` - `build/sbt 'sql/testOnly org.apache.spark.sql.execution.datasources.orc.OrcV1QuerySuite -- -z "SPARK-57455"'` - `build/sbt 'sql/testOnly org.apache.spark.sql.execution.datasources.orc.OrcV2QuerySuite -- -z "SPARK-57455"'` - `build/sbt 'sql/testOnly org.apache.spark.sql.execution.datasources.orc.OrcSourceV1Suite -- -z "SPARK-57455"'` - `build/sbt 'sql/testOnly org.apache.spark.sql.execution.datasources.orc.OrcSourceV2Suite -- -z "SPARK-57455"'` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Cursor Codex 5.3 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
