MaxGekk opened a new pull request, #56909: URL: https://github.com/apache/spark/pull/56909
### What changes were proposed in this pull request? Extend the Spark Connect protobuf protocol to represent nanosecond-capable timestamp types (`TimestampNTZNanos(p)` / `TimestampLTZNanos(p)`, `p` in `[7, 9]`) both as data types and as literal values, and regenerate the Python stubs. - `sql/connect/common/src/main/protobuf/spark/connect/types.proto`: add nested messages `DataType.TimestampNTZNanos` and `DataType.TimestampLTZNanos` (each `optional uint32 precision` (7..9) + `uint32 type_variation_reference`) and two new `oneof kind` arms `timestamp_ntz_nanos = 29` and `timestamp_ltz_nanos = 30`. - `sql/connect/common/src/main/protobuf/spark/connect/expressions.proto`: add nested messages `Expression.Literal.TimestampNTZNanos` and `Expression.Literal.TimestampLTZNanos` (`int64 epoch_micros`, `uint32 nanos_within_micro` (0..999), `optional uint32 precision` (7..9)) and two new `oneof literal_type` arms `timestamp_ntz_nanos = 29` and `timestamp_ltz_nanos = 30` (27/28 remain reserved for geospatial). - Regenerate the Python stubs under `python/pyspark/sql/connect/proto/` (`types_pb2.py(i)`, `expressions_pb2.py(i)`). NTZ and LTZ are distinct kinds/arms even though the physical value is identical, mirroring `timestamp` vs `timestamp_ntz` and the parameterized `Time`. The literal carries microseconds plus the extra nanoseconds within that microsecond (not a single int64 of nanoseconds) because a single int64 of nanoseconds-since-epoch cannot span the supported `[0001-01-01, 9999-12-31]` range; this matches the Catalyst physical value `TimestampNanosVal(epochMicros, nanosWithinMicro)`. Non-negative scalar fields use `uint32` to match their value domain in this cross-language contract. This is the protocol foundation for Spark Connect support of nanosecond timestamps (parent: SPARK-56822). The Scala/Python converters that consume these proto messages are separate sub-tasks and depend on this one. ### Why are the changes needed? Today the Connect `DataType` message has only microsecond timestamp kinds (`timestamp`, `timestamp_ntz`) with no precision field, and the `Expression.Literal` message encodes timestamp literals as a single int64 of microseconds. There is no way to express a nanosecond-capable timestamp type or a sub-microsecond literal over the wire, so no Connect client/server path can carry the new types. The protocol must be extended before any converter, Arrow, or client work can proceed. ### Does this PR introduce _any_ user-facing change? No. This only adds protobuf message definitions; the new types remain gated behind `spark.sql.timestampNanosTypes.enabled` once the consuming paths are implemented. ### How was this patch tested? - `buf build` / `buf lint` succeed for the modified protos (field numbers appended, no reuse/renumber). - `./dev/connect-gen-protos.sh` regenerates the committed Python stubs; `./dev/check-protos.py` reports no drift (pyspark-connect and pyspark-streaming: SUCCESS). - `build/sbt "connect/testOnly *LiteralExpressionProtoConverterSuite"` (44 tests) and `build/sbt "connect-client-jvm/testOnly *ColumnNodeToProtoConverterSuite"` (18 tests) pass, confirming the additive proto fields do not break existing proto plumbing. - scalafmt and `./dev/scalastyle` pass. No functional tests in this PR (there are no consumers of the new fields yet); behavior is covered by the converter and end-to-end sub-tasks. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Cursor (Claude Opus 4.8) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
