This is an automated email from the ASF dual-hosted git repository.
MaxGekk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new bc72d936a702 [SPARK-57160][CONNECT] Add Spark Connect protocol support
for nanosecond-capable timestamp types and literals
bc72d936a702 is described below
commit bc72d936a702cca196a597047b3ca0bf190235d4
Author: Maxim Gekk <[email protected]>
AuthorDate: Wed Jul 1 12:22:15 2026 +0200
[SPARK-57160][CONNECT] Add Spark Connect protocol support for
nanosecond-capable timestamp types and literals
### What changes were proposed in this pull request?
This PR adds the Spark Connect protocol surface for nanosecond timestamps
so they can travel over the wire, both as types and as literals. There is no
behavior change yet -- the converters that consume these messages land in
follow-up sub-tasks of SPARK-56822.
- `types.proto`: two new data-type kinds, `TimestampNTZNanos` and
`TimestampLTZNanos`, each with an optional `precision` (7..9).
- `expressions.proto`: matching literal arms that carry the value as
`epoch_micros` + `nanos_within_micro` (0..999) plus an optional `precision`.
Two components are used instead of a single int64 of nanoseconds because
nanoseconds-since-epoch cannot cover the full `0001..9999` year range; this
mirrors the Catalyst value `TimestampNanosVal`.
- Regenerated the Python stubs under `python/pyspark/sql/connect/proto/`.
NTZ and LTZ are kept as separate kinds/arms (like `timestamp` vs
`timestamp_ntz`), and non-negative fields use `uint32`.
### Why are the changes needed?
Today the Connect `DataType` message has only microsecond timestamp kinds
(`timestamp`, `timestamp_ntz`) with no precision field, and the
`Expression.Literal` message encodes timestamp literals as a single int64 of
microseconds. There is no way to express a nanosecond-capable timestamp type or
a sub-microsecond literal over the wire, so no Connect client/server path can
carry the new types. The protocol must be extended before any converter, Arrow,
or client work can proceed.
### Does this PR introduce _any_ user-facing change?
No. This only adds protobuf message definitions; the new types remain gated
behind `spark.sql.timestampNanosTypes.enabled` once the consuming paths are
implemented.
### How was this patch tested?
- `buf build` / `buf lint` succeed for the modified protos (field numbers
appended, no reuse/renumber).
- `./dev/connect-gen-protos.sh` regenerates the committed Python stubs;
`./dev/check-protos.py` reports no drift (pyspark-connect and
pyspark-streaming: SUCCESS).
- `build/sbt "connect/testOnly *LiteralExpressionProtoConverterSuite"` (44
tests) and `build/sbt "connect-client-jvm/testOnly
*ColumnNodeToProtoConverterSuite"` (18 tests) pass, confirming the additive
proto fields do not break existing proto plumbing.
No functional tests in this PR (there are no consumers of the new fields
yet); behavior is covered by the converter and end-to-end sub-tasks.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor (Claude Opus 4.8)
Closes #56909 from MaxGekk/nanos-proto.
Authored-by: Maxim Gekk <[email protected]>
Signed-off-by: Max Gekk <[email protected]>
---
.../pyspark/sql/connect/proto/expressions_pb2.py | 134 +++++++++++----------
.../pyspark/sql/connect/proto/expressions_pb2.pyi | 114 ++++++++++++++++++
python/pyspark/sql/connect/proto/types_pb2.py | 124 ++++++++++---------
python/pyspark/sql/connect/proto/types_pb2.pyi | 95 +++++++++++++++
.../main/protobuf/spark/connect/expressions.proto | 28 +++++
.../src/main/protobuf/spark/connect/types.proto | 19 +++
6 files changed, 389 insertions(+), 125 deletions(-)
diff --git a/python/pyspark/sql/connect/proto/expressions_pb2.py
b/python/pyspark/sql/connect/proto/expressions_pb2.py
index cf9647b7c674..4f0feaeceac3 100644
--- a/python/pyspark/sql/connect/proto/expressions_pb2.py
+++ b/python/pyspark/sql/connect/proto/expressions_pb2.py
@@ -41,7 +41,7 @@ from pyspark.sql.connect.proto import common_pb2 as
spark_dot_connect_dot_common
DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(
-
b'\n\x1fspark/connect/expressions.proto\x12\rspark.connect\x1a\x19google/protobuf/any.proto\x1a\x19spark/connect/types.proto\x1a\x1aspark/connect/common.proto"\x92\x38\n\nExpression\x12\x37\n\x06\x63ommon\x18\x12
\x01(\x0b\x32\x1f.spark.connect.ExpressionCommonR\x06\x63ommon\x12=\n\x07literal\x18\x01
\x01(\x0b\x32!.spark.connect.Expression.LiteralH\x00R\x07literal\x12\x62\n\x14unresolved_attribute\x18\x02
\x01(\x0b\x32-.spark.connect.Expression.UnresolvedAttributeH\x00R\x13unresolved
[...]
+
b'\n\x1fspark/connect/expressions.proto\x12\rspark.connect\x1a\x19google/protobuf/any.proto\x1a\x19spark/connect/types.proto\x1a\x1aspark/connect/common.proto"\x90<\n\nExpression\x12\x37\n\x06\x63ommon\x18\x12
\x01(\x0b\x32\x1f.spark.connect.ExpressionCommonR\x06\x63ommon\x12=\n\x07literal\x18\x01
\x01(\x0b\x32!.spark.connect.Expression.LiteralH\x00R\x07literal\x12\x62\n\x14unresolved_attribute\x18\x02
\x01(\x0b\x32-.spark.connect.Expression.UnresolvedAttributeH\x00R\x13unresolvedAtt
[...]
)
_globals = globals()
@@ -69,7 +69,7 @@ if not _descriptor._USE_C_DESCRIPTORS:
"struct_type"
]._serialized_options = b"\030\001"
_globals["_EXPRESSION"]._serialized_start = 133
- _globals["_EXPRESSION"]._serialized_end = 7319
+ _globals["_EXPRESSION"]._serialized_end = 7829
_globals["_EXPRESSION_WINDOW"]._serialized_start = 2103
_globals["_EXPRESSION_WINDOW"]._serialized_end = 2886
_globals["_EXPRESSION_WINDOW_WINDOWFRAME"]._serialized_start = 2393
@@ -91,67 +91,71 @@ if not _descriptor._USE_C_DESCRIPTORS:
_globals["_EXPRESSION_CAST_EVALMODE"]._serialized_start = 3595
_globals["_EXPRESSION_CAST_EVALMODE"]._serialized_end = 3693
_globals["_EXPRESSION_LITERAL"]._serialized_start = 3712
- _globals["_EXPRESSION_LITERAL"]._serialized_end = 5918
- _globals["_EXPRESSION_LITERAL_DECIMAL"]._serialized_start = 4762
- _globals["_EXPRESSION_LITERAL_DECIMAL"]._serialized_end = 4879
- _globals["_EXPRESSION_LITERAL_CALENDARINTERVAL"]._serialized_start = 4881
- _globals["_EXPRESSION_LITERAL_CALENDARINTERVAL"]._serialized_end = 4979
- _globals["_EXPRESSION_LITERAL_ARRAY"]._serialized_start = 4982
- _globals["_EXPRESSION_LITERAL_ARRAY"]._serialized_end = 5116
- _globals["_EXPRESSION_LITERAL_MAP"]._serialized_start = 5119
- _globals["_EXPRESSION_LITERAL_MAP"]._serialized_end = 5354
- _globals["_EXPRESSION_LITERAL_STRUCT"]._serialized_start = 5357
- _globals["_EXPRESSION_LITERAL_STRUCT"]._serialized_end = 5490
- _globals["_EXPRESSION_LITERAL_SPECIALIZEDARRAY"]._serialized_start = 5493
- _globals["_EXPRESSION_LITERAL_SPECIALIZEDARRAY"]._serialized_end = 5813
- _globals["_EXPRESSION_LITERAL_TIME"]._serialized_start = 5815
- _globals["_EXPRESSION_LITERAL_TIME"]._serialized_end = 5890
- _globals["_EXPRESSION_UNRESOLVEDATTRIBUTE"]._serialized_start = 5921
- _globals["_EXPRESSION_UNRESOLVEDATTRIBUTE"]._serialized_end = 6107
- _globals["_EXPRESSION_UNRESOLVEDFUNCTION"]._serialized_start = 6110
- _globals["_EXPRESSION_UNRESOLVEDFUNCTION"]._serialized_end = 6368
- _globals["_EXPRESSION_EXPRESSIONSTRING"]._serialized_start = 6370
- _globals["_EXPRESSION_EXPRESSIONSTRING"]._serialized_end = 6420
- _globals["_EXPRESSION_UNRESOLVEDSTAR"]._serialized_start = 6422
- _globals["_EXPRESSION_UNRESOLVEDSTAR"]._serialized_end = 6546
- _globals["_EXPRESSION_UNRESOLVEDREGEX"]._serialized_start = 6548
- _globals["_EXPRESSION_UNRESOLVEDREGEX"]._serialized_end = 6634
- _globals["_EXPRESSION_UNRESOLVEDEXTRACTVALUE"]._serialized_start = 6637
- _globals["_EXPRESSION_UNRESOLVEDEXTRACTVALUE"]._serialized_end = 6769
- _globals["_EXPRESSION_UPDATEFIELDS"]._serialized_start = 6772
- _globals["_EXPRESSION_UPDATEFIELDS"]._serialized_end = 6959
- _globals["_EXPRESSION_ALIAS"]._serialized_start = 6961
- _globals["_EXPRESSION_ALIAS"]._serialized_end = 7081
- _globals["_EXPRESSION_LAMBDAFUNCTION"]._serialized_start = 7084
- _globals["_EXPRESSION_LAMBDAFUNCTION"]._serialized_end = 7242
- _globals["_EXPRESSION_UNRESOLVEDNAMEDLAMBDAVARIABLE"]._serialized_start =
7244
- _globals["_EXPRESSION_UNRESOLVEDNAMEDLAMBDAVARIABLE"]._serialized_end =
7306
- _globals["_EXPRESSIONCOMMON"]._serialized_start = 7321
- _globals["_EXPRESSIONCOMMON"]._serialized_end = 7386
- _globals["_COMMONINLINEUSERDEFINEDFUNCTION"]._serialized_start = 7389
- _globals["_COMMONINLINEUSERDEFINEDFUNCTION"]._serialized_end = 7786
- _globals["_PYTHONUDF"]._serialized_start = 7789
- _globals["_PYTHONUDF"]._serialized_end = 7993
- _globals["_SCALARSCALAUDF"]._serialized_start = 7996
- _globals["_SCALARSCALAUDF"]._serialized_end = 8210
- _globals["_JAVAUDF"]._serialized_start = 8213
- _globals["_JAVAUDF"]._serialized_end = 8362
- _globals["_TYPEDAGGREGATEEXPRESSION"]._serialized_start = 8364
- _globals["_TYPEDAGGREGATEEXPRESSION"]._serialized_end = 8463
- _globals["_CALLFUNCTION"]._serialized_start = 8465
- _globals["_CALLFUNCTION"]._serialized_end = 8573
- _globals["_NAMEDARGUMENTEXPRESSION"]._serialized_start = 8575
- _globals["_NAMEDARGUMENTEXPRESSION"]._serialized_end = 8667
- _globals["_MERGEACTION"]._serialized_start = 8670
- _globals["_MERGEACTION"]._serialized_end = 9182
- _globals["_MERGEACTION_ASSIGNMENT"]._serialized_start = 8892
- _globals["_MERGEACTION_ASSIGNMENT"]._serialized_end = 8998
- _globals["_MERGEACTION_ACTIONTYPE"]._serialized_start = 9001
- _globals["_MERGEACTION_ACTIONTYPE"]._serialized_end = 9168
- _globals["_SUBQUERYEXPRESSION"]._serialized_start = 9185
- _globals["_SUBQUERYEXPRESSION"]._serialized_end = 9894
- _globals["_SUBQUERYEXPRESSION_TABLEARGOPTIONS"]._serialized_start = 9491
- _globals["_SUBQUERYEXPRESSION_TABLEARGOPTIONS"]._serialized_end = 9725
- _globals["_SUBQUERYEXPRESSION_SUBQUERYTYPE"]._serialized_start = 9728
- _globals["_SUBQUERYEXPRESSION_SUBQUERYTYPE"]._serialized_end = 9872
+ _globals["_EXPRESSION_LITERAL"]._serialized_end = 6428
+ _globals["_EXPRESSION_LITERAL_DECIMAL"]._serialized_start = 4968
+ _globals["_EXPRESSION_LITERAL_DECIMAL"]._serialized_end = 5085
+ _globals["_EXPRESSION_LITERAL_CALENDARINTERVAL"]._serialized_start = 5087
+ _globals["_EXPRESSION_LITERAL_CALENDARINTERVAL"]._serialized_end = 5185
+ _globals["_EXPRESSION_LITERAL_ARRAY"]._serialized_start = 5188
+ _globals["_EXPRESSION_LITERAL_ARRAY"]._serialized_end = 5322
+ _globals["_EXPRESSION_LITERAL_MAP"]._serialized_start = 5325
+ _globals["_EXPRESSION_LITERAL_MAP"]._serialized_end = 5560
+ _globals["_EXPRESSION_LITERAL_STRUCT"]._serialized_start = 5563
+ _globals["_EXPRESSION_LITERAL_STRUCT"]._serialized_end = 5696
+ _globals["_EXPRESSION_LITERAL_SPECIALIZEDARRAY"]._serialized_start = 5699
+ _globals["_EXPRESSION_LITERAL_SPECIALIZEDARRAY"]._serialized_end = 6019
+ _globals["_EXPRESSION_LITERAL_TIME"]._serialized_start = 6021
+ _globals["_EXPRESSION_LITERAL_TIME"]._serialized_end = 6096
+ _globals["_EXPRESSION_LITERAL_TIMESTAMPNTZNANOS"]._serialized_start = 6099
+ _globals["_EXPRESSION_LITERAL_TIMESTAMPNTZNANOS"]._serialized_end = 6248
+ _globals["_EXPRESSION_LITERAL_TIMESTAMPLTZNANOS"]._serialized_start = 6251
+ _globals["_EXPRESSION_LITERAL_TIMESTAMPLTZNANOS"]._serialized_end = 6400
+ _globals["_EXPRESSION_UNRESOLVEDATTRIBUTE"]._serialized_start = 6431
+ _globals["_EXPRESSION_UNRESOLVEDATTRIBUTE"]._serialized_end = 6617
+ _globals["_EXPRESSION_UNRESOLVEDFUNCTION"]._serialized_start = 6620
+ _globals["_EXPRESSION_UNRESOLVEDFUNCTION"]._serialized_end = 6878
+ _globals["_EXPRESSION_EXPRESSIONSTRING"]._serialized_start = 6880
+ _globals["_EXPRESSION_EXPRESSIONSTRING"]._serialized_end = 6930
+ _globals["_EXPRESSION_UNRESOLVEDSTAR"]._serialized_start = 6932
+ _globals["_EXPRESSION_UNRESOLVEDSTAR"]._serialized_end = 7056
+ _globals["_EXPRESSION_UNRESOLVEDREGEX"]._serialized_start = 7058
+ _globals["_EXPRESSION_UNRESOLVEDREGEX"]._serialized_end = 7144
+ _globals["_EXPRESSION_UNRESOLVEDEXTRACTVALUE"]._serialized_start = 7147
+ _globals["_EXPRESSION_UNRESOLVEDEXTRACTVALUE"]._serialized_end = 7279
+ _globals["_EXPRESSION_UPDATEFIELDS"]._serialized_start = 7282
+ _globals["_EXPRESSION_UPDATEFIELDS"]._serialized_end = 7469
+ _globals["_EXPRESSION_ALIAS"]._serialized_start = 7471
+ _globals["_EXPRESSION_ALIAS"]._serialized_end = 7591
+ _globals["_EXPRESSION_LAMBDAFUNCTION"]._serialized_start = 7594
+ _globals["_EXPRESSION_LAMBDAFUNCTION"]._serialized_end = 7752
+ _globals["_EXPRESSION_UNRESOLVEDNAMEDLAMBDAVARIABLE"]._serialized_start =
7754
+ _globals["_EXPRESSION_UNRESOLVEDNAMEDLAMBDAVARIABLE"]._serialized_end =
7816
+ _globals["_EXPRESSIONCOMMON"]._serialized_start = 7831
+ _globals["_EXPRESSIONCOMMON"]._serialized_end = 7896
+ _globals["_COMMONINLINEUSERDEFINEDFUNCTION"]._serialized_start = 7899
+ _globals["_COMMONINLINEUSERDEFINEDFUNCTION"]._serialized_end = 8296
+ _globals["_PYTHONUDF"]._serialized_start = 8299
+ _globals["_PYTHONUDF"]._serialized_end = 8503
+ _globals["_SCALARSCALAUDF"]._serialized_start = 8506
+ _globals["_SCALARSCALAUDF"]._serialized_end = 8720
+ _globals["_JAVAUDF"]._serialized_start = 8723
+ _globals["_JAVAUDF"]._serialized_end = 8872
+ _globals["_TYPEDAGGREGATEEXPRESSION"]._serialized_start = 8874
+ _globals["_TYPEDAGGREGATEEXPRESSION"]._serialized_end = 8973
+ _globals["_CALLFUNCTION"]._serialized_start = 8975
+ _globals["_CALLFUNCTION"]._serialized_end = 9083
+ _globals["_NAMEDARGUMENTEXPRESSION"]._serialized_start = 9085
+ _globals["_NAMEDARGUMENTEXPRESSION"]._serialized_end = 9177
+ _globals["_MERGEACTION"]._serialized_start = 9180
+ _globals["_MERGEACTION"]._serialized_end = 9692
+ _globals["_MERGEACTION_ASSIGNMENT"]._serialized_start = 9402
+ _globals["_MERGEACTION_ASSIGNMENT"]._serialized_end = 9508
+ _globals["_MERGEACTION_ACTIONTYPE"]._serialized_start = 9511
+ _globals["_MERGEACTION_ACTIONTYPE"]._serialized_end = 9678
+ _globals["_SUBQUERYEXPRESSION"]._serialized_start = 9695
+ _globals["_SUBQUERYEXPRESSION"]._serialized_end = 10404
+ _globals["_SUBQUERYEXPRESSION_TABLEARGOPTIONS"]._serialized_start = 10001
+ _globals["_SUBQUERYEXPRESSION_TABLEARGOPTIONS"]._serialized_end = 10235
+ _globals["_SUBQUERYEXPRESSION_SUBQUERYTYPE"]._serialized_start = 10238
+ _globals["_SUBQUERYEXPRESSION_SUBQUERYTYPE"]._serialized_end = 10382
# @@protoc_insertion_point(module_scope)
diff --git a/python/pyspark/sql/connect/proto/expressions_pb2.pyi
b/python/pyspark/sql/connect/proto/expressions_pb2.pyi
index 99c9f13b2c4b..c613ade2f43f 100644
--- a/python/pyspark/sql/connect/proto/expressions_pb2.pyi
+++ b/python/pyspark/sql/connect/proto/expressions_pb2.pyi
@@ -729,6 +729,99 @@ class Expression(google.protobuf.message.Message):
self, oneof_group: typing_extensions.Literal["_precision",
b"_precision"]
) -> typing_extensions.Literal["precision"] | None: ...
+ class TimestampNTZNanos(google.protobuf.message.Message):
+ """A TIMESTAMP_NTZ literal with nanosecond-capable precision. The
physical value is carried
+ as microseconds since the UNIX epoch plus the extra nanoseconds
within that microsecond,
+ because a single int64 of nanoseconds cannot span the supported
year range.
+ """
+
+ DESCRIPTOR: google.protobuf.descriptor.Descriptor
+
+ EPOCH_MICROS_FIELD_NUMBER: builtins.int
+ NANOS_WITHIN_MICRO_FIELD_NUMBER: builtins.int
+ PRECISION_FIELD_NUMBER: builtins.int
+ epoch_micros: builtins.int
+ """Microseconds since the UNIX epoch (without timezone
information)."""
+ nanos_within_micro: builtins.int
+ """Additional nanoseconds within epoch_micros, in [0, 999]."""
+ precision: builtins.int
+ """Number of fractional-second digits (7, 8, or 9). If omitted,
defaults to 9 (nanoseconds)."""
+ def __init__(
+ self,
+ *,
+ epoch_micros: builtins.int = ...,
+ nanos_within_micro: builtins.int = ...,
+ precision: builtins.int | None = ...,
+ ) -> None: ...
+ def HasField(
+ self,
+ field_name: typing_extensions.Literal[
+ "_precision", b"_precision", "precision", b"precision"
+ ],
+ ) -> builtins.bool: ...
+ def ClearField(
+ self,
+ field_name: typing_extensions.Literal[
+ "_precision",
+ b"_precision",
+ "epoch_micros",
+ b"epoch_micros",
+ "nanos_within_micro",
+ b"nanos_within_micro",
+ "precision",
+ b"precision",
+ ],
+ ) -> None: ...
+ def WhichOneof(
+ self, oneof_group: typing_extensions.Literal["_precision",
b"_precision"]
+ ) -> typing_extensions.Literal["precision"] | None: ...
+
+ class TimestampLTZNanos(google.protobuf.message.Message):
+ """A TIMESTAMP_LTZ literal with nanosecond-capable precision. See
TimestampNTZNanos for the
+ rationale behind the two-component physical value.
+ """
+
+ DESCRIPTOR: google.protobuf.descriptor.Descriptor
+
+ EPOCH_MICROS_FIELD_NUMBER: builtins.int
+ NANOS_WITHIN_MICRO_FIELD_NUMBER: builtins.int
+ PRECISION_FIELD_NUMBER: builtins.int
+ epoch_micros: builtins.int
+ """Microseconds since the UNIX epoch."""
+ nanos_within_micro: builtins.int
+ """Additional nanoseconds within epoch_micros, in [0, 999]."""
+ precision: builtins.int
+ """Number of fractional-second digits (7, 8, or 9). If omitted,
defaults to 9 (nanoseconds)."""
+ def __init__(
+ self,
+ *,
+ epoch_micros: builtins.int = ...,
+ nanos_within_micro: builtins.int = ...,
+ precision: builtins.int | None = ...,
+ ) -> None: ...
+ def HasField(
+ self,
+ field_name: typing_extensions.Literal[
+ "_precision", b"_precision", "precision", b"precision"
+ ],
+ ) -> builtins.bool: ...
+ def ClearField(
+ self,
+ field_name: typing_extensions.Literal[
+ "_precision",
+ b"_precision",
+ "epoch_micros",
+ b"epoch_micros",
+ "nanos_within_micro",
+ b"nanos_within_micro",
+ "precision",
+ b"precision",
+ ],
+ ) -> None: ...
+ def WhichOneof(
+ self, oneof_group: typing_extensions.Literal["_precision",
b"_precision"]
+ ) -> typing_extensions.Literal["precision"] | None: ...
+
NULL_FIELD_NUMBER: builtins.int
BINARY_FIELD_NUMBER: builtins.int
BOOLEAN_FIELD_NUMBER: builtins.int
@@ -751,6 +844,8 @@ class Expression(google.protobuf.message.Message):
STRUCT_FIELD_NUMBER: builtins.int
SPECIALIZED_ARRAY_FIELD_NUMBER: builtins.int
TIME_FIELD_NUMBER: builtins.int
+ TIMESTAMP_NTZ_NANOS_FIELD_NUMBER: builtins.int
+ TIMESTAMP_LTZ_NANOS_FIELD_NUMBER: builtins.int
DATA_TYPE_FIELD_NUMBER: builtins.int
@property
def null(self) -> pyspark.sql.connect.proto.types_pb2.DataType: ...
@@ -786,6 +881,13 @@ class Expression(google.protobuf.message.Message):
@property
def time(self) -> global___Expression.Literal.Time: ...
@property
+ def timestamp_ntz_nanos(self) ->
global___Expression.Literal.TimestampNTZNanos:
+ """Nanosecond-capable timestamp literals (precision 7..9). NTZ and
LTZ are distinct
+ arms so the literal kind is self-describing.
+ """
+ @property
+ def timestamp_ltz_nanos(self) ->
global___Expression.Literal.TimestampLTZNanos: ...
+ @property
def data_type(self) -> pyspark.sql.connect.proto.types_pb2.DataType:
"""Data type information for the literal.
This field is required only in the root literal message for null
values or
@@ -818,6 +920,8 @@ class Expression(google.protobuf.message.Message):
struct: global___Expression.Literal.Struct | None = ...,
specialized_array: global___Expression.Literal.SpecializedArray |
None = ...,
time: global___Expression.Literal.Time | None = ...,
+ timestamp_ntz_nanos: global___Expression.Literal.TimestampNTZNanos
| None = ...,
+ timestamp_ltz_nanos: global___Expression.Literal.TimestampLTZNanos
| None = ...,
data_type: pyspark.sql.connect.proto.types_pb2.DataType | None =
...,
) -> None: ...
def HasField(
@@ -867,8 +971,12 @@ class Expression(google.protobuf.message.Message):
b"time",
"timestamp",
b"timestamp",
+ "timestamp_ltz_nanos",
+ b"timestamp_ltz_nanos",
"timestamp_ntz",
b"timestamp_ntz",
+ "timestamp_ntz_nanos",
+ b"timestamp_ntz_nanos",
"year_month_interval",
b"year_month_interval",
],
@@ -920,8 +1028,12 @@ class Expression(google.protobuf.message.Message):
b"time",
"timestamp",
b"timestamp",
+ "timestamp_ltz_nanos",
+ b"timestamp_ltz_nanos",
"timestamp_ntz",
b"timestamp_ntz",
+ "timestamp_ntz_nanos",
+ b"timestamp_ntz_nanos",
"year_month_interval",
b"year_month_interval",
],
@@ -952,6 +1064,8 @@ class Expression(google.protobuf.message.Message):
"struct",
"specialized_array",
"time",
+ "timestamp_ntz_nanos",
+ "timestamp_ltz_nanos",
]
| None
): ...
diff --git a/python/pyspark/sql/connect/proto/types_pb2.py
b/python/pyspark/sql/connect/proto/types_pb2.py
index 224c55660842..2f0e5b2181b0 100644
--- a/python/pyspark/sql/connect/proto/types_pb2.py
+++ b/python/pyspark/sql/connect/proto/types_pb2.py
@@ -36,7 +36,7 @@ _sym_db = _symbol_database.Default()
DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(
-
b"\n\x19spark/connect/types.proto\x12\rspark.connect\"\xd8%\n\x08\x44\x61taType\x12\x32\n\x04null\x18\x01
\x01(\x0b\x32\x1c.spark.connect.DataType.NULLH\x00R\x04null\x12\x38\n\x06\x62inary\x18\x02
\x01(\x0b\x32\x1e.spark.connect.DataType.BinaryH\x00R\x06\x62inary\x12;\n\x07\x62oolean\x18\x03
\x01(\x0b\x32\x1f.spark.connect.DataType.BooleanH\x00R\x07\x62oolean\x12\x32\n\x04\x62yte\x18\x04
\x01(\x0b\x32\x1c.spark.connect.DataType.ByteH\x00R\x04\x62yte\x12\x35\n\x05short\x18\x05
\x01(\x [...]
+
b"\n\x19spark/connect/types.proto\x12\rspark.connect\"\x92)\n\x08\x44\x61taType\x12\x32\n\x04null\x18\x01
\x01(\x0b\x32\x1c.spark.connect.DataType.NULLH\x00R\x04null\x12\x38\n\x06\x62inary\x18\x02
\x01(\x0b\x32\x1e.spark.connect.DataType.BinaryH\x00R\x06\x62inary\x12;\n\x07\x62oolean\x18\x03
\x01(\x0b\x32\x1f.spark.connect.DataType.BooleanH\x00R\x07\x62oolean\x12\x32\n\x04\x62yte\x18\x04
\x01(\x0b\x32\x1c.spark.connect.DataType.ByteH\x00R\x04\x62yte\x12\x35\n\x05short\x18\x05
\x01(\x [...]
)
_globals = globals()
@@ -48,63 +48,67 @@ if not _descriptor._USE_C_DESCRIPTORS:
"DESCRIPTOR"
]._serialized_options =
b"\n\036org.apache.spark.connect.protoP\001Z\022internal/generated"
_globals["_DATATYPE"]._serialized_start = 45
- _globals["_DATATYPE"]._serialized_end = 4869
- _globals["_DATATYPE_BOOLEAN"]._serialized_start = 1778
- _globals["_DATATYPE_BOOLEAN"]._serialized_end = 1845
- _globals["_DATATYPE_BYTE"]._serialized_start = 1847
- _globals["_DATATYPE_BYTE"]._serialized_end = 1911
- _globals["_DATATYPE_SHORT"]._serialized_start = 1913
- _globals["_DATATYPE_SHORT"]._serialized_end = 1978
- _globals["_DATATYPE_INTEGER"]._serialized_start = 1980
- _globals["_DATATYPE_INTEGER"]._serialized_end = 2047
- _globals["_DATATYPE_LONG"]._serialized_start = 2049
- _globals["_DATATYPE_LONG"]._serialized_end = 2113
- _globals["_DATATYPE_FLOAT"]._serialized_start = 2115
- _globals["_DATATYPE_FLOAT"]._serialized_end = 2180
- _globals["_DATATYPE_DOUBLE"]._serialized_start = 2182
- _globals["_DATATYPE_DOUBLE"]._serialized_end = 2248
- _globals["_DATATYPE_STRING"]._serialized_start = 2250
- _globals["_DATATYPE_STRING"]._serialized_end = 2346
- _globals["_DATATYPE_BINARY"]._serialized_start = 2348
- _globals["_DATATYPE_BINARY"]._serialized_end = 2414
- _globals["_DATATYPE_NULL"]._serialized_start = 2416
- _globals["_DATATYPE_NULL"]._serialized_end = 2480
- _globals["_DATATYPE_TIMESTAMP"]._serialized_start = 2482
- _globals["_DATATYPE_TIMESTAMP"]._serialized_end = 2551
- _globals["_DATATYPE_DATE"]._serialized_start = 2553
- _globals["_DATATYPE_DATE"]._serialized_end = 2617
- _globals["_DATATYPE_TIMESTAMPNTZ"]._serialized_start = 2619
- _globals["_DATATYPE_TIMESTAMPNTZ"]._serialized_end = 2691
- _globals["_DATATYPE_TIME"]._serialized_start = 2693
- _globals["_DATATYPE_TIME"]._serialized_end = 2806
- _globals["_DATATYPE_CALENDARINTERVAL"]._serialized_start = 2808
- _globals["_DATATYPE_CALENDARINTERVAL"]._serialized_end = 2884
- _globals["_DATATYPE_YEARMONTHINTERVAL"]._serialized_start = 2887
- _globals["_DATATYPE_YEARMONTHINTERVAL"]._serialized_end = 3066
- _globals["_DATATYPE_DAYTIMEINTERVAL"]._serialized_start = 3069
- _globals["_DATATYPE_DAYTIMEINTERVAL"]._serialized_end = 3246
- _globals["_DATATYPE_CHAR"]._serialized_start = 3248
- _globals["_DATATYPE_CHAR"]._serialized_end = 3336
- _globals["_DATATYPE_VARCHAR"]._serialized_start = 3338
- _globals["_DATATYPE_VARCHAR"]._serialized_end = 3429
- _globals["_DATATYPE_DECIMAL"]._serialized_start = 3432
- _globals["_DATATYPE_DECIMAL"]._serialized_end = 3585
- _globals["_DATATYPE_STRUCTFIELD"]._serialized_start = 3588
- _globals["_DATATYPE_STRUCTFIELD"]._serialized_end = 3749
- _globals["_DATATYPE_STRUCT"]._serialized_start = 3751
- _globals["_DATATYPE_STRUCT"]._serialized_end = 3878
- _globals["_DATATYPE_ARRAY"]._serialized_start = 3881
- _globals["_DATATYPE_ARRAY"]._serialized_end = 4043
- _globals["_DATATYPE_MAP"]._serialized_start = 4046
- _globals["_DATATYPE_MAP"]._serialized_end = 4265
- _globals["_DATATYPE_GEOMETRY"]._serialized_start = 4267
- _globals["_DATATYPE_GEOMETRY"]._serialized_end = 4355
- _globals["_DATATYPE_GEOGRAPHY"]._serialized_start = 4357
- _globals["_DATATYPE_GEOGRAPHY"]._serialized_end = 4446
- _globals["_DATATYPE_VARIANT"]._serialized_start = 4448
- _globals["_DATATYPE_VARIANT"]._serialized_end = 4515
- _globals["_DATATYPE_UDT"]._serialized_start = 4518
- _globals["_DATATYPE_UDT"]._serialized_end = 4807
- _globals["_DATATYPE_UNPARSED"]._serialized_start = 4809
- _globals["_DATATYPE_UNPARSED"]._serialized_end = 4861
+ _globals["_DATATYPE"]._serialized_end = 5311
+ _globals["_DATATYPE_BOOLEAN"]._serialized_start = 1964
+ _globals["_DATATYPE_BOOLEAN"]._serialized_end = 2031
+ _globals["_DATATYPE_BYTE"]._serialized_start = 2033
+ _globals["_DATATYPE_BYTE"]._serialized_end = 2097
+ _globals["_DATATYPE_SHORT"]._serialized_start = 2099
+ _globals["_DATATYPE_SHORT"]._serialized_end = 2164
+ _globals["_DATATYPE_INTEGER"]._serialized_start = 2166
+ _globals["_DATATYPE_INTEGER"]._serialized_end = 2233
+ _globals["_DATATYPE_LONG"]._serialized_start = 2235
+ _globals["_DATATYPE_LONG"]._serialized_end = 2299
+ _globals["_DATATYPE_FLOAT"]._serialized_start = 2301
+ _globals["_DATATYPE_FLOAT"]._serialized_end = 2366
+ _globals["_DATATYPE_DOUBLE"]._serialized_start = 2368
+ _globals["_DATATYPE_DOUBLE"]._serialized_end = 2434
+ _globals["_DATATYPE_STRING"]._serialized_start = 2436
+ _globals["_DATATYPE_STRING"]._serialized_end = 2532
+ _globals["_DATATYPE_BINARY"]._serialized_start = 2534
+ _globals["_DATATYPE_BINARY"]._serialized_end = 2600
+ _globals["_DATATYPE_NULL"]._serialized_start = 2602
+ _globals["_DATATYPE_NULL"]._serialized_end = 2666
+ _globals["_DATATYPE_TIMESTAMP"]._serialized_start = 2668
+ _globals["_DATATYPE_TIMESTAMP"]._serialized_end = 2737
+ _globals["_DATATYPE_DATE"]._serialized_start = 2739
+ _globals["_DATATYPE_DATE"]._serialized_end = 2803
+ _globals["_DATATYPE_TIMESTAMPNTZ"]._serialized_start = 2805
+ _globals["_DATATYPE_TIMESTAMPNTZ"]._serialized_end = 2877
+ _globals["_DATATYPE_TIME"]._serialized_start = 2879
+ _globals["_DATATYPE_TIME"]._serialized_end = 2992
+ _globals["_DATATYPE_TIMESTAMPNTZNANOS"]._serialized_start = 2994
+ _globals["_DATATYPE_TIMESTAMPNTZNANOS"]._serialized_end = 3120
+ _globals["_DATATYPE_TIMESTAMPLTZNANOS"]._serialized_start = 3122
+ _globals["_DATATYPE_TIMESTAMPLTZNANOS"]._serialized_end = 3248
+ _globals["_DATATYPE_CALENDARINTERVAL"]._serialized_start = 3250
+ _globals["_DATATYPE_CALENDARINTERVAL"]._serialized_end = 3326
+ _globals["_DATATYPE_YEARMONTHINTERVAL"]._serialized_start = 3329
+ _globals["_DATATYPE_YEARMONTHINTERVAL"]._serialized_end = 3508
+ _globals["_DATATYPE_DAYTIMEINTERVAL"]._serialized_start = 3511
+ _globals["_DATATYPE_DAYTIMEINTERVAL"]._serialized_end = 3688
+ _globals["_DATATYPE_CHAR"]._serialized_start = 3690
+ _globals["_DATATYPE_CHAR"]._serialized_end = 3778
+ _globals["_DATATYPE_VARCHAR"]._serialized_start = 3780
+ _globals["_DATATYPE_VARCHAR"]._serialized_end = 3871
+ _globals["_DATATYPE_DECIMAL"]._serialized_start = 3874
+ _globals["_DATATYPE_DECIMAL"]._serialized_end = 4027
+ _globals["_DATATYPE_STRUCTFIELD"]._serialized_start = 4030
+ _globals["_DATATYPE_STRUCTFIELD"]._serialized_end = 4191
+ _globals["_DATATYPE_STRUCT"]._serialized_start = 4193
+ _globals["_DATATYPE_STRUCT"]._serialized_end = 4320
+ _globals["_DATATYPE_ARRAY"]._serialized_start = 4323
+ _globals["_DATATYPE_ARRAY"]._serialized_end = 4485
+ _globals["_DATATYPE_MAP"]._serialized_start = 4488
+ _globals["_DATATYPE_MAP"]._serialized_end = 4707
+ _globals["_DATATYPE_GEOMETRY"]._serialized_start = 4709
+ _globals["_DATATYPE_GEOMETRY"]._serialized_end = 4797
+ _globals["_DATATYPE_GEOGRAPHY"]._serialized_start = 4799
+ _globals["_DATATYPE_GEOGRAPHY"]._serialized_end = 4888
+ _globals["_DATATYPE_VARIANT"]._serialized_start = 4890
+ _globals["_DATATYPE_VARIANT"]._serialized_end = 4957
+ _globals["_DATATYPE_UDT"]._serialized_start = 4960
+ _globals["_DATATYPE_UDT"]._serialized_end = 5249
+ _globals["_DATATYPE_UNPARSED"]._serialized_start = 5251
+ _globals["_DATATYPE_UNPARSED"]._serialized_end = 5303
# @@protoc_insertion_point(module_scope)
diff --git a/python/pyspark/sql/connect/proto/types_pb2.pyi
b/python/pyspark/sql/connect/proto/types_pb2.pyi
index f2c727b26db9..8a68a7511620 100644
--- a/python/pyspark/sql/connect/proto/types_pb2.pyi
+++ b/python/pyspark/sql/connect/proto/types_pb2.pyi
@@ -314,6 +314,80 @@ class DataType(google.protobuf.message.Message):
self, oneof_group: typing_extensions.Literal["_precision",
b"_precision"]
) -> typing_extensions.Literal["precision"] | None: ...
+ class TimestampNTZNanos(google.protobuf.message.Message):
+ """Timestamp without time zone with nanosecond-capable
fractional-second precision."""
+
+ DESCRIPTOR: google.protobuf.descriptor.Descriptor
+
+ PRECISION_FIELD_NUMBER: builtins.int
+ TYPE_VARIATION_REFERENCE_FIELD_NUMBER: builtins.int
+ precision: builtins.int
+ """Number of fractional-second digits. Valid values are 7, 8, and 9."""
+ type_variation_reference: builtins.int
+ def __init__(
+ self,
+ *,
+ precision: builtins.int | None = ...,
+ type_variation_reference: builtins.int = ...,
+ ) -> None: ...
+ def HasField(
+ self,
+ field_name: typing_extensions.Literal[
+ "_precision", b"_precision", "precision", b"precision"
+ ],
+ ) -> builtins.bool: ...
+ def ClearField(
+ self,
+ field_name: typing_extensions.Literal[
+ "_precision",
+ b"_precision",
+ "precision",
+ b"precision",
+ "type_variation_reference",
+ b"type_variation_reference",
+ ],
+ ) -> None: ...
+ def WhichOneof(
+ self, oneof_group: typing_extensions.Literal["_precision",
b"_precision"]
+ ) -> typing_extensions.Literal["precision"] | None: ...
+
+ class TimestampLTZNanos(google.protobuf.message.Message):
+ """Timestamp with local time zone with nanosecond-capable
fractional-second precision."""
+
+ DESCRIPTOR: google.protobuf.descriptor.Descriptor
+
+ PRECISION_FIELD_NUMBER: builtins.int
+ TYPE_VARIATION_REFERENCE_FIELD_NUMBER: builtins.int
+ precision: builtins.int
+ """Number of fractional-second digits. Valid values are 7, 8, and 9."""
+ type_variation_reference: builtins.int
+ def __init__(
+ self,
+ *,
+ precision: builtins.int | None = ...,
+ type_variation_reference: builtins.int = ...,
+ ) -> None: ...
+ def HasField(
+ self,
+ field_name: typing_extensions.Literal[
+ "_precision", b"_precision", "precision", b"precision"
+ ],
+ ) -> builtins.bool: ...
+ def ClearField(
+ self,
+ field_name: typing_extensions.Literal[
+ "_precision",
+ b"_precision",
+ "precision",
+ b"precision",
+ "type_variation_reference",
+ b"type_variation_reference",
+ ],
+ ) -> None: ...
+ def WhichOneof(
+ self, oneof_group: typing_extensions.Literal["_precision",
b"_precision"]
+ ) -> typing_extensions.Literal["precision"] | None: ...
+
class CalendarInterval(google.protobuf.message.Message):
DESCRIPTOR: google.protobuf.descriptor.Descriptor
@@ -866,6 +940,8 @@ class DataType(google.protobuf.message.Message):
GEOGRAPHY_FIELD_NUMBER: builtins.int
UNPARSED_FIELD_NUMBER: builtins.int
TIME_FIELD_NUMBER: builtins.int
+ TIMESTAMP_NTZ_NANOS_FIELD_NUMBER: builtins.int
+ TIMESTAMP_LTZ_NANOS_FIELD_NUMBER: builtins.int
@property
def null(self) -> global___DataType.NULL: ...
@property
@@ -930,6 +1006,13 @@ class DataType(google.protobuf.message.Message):
"""UnparsedDataType"""
@property
def time(self) -> global___DataType.Time: ...
+ @property
+ def timestamp_ntz_nanos(self) -> global___DataType.TimestampNTZNanos:
+ """Nanosecond-capable timestamp types (precision 7..9). NTZ and LTZ
are distinct kinds
+ even though their physical value is identical, mirroring timestamp vs
timestamp_ntz.
+ """
+ @property
+ def timestamp_ltz_nanos(self) -> global___DataType.TimestampLTZNanos: ...
def __init__(
self,
*,
@@ -961,6 +1044,8 @@ class DataType(google.protobuf.message.Message):
geography: global___DataType.Geography | None = ...,
unparsed: global___DataType.Unparsed | None = ...,
time: global___DataType.Time | None = ...,
+ timestamp_ntz_nanos: global___DataType.TimestampNTZNanos | None = ...,
+ timestamp_ltz_nanos: global___DataType.TimestampLTZNanos | None = ...,
) -> None: ...
def HasField(
self,
@@ -1011,8 +1096,12 @@ class DataType(google.protobuf.message.Message):
b"time",
"timestamp",
b"timestamp",
+ "timestamp_ltz_nanos",
+ b"timestamp_ltz_nanos",
"timestamp_ntz",
b"timestamp_ntz",
+ "timestamp_ntz_nanos",
+ b"timestamp_ntz_nanos",
"udt",
b"udt",
"unparsed",
@@ -1074,8 +1163,12 @@ class DataType(google.protobuf.message.Message):
b"time",
"timestamp",
b"timestamp",
+ "timestamp_ltz_nanos",
+ b"timestamp_ltz_nanos",
"timestamp_ntz",
b"timestamp_ntz",
+ "timestamp_ntz_nanos",
+ b"timestamp_ntz_nanos",
"udt",
b"udt",
"unparsed",
@@ -1120,6 +1213,8 @@ class DataType(google.protobuf.message.Message):
"geography",
"unparsed",
"time",
+ "timestamp_ntz_nanos",
+ "timestamp_ltz_nanos",
]
| None
): ...
diff --git
a/sql/connect/common/src/main/protobuf/spark/connect/expressions.proto
b/sql/connect/common/src/main/protobuf/spark/connect/expressions.proto
index f74c5af11782..e6a071598b9d 100644
--- a/sql/connect/common/src/main/protobuf/spark/connect/expressions.proto
+++ b/sql/connect/common/src/main/protobuf/spark/connect/expressions.proto
@@ -205,6 +205,11 @@ message Expression {
SpecializedArray specialized_array = 25;
Time time = 26;
+
+ // Nanosecond-capable timestamp literals (precision 7..9). NTZ and LTZ
are distinct
+ // arms so the literal kind is self-describing.
+ TimestampNTZNanos timestamp_ntz_nanos = 29;
+ TimestampLTZNanos timestamp_ltz_nanos = 30;
}
// Reserved for Geometry and Geography.
@@ -289,6 +294,29 @@ message Expression {
// The precision of this time, if omitted, uses the default value of
MICROS_PRECISION.
optional int32 precision = 2;
}
+
+ // A TIMESTAMP_NTZ literal with nanosecond-capable precision. The physical
value is carried
+ // as microseconds since the UNIX epoch plus the extra nanoseconds within
that microsecond,
+ // because a single int64 of nanoseconds cannot span the supported year
range.
+ message TimestampNTZNanos {
+ // Microseconds since the UNIX epoch (without timezone information).
+ int64 epoch_micros = 1;
+ // Additional nanoseconds within epoch_micros, in [0, 999].
+ uint32 nanos_within_micro = 2;
+ // Number of fractional-second digits (7, 8, or 9). If omitted, defaults
to 9 (nanoseconds).
+ optional uint32 precision = 3;
+ }
+
+ // A TIMESTAMP_LTZ literal with nanosecond-capable precision. See
TimestampNTZNanos for the
+ // rationale behind the two-component physical value.
+ message TimestampLTZNanos {
+ // Microseconds since the UNIX epoch.
+ int64 epoch_micros = 1;
+ // Additional nanoseconds within epoch_micros, in [0, 999].
+ uint32 nanos_within_micro = 2;
+ // Number of fractional-second digits (7, 8, or 9). If omitted, defaults
to 9 (nanoseconds).
+ optional uint32 precision = 3;
+ }
}
// An unresolved attribute that is not explicitly bound to a specific
column, but the column
diff --git a/sql/connect/common/src/main/protobuf/spark/connect/types.proto
b/sql/connect/common/src/main/protobuf/spark/connect/types.proto
index caaa2340f95d..4a3e13b3d5e7 100644
--- a/sql/connect/common/src/main/protobuf/spark/connect/types.proto
+++ b/sql/connect/common/src/main/protobuf/spark/connect/types.proto
@@ -76,6 +76,11 @@ message DataType {
Unparsed unparsed = 24;
Time time = 28;
+
+ // Nanosecond-capable timestamp types (precision 7..9). NTZ and LTZ are
distinct kinds
+ // even though their physical value is identical, mirroring timestamp vs
timestamp_ntz.
+ TimestampNTZNanos timestamp_ntz_nanos = 29;
+ TimestampLTZNanos timestamp_ltz_nanos = 30;
}
message Boolean {
@@ -136,6 +141,20 @@ message DataType {
uint32 type_variation_reference = 2;
}
+ // Timestamp without time zone with nanosecond-capable fractional-second
precision.
+ message TimestampNTZNanos {
+ // Number of fractional-second digits. Valid values are 7, 8, and 9.
+ optional uint32 precision = 1;
+ uint32 type_variation_reference = 2;
+ }
+
+ // Timestamp with local time zone with nanosecond-capable fractional-second
precision.
+ message TimestampLTZNanos {
+ // Number of fractional-second digits. Valid values are 7, 8, and 9.
+ optional uint32 precision = 1;
+ uint32 type_variation_reference = 2;
+ }
+
message CalendarInterval {
uint32 type_variation_reference = 1;
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
