bhollis-dbx opened a new pull request, #57474:
URL: https://github.com/apache/spark/pull/57474
### What changes were proposed in this pull request?
`from_protobuf` always maps `google.protobuf.Timestamp` -> `TimestampType`
and `google.protobuf.Duration` -> `DayTimeIntervalType`. This is Spark's
well-known-type conversion and there is currently no way to turn it off.
This adds a `from_protobuf` reader option
**`convert.timestamp.duration.to.native`** (default `true`, preserving today's
behavior). When set to `false`, both well-known types fall through to the
generic message path and deserialize as their raw proto shape `struct<seconds:
bigint, nanos: int>`.
The two well-known-type special-cases in `SchemaConverters.structFieldFor`
are the only places these types are selected. Guarding both with the flag makes
`Timestamp`/`Duration` fall through to the existing generic `case MESSAGE =>`
-> `StructType` path. The deserializer needs no change: it dispatches on the
catalyst type, so the existing `(MESSAGE, StructType)` case already produces
the struct. The option mirrors the existing `unwrap.primitive.wrapper.types`
precedent.
Scope note: the `to_protobuf` serializer is intentionally not touched --
this is a read-path option only.
### Why are the changes needed?
A consumer whose downstream schema was defined against the `{seconds,
nanos}` struct representation (e.g. a table populated by a different decode
path) and must stay stable, or that needs the exact `seconds`/`nanos` rather
than a microsecond-truncated interval/timestamp, currently has no way to keep
these well-known types in their raw proto shape.
### Does this PR introduce _any_ user-facing change?
Yes, additively. A new `from_protobuf` reader option
`convert.timestamp.duration.to.native` is introduced. Its default is `true`,
which preserves the existing behavior (`google.protobuf.Timestamp` ->
`TimestampType`, `google.protobuf.Duration` -> `DayTimeIntervalType`). Setting
it to `false` makes those well-known types deserialize as `struct<seconds:
bigint, nanos: int>` instead. Default behavior is unchanged, so this is purely
opt-in.
### How was this patch tested?
Added `ProtobufFunctionsSuite` test
*"convert.timestamp.duration.to.native=false keeps Timestamp/Duration as
struct"*, which round-trips a `Timestamp` and a `Duration` field through
`to_protobuf`/`from_protobuf` with the option set to `false` and asserts:
- the deserialized field type is `struct<seconds: bigint, nanos: int>` (not
`TimestampType`/`DayTimeIntervalType`), and
- the values round-trip correctly (the `Duration` `4.5s` -> `seconds=4,
nanos=500000000`; the `Timestamp` seconds are derived from the input so the
assertion is time-zone independent).
Default behavior (`true`) is unchanged, so existing well-known-type tests
continue to exercise the native-type path. Ran the new test and the existing
WKT round-trip tests locally via sbt; all pass.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
