MaxGekk commented on code in PR #56807:
URL: https://github.com/apache/spark/pull/56807#discussion_r3480906744
##########
connector/protobuf/src/main/scala/org/apache/spark/sql/protobuf/ProtobufDeserializer.scala:
##########
@@ -235,6 +235,11 @@ private[sql] class ProtobufDeserializer(
case (LONG, LongType) =>
(updater, ordinal, value) => updater.setLong(ordinal,
value.asInstanceOf[Long])
+ case (LONG, _: TimeType) =>
+ // The int64 field stores nanoseconds-of-day, matching TimeType's
internal
Review Comment:
The unit difference is a deliberate consequence of each format's constraints
rather than an oversight. Avro has no nanosecond logical type (`time-nanos`) -
see AVRO-4043 - so the Avro connector stores TIME as the `time-micros` logical
type. Protobuf has no logical-type system at all here, so we're free to pick
the representation that loses the least information, and `TimeType` is stored
internally as nanoseconds-of-day. Emitting nanos-of-day makes the Protobuf
mapping lossless and keeps the conversion a pure identity copy (no rounding),
whereas forcing micros-of-day would truncate sub-microsecond values for no
benefit.
Note also that in both connectors the canonical, format-agnostic round-trip
is `TIME -> TIME`; the "raw long" is an implementation detail of each on-disk
encoding that a user only observes when they bypass the logical type (Protobuf)
or read with an external tool. So I'd lean toward keeping each connector
lossless within its own constraints rather than degrading Protobuf to match
Avro's limitation. The mapping and its unit are documented in
`sql-data-sources-protobuf.md`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
