rangadi commented on code in PR #40686:
URL: https://github.com/apache/spark/pull/40686#discussion_r1181136578
##########
connector/protobuf/src/main/scala/org/apache/spark/sql/protobuf/utils/ProtobufOptions.scala:
##########
@@ -46,6 +46,42 @@ private[sql] class ProtobufOptions(
// record has more depth than the allowed value for recursive fields, it
will be truncated
// and corresponding fields are ignored (dropped).
val recursiveFieldMaxDepth: Int =
parameters.getOrElse("recursive.fields.max.depth", "-1").toInt
+
+ // For fields without presence information, there is ambiguity in serialized
protos
Review Comment:
Looks like the policy seems to be `materialize Protobuf V3 scalar fields
that are not explicitly declared optional`. Lets make it clear and simplify
this. We shouldn't have to read anything more.
Also update the example to include two optional fields and illustrate the
difference between optional and regular fields.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
