justaparth commented on code in PR #40686:
URL: https://github.com/apache/spark/pull/40686#discussion_r1179805430
##########
connector/protobuf/src/main/scala/org/apache/spark/sql/protobuf/utils/ProtobufOptions.scala:
##########
@@ -46,9 +47,35 @@ private[sql] class ProtobufOptions(
// record has more depth than the allowed value for recursive fields, it
will be truncated
// and corresponding fields are ignored (dropped).
val recursiveFieldMaxDepth: Int =
parameters.getOrElse("recursive.fields.max.depth", "-1").toInt
+
+ // Whether or not to explicitly materialize the zero values for fields
+ // without field presence information
https://protobuf.dev/programming-guides/field_presence/.
+ // This includes most fields in proto3.
+ //
+ // For example, if we have a proto like
+ // ```
+ // syntax = "proto3";
+ // message Example {
+ // string s = 1;
+ // int64 i = 2;
Review Comment:
ah, i figured out how to link changes:
i think this behavior is shown in this test:
[test](https://github.com/apache/spark/pull/40686/files#diff-f5dc81b1d20dde97f9d1dd26e993f811fa096bf68936ab5e449b9937314585d8R1123-R1159)
If you look at this test, you'll see that we [don't set a
value](https://github.com/apache/spark/pull/40686/files#diff-f5dc81b1d20dde97f9d1dd26e993f811fa096bf68936ab5e449b9937314585d8R1126-R1137)
for the `message` field. And the deserialized
[result](https://github.com/apache/spark/pull/40686/files#diff-f5dc81b1d20dde97f9d1dd26e993f811fa096bf68936ab5e449b9937314585d8R1148)
is null for that field,
[regardless](https://github.com/apache/spark/pull/40686/files#diff-f5dc81b1d20dde97f9d1dd26e993f811fa096bf68936ab5e449b9937314585d8R1168)
of the flag.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
