Ben Hollis created SPARK-58972:
----------------------------------
Summary: from_protobuf: unwrapped primitive wrapper types should
follow wrapper presence, not emit.default.values
Key: SPARK-58972
URL: https://issues.apache.org/jira/browse/SPARK-58972
Project: Spark
Issue Type: Bug
Components: Protobuf, SQL
Affects Versions: 4.1.0
Reporter: Ben Hollis
When from_protobuf is used with unwrap.primitive.wrapper.types=true, the
well-known primitive wrapper types
(google.protobuf.Bool,Int32,UInt32,Int64,UInt64,Float,Double,
String,Bytes}Value) are unwrapped to their inner scalar. A wrapper message uses
_message_ presence: a present wrapper always carries a value, and its inner
scalar field is a proto3 singular scalar that serializes as empty when it holds
the default (0/""/false/empty-bytes).
Today the deserializer reads that inner scalar through the same path used for
bare proto3 scalars, which returns null for an unset/default scalar unless
emit.default.values=true. So a _present_ wrapper whose inner value is the
default (e.g. Int32Value.of(0)) is deserialized as null when
emit.default.values=false. This conflates two different concepts:
emit.default.values governs whether bare proto3 scalar defaults are emitted,
whereas a wrapper's value is determined by the wrapper message's presence.
Correct behavior (matching the canonical proto3 JSON / wrapper mapping): a
present wrapper -- even one whose inner value is the default -- unwraps to the
inner scalar's default; null results only when the wrapper field is absent.
This is independent of emit.default.values.
*Fixing this would also fix an untested crash.* For repeated/map fields of
unwrapped wrappers the container is non-nullable (containsNull=false /
valueContainsNull=false in SchemaConverters). A present-but-empty wrapper
element currently deserializes to null inside that non-null container,
producing a downstream UnsafeWriter NullPointerException. Under the fix such
elements become non-null defaults, so both the crash and the schema mismatch
disappear with no schema change.
Repro (unwrap=true, emit.default.values=false):
Int32Value.of(0) -> currently null, should be 0
repeated Int32Value with an empty element -> currently
null-in-non-null-array (NPE), should be 0
map<int, StringValue> with an empty value -> currently null-in-non-null-map
(NPE), should be ""
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
