[
https://issues.apache.org/jira/browse/SPARK-58972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-58972:
-----------------------------------
Labels: pull-request-available (was: )
> from_protobuf: unwrapped primitive wrapper types should follow wrapper
> presence, not emit.default.values
> --------------------------------------------------------------------------------------------------------
>
> Key: SPARK-58972
> URL: https://issues.apache.org/jira/browse/SPARK-58972
> Project: Spark
> Issue Type: Bug
> Components: Protobuf, SQL
> Affects Versions: 4.1.0
> Reporter: Ben Hollis
> Priority: Major
> Labels: pull-request-available
>
> When from_protobuf is used with unwrap.primitive.wrapper.types=true, the
> well-known primitive wrapper types
> (google.protobuf.Bool,Int32,UInt32,Int64,UInt64,Float,Double,
> String,Bytes}Value) are unwrapped to their inner scalar. A wrapper message
> uses _message_ presence: a present wrapper always carries a value, and its
> inner scalar field is a proto3 singular scalar that serializes as empty when
> it holds the default (0/""/false/empty-bytes).
> Today the deserializer reads that inner scalar through the same path used for
> bare proto3 scalars, which returns null for an unset/default scalar unless
> emit.default.values=true. So a _present_ wrapper whose inner value is the
> default (e.g. Int32Value.of(0)) is deserialized as null when
> emit.default.values=false. This conflates two different concepts:
> emit.default.values governs whether bare proto3 scalar defaults are emitted,
> whereas a wrapper's value is determined by the wrapper message's presence.
> Correct behavior (matching the canonical proto3 JSON / wrapper mapping): a
> present wrapper -- even one whose inner value is the default -- unwraps to
> the inner scalar's default; null results only when the wrapper field is
> absent. This is independent of emit.default.values.
> *Fixing this would also fix an untested crash.* For repeated/map fields of
> unwrapped wrappers the container is non-nullable (containsNull=false /
> valueContainsNull=false in SchemaConverters). A present-but-empty wrapper
> element currently deserializes to null inside that non-null container,
> producing a downstream UnsafeWriter NullPointerException. Under the fix such
> elements become non-null defaults, so both the crash and the schema mismatch
> disappear with no schema change.
> Repro (unwrap=true, emit.default.values=false):
> Int32Value.of(0) -> currently null, should be 0
> repeated Int32Value with an empty element -> currently
> null-in-non-null-array (NPE), should be 0
> map<int, StringValue> with an empty value -> currently
> null-in-non-null-map (NPE), should be ""
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
