[
https://issues.apache.org/jira/browse/SPARK-61992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-61992.
----------------------------------
Fix Version/s: connect-rust-4.4.0
Resolution: Fixed
Issue resolved by pull request 844
[https://github.com/apache/spark-connect-rust/pull/844]
> Take the map fields of a protobuf message's constructor as upb's
> InitMapAttributes, and make the runtime's DecodeError a subclass of protobuf's
> -----------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61992
> URL: https://issues.apache.org/jira/browse/SPARK-61992
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.4.0
>
>
> Two documented gaps of the drop-in's protobuf runtime against PySpark 4.3.0
> with upb (protobuf 7.36.2):
> - A map field of a message's constructor takes only a mapping, and raises
> "TypeError: ... is not iterable" for any other value. upb's InitMapAttributes
> takes, for a map of scalars, anything that MutableMapping.update takes:
> pairs, an iterator of pairs, lists of 2-character strings and [], with
> Python's ValueError for a pair of another shape, and a tuple as the arguments
> of update. For a map of messages it takes the keys that the value iterates,
> each of value[key]: a dict sets the fields of the key's message, and anything
> else goes through CopyFrom.
> - The runtime's DecodeError is not a google.protobuf.message.DecodeError, so
> "except google.protobuf.message.DecodeError" does not catch a failed parse,
> where it catches upb's. The error also does not pickle (attribute lookup
> DecodeError on pyspark.sql.connect.proto failed).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]