[ 
https://issues.apache.org/jira/browse/SPARK-61992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon resolved SPARK-61992.
----------------------------------
    Fix Version/s: connect-rust-4.4.0
       Resolution: Fixed

Issue resolved by pull request 844
[https://github.com/apache/spark-connect-rust/pull/844]

> Take the map fields of a protobuf message's constructor as upb's 
> InitMapAttributes, and make the runtime's DecodeError a subclass of protobuf's
> -----------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61992
>                 URL: https://issues.apache.org/jira/browse/SPARK-61992
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: 5.0.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.4.0
>
>
> Two documented gaps of the drop-in's protobuf runtime against PySpark 4.3.0 
> with upb (protobuf 7.36.2):
> - A map field of a message's constructor takes only a mapping, and raises 
> "TypeError: ... is not iterable" for any other value. upb's InitMapAttributes 
> takes, for a map of scalars, anything that MutableMapping.update takes: 
> pairs, an iterator of pairs, lists of 2-character strings and [], with 
> Python's ValueError for a pair of another shape, and a tuple as the arguments 
> of update. For a map of messages it takes the keys that the value iterates, 
> each of value[key]: a dict sets the fields of the key's message, and anything 
> else goes through CopyFrom.
> - The runtime's DecodeError is not a google.protobuf.message.DecodeError, so 
> "except google.protobuf.message.DecodeError" does not catch a failed parse, 
> where it catches upb's. The error also does not pickle (attribute lookup 
> DecodeError on pyspark.sql.connect.proto failed).



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to