[ 
https://issues.apache.org/jira/browse/SPARK-62019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon resolved SPARK-62019.
----------------------------------
    Fix Version/s: connect-rust-4.4.0
       Resolution: Fixed

Issue resolved by pull request 860
[https://github.com/apache/spark-connect-rust/pull/860]

> Compare the messages of the Rust protobuf runtime as upb's: floats of their 
> bits, unknown fields of their contents, maps as mappings, and take only 
> protobuf's messages of its default pool
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-62019
>                 URL: https://issues.apache.org/jira/browse/SPARK-62019
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: 5.0.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.4.0
>
>
> Problem
> pyspark.sql.connect.proto is the drop-in's protobuf runtime in Rust 
> (crates/pyspark-rs/src/pbmsg),
> which follows upb, the implementation of a protobuf install that 
> pyspark-client 4.3.0 uses. Its
> equality and its type checks differ from upb's:
> 1. == of two messages compared float and double fields by value. upb 
> (upb_Message_IsEqual) compares
>    them by their bits. So a message of a NaN never equaled its copy or its 
> parse, `in` and
>    `remove` of a container missed it, and -0.0 equaled 0.0 and the unset 
> default of a field
>    without presence.
> 2. == compared the unknown fields of two messages byte for byte. upb compares 
> their fields, sorted
>    by their tags, of their values. So the same unknown fields in another 
> order, or a varint of
>    another encoding, were unequal.
> 3. CopyFrom, MergeFrom, the constructors and the containers took any object 
> that has the
>    DESCRIPTOR.full_name of the type and a SerializeToString: protobuf's 
> message of a type of the
>    same name of another pool, or a lookalike. upb takes a message of its 
> class only, and refuses
>    those with the TypeError of another class. == already refused them, since 
> SPARK-61958.
> 4. == of two maps of other key or value types (a plugin's maps of int32 and 
> of int64 keys) was
>    False for equal keys and values, and a map of a NaN value equaled itself. 
> upb's == of maps is
>    Mapping.__eq__, of the dicts of their items.
> docs/compatibility.md and dev/parity/known-differences.md also gave no reason 
> why a plugin's
> file that imports no proto of Spark Connect (Delta Lake's base.proto) keeps 
> protobuf's classes.
> Repro (master 9cf73197)
>   >>> import copy; import pyspark.sql.connect.proto as pb
>   >>> m = pb.Expression.Literal(double=float("nan")); m == copy.deepcopy(m)
>   False
>   >>> pb.Sample(lower_bound=-0.0) == pb.Sample()
>   True
>   >>> a = 
> pb.Expression.Literal.FromString(bytes.fromhex("a81f05b21f03616263"))
>   >>> b = 
> pb.Expression.Literal.FromString(bytes.fromhex("b21f03616263a81f05"))
>   >>> a == b             # the same unknown fields 501 and 502, in the other 
> order
>   False
>   >>> other = <protobuf's class of spark.connect.StorageLevel of another 
> DescriptorPool>
>   >>> m = pb.StorageLevel(); m.CopyFrom(other(use_disk=True)); m.use_disk
>   True
> Expected (as with pyspark-client 4.3.0)
>   True, False, True, and CopyFrom raises TypeError: Parameter to CopyFrom() 
> must be instance of
>   same class: expected <class '...StorageLevel'> got <class 'StorageLevel'>.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to