[ 
https://issues.apache.org/jira/browse/SPARK-62010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon reassigned SPARK-62010:
------------------------------------

    Assignee: Hyukjin Kwon

> Make the maps of the Rust protobuf runtime merge their arguments, fail an 
> iteration that they change, and order integer keys as upb's maps do
> ---------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-62010
>                 URL: https://issues.apache.org/jira/browse/SPARK-62010
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: 5.0.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>
> Type: Bug    Component: Connect
> Problem
> pyspark.sql.connect.proto is the drop-in's protobuf runtime in Rust 
> (crates/pyspark-rs/src/pbmsg),
> which follows upb, the implementation of a protobuf install that 
> pyspark-client 4.3.0 uses. Its map
> containers differ from upb's in four ways:
> 1. MapContainer.MergeFrom of a map of the runtime of other key or value types 
> took its entries
>    as they were, without converting them. So a map held keys and values of 
> other types:
>    sql.args.MergeFrom(sql.named_arguments) put an Expression into the map of 
> Literals, which
>    serialized as a Literal of other fields, and options.MergeFrom(sql.args) 
> put a message into a
>    map of strings. A map of int32 keys merged from a map of uint32 keys held 
> two entries of the
>    key 4, and one of bool keys held True. upb converts each entry, with its 
> errors.
> 2. MapContainer.MergeFrom of any other argument but a dict iterated it as 
> (key, value) pairs. upb
>    takes a mapping by its items (protobuf's maps, MappingProxyType, UserDict, 
> an object with keys),
>    as MutableMapping.update does. So the map of a message of protobuf's class 
> of the type, which a
>    plugin's module that keeps protobuf's classes gives (Delta Lake's 
> base_pb2), merged
>    {"ab": "zz", "cd": "yy"} as {"a": "b", "c": "d"}, and failed for keys of 
> other lengths. Of a
>    map of messages, upb reads each key's value and copies it into the key's 
> message (CopyFrom), or
>    sets its fields of a dict, and raises "Argument for field ... is not 
> iterable".
>    upb's MergeFrom is a dict check, then upb's InitMapAttributes, which 
> SPARK-61992 gave the
>    runtime's constructor: it also copies a message into the message that the 
> map has for its
>    key, so that a message read before (x = map[key]) has the new fields, 
> where the runtime
>    replaced it. And map.update(None) did nothing, where upb's 
> MutableMapping.update raises the
>    TypeError of iterating None.
> 3. A map changed during its iteration did not fail the iteration. upb raises
>    "RuntimeError: Map modified during iteration." at the next step, once a 
> method of the map that
>    may change its keys ran. The runtime iterated a copy of the keys, so 
> items() and values() read
>    the value of a key that the loop had removed, which inserted the key 
> again, of the default
>    value, into the message.
> 4. The keys of a signed integer type were ordered as signed numbers in the 
> serialization and in
>    str(). upb orders them as unsigned numbers (0, 1, 3, 100, -5, -1), so the 
> bytes of
>    SerializeToString(deterministic=True) differed. The protos of Spark 
> Connect have maps of string
>    keys only, so this affects the maps of plugins' protos.
> docs/compatibility.md also:
> - described upb's limit of 100 nested messages as a limit of its parse only;
> - described the serialization order of maps and the outcome of a truncated 
> varint imprecisely;
> - said that no message that the runtime makes of a parse or a copy is tracked 
> by the garbage
>   collector, where the top-level messages are;
> - gave the server's limit of the depth of a plan as 100 levels, where it is 
> 1,024 nested messages
>   (spark.connect.grpc.marshallerRecursionLimit), from about 510 nested 
> expressions.
> Repro (master 9cf73197; scan is a message of a plugin's module of the 
> runtime's classes, whose
> field table is of a type of base_pb2, a plugin's module that keeps protobuf's 
> classes, as Delta
> Lake's DeltaScan holds a DeltaTable)
>   >>> s = pb.SQL(); s.args.MergeFrom(pb.SQL(named_arguments={"k": 
> {"unresolved_attribute": {"unparsed_identifier": "c"}}}).named_arguments)
>   >>> pb.SQL.FromString(s.SerializeToString()).args["k"]
>   binary: "\n\001c"                   (the Expression's bytes, read as a 
> Literal)
>   >>> d = pb.Read.DataSource(); 
> d.options.MergeFrom(types.MappingProxyType({"ab": "zz"})); dict(d.options)
>   {'a': 'b'}
>   >>> scan.table.options.MergeFrom(base_pb2.Table(options={"ab": 
> "zz"}).options); dict(scan.table.options)
>   {'a': 'b'}
>   >>> d = pb.Read.DataSource(options={"k0": "v0", "k1": "v1", "k2": "v2"})
>   >>> for k, v in d.options.items():
>   ...     if k == "k0": del d.options["k1"]
>   >>> dict(d.options)
>   {'k0': 'v0', 'k2': 'v2', 'k1': ''}
> Expected (as with pyspark-client 4.3.0)
>   The first MergeFrom raises TypeError: Parameter to CopyFrom() must be 
> instance of same class:
>   expected <class '...Expression.Literal'> got <class '...Expression'>.
>   The two other MergeFrom calls give {'ab': 'zz'}.
>   The loop raises RuntimeError: Map modified during iteration., and k1 stays 
> deleted.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to