Hyukjin Kwon created SPARK-62010:
------------------------------------

             Summary: Make the maps of the Rust protobuf runtime merge their 
arguments, fail an iteration that they change, and order integer keys as upb's 
maps do
                 Key: SPARK-62010
                 URL: https://issues.apache.org/jira/browse/SPARK-62010
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: 5.0.0
            Reporter: Hyukjin Kwon


Type: Bug    Component: Connect

Problem
pyspark.sql.connect.proto is the drop-in's protobuf runtime in Rust 
(crates/pyspark-rs/src/pbmsg),
which follows upb, the implementation of a protobuf install that pyspark-client 
4.3.0 uses. Its map
containers differ from upb's in four ways:

1. MapContainer.MergeFrom of a map of the runtime of other key or value types 
took its entries
   as they were, without converting them. So a map held keys and values of 
other types:
   sql.args.MergeFrom(sql.named_arguments) put an Expression into the map of 
Literals, which
   serialized as a Literal of other fields, and options.MergeFrom(sql.args) put 
a message into a
   map of strings. A map of int32 keys merged from a map of uint32 keys held 
two entries of the
   key 4, and one of bool keys held True. upb converts each entry, with its 
errors.
2. MapContainer.MergeFrom of any other argument but a dict iterated it as (key, 
value) pairs. upb
   takes a mapping by its items (protobuf's maps, MappingProxyType, UserDict, 
an object with keys),
   as MutableMapping.update does. So the map of a message of protobuf's class 
of the type, which a
   plugin's module that keeps protobuf's classes gives (Delta Lake's base_pb2), 
merged
   {"ab": "zz", "cd": "yy"} as {"a": "b", "c": "d"}, and failed for keys of 
other lengths. Of a
   map of messages, upb reads each key's value and copies it into the key's 
message (CopyFrom), or
   sets its fields of a dict, and raises "Argument for field ... is not 
iterable".
   upb's MergeFrom is a dict check, then upb's InitMapAttributes, which 
SPARK-61992 gave the
   runtime's constructor: it also copies a message into the message that the 
map has for its
   key, so that a message read before (x = map[key]) has the new fields, where 
the runtime
   replaced it. And map.update(None) did nothing, where upb's 
MutableMapping.update raises the
   TypeError of iterating None.
3. A map changed during its iteration did not fail the iteration. upb raises
   "RuntimeError: Map modified during iteration." at the next step, once a 
method of the map that
   may change its keys ran. The runtime iterated a copy of the keys, so items() 
and values() read
   the value of a key that the loop had removed, which inserted the key again, 
of the default
   value, into the message.
4. The keys of a signed integer type were ordered as signed numbers in the 
serialization and in
   str(). upb orders them as unsigned numbers (0, 1, 3, 100, -5, -1), so the 
bytes of
   SerializeToString(deterministic=True) differed. The protos of Spark Connect 
have maps of string
   keys only, so this affects the maps of plugins' protos.

docs/compatibility.md also:
- described upb's limit of 100 nested messages as a limit of its parse only;
- described the serialization order of maps and the outcome of a truncated 
varint imprecisely;
- said that no message that the runtime makes of a parse or a copy is tracked 
by the garbage
  collector, where the top-level messages are;
- gave the server's limit of the depth of a plan as 100 levels, where it is 
1,024 nested messages
  (spark.connect.grpc.marshallerRecursionLimit), from about 510 nested 
expressions.

Repro (master 9cf73197; scan is a message of a plugin's module of the runtime's 
classes, whose
field table is of a type of base_pb2, a plugin's module that keeps protobuf's 
classes, as Delta
Lake's DeltaScan holds a DeltaTable)
  >>> s = pb.SQL(); s.args.MergeFrom(pb.SQL(named_arguments={"k": 
{"unresolved_attribute": {"unparsed_identifier": "c"}}}).named_arguments)
  >>> pb.SQL.FromString(s.SerializeToString()).args["k"]
  binary: "\n\001c"                   (the Expression's bytes, read as a 
Literal)
  >>> d = pb.Read.DataSource(); 
d.options.MergeFrom(types.MappingProxyType({"ab": "zz"})); dict(d.options)
  {'a': 'b'}
  >>> scan.table.options.MergeFrom(base_pb2.Table(options={"ab": 
"zz"}).options); dict(scan.table.options)
  {'a': 'b'}
  >>> d = pb.Read.DataSource(options={"k0": "v0", "k1": "v1", "k2": "v2"})
  >>> for k, v in d.options.items():
  ...     if k == "k0": del d.options["k1"]
  >>> dict(d.options)
  {'k0': 'v0', 'k2': 'v2', 'k1': ''}

Expected (as with pyspark-client 4.3.0)
  The first MergeFrom raises TypeError: Parameter to CopyFrom() must be 
instance of same class:
  expected <class '...Expression.Literal'> got <class '...Expression'>.
  The two other MergeFrom calls give {'ab': 'zz'}.
  The loop raises RuntimeError: Map modified during iteration., and k1 stays 
deleted.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to