Hyukjin Kwon created SPARK-62010:
------------------------------------
Summary: Make the maps of the Rust protobuf runtime merge their
arguments, fail an iteration that they change, and order integer keys as upb's
maps do
Key: SPARK-62010
URL: https://issues.apache.org/jira/browse/SPARK-62010
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
Type: Bug Component: Connect
Problem
pyspark.sql.connect.proto is the drop-in's protobuf runtime in Rust
(crates/pyspark-rs/src/pbmsg),
which follows upb, the implementation of a protobuf install that pyspark-client
4.3.0 uses. Its map
containers differ from upb's in four ways:
1. MapContainer.MergeFrom of a map of the runtime of other key or value types
took its entries
as they were, without converting them. So a map held keys and values of
other types:
sql.args.MergeFrom(sql.named_arguments) put an Expression into the map of
Literals, which
serialized as a Literal of other fields, and options.MergeFrom(sql.args) put
a message into a
map of strings. A map of int32 keys merged from a map of uint32 keys held
two entries of the
key 4, and one of bool keys held True. upb converts each entry, with its
errors.
2. MapContainer.MergeFrom of any other argument but a dict iterated it as (key,
value) pairs. upb
takes a mapping by its items (protobuf's maps, MappingProxyType, UserDict,
an object with keys),
as MutableMapping.update does. So the map of a message of protobuf's class
of the type, which a
plugin's module that keeps protobuf's classes gives (Delta Lake's base_pb2),
merged
{"ab": "zz", "cd": "yy"} as {"a": "b", "c": "d"}, and failed for keys of
other lengths. Of a
map of messages, upb reads each key's value and copies it into the key's
message (CopyFrom), or
sets its fields of a dict, and raises "Argument for field ... is not
iterable".
upb's MergeFrom is a dict check, then upb's InitMapAttributes, which
SPARK-61992 gave the
runtime's constructor: it also copies a message into the message that the
map has for its
key, so that a message read before (x = map[key]) has the new fields, where
the runtime
replaced it. And map.update(None) did nothing, where upb's
MutableMapping.update raises the
TypeError of iterating None.
3. A map changed during its iteration did not fail the iteration. upb raises
"RuntimeError: Map modified during iteration." at the next step, once a
method of the map that
may change its keys ran. The runtime iterated a copy of the keys, so items()
and values() read
the value of a key that the loop had removed, which inserted the key again,
of the default
value, into the message.
4. The keys of a signed integer type were ordered as signed numbers in the
serialization and in
str(). upb orders them as unsigned numbers (0, 1, 3, 100, -5, -1), so the
bytes of
SerializeToString(deterministic=True) differed. The protos of Spark Connect
have maps of string
keys only, so this affects the maps of plugins' protos.
docs/compatibility.md also:
- described upb's limit of 100 nested messages as a limit of its parse only;
- described the serialization order of maps and the outcome of a truncated
varint imprecisely;
- said that no message that the runtime makes of a parse or a copy is tracked
by the garbage
collector, where the top-level messages are;
- gave the server's limit of the depth of a plan as 100 levels, where it is
1,024 nested messages
(spark.connect.grpc.marshallerRecursionLimit), from about 510 nested
expressions.
Repro (master 9cf73197; scan is a message of a plugin's module of the runtime's
classes, whose
field table is of a type of base_pb2, a plugin's module that keeps protobuf's
classes, as Delta
Lake's DeltaScan holds a DeltaTable)
>>> s = pb.SQL(); s.args.MergeFrom(pb.SQL(named_arguments={"k":
{"unresolved_attribute": {"unparsed_identifier": "c"}}}).named_arguments)
>>> pb.SQL.FromString(s.SerializeToString()).args["k"]
binary: "\n\001c" (the Expression's bytes, read as a
Literal)
>>> d = pb.Read.DataSource();
d.options.MergeFrom(types.MappingProxyType({"ab": "zz"})); dict(d.options)
{'a': 'b'}
>>> scan.table.options.MergeFrom(base_pb2.Table(options={"ab":
"zz"}).options); dict(scan.table.options)
{'a': 'b'}
>>> d = pb.Read.DataSource(options={"k0": "v0", "k1": "v1", "k2": "v2"})
>>> for k, v in d.options.items():
... if k == "k0": del d.options["k1"]
>>> dict(d.options)
{'k0': 'v0', 'k2': 'v2', 'k1': ''}
Expected (as with pyspark-client 4.3.0)
The first MergeFrom raises TypeError: Parameter to CopyFrom() must be
instance of same class:
expected <class '...Expression.Literal'> got <class '...Expression'>.
The two other MergeFrom calls give {'ab': 'zz'}.
The loop raises RuntimeError: Map modified during iteration., and k1 stays
deleted.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]