Hyukjin Kwon created SPARK-62056:
------------------------------------
Summary: Take the arguments of the methods of the maps and
repeated containers of the Rust protobuf runtime as upb's, with upb's errors
Key: SPARK-62056
URL: https://issues.apache.org/jira/browse/SPARK-62056
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
Problem
pyspark.sql.connect.proto is the drop-in's protobuf runtime in Rust
(crates/pyspark-rs/src/pbmsg),
which follows upb, the implementation of a protobuf install that pyspark-client
4.3.0 uses. The
methods of its containers took their arguments as PyO3's defaults make them,
with PyO3's errors,
where upb's take them as their C functions and collections.abc's Python methods
do:
- upb's maps are of its C types ScalarMapContainer and MessageMapContainer.
- MergeFrom, clear, GetEntryClass and get_or_create take positional arguments
only, with
CPython's errors naming the class: "ScalarMapContainer.MergeFrom() takes
exactly one argument
(0 given)", "... takes no keyword arguments".
- setdefault takes no keyword arguments and counts its arguments as
PyArg_UnpackTuple does. A
map of messages refuses it whatever its arguments.
- get parses its key and default as PyArg_ParseTupleAndKeywords("O|O") does.
- pop, popitem, update, keys, values and items are MutableMapping's and
Mapping's, in Python.
- upb's repeated containers take positional arguments only:
- append, extend, remove and MergeFrom take exactly one;
- insert and pop parse theirs as PyArg_ParseTuple's "nO" and "|n";
- __deepcopy__ takes no keyword arguments.
The runtime's methods took keyword arguments, so these all succeeded where upb
raises TypeError:
MergeFrom(other=...), append(value=...), insert(key=..., value=...),
pop(key=...),
setdefault(key, default=...), __deepcopy__(memo=...). Their errors were PyO3's,
and the maps'
errors named MapContainer: "MapContainer.MergeFrom() missing 1 required
positional argument:
'other'".
Repro (master 80046f2f)
>>> import pyspark.sql.connect.proto as pb
>>> d = pb.Read.DataSource(); d.options.MergeFrom()
TypeError: MapContainer.MergeFrom() missing 1 required positional argument:
'other'
>>> d.options.MergeFrom(other=[("a", "b")]); dict(d.options)
{'a': 'b'}
>>> q = pb.StatApproxQuantile(); q.probabilities.append(value=0.5);
list(q.probabilities)
[0.5]
>>> q.probabilities.insert(0)
TypeError: RepeatedScalarContainer.insert() missing 1 required positional
argument: 'value'
Expected (as with pyspark-client 4.3.0)
TypeError: ScalarMapContainer.MergeFrom() takes exactly one argument (0 given)
TypeError: ScalarMapContainer.MergeFrom() takes no keyword arguments
TypeError: RepeatedScalarContainer.append() takes no keyword arguments
TypeError: function takes exactly 2 arguments (1 given)
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]