Hyukjin Kwon created SPARK-62056:
------------------------------------

             Summary: Take the arguments of the methods of the maps and 
repeated containers of the Rust protobuf runtime as upb's, with upb's errors
                 Key: SPARK-62056
                 URL: https://issues.apache.org/jira/browse/SPARK-62056
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: 5.0.0
            Reporter: Hyukjin Kwon


Problem
pyspark.sql.connect.proto is the drop-in's protobuf runtime in Rust 
(crates/pyspark-rs/src/pbmsg),
which follows upb, the implementation of a protobuf install that pyspark-client 
4.3.0 uses. The
methods of its containers took their arguments as PyO3's defaults make them, 
with PyO3's errors,
where upb's take them as their C functions and collections.abc's Python methods 
do:

- upb's maps are of its C types ScalarMapContainer and MessageMapContainer.
  - MergeFrom, clear, GetEntryClass and get_or_create take positional arguments 
only, with
    CPython's errors naming the class: "ScalarMapContainer.MergeFrom() takes 
exactly one argument
    (0 given)", "... takes no keyword arguments".
  - setdefault takes no keyword arguments and counts its arguments as 
PyArg_UnpackTuple does. A
    map of messages refuses it whatever its arguments.
  - get parses its key and default as PyArg_ParseTupleAndKeywords("O|O") does.
  - pop, popitem, update, keys, values and items are MutableMapping's and 
Mapping's, in Python.
- upb's repeated containers take positional arguments only:
  - append, extend, remove and MergeFrom take exactly one;
  - insert and pop parse theirs as PyArg_ParseTuple's "nO" and "|n";
  - __deepcopy__ takes no keyword arguments.

The runtime's methods took keyword arguments, so these all succeeded where upb 
raises TypeError:
MergeFrom(other=...), append(value=...), insert(key=..., value=...), 
pop(key=...),
setdefault(key, default=...), __deepcopy__(memo=...). Their errors were PyO3's, 
and the maps'
errors named MapContainer: "MapContainer.MergeFrom() missing 1 required 
positional argument:
'other'".

Repro (master 80046f2f)
  >>> import pyspark.sql.connect.proto as pb
  >>> d = pb.Read.DataSource(); d.options.MergeFrom()
  TypeError: MapContainer.MergeFrom() missing 1 required positional argument: 
'other'
  >>> d.options.MergeFrom(other=[("a", "b")]); dict(d.options)
  {'a': 'b'}
  >>> q = pb.StatApproxQuantile(); q.probabilities.append(value=0.5); 
list(q.probabilities)
  [0.5]
  >>> q.probabilities.insert(0)
  TypeError: RepeatedScalarContainer.insert() missing 1 required positional 
argument: 'value'

Expected (as with pyspark-client 4.3.0)
  TypeError: ScalarMapContainer.MergeFrom() takes exactly one argument (0 given)
  TypeError: ScalarMapContainer.MergeFrom() takes no keyword arguments
  TypeError: RepeatedScalarContainer.append() takes no keyword arguments
  TypeError: function takes exactly 2 arguments (1 given)




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to