[
https://issues.apache.org/jira/browse/SPARK-62056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-62056.
----------------------------------
Fix Version/s: connect-rust-4.4.0
Resolution: Fixed
Issue resolved by pull request 873
[https://github.com/apache/spark-connect-rust/pull/873]
> Take the arguments of the methods of the maps and repeated containers of the
> Rust protobuf runtime as upb's, with upb's errors
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-62056
> URL: https://issues.apache.org/jira/browse/SPARK-62056
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.4.0
>
>
> Problem
> pyspark.sql.connect.proto is the drop-in's protobuf runtime in Rust
> (crates/pyspark-rs/src/pbmsg),
> which follows upb, the implementation of a protobuf install that
> pyspark-client 4.3.0 uses. The
> methods of its containers took their arguments as PyO3's defaults make them,
> with PyO3's errors,
> where upb's take them as their C functions and collections.abc's Python
> methods do:
> - upb's maps are of its C types ScalarMapContainer and MessageMapContainer.
> - MergeFrom, clear, GetEntryClass and get_or_create take positional
> arguments only, with
> CPython's errors naming the class: "ScalarMapContainer.MergeFrom() takes
> exactly one argument
> (0 given)", "... takes no keyword arguments".
> - setdefault takes no keyword arguments and counts its arguments as
> PyArg_UnpackTuple does. A
> map of messages refuses it whatever its arguments.
> - get parses its key and default as PyArg_ParseTupleAndKeywords("O|O") does.
> - pop, popitem, update, keys, values and items are MutableMapping's and
> Mapping's, in Python.
> - upb's repeated containers take positional arguments only:
> - append, extend, remove and MergeFrom take exactly one;
> - insert and pop parse theirs as PyArg_ParseTuple's "nO" and "|n";
> - __deepcopy__ takes no keyword arguments.
> The runtime's methods took keyword arguments, so these all succeeded where
> upb raises TypeError:
> MergeFrom(other=...), append(value=...), insert(key=..., value=...),
> pop(key=...),
> setdefault(key, default=...), __deepcopy__(memo=...). Their errors were
> PyO3's, and the maps'
> errors named MapContainer: "MapContainer.MergeFrom() missing 1 required
> positional argument:
> 'other'".
> Repro (master 80046f2f)
> >>> import pyspark.sql.connect.proto as pb
> >>> d = pb.Read.DataSource(); d.options.MergeFrom()
> TypeError: MapContainer.MergeFrom() missing 1 required positional argument:
> 'other'
> >>> d.options.MergeFrom(other=[("a", "b")]); dict(d.options)
> {'a': 'b'}
> >>> q = pb.StatApproxQuantile(); q.probabilities.append(value=0.5);
> list(q.probabilities)
> [0.5]
> >>> q.probabilities.insert(0)
> TypeError: RepeatedScalarContainer.insert() missing 1 required positional
> argument: 'value'
> Expected (as with pyspark-client 4.3.0)
> TypeError: ScalarMapContainer.MergeFrom() takes exactly one argument (0
> given)
> TypeError: ScalarMapContainer.MergeFrom() takes no keyword arguments
> TypeError: RepeatedScalarContainer.append() takes no keyword arguments
> TypeError: function takes exactly 2 arguments (1 given)
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]