[
https://issues.apache.org/jira/browse/SPARK-59117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59117:
-----------------------------------
Labels: pull-request-available (was: )
> Set __module__ on the Rust drop-in's PyO3 classes to the reference pyspark
> path
> -------------------------------------------------------------------------------
>
> Key: SPARK-59117
> URL: https://issues.apache.org/jira/browse/SPARK-59117
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.2.0
> Reporter: L. C. Hsieh
> Assignee: L. C. Hsieh
> Priority: Major
> Labels: pull-request-available
>
> The PyO3 classes exposed by the Rust drop-in (pyspark._pyspark) all report
> __module__ == "builtins", because that is PyO3's default when #[pyclass]
> carries no
> `module` argument. None of the 99 #[pyclass] attributes in
> crates/pyspark-rs/src set
> it. The reference client reports the module that defines each class
> (pyspark.sql.types.IntegerType, etc.), so this is a behavioral divergence
> from
> PySpark, not just a cosmetic one.
> Two consequences:
> 1. The class ITSELF is unpicklable. pickle serializes a class BY REFERENCE
> — it
> writes __module__ + __qualname__ and re-imports on load — so the lookup
> lands in
> `builtins` and fails:
> >>> import pickle
> >>> from pyspark.sql.types import IntegerType
> >>> pickle.dumps(IntegerType)
> PicklingError: Can't pickle <class 'builtins.IntegerType'>:
> attribute lookup IntegerType on builtins failed
> Real pyspark pickles these classes fine. Reproduced for DataFrame,
> Column, and
> every DataType class.
> 2. Error messages, reprs, and any __module__-based dispatch show
> "builtins.DataFrame" instead of the pyspark path. That is user-visible
> and
> undercuts the drop-in premise.
> Pickling of type INSTANCES already works, but only because each DataType
> hand-writes
> a __reduce__ that routes through
> pyspark.sql.types._parse_datatype_json_string
> (crates/pyspark-rs/src/types.rs). In other words the cost of the wrong
> __module__ is
> already being paid — worked around per class rather than fixed at the root.
> Those
> __reduce__ impls must be KEPT regardless: they carry the type's parameters
> (DecimalType(12,3), ArrayType(...)), which a by-reference pickle of the
> bare class
> would not restore.
> Fix: add an explicit `module = "..."` to every #[pyclass] (including the
> abstract_type! macro that generates the 8 intermediate type bases), and set
> the
> module on create_exception! for SkipRestOfInputTableException.
> The class-to-module mapping is derived from where the drop-in skin
> (python/pyspark/**) already re-exports each class, cross-checked against
> Apache Spark
> v4.2.0 (commit 32f72996011 — the same commit pinned in
> crates/spark-connect-proto/proto/PROTO_SHA.txt).
> Two mapping notes worth recording:
> * DataFrame and Column map to pyspark.sql.connect.*, NOT pyspark.sql.*.
> Upstream's
> pyspark/sql/dataframe.py:78 and pyspark/sql/column.py:41 hold the
> ABSTRACT
> parent; the concrete class a Connect session hands the user is
> pyspark/sql/connect/dataframe.py:112 and .../connect/column.py:109,
> which is what
> type(df).__module__ reports on a real Connect session.
> * RustRpcError keeps __module__ == "_pyspark": it is an internal
> transport error
> with no upstream counterpart, so asserting a pyspark path for it would
> be false.
> This is the same "declared but never applied" class of gap as SPARK-59032
> (parsed
> gRPC keepalive never applied to the channel) and SPARK-59037 (the 128 MiB
> max message
> size declared but unused): the intent existed, but the wiring that makes it
> take
> effect was missing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
