This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 7b0488d4 [SPARK-61170][CONNECT][PYTHON] Add the Python 2 operators
__div__, __rdiv__ and __nonzero__, and __getnewargs__, to Column in the Python
client, as PySpark has them
7b0488d4 is described below
commit 7b0488d4c8f1a16a5d877ff66e0af5bd31f445f7
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 17:26:40 2026 +0900
[SPARK-61170][CONNECT][PYTHON] Add the Python 2 operators __div__, __rdiv__
and __nonzero__, and __getnewargs__, to Column in the Python client, as PySpark
has them
### What changes were proposed in this pull request?
[SPARK-61170](https://issues.apache.org/jira/browse/SPARK-61170): PySpark's
Column, also of Spark Connect, has the operators of the names of Python 2, and
`__getnewargs__`. The client's Column now has the same four methods, as
PySpark's:
- `__div__` and `__rdiv__` divide as `__truediv__` and `__rtruediv__`, of
the origins of their own names. pandas-on-Spark calls `__div__`: its
`rfloordiv` does `F.lit(np.inf).__div__(right)`.
- `__nonzero__` raises `CANNOT_CONVERT_COLUMN_INTO_BOOL`, as `__bool__`.
- `__getnewargs__` gives the expression of the column (`_expr`).
### Why are the changes needed?
The client's Column had none of them. Its `__getattr__` refuses a name that
starts with `__`, as PySpark's, so `F.lit(1).__div__(x)` raised:
```
PySparkAttributeError: [ATTRIBUTE_NOT_SUPPORTED] Attribute `__div__` is not
supported.
```
Apache Spark's `test_rfloordiv` of pandas-on-Spark's
`computation/test_parity_binary_ops.py` failed on this client.
### Does this PR introduce _any_ user-facing change?
Yes. Columns have the methods of PySpark 4.3.0's above.
### How was this patch tested?
- A new offline test checks:
- the reprs and the plan of `__div__` and `__rdiv__`, which are those of
`/`
- the error of `__nonzero__`
- the expression of `__getnewargs__`
It passes with official PySpark 4.3.0, and fails without this change.
- The reprs and the rows of `__div__` and `__rdiv__` against a local server
are those of official PySpark 4.3.0.
- The dunder methods of `Column` are those of official PySpark 4.3.0, but
for `__dict__` and `__weakref__` (SPARK-61171) and `__getattr__`, which PyO3
implements as a slot.
- The lib tests of both crates (597 passed), the build of all tests, also
with `wasm-udf`, the offline Python suites (2223 passed, 8 skipped), and the
live drop-in and Arrow suites (206 passed) and live cases (10 passed) against a
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #359 from HyukjinKwon/SPARK-61170-column-python2-operators.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/column.rs | 24 ++++++++++++++++++++++++
python/tests/test_dropin_offline.py | 18 ++++++++++++++++++
2 files changed, 42 insertions(+)
diff --git a/crates/pyspark-rs/src/column.rs b/crates/pyspark-rs/src/column.rs
index 2225c1c6..e948a22b 100644
--- a/crates/pyspark-rs/src/column.rs
+++ b/crates/pyspark-rs/src/column.rs
@@ -591,6 +591,14 @@ impl PyColumn {
Ok(PyColumn::new(self.column.clone().div(other_col)))
}
+ /// Division (/), of the name of Python 2, as PySpark's Column has it,
which pandas-on-Spark
+ /// calls: as `__truediv__`.
+ fn __div__(&self, py: Python<'_>, other: Py<PyAny>) -> PyResult<PyColumn> {
+ let _origin = crate::origin::with_origin("__div__");
+ let other_col = to_column(&other.bind(py))?;
+ Ok(PyColumn::new(self.column.clone().div(other_col)))
+ }
+
/// Modulo (%)
fn __mod__(&self, py: Python<'_>, other: Py<PyAny>) -> PyResult<PyColumn> {
let _origin = crate::origin::with_origin("__mod__");
@@ -721,6 +729,12 @@ impl PyColumn {
let o = to_column(&other.bind(py))?;
Ok(PyColumn::new(o.div(self.column.clone())))
}
+ /// Reverse division, of the name of Python 2, as PySpark's Column has it:
as `__rtruediv__`.
+ fn __rdiv__(&self, py: Python<'_>, other: Py<PyAny>) -> PyResult<PyColumn>
{
+ let _origin = crate::origin::with_origin("__rdiv__");
+ let o = to_column(&other.bind(py))?;
+ Ok(PyColumn::new(o.div(self.column.clone())))
+ }
fn __rmod__(&self, py: Python<'_>, other: Py<PyAny>) -> PyResult<PyColumn>
{
let _origin = crate::origin::with_origin("__rmod__");
let o = to_column(&other.bind(py))?;
@@ -793,6 +807,16 @@ impl PyColumn {
))
}
+ /// `__bool__` of the name of Python 2, as PySpark's Column has it.
+ fn __nonzero__(&self, py: Python<'_>) -> PyResult<bool> {
+ self.__bool__(py)
+ }
+
+ /// The arguments of `__new__` of this column, as PySpark Connect's: its
expression.
+ fn __getnewargs__(&self) -> (PyColumnExpression,) {
+ (self._expr(),)
+ }
+
// --- named Column methods (PySpark camelCase) ---
fn between(
&self,
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index c955ada6..799cfc4f 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -4250,6 +4250,24 @@ def test_udtf_variants():
assert SimpleUDTF.deterministic is False
+def test_columns_have_the_python_2_operators_of_pyspark():
+ # As PySpark's Column: __div__ and __rdiv__ divide, as __truediv__ and
__rtruediv__, which
+ # pandas-on-Spark calls, __nonzero__ is __bool__, and __getnewargs__ gives
the expression.
+ from pyspark.errors import PySparkValueError
+ from pyspark.sql import functions as F
+
+ column = F.col("a")
+ assert repr(column.__div__(2)) == repr(column / 2) == "Column<'/(a, 2)'>"
+ assert repr(column.__rdiv__(2)) == repr(2 / column) == "Column<'/(2, a)'>"
+ plan = column.__div__(2).to_plan(None)
+ assert plan.unresolved_function.function_name == "/"
+ assert [a.literal.integer for a in plan.unresolved_function.arguments[1:]]
== [2]
+ with pytest.raises(PySparkValueError) as error:
+ column.__nonzero__()
+ assert error.value.getCondition() == "CANNOT_CONVERT_COLUMN_INTO_BOOL"
+ assert [repr(e) for e in (column + 1).__getnewargs__()] == ["+(a, 1)"]
+
+
def test_column_with_alias_chain():
"""Test chaining column methods."""
from pyspark.sql import functions as F
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]