This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 7b0488d4 [SPARK-61170][CONNECT][PYTHON] Add the Python 2 operators 
__div__, __rdiv__ and __nonzero__, and __getnewargs__, to Column in the Python 
client, as PySpark has them
7b0488d4 is described below

commit 7b0488d4c8f1a16a5d877ff66e0af5bd31f445f7
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 17:26:40 2026 +0900

    [SPARK-61170][CONNECT][PYTHON] Add the Python 2 operators __div__, __rdiv__ 
and __nonzero__, and __getnewargs__, to Column in the Python client, as PySpark 
has them
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61170](https://issues.apache.org/jira/browse/SPARK-61170): PySpark's 
Column, also of Spark Connect, has the operators of the names of Python 2, and 
`__getnewargs__`. The client's Column now has the same four methods, as 
PySpark's:
    - `__div__` and `__rdiv__` divide as `__truediv__` and `__rtruediv__`, of 
the origins of their own names. pandas-on-Spark calls `__div__`: its 
`rfloordiv` does `F.lit(np.inf).__div__(right)`.
    - `__nonzero__` raises `CANNOT_CONVERT_COLUMN_INTO_BOOL`, as `__bool__`.
    - `__getnewargs__` gives the expression of the column (`_expr`).
    
    ### Why are the changes needed?
    
    The client's Column had none of them. Its `__getattr__` refuses a name that 
starts with `__`, as PySpark's, so `F.lit(1).__div__(x)` raised:
    
    ```
    PySparkAttributeError: [ATTRIBUTE_NOT_SUPPORTED] Attribute `__div__` is not 
supported.
    ```
    
    Apache Spark's `test_rfloordiv` of pandas-on-Spark's 
`computation/test_parity_binary_ops.py` failed on this client.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. Columns have the methods of PySpark 4.3.0's above.
    
    ### How was this patch tested?
    
    - A new offline test checks:
      - the reprs and the plan of `__div__` and `__rdiv__`, which are those of 
`/`
      - the error of `__nonzero__`
      - the expression of `__getnewargs__`
    
      It passes with official PySpark 4.3.0, and fails without this change.
    - The reprs and the rows of `__div__` and `__rdiv__` against a local server 
are those of official PySpark 4.3.0.
    - The dunder methods of `Column` are those of official PySpark 4.3.0, but 
for `__dict__` and `__weakref__` (SPARK-61171) and `__getattr__`, which PyO3 
implements as a slot.
    - The lib tests of both crates (597 passed), the build of all tests, also 
with `wasm-udf`, the offline Python suites (2223 passed, 8 skipped), and the 
live drop-in and Arrow suites (206 passed) and live cases (10 passed) against a 
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #359 from HyukjinKwon/SPARK-61170-column-python2-operators.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/column.rs     | 24 ++++++++++++++++++++++++
 python/tests/test_dropin_offline.py | 18 ++++++++++++++++++
 2 files changed, 42 insertions(+)

diff --git a/crates/pyspark-rs/src/column.rs b/crates/pyspark-rs/src/column.rs
index 2225c1c6..e948a22b 100644
--- a/crates/pyspark-rs/src/column.rs
+++ b/crates/pyspark-rs/src/column.rs
@@ -591,6 +591,14 @@ impl PyColumn {
         Ok(PyColumn::new(self.column.clone().div(other_col)))
     }
 
+    /// Division (/), of the name of Python 2, as PySpark's Column has it, 
which pandas-on-Spark
+    /// calls: as `__truediv__`.
+    fn __div__(&self, py: Python<'_>, other: Py<PyAny>) -> PyResult<PyColumn> {
+        let _origin = crate::origin::with_origin("__div__");
+        let other_col = to_column(&other.bind(py))?;
+        Ok(PyColumn::new(self.column.clone().div(other_col)))
+    }
+
     /// Modulo (%)
     fn __mod__(&self, py: Python<'_>, other: Py<PyAny>) -> PyResult<PyColumn> {
         let _origin = crate::origin::with_origin("__mod__");
@@ -721,6 +729,12 @@ impl PyColumn {
         let o = to_column(&other.bind(py))?;
         Ok(PyColumn::new(o.div(self.column.clone())))
     }
+    /// Reverse division, of the name of Python 2, as PySpark's Column has it: 
as `__rtruediv__`.
+    fn __rdiv__(&self, py: Python<'_>, other: Py<PyAny>) -> PyResult<PyColumn> 
{
+        let _origin = crate::origin::with_origin("__rdiv__");
+        let o = to_column(&other.bind(py))?;
+        Ok(PyColumn::new(o.div(self.column.clone())))
+    }
     fn __rmod__(&self, py: Python<'_>, other: Py<PyAny>) -> PyResult<PyColumn> 
{
         let _origin = crate::origin::with_origin("__rmod__");
         let o = to_column(&other.bind(py))?;
@@ -793,6 +807,16 @@ impl PyColumn {
         ))
     }
 
+    /// `__bool__` of the name of Python 2, as PySpark's Column has it.
+    fn __nonzero__(&self, py: Python<'_>) -> PyResult<bool> {
+        self.__bool__(py)
+    }
+
+    /// The arguments of `__new__` of this column, as PySpark Connect's: its 
expression.
+    fn __getnewargs__(&self) -> (PyColumnExpression,) {
+        (self._expr(),)
+    }
+
     // --- named Column methods (PySpark camelCase) ---
     fn between(
         &self,
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index c955ada6..799cfc4f 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -4250,6 +4250,24 @@ def test_udtf_variants():
     assert SimpleUDTF.deterministic is False
 
 
+def test_columns_have_the_python_2_operators_of_pyspark():
+    # As PySpark's Column: __div__ and __rdiv__ divide, as __truediv__ and 
__rtruediv__, which
+    # pandas-on-Spark calls, __nonzero__ is __bool__, and __getnewargs__ gives 
the expression.
+    from pyspark.errors import PySparkValueError
+    from pyspark.sql import functions as F
+
+    column = F.col("a")
+    assert repr(column.__div__(2)) == repr(column / 2) == "Column<'/(a, 2)'>"
+    assert repr(column.__rdiv__(2)) == repr(2 / column) == "Column<'/(2, a)'>"
+    plan = column.__div__(2).to_plan(None)
+    assert plan.unresolved_function.function_name == "/"
+    assert [a.literal.integer for a in plan.unresolved_function.arguments[1:]] 
== [2]
+    with pytest.raises(PySparkValueError) as error:
+        column.__nonzero__()
+    assert error.value.getCondition() == "CANNOT_CONVERT_COLUMN_INTO_BOOL"
+    assert [repr(e) for e in (column + 1).__getnewargs__()] == ["+(a, 1)"]
+
+
 def test_column_with_alias_chain():
     """Test chaining column methods."""
     from pyspark.sql import functions as F


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to