Hyukjin Kwon created SPARK-61170:
------------------------------------
Summary: Add the Python 2 operators __div__, __rdiv__ and
__nonzero__, and __getnewargs__, to Column in the Python client, as PySpark has
them
Key: SPARK-61170
URL: https://issues.apache.org/jira/browse/SPARK-61170
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
PySpark's Column (also of Spark Connect) has the operators of the names of
Python 2: __div__ and __rdiv__, which divide as __truediv__ and __rtruediv__,
and __nonzero__, which raises CANNOT_CONVERT_COLUMN_INTO_BOOL as __bool__; and
__getnewargs__, which gives its expression. pandas-on-Spark calls __div__: its
rfloordiv does F.lit(np.inf).__div__(right).
The Python client's Column has none of them. As its __getattr__ refuses a name
that starts with "__", F.lit(1).__div__(x) raises:
PySparkAttributeError: [ATTRIBUTE_NOT_SUPPORTED] Attribute `__div__` is not
supported.
Apache Spark's test_rfloordiv of pandas-on-Spark's
computation/test_parity_binary_ops.py fails on this client.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
