[
https://issues.apache.org/jira/browse/SPARK-43877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haejoon Lee updated SPARK-43877:
--------------------------------
Parent: (was: SPARK-42497)
Issue Type: Improvement (was: Sub-task)
> Fix behavior difference for compare binary functions.
> -----------------------------------------------------
>
> Key: SPARK-43877
> URL: https://issues.apache.org/jira/browse/SPARK-43877
> Project: Spark
> Issue Type: Improvement
> Components: Pandas API on Spark, PySpark
> Affects Versions: 3.5.0
> Reporter: Haejoon Lee
> Priority: Major
>
> In [https://github.com/apache/spark/pull/41362,] we add `result =
> result.fillna(False)` for filling the gap between pandas <> pandas API on
> Spark, but it should be internally fixed from Spark Connect side. Please
> refer to the reproducible code below:
>
> {code:java}
> import pandas as pd
> import pyspark.pandas as ps
> from pyspark.sql.utils import pyspark_column_op
> pser = pd.Series([None, None, None])
> psser = ps.from_pandas(pser)
> pyspark_column_op("__ge__")(psser, psser)
> # Wrong result:
> # 0 None
> # 1 None
> # 2 None
> # dtype: object
> # Expected result:
> pser > pser
> # 0 False
> # 1 False
> # 2 False
> dtype: bool{code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
