[ 
https://issues.apache.org/jira/browse/SPARK-60648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-60648:
-----------------------------------
    Labels: pull-request-available  (was: )

> pandas-on-Spark comparison between a boolean Series and a float Series raises 
> a SQL type error
> ----------------------------------------------------------------------------------------------
>
>                 Key: SPARK-60648
>                 URL: https://issues.apache.org/jira/browse/SPARK-60648
>             Project: Spark
>          Issue Type: Bug
>          Components: Pandas API on Spark, PySpark
>    Affects Versions: 4.2.0
>         Environment: Platform: Ubuntu 24.04 Linux-6.17.0-29-generic x86_64
> Python: 3.11.17
> pyspark 4.2.0
> findspark 2.0.1
> pandas 3.0.6
> numpy 2.4.6
>            Reporter: asddfl
>            Priority: Critical
>              Labels: pull-request-available
>
> Comparing a boolean Series derived from c0 != 0 with the original float 
> Series via .gt() succeeds in Pandas but fails in the pandas API on Spark. The 
> generated SQL compares BOOLEAN and DOUBLE operands without compatible numeric 
> promotion.
> {code:python}
> import pandas as pd
> from pyspark.sql import SparkSession
> import pyspark.pandas as ps
> t0 = pd.DataFrame({'c0': [1.0]})
> print("Pandas:", flush=True)
> result = (t0['c0'] != 0).gt(t0['c0'])
> print(result, flush=True)
> spark = (
>     SparkSession.builder
>     .master("local[2]")
>     .config("spark.sql.ansi.enabled", "false")
>     .config("spark.sql.shuffle.partitions", "2")
>     .getOrCreate()
> )
> t0 = ps.DataFrame({'c0': [1.0]})
> print("PySpark Pandas:", flush=True)
> result = (t0['c0'] != 0).gt(t0['c0'])
> print(result, flush=True)
> spark.stop()
> {code}
> {code:bash}
> Pandas:
> 0    False
> Name: c0, dtype: bool
> PySpark Pandas:
> Traceback (most recent call last):
>   File "/test.py", line 24, in <module>
>     result = (t0['c0'] != 0).gt(t0['c0'])
>              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File 
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/pandas/series.py", line 
> 815, in gt
>     return self > other
>            ^^^^^^^^^^^^
>   File "/opt/conda/envs/lib/python3.11/site-packages/pyspark/pandas/base.py", 
> line 507, in __gt__
>     return self._dtype_op.gt(self, other)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File 
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/pandas/data_type_ops/boolean_ops.py",
>  line 395, in gt
>     return column_op(PySparkColumn.__gt__)(left, right)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/opt/conda/envs/lib/python3.11/site-packages/pyspark/pandas/base.py", 
> line 246, in wrapper
>     self._internal.spark_frame.select(scol).schema[0],
>     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File 
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/sql/classic/dataframe.py",
>  line 1101, in select
>     jdf = self._jdf.select(self._jcols(*cols))
>           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/opt/conda/envs/lib/python3.11/site-packages/py4j/java_gateway.py", 
> line 1362, in __call__
>     return_value = get_return_value(
>                    ^^^^^^^^^^^^^^^^^
>   File 
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/errors/exceptions/captured.py",
>  line 257, in deco
>     raise converted from None
> pyspark.errors.exceptions.captured.AnalysisException: 
> [DATATYPE_MISMATCH.BINARY_OP_DIFF_TYPES] Cannot resolve "(CASE WHEN ((NOT (c0 
> = 0)) IS NULL) THEN true ELSE (NOT (c0 = 0)) END > c0)" due to data type 
> mismatch: the left and right operands of the binary operator have 
> incompatible types ("BOOLEAN" and "DOUBLE"). SQLSTATE: 42K09;
> 'Project [unresolvedalias((CASE WHEN isnull(NOT (c0#1 = cast(0 as double))) 
> THEN true ELSE NOT (c0#1 = cast(0 as double)) END > c0#1))]
> +- Project [__index_level_0__#0L, c0#1, monotonically_increasing_id() AS 
> __natural_order__#2L]
>    +- LocalRelation [__index_level_0__#0L, c0#1]
> {code}
> *Expected behavior*
> For c0 = 1.0, return a boolean Series containing False: the comparison is 
> True > 1.0, which is false in Pandas.
> *Actual behavior*
> The call raises AnalysisException with condition 
> DATATYPE_MISMATCH.BINARY_OP_DIFF_TYPES and SQLSTATE 42K09. The error 
> identifies BOOLEAN and DOUBLE as incompatible operand types.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to