[
https://issues.apache.org/jira/browse/SPARK-60648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60648:
-----------------------------------
Labels: pull-request-available (was: )
> pandas-on-Spark comparison between a boolean Series and a float Series raises
> a SQL type error
> ----------------------------------------------------------------------------------------------
>
> Key: SPARK-60648
> URL: https://issues.apache.org/jira/browse/SPARK-60648
> Project: Spark
> Issue Type: Bug
> Components: Pandas API on Spark, PySpark
> Affects Versions: 4.2.0
> Environment: Platform: Ubuntu 24.04 Linux-6.17.0-29-generic x86_64
> Python: 3.11.17
> pyspark 4.2.0
> findspark 2.0.1
> pandas 3.0.6
> numpy 2.4.6
> Reporter: asddfl
> Priority: Critical
> Labels: pull-request-available
>
> Comparing a boolean Series derived from c0 != 0 with the original float
> Series via .gt() succeeds in Pandas but fails in the pandas API on Spark. The
> generated SQL compares BOOLEAN and DOUBLE operands without compatible numeric
> promotion.
> {code:python}
> import pandas as pd
> from pyspark.sql import SparkSession
> import pyspark.pandas as ps
> t0 = pd.DataFrame({'c0': [1.0]})
> print("Pandas:", flush=True)
> result = (t0['c0'] != 0).gt(t0['c0'])
> print(result, flush=True)
> spark = (
> SparkSession.builder
> .master("local[2]")
> .config("spark.sql.ansi.enabled", "false")
> .config("spark.sql.shuffle.partitions", "2")
> .getOrCreate()
> )
> t0 = ps.DataFrame({'c0': [1.0]})
> print("PySpark Pandas:", flush=True)
> result = (t0['c0'] != 0).gt(t0['c0'])
> print(result, flush=True)
> spark.stop()
> {code}
> {code:bash}
> Pandas:
> 0 False
> Name: c0, dtype: bool
> PySpark Pandas:
> Traceback (most recent call last):
> File "/test.py", line 24, in <module>
> result = (t0['c0'] != 0).gt(t0['c0'])
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/pandas/series.py", line
> 815, in gt
> return self > other
> ^^^^^^^^^^^^
> File "/opt/conda/envs/lib/python3.11/site-packages/pyspark/pandas/base.py",
> line 507, in __gt__
> return self._dtype_op.gt(self, other)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/pandas/data_type_ops/boolean_ops.py",
> line 395, in gt
> return column_op(PySparkColumn.__gt__)(left, right)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/opt/conda/envs/lib/python3.11/site-packages/pyspark/pandas/base.py",
> line 246, in wrapper
> self._internal.spark_frame.select(scol).schema[0],
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/sql/classic/dataframe.py",
> line 1101, in select
> jdf = self._jdf.select(self._jcols(*cols))
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/opt/conda/envs/lib/python3.11/site-packages/py4j/java_gateway.py",
> line 1362, in __call__
> return_value = get_return_value(
> ^^^^^^^^^^^^^^^^^
> File
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/errors/exceptions/captured.py",
> line 257, in deco
> raise converted from None
> pyspark.errors.exceptions.captured.AnalysisException:
> [DATATYPE_MISMATCH.BINARY_OP_DIFF_TYPES] Cannot resolve "(CASE WHEN ((NOT (c0
> = 0)) IS NULL) THEN true ELSE (NOT (c0 = 0)) END > c0)" due to data type
> mismatch: the left and right operands of the binary operator have
> incompatible types ("BOOLEAN" and "DOUBLE"). SQLSTATE: 42K09;
> 'Project [unresolvedalias((CASE WHEN isnull(NOT (c0#1 = cast(0 as double)))
> THEN true ELSE NOT (c0#1 = cast(0 as double)) END > c0#1))]
> +- Project [__index_level_0__#0L, c0#1, monotonically_increasing_id() AS
> __natural_order__#2L]
> +- LocalRelation [__index_level_0__#0L, c0#1]
> {code}
> *Expected behavior*
> For c0 = 1.0, return a boolean Series containing False: the comparison is
> True > 1.0, which is false in Pandas.
> *Actual behavior*
> The call raises AnalysisException with condition
> DATATYPE_MISMATCH.BINARY_OP_DIFF_TYPES and SQLSTATE 42K09. The error
> identifies BOOLEAN and DOUBLE as incompatible operand types.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]