[
https://issues.apache.org/jira/browse/SPARK-60559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
asddfl updated SPARK-60559:
---------------------------
Priority: Critical (was: Major)
> pandas-on-Spark assign loses resolved attributes from independently filtered
> Series
> -----------------------------------------------------------------------------------
>
> Key: SPARK-60559
> URL: https://issues.apache.org/jira/browse/SPARK-60559
> Project: Spark
> Issue Type: Bug
> Components: Pandas API on Spark, PySpark
> Affects Versions: 4.2.0
> Environment: Platform: Ubuntu 24.04 Linux-6.17.0-29-generic x86_64
> Python: 3.11.17
> pyspark 4.2.0
> findspark 2.0.1
> pandas 3.0.6
> numpy 2.4.6
> Reporter: asddfl
> Priority: Critical
>
> Comparing two independently filtered Series and assigning the comparison to
> another filtered view of the same DataFrame fails in the pandas API on Spark.
> The example has one row with the same index throughout, and
> compute.ops_on_diff_frames is explicitly enabled. Pandas evaluates the
> expression successfully.
> {code:python}
> import pandas as pd
> from pyspark.sql import SparkSession
> import pyspark.pandas as ps
> t0 = pd.DataFrame({'c0': [1]})
> print("Pandas:", flush=True)
> result = t0[t0['c0'] != 0].assign(
> c1=t0.loc[t0['c0'] != 0, 'c0'] < t0.loc[t0['c0'] != 0, 'c0']
> )
> print(result, flush=True)
> spark = (
> SparkSession.builder
> .master("local[2]")
> .config("spark.sql.ansi.enabled", "false")
> .config("spark.sql.shuffle.partitions", "2")
> .getOrCreate()
> )
> ps.set_option("compute.ops_on_diff_frames", True)
> t0 = ps.DataFrame({'c0': [1]})
> print("PySpark Pandas:", flush=True)
> result = t0[t0['c0'] != 0].assign(
> c1=t0.loc[t0['c0'] != 0, 'c0'] < t0.loc[t0['c0'] != 0, 'c0']
> )
> print(result, flush=True)
> spark.stop()
> {code}
> {code:bash}
> Pandas:
> c0 c1
> 0 1 False
> PySpark Pandas:
> Traceback (most recent call last):
> File "/test.py", line 30, in <module>
> print(result, flush=True)
> File
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/pandas/frame.py", line
> 13997, in __repr__
> pdf = cast("DataFrame",
> self._get_or_create_repr_pandas_cache(max_display_count))
>
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/pandas/frame.py", line
> 13988, in _get_or_create_repr_pandas_cache
> self, "_repr_pandas_cache", {n: self.head(n + 1)._to_internal_pandas()}
> ^^^^^^^^^^^^^^^^
> File
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/pandas/frame.py", line
> 6654, in head
> sdf = self._internal.resolved_copy.spark_frame
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/pandas/utils.py", line
> 608, in wrapped_lazy_property
> setattr(self, attr_name, fn(self))
> ^^^^^^^^
> File
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/pandas/internal.py",
> line 1211, in resolved_copy
> sdf = self.spark_frame.select(self.spark_columns + list(HIDDEN_COLUMNS))
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/sql/classic/dataframe.py",
> line 1101, in select
> jdf = self._jdf.select(self._jcols(*cols))
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/opt/conda/envs/lib/python3.11/site-packages/py4j/java_gateway.py",
> line 1362, in __call__
> return_value = get_return_value(
> ^^^^^^^^^^^^^^^^^
> File
> "/opt/conda/envs/lib/python3.11/site-packages/pyspark/errors/exceptions/captured.py",
> line 257, in deco
> raise converted from None
> pyspark.errors.exceptions.captured.AnalysisException:
> [MISSING_ATTRIBUTES.RESOLVED_ATTRIBUTE_MISSING_FROM_INPUT] Resolved
> attribute(s) "__this_c0", "__that_0" missing from "__index_level_0__", "c0",
> "__natural_order__" in operator !Project [__index_level_0__#0L, c0#1L, CASE
> WHEN isnull(CASE WHEN isnull((__this_c0#19L < __that_0#22L)) THEN false ELSE
> (__this_c0#19L < __that_0#22L) END) THEN false ELSE CASE WHEN
> isnull((__this_c0#19L < __that_0#22L)) THEN false ELSE (__this_c0#19L <
> __that_0#22L) END END AS c1#49, __natural_order__#9L]. SQLSTATE: XX000;
> !Project [__index_level_0__#0L, c0#1L, CASE WHEN isnull(CASE WHEN
> isnull((__this_c0#19L < __that_0#22L)) THEN false ELSE (__this_c0#19L <
> __that_0#22L) END) THEN false ELSE CASE WHEN isnull((__this_c0#19L <
> __that_0#22L)) THEN false ELSE (__this_c0#19L < __that_0#22L) END END AS
> c1#49, __natural_order__#9L]
> +- Project [__index_level_0__#0L, c0#1L, monotonically_increasing_id() AS
> __natural_order__#9L]
> +- Project [__index_level_0__#0L, c0#1L]
> +- Filter CASE WHEN isnull(CASE WHEN isnull(NOT (c0#1L = cast(0 as
> bigint))) THEN true ELSE NOT (c0#1L = cast(0 as bigint)) END) THEN false ELSE
> cast(CASE WHEN isnull(NOT (c0#1L = cast(0 as bigint))) THEN true ELSE NOT
> (c0#1L = cast(0 as bigint)) END as boolean) END
> +- Project [__index_level_0__#0L, c0#1L,
> monotonically_increasing_id() AS __natural_order__#2L]
> +- LocalRelation [__index_level_0__#0L, c0#1L]
> {code}
> *Expected behavior*
> Return a one-row DataFrame with c0 = 1 and c1 = False, matching Pandas.
> *Actual behavior*
> The assignment raises AnalysisException with condition
> MISSING_ATTRIBUTES.RESOLVED_ATTRIBUTE_MISSING_FROM_INPUT. The message reports
> that resolved attributes __this_c0 and __that_0 are missing from the input of
> the projection.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
