asddfl created SPARK-60559:
------------------------------
Summary: pandas-on-Spark assign loses resolved attributes from
independently filtered Series
Key: SPARK-60559
URL: https://issues.apache.org/jira/browse/SPARK-60559
Project: Spark
Issue Type: Bug
Components: Pandas API on Spark, PySpark
Affects Versions: 4.2.0
Environment: Platform: Ubuntu 24.04 Linux-6.17.0-29-generic x86_64
Python: 3.11.17
pyspark 4.2.0
findspark 2.0.1
pandas 3.0.6
numpy 2.4.6
Reporter: asddfl
Comparing two independently filtered Series and assigning the comparison to
another filtered view of the same DataFrame fails in the pandas API on Spark.
The example has one row with the same index throughout, and
compute.ops_on_diff_frames is explicitly enabled. Pandas evaluates the
expression successfully.
{code:python}
import pandas as pd
from pyspark.sql import SparkSession
import pyspark.pandas as ps
t0 = pd.DataFrame({'c0': [1]})
print("Pandas:", flush=True)
result = t0[t0['c0'] != 0].assign(
c1=t0.loc[t0['c0'] != 0, 'c0'] < t0.loc[t0['c0'] != 0, 'c0']
)
print(result, flush=True)
spark = (
SparkSession.builder
.master("local[2]")
.config("spark.sql.ansi.enabled", "false")
.config("spark.sql.shuffle.partitions", "2")
.getOrCreate()
)
ps.set_option("compute.ops_on_diff_frames", True)
t0 = ps.DataFrame({'c0': [1]})
print("PySpark Pandas:", flush=True)
result = t0[t0['c0'] != 0].assign(
c1=t0.loc[t0['c0'] != 0, 'c0'] < t0.loc[t0['c0'] != 0, 'c0']
)
print(result, flush=True)
spark.stop()
{code}
{code:bash}
Pandas:
c0 c1
0 1 False
PySpark Pandas:
Traceback (most recent call last):
File "/test.py", line 30, in <module>
print(result, flush=True)
File "/opt/conda/envs/lib/python3.11/site-packages/pyspark/pandas/frame.py",
line 13997, in __repr__
pdf = cast("DataFrame",
self._get_or_create_repr_pandas_cache(max_display_count))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/lib/python3.11/site-packages/pyspark/pandas/frame.py",
line 13988, in _get_or_create_repr_pandas_cache
self, "_repr_pandas_cache", {n: self.head(n + 1)._to_internal_pandas()}
^^^^^^^^^^^^^^^^
File "/opt/conda/envs/lib/python3.11/site-packages/pyspark/pandas/frame.py",
line 6654, in head
sdf = self._internal.resolved_copy.spark_frame
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/lib/python3.11/site-packages/pyspark/pandas/utils.py",
line 608, in wrapped_lazy_property
setattr(self, attr_name, fn(self))
^^^^^^^^
File
"/opt/conda/envs/lib/python3.11/site-packages/pyspark/pandas/internal.py", line
1211, in resolved_copy
sdf = self.spark_frame.select(self.spark_columns + list(HIDDEN_COLUMNS))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/opt/conda/envs/lib/python3.11/site-packages/pyspark/sql/classic/dataframe.py",
line 1101, in select
jdf = self._jdf.select(self._jcols(*cols))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/lib/python3.11/site-packages/py4j/java_gateway.py",
line 1362, in __call__
return_value = get_return_value(
^^^^^^^^^^^^^^^^^
File
"/opt/conda/envs/lib/python3.11/site-packages/pyspark/errors/exceptions/captured.py",
line 257, in deco
raise converted from None
pyspark.errors.exceptions.captured.AnalysisException:
[MISSING_ATTRIBUTES.RESOLVED_ATTRIBUTE_MISSING_FROM_INPUT] Resolved
attribute(s) "__this_c0", "__that_0" missing from "__index_level_0__", "c0",
"__natural_order__" in operator !Project [__index_level_0__#0L, c0#1L, CASE
WHEN isnull(CASE WHEN isnull((__this_c0#19L < __that_0#22L)) THEN false ELSE
(__this_c0#19L < __that_0#22L) END) THEN false ELSE CASE WHEN
isnull((__this_c0#19L < __that_0#22L)) THEN false ELSE (__this_c0#19L <
__that_0#22L) END END AS c1#49, __natural_order__#9L]. SQLSTATE: XX000;
!Project [__index_level_0__#0L, c0#1L, CASE WHEN isnull(CASE WHEN
isnull((__this_c0#19L < __that_0#22L)) THEN false ELSE (__this_c0#19L <
__that_0#22L) END) THEN false ELSE CASE WHEN isnull((__this_c0#19L <
__that_0#22L)) THEN false ELSE (__this_c0#19L < __that_0#22L) END END AS c1#49,
__natural_order__#9L]
+- Project [__index_level_0__#0L, c0#1L, monotonically_increasing_id() AS
__natural_order__#9L]
+- Project [__index_level_0__#0L, c0#1L]
+- Filter CASE WHEN isnull(CASE WHEN isnull(NOT (c0#1L = cast(0 as
bigint))) THEN true ELSE NOT (c0#1L = cast(0 as bigint)) END) THEN false ELSE
cast(CASE WHEN isnull(NOT (c0#1L = cast(0 as bigint))) THEN true ELSE NOT
(c0#1L = cast(0 as bigint)) END as boolean) END
+- Project [__index_level_0__#0L, c0#1L, monotonically_increasing_id()
AS __natural_order__#2L]
+- LocalRelation [__index_level_0__#0L, c0#1L]
{code}
*Expected behavior*
Return a one-row DataFrame with c0 = 1 and c1 = False, matching Pandas.
*Actual behavior*
The assignment raises AnalysisException with condition
MISSING_ATTRIBUTES.RESOLVED_ATTRIBUTE_MISSING_FROM_INPUT. The message reports
that resolved attributes __this_c0 and __that_0 are missing from the input of
the projection.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
