asddfl created SPARK-60649:
------------------------------
Summary: pandas-on-Spark GroupBy.apply callback is missing the
group name attribute
Key: SPARK-60649
URL: https://issues.apache.org/jira/browse/SPARK-60649
Project: Spark
Issue Type: Bug
Components: Pandas API on Spark, PySpark
Affects Versions: 4.2.0
Environment: Platform: Ubuntu 24.04 Linux-6.17.0-29-generic x86_64
Python: 3.11.17
pyspark 4.2.0
findspark 2.0.1
pandas 3.0.6
numpy 2.4.6
Reporter: asddfl
A GroupBy.apply callback that reads the group's name attribute succeeds in
Pandas but raises AttributeError in the pandas API on Spark. The callback
returns a native pandas.Series on both sides.
{code:python}
import pandas as pd
from pyspark.sql import SparkSession
import pyspark.pandas as ps
t0 = pd.DataFrame({'c0': [1]})
print("Pandas:", flush=True)
result = t0.groupby('c0').apply(lambda t1: pd.Series({'c1': t1.name}))
print(result, flush=True)
spark = (
SparkSession.builder
.master("local[2]")
.config("spark.sql.ansi.enabled", "false")
.config("spark.sql.shuffle.partitions", "2")
.getOrCreate()
)
t0 = ps.DataFrame({'c0': [1]})
print("PySpark Pandas:", flush=True)
result = t0.groupby('c0').apply(lambda t1: pd.Series({'c1': t1.name}))
print(result, flush=True)
spark.stop()
{code}
{code:bash}
Pandas:
c1
c0
1 1
PySpark Pandas:
Traceback (most recent call last):
File "/test.py", line 24, in <module>
result = t0.groupby('c0').apply(lambda t1: pd.Series({'c1': t1.name}))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/opt/conda/envs/lib/python3.11/site-packages/pyspark/pandas/groupby.py", line
2061, in apply
pser_or_pdf = grouped.apply(pandas_apply, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/opt/conda/envs/lib/python3.11/site-packages/pandas/core/groupby/groupby.py",
line 1650, in apply
return self._python_apply_general(f, self._obj_with_exclusions)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/opt/conda/envs/lib/python3.11/site-packages/pandas/core/groupby/groupby.py",
line 1687, in _python_apply_general
values, mutated = self._grouper.apply_groupwise(f, data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/opt/conda/envs/lib/python3.11/site-packages/pandas/core/groupby/ops.py", line
1035, in apply_groupwise
res = f(group)
^^^^^^^^
File
"/opt/conda/envs/lib/python3.11/site-packages/pyspark/pandas/groupby.py", line
2038, in pandas_apply
return f(pdf.drop(groupkey_names, axis=1), *a, **k)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/artifact/TDiFf-plus-bughub/pandas/pyspark/err_bug10_min.py", line 24,
in <lambda>
result = t0.groupby('c0').apply(lambda t1: pd.Series({'c1': t1.name}))
^^^^^^^
File "/opt/conda/envs/lib/python3.11/site-packages/pandas/core/generic.py",
line 6194, in __getattr__
return object.__getattribute__(self, name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'DataFrame' object has no attribute 'name'. Did you mean:
'rename'?
{code}
*Expected behavior*
Return one row indexed by the c0 group key 1, with c1 = 1. The callback should
be able to read the group key through t1.name as it does in Pandas.
*Actual behavior*
The callback raises AttributeError: 'DataFrame' object has no attribute 'name'
while evaluating t1.name, before the returned Series can be constructed.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
