asddfl created SPARK-60523:
------------------------------
Summary: pandas-on-Spark np.gcd with a left-hand scalar raises
AttributeError
Key: SPARK-60523
URL: https://issues.apache.org/jira/browse/SPARK-60523
Project: Spark
Issue Type: Bug
Components: Pandas API on Spark, PySpark
Affects Versions: 4.2.0
Environment: Platform: Ubuntu 24.04 Linux-6.17.0-29-generic x86_64
Python: 3.11.17
pyspark 4.2.0
findspark 2.0.1
pandas 3.0.6
numpy 2.4.6
Reporter: asddfl
When using the pandas API on Spark (pyspark.pandas), calling np.gcd with a
Python integer as its left operand and an integer Series as its right operand
raises AttributeError. The equivalent Pandas/NumPy call succeeds for the same
data.
The example contains one integer value and no missing values. The failure
occurs in NumPy ufunc dispatch before a result can be produced.
{code:python}
# Minimized from err_bug1.py: A left-hand scalar is incorrectly treated as a
distributed Series.
import pandas as pd
import numpy as np
from pyspark.sql import SparkSession
import pyspark.pandas as ps
t0 = pd.DataFrame({'c0': [1]})
print("Pandas:", flush=True)
result = np.gcd(2, t0['c0'])
print(result, flush=True)
spark = (
SparkSession.builder
.master("local[2]")
.config("spark.sql.ansi.enabled", "false")
.config("spark.sql.shuffle.partitions", "2")
.getOrCreate()
)
t0 = ps.DataFrame({'c0': [1]})
print("PySpark Pandas:", flush=True)
result = np.gcd(2, t0['c0'])
print(result, flush=True)
spark.stop()
{code}
{code:bash}
Pandas:
0 1
Name: c0, dtype: int64
PySpark Pandas:
Traceback (most recent call last):
File "/test.py", line 25, in <module>
result = np.gcd(2, t0['c0'])
^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/lib/python3.11/site-packages/pyspark/pandas/base.py",
line 555, in __array_ufunc__
result = numpy_compat.maybe_dispatch_ufunc_to_spark_func(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/opt/conda/envs/lib/python3.11/site-packages/pyspark/pandas/numpy_compat.py",
line 226, in maybe_dispatch_ufunc_to_spark_func
return column_op(convert_arguments)(*inputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/lib/python3.11/site-packages/pyspark/pandas/base.py",
line 238, in wrapper
if all(not should_alignment_for_column_op(self, col) for col in cols):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/lib/python3.11/site-packages/pyspark/pandas/base.py",
line 238, in <genexpr>
if all(not should_alignment_for_column_op(self, col) for col in cols):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/lib/python3.11/site-packages/pyspark/pandas/base.py",
line 82, in should_alignment_for_column_op
return self._internal.spark_frame is not other._internal.spark_frame
^^^^^^^^^^^^^^
AttributeError: 'int' object has no attribute '_internal'
{code}
*Expected behavior*
np.gcd(2, a Series containing 1) should return an integer Series containing 1,
matching Pandas. A scalar left operand should not be treated as a distributed
Series.
*Actual behavior*
The pandas-on-Spark call raises AttributeError: 'int' object has no attribute
'_internal'. The traceback passes through maybe_dispatch_ufunc_to_spark_func
and column_op, then should_alignment_for_column_op accesses self._internal when
self is the Python integer.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
