asddfl created SPARK-60560:
------------------------------
Summary: pandas-on-Spark log(0).notna() filter drops a row
retained by Pandas
Key: SPARK-60560
URL: https://issues.apache.org/jira/browse/SPARK-60560
Project: Spark
Issue Type: Bug
Components: Pandas API on Spark, PySpark
Affects Versions: 4.2.0
Environment: Platform: Ubuntu 24.04 Linux-6.17.0-29-generic x86_64
Python: 3.11.17
pyspark 4.2.0
findspark 2.0.1
pandas 3.0.6
numpy 2.4.6
Reporter: asddfl
Filtering a float64 DataFrame with np.log(c0).notna() produces different row
sets for c0 = 0.0. Pandas/NumPy log(0.0) is negative infinity, which is not a
missing value, so Pandas retains the row. The pandas-on-Spark query returns an
empty DataFrame.
This changes filtering behavior for a valid floating-point input even though
neither query raises an exception.
{code:python}
import pandas as pd
import numpy as np
from pyspark.sql import SparkSession
import pyspark.pandas as ps
t0 = pd.DataFrame({'c0': [0.0]})
print("Pandas:", flush=True)
result = t0[np.log(t0['c0']).notna()]
print(result, flush=True)
spark = (
SparkSession.builder
.master("local[2]")
.config("spark.sql.ansi.enabled", "false")
.config("spark.sql.shuffle.partitions", "2")
.getOrCreate()
)
ps.set_option("compute.ops_on_diff_frames", True)
t0 = ps.DataFrame({'c0': [0.0]})
print("PySpark Pandas:", flush=True)
result = t0[np.log(t0['c0']).notna()]
print(result, flush=True)
spark.stop()
{code}
{code:bash}
Pandas:
c0
0 0.0
PySpark Pandas:
Empty DataFrame
Columns: [c0]
Index: []
{code}
*Expected behavior*
Retain the row containing c0 = 0.0. The negative-infinity result of log(0.0)
should satisfy notna(), as it does in Pandas/NumPy.
*Actual behavior*
Pandas returns one row with c0 = 0.0. pandas-on-Spark returns an empty
DataFrame with the same c0 column.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
