[ https://issues.apache.org/jira/browse/SPARK-46189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ruifeng Zheng updated SPARK-46189: ---------------------------------- Fix Version/s: 3.4.3 > Various Pandas functions fail in interpreted mode > ------------------------------------------------- > > Key: SPARK-46189 > URL: https://issues.apache.org/jira/browse/SPARK-46189 > Project: Spark > Issue Type: Bug > Components: Pandas API on Spark, SQL > Affects Versions: 3.4.1, 3.5.0 > Reporter: Bruce Robbins > Assignee: Bruce Robbins > Priority: Major > Labels: pull-request-available > Fix For: 4.0.0, 3.5.1, 3.4.3 > > > Various Pandas functions ({{kurt}}, {{var}}, {{skew}}, {{cov}}, and > {{stddev}}) fail with an unboxing-related exception when run in interpreted > mode. > Here are some reproduction cases for pyspark interactive mode: > {noformat} > spark.sql("set spark.sql.codegen.wholeStage=false") > spark.sql("set spark.sql.codegen.factoryMode=NO_CODEGEN") > import numpy as np > import pandas as pd > import pyspark.pandas as ps > pser = pd.Series([1, 2, 3, 7, 9, 8], index=np.random.rand(6), name="a") > psser = ps.from_pandas(pser) > # each of the following actions gets an unboxing error > psser.kurt() > psser.var() > psser.skew() > # set up for covariance test > pdf = pd.DataFrame([(1, 2), (0, 3), (2, 0), (1, 1)], columns=["a", "b"]) > psdf = ps.from_pandas(pdf) > # this gets an unboxing error > psdf.cov() > # set up for stddev resr > from pyspark.pandas.spark import functions as SF > from pyspark.sql.functions import col > from pyspark.sql import Row > df = spark.createDataFrame([Row(a=1), Row(a=2), Row(a=3), Row(a=7), Row(a=9), > Row(a=8)]) > # this gets an unboxing error > df.select(SF.stddev(col("a"), 1)).collect() > {noformat} > Exception from the first case ({{psser.kurt()}}) is > {noformat} > java.lang.ClassCastException: class java.lang.Integer cannot be cast to class > java.lang.Double (java.lang.Integer and java.lang.Double are in module > java.base of loader 'bootstrap') > at scala.runtime.BoxesRunTime.unboxToDouble(BoxesRunTime.java:112) > at > org.apache.spark.sql.catalyst.types.PhysicalDoubleType$$anonfun$2.compare(PhysicalDataType.scala:184) > at scala.math.Ordering.lt(Ordering.scala:98) > at scala.math.Ordering.lt$(Ordering.scala:98) > at > org.apache.spark.sql.catalyst.types.PhysicalDoubleType$$anonfun$2.lt(PhysicalDataType.scala:184) > at > org.apache.spark.sql.catalyst.expressions.LessThan.nullSafeEval(predicates.scala:1196) > {noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org