Hyukjin Kwon created SPARK-31915: ------------------------------------ Summary: Remove projection that adds grouping keys in grouped and cogrouped pandas UDFs Key: SPARK-31915 URL: https://issues.apache.org/jira/browse/SPARK-31915 Project: Spark Issue Type: Bug Components: PySpark, SQL Affects Versions: 3.0.0 Reporter: Hyukjin Kwon
Currently, grouped and cogrouped pandas UDFs in Spark unnecessarily projects the grouping keys. This results in case-sensitivity resolution failure when the project contains columns such as "Column" and "column" as they are considered different but ambiguous columns. It results as below: {code} from pyspark.sql.functions import * df = spark.createDataFrame([[1, 1]], ["column", "Score"]) @pandas_udf("column integer, Score float", PandasUDFType.GROUPED_MAP) def my_pandas_udf(pdf): return pdf.assign(Score=0.5) df.groupby('COLUMN').apply(my_pandas_udf).show() {code} {code} pyspark.sql.utils.AnalysisException: Reference 'COLUMN' is ambiguous, could be: COLUMN, COLUMN.; {code} {code} pyspark.sql.utils.AnalysisException: cannot resolve '`COLUMN`' given input columns: [COLUMN, COLUMN, value, value];; 'FlatMapCoGroupsInPandas ['COLUMN], ['COLUMN], <lambda>(column#9L, value#10L, column#13L, value#14L), [column#22L, value#23L] :- Project [COLUMN#9L, column#9L, value#10L] : +- LogicalRDD [column#9L, value#10L], false +- Project [COLUMN#13L, column#13L, value#14L] +- LogicalRDD [column#13L, value#14L], false {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org