[
https://issues.apache.org/jira/browse/SPARK-61880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-61880.
----------------------------------
Resolution: Fixed
Issue resolved by pull request 790
[https://github.com/apache/spark-connect-rust/pull/790]
> Raise the argument errors of the DataFrame, Column, GroupedData and functions
> APIs where PySpark raises them, when the plan is built, with PySpark's errors
> -----------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61880
> URL: https://issues.apache.org/jira/browse/SPARK-61880
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.4.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.4.0
>
>
> Many DataFrame, Column, GroupedData and functions APIs of the drop-in convert
> their arguments when they are called, where PySpark 4.3.0 sets them into the
> plan's protobuf messages when it builds the plan:
> - A str that UTF-8 cannot encode, such as a lone surrogate, fails at the call
> in F.col, lit, expr, df[name], colRegex, metadataColumn, alias, selectExpr,
> filter, withColumnRenamed, withColumnsRenamed, withMetadata, hint, drop,
> dropDuplicates, toDF, withWatermark, unpivot, transpose, observe, groupBy,
> rollup, cube, groupingSets, pivot, sort, orderBy, sortWithinPartitions,
> describe, summary, na, crosstab, freqItems, sampleBy, join(on=[...]),
> repartitionById and the column arguments and keyword names of UDFs, where
> PySpark fails at the first request with upb's UnicodeEncodeError. Some raise
> other errors: agg({name: "max"}) AttributeError, join(df, name)
> AttributeError, repartition(name) NOT_EXPECTED_TYPE, explain and
> dropna(how=...) UnicodeEncodeError where PySpark raises VALUE_NOT_ALLOWED,
> cast of such a type name TypeError; spark.addTag of such a str is refused
> where PySpark accepts it and its later executions fail.
> - Values that PySpark sets only at plan build are converted at the call:
> DataFrame.alias(5), withColumnRenamed(1, "z"), withColumnsRenamed({"a": 1}),
> unpivot of an int name, describe(b"ab"), freqItems(["a", 1]), fillna(2**70),
> dropna(thresh=2**40), na.replace(subset=[1]), join(on=["a", 1]),
> spark.range(2**70) and tvf.range; bytes, which protobuf's string fields take,
> are refused.
> - withColumn and withMetadata of a name that is not a str raise TypeError
> where PySpark raises AssertionError; F.col(5) raises the AttributeError of
> endswith where PySpark raises TypeError; F.expr(5), filter(5),
> selectExpr(b"a") and agg({b"a": ...}) raise other errors; a cast to a
> DataType subclass is sent as VOID where PySpark raises UNSUPPORTED_OPERATION
> at plan build.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]