Hyukjin Kwon created SPARK-61092:
------------------------------------
Summary: Take the arguments of withMetadata, withWatermark and
randomSplit, and the attributes of a DataFrame, in the Python client as PySpark
does
Key: SPARK-61092
URL: https://issues.apache.org/jira/browse/SPARK-61092
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
In PySpark Connect, these methods of a DataFrame take their arguments, and
raise their errors, as follows:
* {{withMetadata(columnName, metadata)}} takes a dict of any JSON values
({{json.dumps(metadata)}}), and raises {{NOT_EXPECTED_TYPE}} for a metadata
that is not a dict.
* {{withWatermark(eventTime, delayThreshold)}} raises {{NOT_EXPECTED_TYPE}} for
an empty one or one that is not a str.
* {{randomSplit(weights, seed)}} raises {{VALUE_NOT_POSITIVE}} for a negative
weight ({{arg_name}} "weights") and for weights whose sum is not positive
({{arg_name}} "sum(weights)").
* {{df.name}} is the column {{name}} for any name but a dunder one (unless
{{PYSPARK_VALIDATE_COLUMN_NAME_LEGACY}} is 1), for which a name that is not a
column raises {{ATTRIBUTE_NOT_SUPPORTED}}; {{_jseq}}, {{_jdf}}, {{_jmap}},
{{_jcols}} and {{rdd}} raise {{JVM_ATTRIBUTE_NOT_SUPPORTED}}.
The Rust-backed Python client takes only str values of metadata, raises plain
{{TypeError}}s for the others, sends an empty event time to the server
({{PARSE_EMPTY_STATEMENT}}), sends negative weights to the server
({{IllegalArgumentException}}), and raises an {{AttributeError}} for a name
that starts with "_". Take them as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
