[
https://issues.apache.org/jira/browse/SPARK-61092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61092:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Take the arguments of withMetadata, withWatermark and randomSplit, and the
> attributes of a DataFrame, in the Python client as PySpark does
> ------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61092
> URL: https://issues.apache.org/jira/browse/SPARK-61092
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> In PySpark Connect, these methods of a DataFrame take their arguments, and
> raise their errors, as follows:
> * {{withMetadata(columnName, metadata)}} takes a dict of any JSON values
> ({{json.dumps(metadata)}}), and raises {{NOT_EXPECTED_TYPE}} for a metadata
> that is not a dict.
> * {{withWatermark(eventTime, delayThreshold)}} raises {{NOT_EXPECTED_TYPE}}
> for an empty one or one that is not a str.
> * {{randomSplit(weights, seed)}} raises {{VALUE_NOT_POSITIVE}} for a negative
> weight ({{arg_name}} "weights") and for weights whose sum is not positive
> ({{arg_name}} "sum(weights)").
> * {{df.name}} is the column {{name}} for any name but a dunder one (unless
> {{PYSPARK_VALIDATE_COLUMN_NAME_LEGACY}} is 1), for which a name that is not a
> column raises {{ATTRIBUTE_NOT_SUPPORTED}}; {{_jseq}}, {{_jdf}}, {{_jmap}},
> {{_jcols}} and {{rdd}} raise {{JVM_ATTRIBUTE_NOT_SUPPORTED}}.
> The Rust-backed Python client takes only str values of metadata, raises plain
> {{TypeError}}s for the others, sends an empty event time to the server
> ({{PARSE_EMPTY_STATEMENT}}), sends negative weights to the server
> ({{IllegalArgumentException}}), and raises an {{AttributeError}} for a name
> that starts with "_". Take them as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
