[
https://issues.apache.org/jira/browse/SPARK-61092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-61092.
----------------------------------
Resolution: Fixed
Issue resolved by pull request 296
[https://github.com/apache/spark-connect-rust/pull/296]
> Take a client seed in sample and randomSplit, and the arguments of
> withMetadata, withWatermark and randomSplit and the attributes of a
> DataFrame, in the Python client as PySpark does
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61092
> URL: https://issues.apache.org/jira/browse/SPARK-61092
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> In PySpark Connect, these methods of a DataFrame take their arguments, and
> raise their errors, as follows:
> * {{sample}} and {{randomSplit}} without a seed take a random seed in the
> client ({{random.randint(0, sys.maxsize)}}), so every action of a sample
> returns the same rows, and the splits of {{randomSplit}}, of one seed and a
> deterministic order ({{deterministic_order}} of the {{Sample}}), are disjoint
> and have every row.
> * {{randomSplit(weights, seed)}} raises {{VALUE_NOT_POSITIVE}} for a negative
> weight ({{arg_name}} "weights") and for weights whose sum is not positive
> ({{arg_name}} "sum(weights)").
> * {{withMetadata(columnName, metadata)}} takes a dict of any JSON values
> ({{json.dumps(metadata)}}), and raises {{NOT_EXPECTED_TYPE}} for a metadata
> that is not a dict.
> * {{withWatermark(eventTime, delayThreshold)}} raises {{NOT_EXPECTED_TYPE}}
> for an empty one or one that is not a str.
> * {{df.name}} is the column {{name}} for any name but a dunder one (unless
> {{PYSPARK_VALIDATE_COLUMN_NAME_LEGACY}} is 1), for which a name that is not a
> column raises {{ATTRIBUTE_NOT_SUPPORTED}}; {{_jseq}}, {{_jdf}}, {{_jmap}},
> {{_jcols}} and {{rdd}} raise {{JVM_ATTRIBUTE_NOT_SUPPORTED}}.
> The Rust-backed Python client sends no seed, so each action of
> {{df.sample(0.5)}} returns other rows, and the splits of {{randomSplit([0.5,
> 0.5])}} miss and repeat rows (2003 rows of 2000). It sends negative weights
> to the server ({{IllegalArgumentException}}), takes only str values of
> metadata, raises plain {{TypeError}}s for the others, sends an empty event
> time to the server ({{PARSE_EMPTY_STATEMENT}}), and raises an
> {{AttributeError}} for a name that starts with "_". Take them as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
