[ 
https://issues.apache.org/jira/browse/SPARK-61994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon resolved SPARK-61994.
----------------------------------
    Fix Version/s: connect-rust-4.4.0
       Resolution: Fixed

Issue resolved by pull request 845
[https://github.com/apache/spark-connect-rust/pull/845]

> Add typed Rust forms of the optional arguments of PySpark's functions
> ---------------------------------------------------------------------
>
>                 Key: SPARK-61994
>                 URL: https://issues.apache.org/jira/browse/SPARK-61994
>             Project: Spark
>          Issue Type: Improvement
>          Components: Connect
>    Affects Versions: 5.0.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.4.0
>
>
> The Rust API's functions module (spark_connect::functions) has no typed form 
> for the optional arguments of about 106 of PySpark 4.3.0's functions, so a 
> Rust caller cannot send their requests but through call_function or expr: the 
> formats of to_date, to_timestamp, try_to_* and from_unixtime, the ignorenulls 
> of first, last, any_value, first_value, last_value and nth_value, 
> sort_array(asc=False), log(base, col), split(limit), substr(len), 
> locate(pos), the options of from_json, to_json and the csv and xml functions, 
> array_join(null_replacement), array_sort(comparator), sequence(step), 
> make_interval and the interval constructors of non-zero parts, make_timestamp 
> of a time zone or of a date and a time, percentile_approx(accuracy), the 
> mode, padding, iv and aad of aes_*, mask, like and ilike(escapeChar), listagg 
> and string_agg(delimiter), max_by and min_by(k), current_time(precision), the 
> parameters of the sketch functions, and window(slideDuration).
> Add a form of each, which sends PySpark's request, with golden tests of the 
> requests that the official client sends, and a check of the tooling tests 
> that fails on an optional parameter of a PySpark function that no Rust form 
> takes.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to