[
https://issues.apache.org/jira/browse/SPARK-61994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-61994:
------------------------------------
Assignee: Hyukjin Kwon
> Add typed Rust forms of the optional arguments of PySpark's functions
> ---------------------------------------------------------------------
>
> Key: SPARK-61994
> URL: https://issues.apache.org/jira/browse/SPARK-61994
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
>
> The Rust API's functions module (spark_connect::functions) has no typed form
> for the optional arguments of about 106 of PySpark 4.3.0's functions, so a
> Rust caller cannot send their requests but through call_function or expr: the
> formats of to_date, to_timestamp, try_to_* and from_unixtime, the ignorenulls
> of first, last, any_value, first_value, last_value and nth_value,
> sort_array(asc=False), log(base, col), split(limit), substr(len),
> locate(pos), the options of from_json, to_json and the csv and xml functions,
> array_join(null_replacement), array_sort(comparator), sequence(step),
> make_interval and the interval constructors of non-zero parts, make_timestamp
> of a time zone or of a date and a time, percentile_approx(accuracy), the
> mode, padding, iv and aad of aes_*, mask, like and ilike(escapeChar), listagg
> and string_agg(delimiter), max_by and min_by(k), current_time(precision), the
> parameters of the sketch functions, and window(slideDuration).
> Add a form of each, which sends PySpark's request, with golden tests of the
> requests that the official client sends, and a check of the tooling tests
> that fails on an optional parameter of a PySpark function that no Rust form
> takes.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]