Hyukjin Kwon created SPARK-61994:
------------------------------------
Summary: Add typed Rust forms of the optional arguments of
PySpark's functions
Key: SPARK-61994
URL: https://issues.apache.org/jira/browse/SPARK-61994
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
The Rust API's functions module (spark_connect::functions) has no typed form
for the optional arguments of about 106 of PySpark 4.3.0's functions, so a Rust
caller cannot send their requests but through call_function or expr: the
formats of to_date, to_timestamp, try_to_* and from_unixtime, the ignorenulls
of first, last, any_value, first_value, last_value and nth_value,
sort_array(asc=False), log(base, col), split(limit), substr(len), locate(pos),
the options of from_json, to_json and the csv and xml functions,
array_join(null_replacement), array_sort(comparator), sequence(step),
make_interval and the interval constructors of non-zero parts, make_timestamp
of a time zone or of a date and a time, percentile_approx(accuracy), the mode,
padding, iv and aad of aes_*, mask, like and ilike(escapeChar), listagg and
string_agg(delimiter), max_by and min_by(k), current_time(precision), the
parameters of the sketch functions, and window(slideDuration).
Add a form of each, which sends PySpark's request, with golden tests of the
requests that the official client sends, and a check of the tooling tests that
fails on an optional parameter of a PySpark function that no Rust form takes.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]