Hyukjin Kwon created SPARK-61994:
------------------------------------

             Summary: Add typed Rust forms of the optional arguments of 
PySpark's functions
                 Key: SPARK-61994
                 URL: https://issues.apache.org/jira/browse/SPARK-61994
             Project: Spark
          Issue Type: Improvement
          Components: Connect
    Affects Versions: 5.0.0
            Reporter: Hyukjin Kwon


The Rust API's functions module (spark_connect::functions) has no typed form 
for the optional arguments of about 106 of PySpark 4.3.0's functions, so a Rust 
caller cannot send their requests but through call_function or expr: the 
formats of to_date, to_timestamp, try_to_* and from_unixtime, the ignorenulls 
of first, last, any_value, first_value, last_value and nth_value, 
sort_array(asc=False), log(base, col), split(limit), substr(len), locate(pos), 
the options of from_json, to_json and the csv and xml functions, 
array_join(null_replacement), array_sort(comparator), sequence(step), 
make_interval and the interval constructors of non-zero parts, make_timestamp 
of a time zone or of a date and a time, percentile_approx(accuracy), the mode, 
padding, iv and aad of aes_*, mask, like and ilike(escapeChar), listagg and 
string_agg(delimiter), max_by and min_by(k), current_time(precision), the 
parameters of the sketch functions, and window(slideDuration).

Add a form of each, which sends PySpark's request, with golden tests of the 
requests that the official client sends, and a check of the tooling tests that 
fails on an optional parameter of a PySpark function that no Rust form takes.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to