Hyukjin Kwon created SPARK-60993:
------------------------------------
Summary: Send the optional arguments of the SQL functions of the
Python client as PySpark does
Key: SPARK-60993
URL: https://issues.apache.org/jira/browse/SPARK-60993
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
In PySpark Connect, the functions of {{pyspark.sql.functions}} send their
optional arguments: {{to_date(col, format)}} sends the format, {{log(base,
col)}} the base, {{sort_array(col, asc=False)}} the order. The Rust-backed
Python client takes these arguments, as its wrappers have PySpark's signatures,
but for about 100 functions its dispatch sends only the leading ones, or a
default instead, so the results are silently wrong. Against a Spark 4.3.0
server:
||Call||PySpark 4.3.0||Rust-backed client||
|{{to_date('02/01/2024', 'dd/MM/yyyy')}}|2024-01-02|fails with
{{CAST_INVALID_INPUT}}|
|{{log(2.0, 8.0)}}|3.0|0.693, the {{ln}} of 2.0|
|{{sort_array(array(2, 1, 3), asc=False)}}|[3, 2, 1]|[1, 2, 3]|
|{{split('a,b,c', ',', 2)}}|['a', 'b,c']|['a', 'b', 'c']|
|{{substr('a,b,c', lit(1), lit(2))}}|'a,'|'a,b,c'|
|{{like('a_c', lit('a!_c'), lit('!'))}}|true|false|
|{{unix_timestamp(timestamp_seconds(1700000000))}}|1700000000|the current time|
|{{months_between('2024-03-31', '2024-02-28',
roundOff=False)}}|1.096774193548387|1.09677419|
|{{first(v, ignorenulls=True)}} over NULL and 5|5|NULL|
The others include the format of {{to_timestamp}} and the other parsing
functions, the mode, padding, IV and AAD of {{aes_encrypt}} and
{{aes_decrypt}}, the seed of {{rand}}, {{randn}}, {{shuffle}} and {{uniform}},
the delimiter of {{listagg}} and {{string_agg}}, the sketch sizes of the
{{hll_}}, {{kll_}}, {{theta_}} and {{tuple_}} functions, the {{ignoreNulls}} of
{{any_value}}, {{first_value}} and {{last_value}}, and the {{k}} of {{max_by}}
and {{min_by}}. SPARK-60952 fixed four functions. Send the optional arguments
as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
