Hyukjin Kwon created SPARK-60952:
------------------------------------
Summary: Send the optional arguments of hmac, variant_set,
vector_norm and counter_diff in Spark Connect Rust
Key: SPARK-60952
URL: https://issues.apache.org/jira/browse/SPARK-60952
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
PySpark 4.3.0 gives these functions optional arguments, which the Rust-backed
Python client accepts in their signatures but does not send, so the results
silently differ:
* {{hmac(key, message, algorithm)}}: {{F.hmac(k, m, F.lit("SHA-1"))}} is
{{hmac(k, m, SHA-1)}}; the client sends {{hmac(k, m)}}, which computes an
HMAC-SHA256.
* {{variant_set(v, path, value, create_if_missing)}} and {{try_variant_set}}:
the client drops {{create_if_missing}}, so {{variant_set(v, "$.b", lit(2),
False)}} inserts the missing path.
* {{vector_norm(vector, degree)}} and {{vector_normalize(vector, degree)}}: the
client drops {{degree}} and computes the L2 norm.
* {{counter_diff(value, startTime)}}: the client drops {{startTime}}.
Send the optional arguments as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
