[
https://issues.apache.org/jira/browse/SPARK-60952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-60952:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Send the optional arguments of hmac, variant_set, vector_norm and
> counter_diff in Spark Connect Rust
> ----------------------------------------------------------------------------------------------------
>
> Key: SPARK-60952
> URL: https://issues.apache.org/jira/browse/SPARK-60952
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> PySpark 4.3.0 gives these functions optional arguments, which the Rust-backed
> Python client accepts in their signatures but does not send, so the results
> silently differ:
> * {{hmac(key, message, algorithm)}}: {{F.hmac(k, m, F.lit("SHA-1"))}} is
> {{hmac(k, m, SHA-1)}}; the client sends {{hmac(k, m)}}, which computes an
> HMAC-SHA256.
> * {{variant_set(v, path, value, create_if_missing)}} and {{try_variant_set}}:
> the client drops {{create_if_missing}}, so {{variant_set(v, "$.b", lit(2),
> False)}} inserts the missing path.
> * {{vector_norm(vector, degree)}} and {{vector_normalize(vector, degree)}}:
> the client drops {{degree}} and computes the L2 norm.
> * {{counter_diff(value, startTime)}}: the client drops {{startTime}}.
> Send the optional arguments as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
