[ 
https://issues.apache.org/jira/browse/SPARK-61941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon reassigned SPARK-61941:
------------------------------------

    Assignee: Hyukjin Kwon

> Take any bytes-like blob in SparkConnectClient.cache_artifact, raise 
> PySpark's errors of the return types of register_udf and register_udtf, and 
> make global_user_context_extensions PySpark's list attribute
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61941
>                 URL: https://issues.apache.org/jira/browse/SPARK-61941
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.4.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.4.0
>
>
> Problem
> Follow-ups of SPARK-61856 (#779), whose review compared the drop-in's 
> spark.client with
> pyspark-client 4.3.0:
> 1. cache_artifact(s) accepted only bytes and bytearray; PySpark hashes each 
> blob with hashlib
>    first, so it takes any bytes-like object and raises hashlib's TypeError 
> for another.
> 2. register_udf(f, None or 5) and register_udtf(cls, 5, name) raised a 
> TypeError of the transport;
>    PySpark raises PySparkValueError [UNSUPPORTED_OPERATION] "data type None 
> is not supported."
> 3. global_user_context_extensions was a read-only property; PySpark's is a 
> list attribute whose
>    changes, and a list that it is set to, its requests carry.
> 4. Undocumented: PySpark leaves the errors of clone/cloneSession and 
> copy_from_local_to_fs/
>    copyFromLocalToFs to grpc (RpcError), which the drop-in converts; the core 
> client's retry policy
>    is shared by its clones (set_retry_policy of a clone changes the 
> original), and a client of a new
>    session gets the default policy.
> Repro (Spark 4.3.0 server)
>     c = spark.client
>     c.cache_artifact(memoryview(b"x"))            # drop-in: TypeError; 
> PySpark: the SHA-256 hash
>     c.cache_artifact("x")                         # PySpark: TypeError: 
> Strings must be encoded before hashing
>     c.register_udf(lambda x: x, None)             # PySpark: 
> PySparkValueError [UNSUPPORTED_OPERATION]
>     c.global_user_context_extensions.append((i, any_message))  # PySpark: the 
> next request carries it
> Expected: the results and errors of pyspark-client 4.3.0, and the remaining 
> differences documented.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to