Hyukjin Kwon created SPARK-61941:
------------------------------------

             Summary: Take any bytes-like blob in 
SparkConnectClient.cache_artifact, raise PySpark's errors of the return types 
of register_udf and register_udtf, and make global_user_context_extensions 
PySpark's list attribute
                 Key: SPARK-61941
                 URL: https://issues.apache.org/jira/browse/SPARK-61941
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.4.0
            Reporter: Hyukjin Kwon


Problem
Follow-ups of SPARK-61856 (#779), whose review compared the drop-in's 
spark.client with
pyspark-client 4.3.0:
1. cache_artifact(s) accepted only bytes and bytearray; PySpark hashes each 
blob with hashlib
   first, so it takes any bytes-like object and raises hashlib's TypeError for 
another.
2. register_udf(f, None or 5) and register_udtf(cls, 5, name) raised a 
TypeError of the transport;
   PySpark raises PySparkValueError [UNSUPPORTED_OPERATION] "data type None is 
not supported."
3. global_user_context_extensions was a read-only property; PySpark's is a list 
attribute whose
   changes, and a list that it is set to, its requests carry.
4. Undocumented: PySpark leaves the errors of clone/cloneSession and 
copy_from_local_to_fs/
   copyFromLocalToFs to grpc (RpcError), which the drop-in converts; the core 
client's retry policy
   is shared by its clones (set_retry_policy of a clone changes the original), 
and a client of a new
   session gets the default policy.

Repro (Spark 4.3.0 server)
    c = spark.client
    c.cache_artifact(memoryview(b"x"))            # drop-in: TypeError; 
PySpark: the SHA-256 hash
    c.cache_artifact("x")                         # PySpark: TypeError: Strings 
must be encoded before hashing
    c.register_udf(lambda x: x, None)             # PySpark: PySparkValueError 
[UNSUPPORTED_OPERATION]
    c.global_user_context_extensions.append((i, any_message))  # PySpark: the 
next request carries it

Expected: the results and errors of pyspark-client 4.3.0, and the remaining 
differences documented.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to