Hyukjin Kwon created SPARK-61941:
------------------------------------
Summary: Take any bytes-like blob in
SparkConnectClient.cache_artifact, raise PySpark's errors of the return types
of register_udf and register_udtf, and make global_user_context_extensions
PySpark's list attribute
Key: SPARK-61941
URL: https://issues.apache.org/jira/browse/SPARK-61941
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.4.0
Reporter: Hyukjin Kwon
Problem
Follow-ups of SPARK-61856 (#779), whose review compared the drop-in's
spark.client with
pyspark-client 4.3.0:
1. cache_artifact(s) accepted only bytes and bytearray; PySpark hashes each
blob with hashlib
first, so it takes any bytes-like object and raises hashlib's TypeError for
another.
2. register_udf(f, None or 5) and register_udtf(cls, 5, name) raised a
TypeError of the transport;
PySpark raises PySparkValueError [UNSUPPORTED_OPERATION] "data type None is
not supported."
3. global_user_context_extensions was a read-only property; PySpark's is a list
attribute whose
changes, and a list that it is set to, its requests carry.
4. Undocumented: PySpark leaves the errors of clone/cloneSession and
copy_from_local_to_fs/
copyFromLocalToFs to grpc (RpcError), which the drop-in converts; the core
client's retry policy
is shared by its clones (set_retry_policy of a clone changes the original),
and a client of a new
session gets the default policy.
Repro (Spark 4.3.0 server)
c = spark.client
c.cache_artifact(memoryview(b"x")) # drop-in: TypeError;
PySpark: the SHA-256 hash
c.cache_artifact("x") # PySpark: TypeError: Strings
must be encoded before hashing
c.register_udf(lambda x: x, None) # PySpark: PySparkValueError
[UNSUPPORTED_OPERATION]
c.global_user_context_extensions.append((i, any_message)) # PySpark: the
next request carries it
Expected: the results and errors of pyspark-client 4.3.0, and the remaining
differences documented.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]