[
https://issues.apache.org/jira/browse/SPARK-61941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61941:
---------------------------------
Fix Version/s: connect-rust-4.4.0
> Take any bytes-like blob in SparkConnectClient.cache_artifact, raise
> PySpark's errors of the return types of register_udf and register_udtf, and
> make global_user_context_extensions PySpark's list attribute
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61941
> URL: https://issues.apache.org/jira/browse/SPARK-61941
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.4.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.4.0
>
>
> Problem
> Follow-ups of SPARK-61856 (#779), whose review compared the drop-in's
> spark.client with
> pyspark-client 4.3.0:
> 1. cache_artifact(s) accepted only bytes and bytearray; PySpark hashes each
> blob with hashlib
> first, so it takes any bytes-like object and raises hashlib's TypeError
> for another.
> 2. register_udf(f, None or 5) and register_udtf(cls, 5, name) raised a
> TypeError of the transport;
> PySpark raises PySparkValueError [UNSUPPORTED_OPERATION] "data type None
> is not supported."
> 3. global_user_context_extensions was a read-only property; PySpark's is a
> list attribute whose
> changes, and a list that it is set to, its requests carry.
> 4. Undocumented: PySpark leaves the errors of clone/cloneSession and
> copy_from_local_to_fs/
> copyFromLocalToFs to grpc (RpcError), which the drop-in converts; the core
> client's retry policy
> is shared by its clones (set_retry_policy of a clone changes the
> original), and a client of a new
> session gets the default policy.
> Repro (Spark 4.3.0 server)
> c = spark.client
> c.cache_artifact(memoryview(b"x")) # drop-in: TypeError;
> PySpark: the SHA-256 hash
> c.cache_artifact("x") # PySpark: TypeError:
> Strings must be encoded before hashing
> c.register_udf(lambda x: x, None) # PySpark:
> PySparkValueError [UNSUPPORTED_OPERATION]
> c.global_user_context_extensions.append((i, any_message)) # PySpark: the
> next request carries it
> Expected: the results and errors of pyspark-client 4.3.0, and the remaining
> differences documented.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]