[ 
https://issues.apache.org/jira/browse/SPARK-61918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon updated SPARK-61918:
---------------------------------
    Fix Version/s: connect-rust-4.4.0

> Let other threads and signals interrupt the Rust protobuf runtime on large 
> messages, and match upb's __array__, get_or_create, append, bytes field names 
> and nested pickles
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61918
>                 URL: https://issues.apache.org/jira/browse/SPARK-61918
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.4.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>             Fix For: connect-rust-4.4.0
>
>
> Follow-ups of SPARK-61855 for the drop-in's Rust protobuf runtime 
> (pyspark.sql.connect.proto) against upb (protobuf 7.36.2):
> - Parsing, copying (the constructor, CopyFrom, deepcopy), merging and 
> serializing large messages hold the GIL with no checkpoint: with 1M strings, 
> another Python thread waits about 1.2 s at a parse and 1.8 s at a copy, where 
> upb's waits 45-100 ms, and a signal waits as long; the ML results that the 
> vendored pyspark.ml parses (StringIndexerModel.labels, 
> CountVectorizerModel.vocabulary, from_vocabulary) pause the program for up to 
> 3 s.
> - RepeatedScalarContainer has no __array__, so np.asarray of a float or int32 
> field gives float64 or int64, and is 100x slower than upb's.
> - Message maps have no get_or_create; RepeatedCompositeContainer.append 
> returns None; HasField, ClearField and WhichOneof refuse bytes names; dir() 
> lists Extensions; and PySpark's pickles of nested message types do not load.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to