[
https://issues.apache.org/jira/browse/SPARK-61918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61918:
-----------------------------------
Labels: pull-request-available (was: )
> Let other threads and signals interrupt the Rust protobuf runtime on large
> messages, and match upb's __array__, get_or_create, append, bytes field names
> and nested pickles
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61918
> URL: https://issues.apache.org/jira/browse/SPARK-61918
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.4.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.4.0
>
>
> Follow-ups of SPARK-61855 for the drop-in's Rust protobuf runtime
> (pyspark.sql.connect.proto) against upb (protobuf 7.36.2):
> - Parsing, copying (the constructor, CopyFrom, deepcopy), merging and
> serializing large messages hold the GIL with no checkpoint: with 1M strings,
> another Python thread waits about 1.2 s at a parse and 1.8 s at a copy, where
> upb's waits 45-100 ms, and a signal waits as long; the ML results that the
> vendored pyspark.ml parses (StringIndexerModel.labels,
> CountVectorizerModel.vocabulary, from_vocabulary) pause the program for up to
> 3 s.
> - RepeatedScalarContainer has no __array__, so np.asarray of a float or int32
> field gives float64 or int64, and is 100x slower than upb's.
> - Message maps have no get_or_create; RepeatedCompositeContainer.append
> returns None; HasField, ClearField and WhichOneof refuse bytes names; dir()
> lists Extensions; and PySpark's pickles of nested message types do not load.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]