Hyukjin Kwon created SPARK-61918:
------------------------------------
Summary: Let other threads and signals interrupt the Rust protobuf
runtime on large messages, and match upb's __array__, get_or_create, append,
bytes field names and nested pickles
Key: SPARK-61918
URL: https://issues.apache.org/jira/browse/SPARK-61918
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.4.0
Reporter: Hyukjin Kwon
Follow-ups of SPARK-61855 for the drop-in's Rust protobuf runtime
(pyspark.sql.connect.proto) against upb (protobuf 7.36.2):
- Parsing, copying (the constructor, CopyFrom, deepcopy), merging and
serializing large messages hold the GIL with no checkpoint: with 1M strings,
another Python thread waits about 1.2 s at a parse and 1.8 s at a copy, where
upb's waits 45-100 ms, and a signal waits as long; the ML results that the
vendored pyspark.ml parses (StringIndexerModel.labels,
CountVectorizerModel.vocabulary, from_vocabulary) pause the program for up to 3
s.
- RepeatedScalarContainer has no __array__, so np.asarray of a float or int32
field gives float64 or int64, and is 100x slower than upb's.
- Message maps have no get_or_create; RepeatedCompositeContainer.append returns
None; HasField, ClearField and WhichOneof refuse bytes names; dir() lists
Extensions; and PySpark's pickles of nested message types do not load.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]