Hyukjin Kwon created SPARK-61918:
------------------------------------

             Summary: Let other threads and signals interrupt the Rust protobuf 
runtime on large messages, and match upb's __array__, get_or_create, append, 
bytes field names and nested pickles
                 Key: SPARK-61918
                 URL: https://issues.apache.org/jira/browse/SPARK-61918
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.4.0
            Reporter: Hyukjin Kwon


Follow-ups of SPARK-61855 for the drop-in's Rust protobuf runtime 
(pyspark.sql.connect.proto) against upb (protobuf 7.36.2):
- Parsing, copying (the constructor, CopyFrom, deepcopy), merging and 
serializing large messages hold the GIL with no checkpoint: with 1M strings, 
another Python thread waits about 1.2 s at a parse and 1.8 s at a copy, where 
upb's waits 45-100 ms, and a signal waits as long; the ML results that the 
vendored pyspark.ml parses (StringIndexerModel.labels, 
CountVectorizerModel.vocabulary, from_vocabulary) pause the program for up to 3 
s.
- RepeatedScalarContainer has no __array__, so np.asarray of a float or int32 
field gives float64 or int64, and is 100x slower than upb's.
- Message maps have no get_or_create; RepeatedCompositeContainer.append returns 
None; HasField, ClearField and WhichOneof refuse bytes names; dir() lists 
Extensions; and PySpark's pickles of nested message types do not load.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to