Hyukjin Kwon created SPARK-61952:
------------------------------------

             Summary: Index the keys of the maps of the Rust protobuf runtime, 
let one interrupt budget cover each operation, and give its messages protobuf's 
UnknownFieldSet, as upb's
                 Key: SPARK-61952
                 URL: https://issues.apache.org/jira/browse/SPARK-61952
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.4.0
            Reporter: Hyukjin Kwon


A follow-up of SPARK-61918, from a round-3 review of the drop-in's Rust 
protobuf runtime (pyspark.sql.connect.proto) against upb (protobuf 7.36.2):

1. Map containers looked up their keys linearly, so a large map was quadratic 
and held the GIL. At 30k entries, the parse, the constructor, update, MergeFrom 
and == each took about 1.4 s, with every other thread paused that long, where 
upb takes milliseconds. A 1M-entry map did not finish in 300 s.

2. The interrupt budget of SPARK-61918 was per call. So a constructor given a 
list of many messages, such as Literal.Array(elements=<1M>), started a fresh 
budget per element and never yielded: another thread waited 1.4 s, and a signal 
handler ran 1.35 s late. It also raised and swallowed an AttributeError per 
element. The same held for extend and for the deepcopy of a repeated field.

3. google.protobuf.unknown_fields.UnknownFieldSet(msg), and 
text_format.MessageToString(msg, print_unknown_fields=True), raised "TypeError: 
Expected a message object" for every message of the runtime, where upb gives 
the set and prints it.

4. docs/compatibility.md gave stale figures: 0.25 s for the parse of a million 
messages, and 0.07 s for SerializeToString. It also said that the constructor 
yields every 1,024 messages, which was not true of lists and maps.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to