Hyukjin Kwon created SPARK-61952:
------------------------------------
Summary: Index the keys of the maps of the Rust protobuf runtime,
let one interrupt budget cover each operation, and give its messages protobuf's
UnknownFieldSet, as upb's
Key: SPARK-61952
URL: https://issues.apache.org/jira/browse/SPARK-61952
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.4.0
Reporter: Hyukjin Kwon
A follow-up of SPARK-61918, from a round-3 review of the drop-in's Rust
protobuf runtime (pyspark.sql.connect.proto) against upb (protobuf 7.36.2):
1. Map containers looked up their keys linearly, so a large map was quadratic
and held the GIL. At 30k entries, the parse, the constructor, update, MergeFrom
and == each took about 1.4 s, with every other thread paused that long, where
upb takes milliseconds. A 1M-entry map did not finish in 300 s.
2. The interrupt budget of SPARK-61918 was per call. So a constructor given a
list of many messages, such as Literal.Array(elements=<1M>), started a fresh
budget per element and never yielded: another thread waited 1.4 s, and a signal
handler ran 1.35 s late. It also raised and swallowed an AttributeError per
element. The same held for extend and for the deepcopy of a repeated field.
3. google.protobuf.unknown_fields.UnknownFieldSet(msg), and
text_format.MessageToString(msg, print_unknown_fields=True), raised "TypeError:
Expected a message object" for every message of the runtime, where upb gives
the set and prints it.
4. docs/compatibility.md gave stale figures: 0.25 s for the parse of a million
messages, and 0.07 s for SerializeToString. It also said that the constructor
yields every 1,024 messages, which was not true of lists and maps.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]