Ben Hollis created SPARK-58175:
----------------------------------
Summary: Cache descriptor trees built from a binary
FileDescriptorSet in from_protobuf/to_protobuf
Key: SPARK-58175
URL: https://issues.apache.org/jira/browse/SPARK-58175
Project: Spark
Issue Type: Improvement
Components: Protobuf
Affects Versions: 4.2.0
Reporter: Ben Hollis
from_protobuf / to_protobuf used with a binary FileDescriptorSet hold their
descriptor as a "@transient lazy val", so each task instance in an executor JVM
independently rebuilds the entire descriptor object graph from the descriptor
bytes, with no caching. Under high fan-out (many partitions -> many concurrent
tasks in one JVM) with large descriptor sets (hundreds of KB, thousands of
transitive fields), this produces many redundant, concurrently-retained copies
of the same descriptor tree (tens of millions of Descriptors$FieldDescriptor
instances), driving sustained executor GC pressure.
Proposal: add a process-wide bounded cache so concurrent task instances share
one descriptor tree per FileDescriptorSet instead of each rebuilding it. Cache
at the parseFileDescriptorSet layer (which materializes the FieldDescriptor
graph) so a given set is parsed at most once per JVM regardless of entry point
(both the descriptor-tree build and the convertAnyFieldsToJson TypeRegistry
build share one parsed graph), with descriptor trees additionally cached on top
of the parse.
Key the caches on the message name, a content hash of the descriptor bytes, and
whether extension support is enabled; collapse concurrent misses into a single
build; do not cache failed loads. Bound the size by a new internal config
spark.sql.protobuf.descriptorCacheSize (default 8; 0 disables caching as a
kill-switch).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
