Hyukjin Kwon created SPARK-61925:
------------------------------------
Summary: Skip the protobuf builder hook when protobuf has no
builder function, and name the runtime classes of plugin files built before the
hook by their modules
Key: SPARK-61925
URL: https://issues.apache.org/jira/browse/SPARK-61925
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.4.0
Reporter: Hyukjin Kwon
SPARK-61867 wraps protobuf's builder.BuildTopDescriptorsAndMessages, so that
the protoc-generated modules of Spark Connect plugins get the classes of the
drop-in's Rust protobuf runtime. Two gaps:
- pyspark/sql/connect/proto/_rust.py reads
builder.BuildTopDescriptorsAndMessages outside its guard, so a protobuf without
that function would make `import pyspark.sql.connect.proto` raise
AttributeError, where it should only skip the hook, as it does on a protobuf
without the builder. No released protobuf lacks it today (checked 3.19.6 to
7.36.2).
- A plugin file built before the hook, for example Delta Lake's base_pb2 when
`from delta.tables import DeltaTable` is imported before
pyspark.sql.connect.proto, gives its runtime classes protoc's module name, such
as delta.connect.base_pb2.DeltaTable, a module that does not exist, where upb
names delta.connect.proto.base_pb2.DeltaTable. It shows in reprs and error
messages.
The hook should be skipped when protobuf has no such function, and the classes
should be named by the module that holds the file's DESCRIPTOR.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]