Hyukjin Kwon created SPARK-61925:
------------------------------------

             Summary: Skip the protobuf builder hook when protobuf has no 
builder function, and name the runtime classes of plugin files built before the 
hook by their modules
                 Key: SPARK-61925
                 URL: https://issues.apache.org/jira/browse/SPARK-61925
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.4.0
            Reporter: Hyukjin Kwon


SPARK-61867 wraps protobuf's builder.BuildTopDescriptorsAndMessages, so that 
the protoc-generated modules of Spark Connect plugins get the classes of the 
drop-in's Rust protobuf runtime. Two gaps:

- pyspark/sql/connect/proto/_rust.py reads 
builder.BuildTopDescriptorsAndMessages outside its guard, so a protobuf without 
that function would make `import pyspark.sql.connect.proto` raise 
AttributeError, where it should only skip the hook, as it does on a protobuf 
without the builder. No released protobuf lacks it today (checked 3.19.6 to 
7.36.2).
- A plugin file built before the hook, for example Delta Lake's base_pb2 when 
`from delta.tables import DeltaTable` is imported before 
pyspark.sql.connect.proto, gives its runtime classes protoc's module name, such 
as delta.connect.base_pb2.DeltaTable, a module that does not exist, where upb 
names delta.connect.proto.base_pb2.DeltaTable. It shows in reprs and error 
messages.

The hook should be skipped when protobuf has no such function, and the classes 
should be named by the module that holds the file's DESCRIPTOR.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to