Hyukjin Kwon created SPARK-62089:
------------------------------------

             Summary: Make the messages of the Rust protobuf runtime of 
pyspark.sql.connect.proto, and of the plugins' generated modules, behave as 
upb's where protobuf is installed, without requiring it
                 Key: SPARK-62089
                 URL: https://issues.apache.org/jira/browse/SPARK-62089
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: 5.0.0
            Reporter: Hyukjin Kwon


pyspark.sql.connect.proto in the drop-in (pyspark-client-rust) is its own 
protobuf runtime in Rust, which does not require protobuf. The known 
differences of it from upb, the implementation of a protobuf install that 
pyspark-client 4.3.0 uses, and of the plugins' protoc-generated modules (such 
as Delta Lake's Connect client) that import the protos of Spark Connect, are 
closed where protobuf is installed (as a plugin needs it), keeping the runtime 
without it:

- the runtime's classes are of upb's shape: a metaclass whose classes subclass 
google.protobuf.message.Message (and protobuf's well-known-type mixins, 
WKTBASES), of protobuf's descriptors of its default pool, refusing subclassing 
with upb's TypeError, with upb's ScalarMapContainer and MessageMapContainer, 
protobuf's DecodeError and EncodeError, and protobuf's pickles (nested types 
through _InternalConstructMessage; Any and Timestamp byte-identical to upb's);
- protobuf's message_factory, symbol_database and the builder give the 
runtime's classes of the plugins' files, of proto2 and editions too (presence, 
defaults, packing, closed enums, required fields, groups), of the well-known 
types (Duration, Struct, ...) and of modules of protoc before 3.20, and a 
plugin's dependencies of its own package are taken over (Delta Lake's 
base.proto), never other packages' or Google's, with a give-back when a file 
the runtime cannot hold imports them;
- upb's 100-level depth limit (and SetAllowOversizeProtos, pure-Python 
SetRecursionLimit), upb's decoding of corrupt input (the parse states that a 
failure leaves), and PySpark's error of a response that the client cannot 
decode ("Exception deserializing response!");
- docstring-only *_pb2.py source files, so that pyright finds the sources of 
the modules of messages, as with pyspark-client.

What remains (protobuf extensions, the cost of one Python object per message, a 
few corner cases of the plugins' layouts) is documented in 
docs/compatibility.md and dev/parity/known-differences.md.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to