[ 
https://issues.apache.org/jira/browse/SPARK-62089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-62089:
-----------------------------------
    Labels: pull-request-available  (was: )

> Make the messages of the Rust protobuf runtime of pyspark.sql.connect.proto, 
> and of the plugins' generated modules, behave as upb's where protobuf is 
> installed, without requiring it
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-62089
>                 URL: https://issues.apache.org/jira/browse/SPARK-62089
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: 5.0.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>
> pyspark.sql.connect.proto in the drop-in (pyspark-client-rust) is its own 
> protobuf runtime in Rust, which does not require protobuf. The known 
> differences of it from upb, the implementation of a protobuf install that 
> pyspark-client 4.3.0 uses, and of the plugins' protoc-generated modules (such 
> as Delta Lake's Connect client) that import the protos of Spark Connect, are 
> closed where protobuf is installed (as a plugin needs it), keeping the 
> runtime without it:
> - the runtime's classes are of upb's shape: a metaclass whose classes 
> subclass google.protobuf.message.Message (and protobuf's well-known-type 
> mixins, WKTBASES), of protobuf's descriptors of its default pool, refusing 
> subclassing with upb's TypeError, with upb's ScalarMapContainer and 
> MessageMapContainer, protobuf's DecodeError and EncodeError, and protobuf's 
> pickles (nested types through _InternalConstructMessage; Any and Timestamp 
> byte-identical to upb's);
> - protobuf's message_factory, symbol_database and the builder give the 
> runtime's classes of the plugins' files, of proto2 and editions too 
> (presence, defaults, packing, closed enums, required fields, groups), of the 
> well-known types (Duration, Struct, ...) and of modules of protoc before 
> 3.20, and a plugin's dependencies of its own package are taken over (Delta 
> Lake's base.proto), never other packages' or Google's, with a give-back when 
> a file the runtime cannot hold imports them;
> - upb's 100-level depth limit (and SetAllowOversizeProtos, pure-Python 
> SetRecursionLimit), upb's decoding of corrupt input (the parse states that a 
> failure leaves), and PySpark's error of a response that the client cannot 
> decode ("Exception deserializing response!");
> - docstring-only *_pb2.py source files, so that pyright finds the sources of 
> the modules of messages, as with pyspark-client.
> What remains (protobuf extensions, the cost of one Python object per message, 
> a few corner cases of the plugins' layouts) is documented in 
> docs/compatibility.md and dev/parity/known-differences.md.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to