[
https://issues.apache.org/jira/browse/SPARK-62089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-62089:
-----------------------------------
Labels: pull-request-available (was: )
> Make the messages of the Rust protobuf runtime of pyspark.sql.connect.proto,
> and of the plugins' generated modules, behave as upb's where protobuf is
> installed, without requiring it
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-62089
> URL: https://issues.apache.org/jira/browse/SPARK-62089
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
>
> pyspark.sql.connect.proto in the drop-in (pyspark-client-rust) is its own
> protobuf runtime in Rust, which does not require protobuf. The known
> differences of it from upb, the implementation of a protobuf install that
> pyspark-client 4.3.0 uses, and of the plugins' protoc-generated modules (such
> as Delta Lake's Connect client) that import the protos of Spark Connect, are
> closed where protobuf is installed (as a plugin needs it), keeping the
> runtime without it:
> - the runtime's classes are of upb's shape: a metaclass whose classes
> subclass google.protobuf.message.Message (and protobuf's well-known-type
> mixins, WKTBASES), of protobuf's descriptors of its default pool, refusing
> subclassing with upb's TypeError, with upb's ScalarMapContainer and
> MessageMapContainer, protobuf's DecodeError and EncodeError, and protobuf's
> pickles (nested types through _InternalConstructMessage; Any and Timestamp
> byte-identical to upb's);
> - protobuf's message_factory, symbol_database and the builder give the
> runtime's classes of the plugins' files, of proto2 and editions too
> (presence, defaults, packing, closed enums, required fields, groups), of the
> well-known types (Duration, Struct, ...) and of modules of protoc before
> 3.20, and a plugin's dependencies of its own package are taken over (Delta
> Lake's base.proto), never other packages' or Google's, with a give-back when
> a file the runtime cannot hold imports them;
> - upb's 100-level depth limit (and SetAllowOversizeProtos, pure-Python
> SetRecursionLimit), upb's decoding of corrupt input (the parse states that a
> failure leaves), and PySpark's error of a response that the client cannot
> decode ("Exception deserializing response!");
> - docstring-only *_pb2.py source files, so that pyright finds the sources of
> the modules of messages, as with pyspark-client.
> What remains (protobuf extensions, the cost of one Python object per message,
> a few corner cases of the plugins' layouts) is documented in
> docs/compatibility.md and dev/parity/known-differences.md.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]