[
https://issues.apache.org/jira/browse/SPARK-61958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-61958:
------------------------------------
Assignee: Hyukjin Kwon
> Make the messages of the Rust protobuf runtime equal protobuf's messages of
> their type by their contents, as upb's, and document the plugin modules that
> keep protobuf's classes
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61958
> URL: https://issues.apache.org/jira/browse/SPARK-61958
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.4.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.4.0
>
>
> Problem
> Since SPARK-61867 (#785), the protoc-generated modules of a Spark Connect
> plugin that import Spark
> Connect's protos, such as delta-spark 4.0.0's delta.connect.proto, get the
> classes of the drop-in's
> Rust protobuf runtime. Other modules of a plugin keep protobuf's classes:
> - a file that imports no proto of Spark Connect, such as Delta's base.proto;
> - a file that is proto2 or of editions, or that imports a well-known type
> other than Any and
> Timestamp;
> - a module of protoc before 3.20.
> protobuf's own lookups (message_factory, symbol_database, upb's pickles of
> nested messages) give
> protobuf's classes too. upb has one class of each type of a pool. The drop-in
> has two classes of
> such a type, and:
> 1. A message of the runtime never equals protobuf's message of its type,
> whatever its contents:
> its == gives NotImplemented of any object that is not of the runtime.
> - So `relation.scan.table == table` is False for Delta's DeltaScan(table).
> - `in`, `remove` and `==` of the containers miss such messages.
> - This regressed with #785. Before it, all of a plugin's classes were
> protobuf's.
> 2. A constructor field refuses protobuf's message of its type when the type
> has a field named
> `keys`: `Expression.Literal(map=<protobuf's Literal.Map>)` raises
> AttributeError: items, as the
> message is taken for a mapping. CopyFrom takes it.
> 3. A module that keeps protobuf's classes and imports a module that has the
> runtime's classes
> refuses that module's messages (TypeError). An example is a proto2 file of
> the plugin that
> imports its proto3 file that imports Spark Connect. upb takes them, and so
> did the drop-in
> before #785. docs/compatibility.md and dev/parity/known-differences.md say
> that such a file
> keeps protobuf's classes, "which take the package's messages only through
> bytes or dicts, as
> before". They don't mention the plugin's own modules.
> 4. A module of protoc before 3.20 (protoc 3.19, grpcio-tools 1.44 and
> earlier) makes its classes
> with reflection.GeneratedProtocolMessageType, not with protobuf's builder.
> So the drop-in's hook
> never sees it, and its fields refuse the drop-in's messages (TypeError),
> where upb takes
> PySpark's. The docs say the drop-in wraps
> builder.BuildTopDescriptorsAndMessages, "which the
> generated modules call".
> Repro (master 2bee7dc9, upb or pure-Python protobuf 7.36.2, with delta-spark
> 4.0.0's modules)
> >>> from delta.connect.plan import DeltaScan
> >>> import delta.connect.proto as dproto
> >>> t = dproto.DeltaTable(); t.path.path = "/tmp/t"
> >>> DeltaScan(t).to_delta_relation(None).scan.table == t
> False # True with pyspark-client 4.3.0
> >>> from google.protobuf import descriptor_pool, message_factory
> >>> Map =
> message_factory.GetMessageClass(descriptor_pool.Default().FindMessageTypeByName(
> ... "spark.connect.Expression.Literal.Map"))
> >>> pb2.Expression.Literal(map=Map(keys=[{"integer": 1}],
> values=[{"integer": 2}]))
> AttributeError: items # pyspark-client 4.3.0 builds it
> >>> b_pb2.B(a=a_pb2.A(name="n")) # b.proto: proto3, imports a.proto and
> duration.proto
> TypeError: Parameter to initialize message field must be dict or instance
> of same class:
> expected <class 'xplug.a_pb2.A'> got <class 'xplug.a_pb2.A'>. #
> 0a0312016e on pyspark-client
> >>> old_style_pb2.OldStyleRelation(input=pb2.Relation(sql={"query": "select
> 1"})) # protoc 3.19
> TypeError: Parameter to initialize message field must be dict or instance
> of same class: ...
> Expected
> - A message of the runtime equals protobuf's message of its type of
> protobuf's default pool, of
> the same contents, on either side of == and !=, as upb's messages of one
> class do. So `in`,
> `remove` and == of the containers find it. A message of a type of another
> pool, of another
> class for upb too, is still not equal.
> - A field takes protobuf's message of its type as a message before a mapping.
> - The docs say which modules keep protobuf's classes and how they take the
> runtime's messages
> (bytes or dicts): those of the plugin's own modules too, and the modules of
> protoc before 3.20.
> Tests pin it.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]