Hyukjin Kwon created SPARK-61958:
------------------------------------
Summary: Make the messages of the Rust protobuf runtime equal
protobuf's messages of their type by their contents, as upb's, and document the
plugin modules that keep protobuf's classes
Key: SPARK-61958
URL: https://issues.apache.org/jira/browse/SPARK-61958
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.4.0
Reporter: Hyukjin Kwon
Problem
Since SPARK-61867 (#785), the protoc-generated modules of a Spark Connect
plugin that import Spark
Connect's protos, such as delta-spark 4.0.0's delta.connect.proto, get the
classes of the drop-in's
Rust protobuf runtime. Other modules of a plugin keep protobuf's classes:
- a file that imports no proto of Spark Connect, such as Delta's base.proto;
- a file that is proto2 or of editions, or that imports a well-known type other
than Any and
Timestamp;
- a module of protoc before 3.20.
protobuf's own lookups (message_factory, symbol_database, upb's pickles of
nested messages) give
protobuf's classes too. upb has one class of each type of a pool. The drop-in
has two classes of
such a type, and:
1. A message of the runtime never equals protobuf's message of its type,
whatever its contents:
its == gives NotImplemented of any object that is not of the runtime.
- So `relation.scan.table == table` is False for Delta's DeltaScan(table).
- `in`, `remove` and `==` of the containers miss such messages.
- This regressed with #785. Before it, all of a plugin's classes were
protobuf's.
2. A constructor field refuses protobuf's message of its type when the type has
a field named
`keys`: `Expression.Literal(map=<protobuf's Literal.Map>)` raises
AttributeError: items, as the
message is taken for a mapping. CopyFrom takes it.
3. A module that keeps protobuf's classes and imports a module that has the
runtime's classes
refuses that module's messages (TypeError). An example is a proto2 file of
the plugin that
imports its proto3 file that imports Spark Connect. upb takes them, and so
did the drop-in
before #785. docs/compatibility.md and dev/parity/known-differences.md say
that such a file
keeps protobuf's classes, "which take the package's messages only through
bytes or dicts, as
before". They don't mention the plugin's own modules.
4. A module of protoc before 3.20 (protoc 3.19, grpcio-tools 1.44 and earlier)
makes its classes
with reflection.GeneratedProtocolMessageType, not with protobuf's builder.
So the drop-in's hook
never sees it, and its fields refuse the drop-in's messages (TypeError),
where upb takes
PySpark's. The docs say the drop-in wraps
builder.BuildTopDescriptorsAndMessages, "which the
generated modules call".
Repro (master 2bee7dc9, upb or pure-Python protobuf 7.36.2, with delta-spark
4.0.0's modules)
>>> from delta.connect.plan import DeltaScan
>>> import delta.connect.proto as dproto
>>> t = dproto.DeltaTable(); t.path.path = "/tmp/t"
>>> DeltaScan(t).to_delta_relation(None).scan.table == t
False # True with pyspark-client 4.3.0
>>> from google.protobuf import descriptor_pool, message_factory
>>> Map =
message_factory.GetMessageClass(descriptor_pool.Default().FindMessageTypeByName(
... "spark.connect.Expression.Literal.Map"))
>>> pb2.Expression.Literal(map=Map(keys=[{"integer": 1}], values=[{"integer":
2}]))
AttributeError: items # pyspark-client 4.3.0 builds it
>>> b_pb2.B(a=a_pb2.A(name="n")) # b.proto: proto3, imports a.proto and
duration.proto
TypeError: Parameter to initialize message field must be dict or instance of
same class:
expected <class 'xplug.a_pb2.A'> got <class 'xplug.a_pb2.A'>. # 0a0312016e
on pyspark-client
>>> old_style_pb2.OldStyleRelation(input=pb2.Relation(sql={"query": "select
1"})) # protoc 3.19
TypeError: Parameter to initialize message field must be dict or instance of
same class: ...
Expected
- A message of the runtime equals protobuf's message of its type of protobuf's
default pool, of
the same contents, on either side of == and !=, as upb's messages of one
class do. So `in`,
`remove` and == of the containers find it. A message of a type of another
pool, of another
class for upb too, is still not equal.
- A field takes protobuf's message of its type as a message before a mapping.
- The docs say which modules keep protobuf's classes and how they take the
runtime's messages
(bytes or dicts): those of the plugin's own modules too, and the modules of
protoc before 3.20.
Tests pin it.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]