Hyukjin Kwon created SPARK-61958:
------------------------------------

             Summary: Make the messages of the Rust protobuf runtime equal 
protobuf's messages of their type by their contents, as upb's, and document the 
plugin modules that keep protobuf's classes
                 Key: SPARK-61958
                 URL: https://issues.apache.org/jira/browse/SPARK-61958
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.4.0
            Reporter: Hyukjin Kwon


Problem
Since SPARK-61867 (#785), the protoc-generated modules of a Spark Connect 
plugin that import Spark
Connect's protos, such as delta-spark 4.0.0's delta.connect.proto, get the 
classes of the drop-in's
Rust protobuf runtime. Other modules of a plugin keep protobuf's classes:
- a file that imports no proto of Spark Connect, such as Delta's base.proto;
- a file that is proto2 or of editions, or that imports a well-known type other 
than Any and
  Timestamp;
- a module of protoc before 3.20.
protobuf's own lookups (message_factory, symbol_database, upb's pickles of 
nested messages) give
protobuf's classes too. upb has one class of each type of a pool. The drop-in 
has two classes of
such a type, and:
1. A message of the runtime never equals protobuf's message of its type, 
whatever its contents:
   its == gives NotImplemented of any object that is not of the runtime.
   - So `relation.scan.table == table` is False for Delta's DeltaScan(table).
   - `in`, `remove` and `==` of the containers miss such messages.
   - This regressed with #785. Before it, all of a plugin's classes were 
protobuf's.
2. A constructor field refuses protobuf's message of its type when the type has 
a field named
   `keys`: `Expression.Literal(map=<protobuf's Literal.Map>)` raises 
AttributeError: items, as the
   message is taken for a mapping. CopyFrom takes it.
3. A module that keeps protobuf's classes and imports a module that has the 
runtime's classes
   refuses that module's messages (TypeError). An example is a proto2 file of 
the plugin that
   imports its proto3 file that imports Spark Connect. upb takes them, and so 
did the drop-in
   before #785. docs/compatibility.md and dev/parity/known-differences.md say 
that such a file
   keeps protobuf's classes, "which take the package's messages only through 
bytes or dicts, as
   before". They don't mention the plugin's own modules.
4. A module of protoc before 3.20 (protoc 3.19, grpcio-tools 1.44 and earlier) 
makes its classes
   with reflection.GeneratedProtocolMessageType, not with protobuf's builder. 
So the drop-in's hook
   never sees it, and its fields refuse the drop-in's messages (TypeError), 
where upb takes
   PySpark's. The docs say the drop-in wraps 
builder.BuildTopDescriptorsAndMessages, "which the
   generated modules call".

Repro (master 2bee7dc9, upb or pure-Python protobuf 7.36.2, with delta-spark 
4.0.0's modules)
  >>> from delta.connect.plan import DeltaScan
  >>> import delta.connect.proto as dproto
  >>> t = dproto.DeltaTable(); t.path.path = "/tmp/t"
  >>> DeltaScan(t).to_delta_relation(None).scan.table == t
  False                                   # True with pyspark-client 4.3.0
  >>> from google.protobuf import descriptor_pool, message_factory
  >>> Map = 
message_factory.GetMessageClass(descriptor_pool.Default().FindMessageTypeByName(
  ...     "spark.connect.Expression.Literal.Map"))
  >>> pb2.Expression.Literal(map=Map(keys=[{"integer": 1}], values=[{"integer": 
2}]))
  AttributeError: items                   # pyspark-client 4.3.0 builds it
  >>> b_pb2.B(a=a_pb2.A(name="n"))   # b.proto: proto3, imports a.proto and 
duration.proto
  TypeError: Parameter to initialize message field must be dict or instance of 
same class:
  expected <class 'xplug.a_pb2.A'> got <class 'xplug.a_pb2.A'>.    # 0a0312016e 
on pyspark-client
  >>> old_style_pb2.OldStyleRelation(input=pb2.Relation(sql={"query": "select 
1"}))  # protoc 3.19
  TypeError: Parameter to initialize message field must be dict or instance of 
same class: ...

Expected
- A message of the runtime equals protobuf's message of its type of protobuf's 
default pool, of
  the same contents, on either side of == and !=, as upb's messages of one 
class do. So `in`,
  `remove` and == of the containers find it. A message of a type of another 
pool, of another
  class for upb too, is still not equal.
- A field takes protobuf's message of its type as a message before a mapping.
- The docs say which modules keep protobuf's classes and how they take the 
runtime's messages
  (bytes or dicts): those of the plugin's own modules too, and the modules of 
protoc before 3.20.
  Tests pin it.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to