Hyukjin Kwon created SPARK-61967:
------------------------------------

             Summary: Vendor PySpark's pyspark.sql.connect.avro, protobuf and 
streaming packages verbatim, and remove the native resource profile classes 
that nothing reaches
                 Key: SPARK-61967
                 URL: https://issues.apache.org/jira/browse/SPARK-61967
             Project: Spark
          Issue Type: Improvement
          Components: Connect
    Affects Versions: connect-rust-4.4.0
            Reporter: Hyukjin Kwon


Problem
1. The __init__.py of pyspark.sql.connect.avro, pyspark.sql.connect.protobuf and
   pyspark.sql.connect.streaming are the drop-in's own. They re-export names 
that PySpark 4.3.0's
   packages do not have, each with an __all__:
   - from_avro and to_avro;
   - from_protobuf and to_protobuf;
   - StreamingQuery, StreamingQueryManager, DataStreamReader, DataStreamWriter,
     StreamingQueryListener, StatefulProcessor, StatefulProcessorHandle, 
TwsTester and
     StreamingQueryException.
   PySpark's files hold a docstring or the license header only. So code that 
imports these names
   from the packages works on the drop-in and fails on pyspark-client, as
   `from pyspark.sql.connect.resource import ResourceProfile` did before 
SPARK-61846 made that
   package PySpark's file. The three files are not in the drift gate.
2. Since SPARK-61846, pyspark.resource and pyspark.sql.connect.resource are 
PySpark's modules,
   vendored verbatim. The extension still registers its own ResourceProfile,
   ResourceProfileBuilder, ExecutorResourceRequests and TaskResourceRequests
   (crates/pyspark-rs/src/resource.rs), which no module exports any more:
   - their __module__ says pyspark.resource.profile, whose classes of those 
names are others;
   - so a native ResourceProfile does not pickle.
   AGENTS.md still classifies resource/profile.py and serializers.py as 
Rust-backed shims to
   keep, though both are PySpark's files in the drift gate now. The parity 
ledger still says
   that PyResourceProfile of resource.rs implements the connect ResourceProfile.

Repro (master 81751efd, with the drop-in on the path)
  >>> import pyspark.sql.connect.avro as avro, pyspark.sql.connect.streaming as 
streaming
  >>> avro.__all__, hasattr(streaming, "StreamingQuery")
  (['from_avro', 'to_avro'], True)
  >>> from pyspark.sql.connect.protobuf import from_protobuf      # no error
  >>> import pickle, pyspark._pyspark as native, pyspark.resource.profile as 
profile
  >>> native.ResourceProfile.__module__, native.ResourceProfile is 
profile.ResourceProfile
  ('pyspark.resource.profile', False)
  >>> pickle.dumps(native.ResourceProfileBuilder().build())
  TypeError: cannot pickle 'ResourceProfile' object
With pyspark-client 4.3.0, avro has no __all__, streaming has no 
StreamingQuery, and the
import raises ImportError: cannot import name 'from_protobuf' from 
'pyspark.sql.connect.protobuf'.

Expected
The three packages are PySpark 4.3.0's files, byte for byte, and in the drift 
gate, so they
re-export nothing, as PySpark's. The extension has no resource profile classes 
of its own.
AGENTS.md and the parity ledger describe the vendored files.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to