[ 
https://issues.apache.org/jira/browse/SPARK-61967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon reassigned SPARK-61967:
------------------------------------

    Assignee: Hyukjin Kwon

> Vendor PySpark's pyspark.sql.connect.avro, protobuf and streaming packages 
> verbatim, and remove the native resource profile classes that nothing reaches
> --------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61967
>                 URL: https://issues.apache.org/jira/browse/SPARK-61967
>             Project: Spark
>          Issue Type: Improvement
>          Components: Connect
>    Affects Versions: connect-rust-4.4.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.4.0
>
>
> Problem
> 1. The __init__.py of pyspark.sql.connect.avro, pyspark.sql.connect.protobuf 
> and
>    pyspark.sql.connect.streaming are the drop-in's own. They re-export names 
> that PySpark 4.3.0's
>    packages do not have, each with an __all__:
>    - from_avro and to_avro;
>    - from_protobuf and to_protobuf;
>    - StreamingQuery, StreamingQueryManager, DataStreamReader, 
> DataStreamWriter,
>      StreamingQueryListener, StatefulProcessor, StatefulProcessorHandle, 
> TwsTester and
>      StreamingQueryException.
>    PySpark's files hold a docstring or the license header only. So code that 
> imports these names
>    from the packages works on the drop-in and fails on pyspark-client, as
>    `from pyspark.sql.connect.resource import ResourceProfile` did before 
> SPARK-61846 made that
>    package PySpark's file. The three files are not in the drift gate.
> 2. Since SPARK-61846, pyspark.resource and pyspark.sql.connect.resource are 
> PySpark's modules,
>    vendored verbatim. The extension still registers its own ResourceProfile,
>    ResourceProfileBuilder, ExecutorResourceRequests and TaskResourceRequests
>    (crates/pyspark-rs/src/resource.rs), which no module exports any more:
>    - their __module__ says pyspark.resource.profile, whose classes of those 
> names are others;
>    - so a native ResourceProfile does not pickle.
>    AGENTS.md still classifies resource/profile.py and serializers.py as 
> Rust-backed shims to
>    keep, though both are PySpark's files in the drift gate now. The parity 
> ledger still says
>    that PyResourceProfile of resource.rs implements the connect 
> ResourceProfile.
> Repro (master 81751efd, with the drop-in on the path)
>   >>> import pyspark.sql.connect.avro as avro, pyspark.sql.connect.streaming 
> as streaming
>   >>> avro.__all__, hasattr(streaming, "StreamingQuery")
>   (['from_avro', 'to_avro'], True)
>   >>> from pyspark.sql.connect.protobuf import from_protobuf      # no error
>   >>> import pickle, pyspark._pyspark as native, pyspark.resource.profile as 
> profile
>   >>> native.ResourceProfile.__module__, native.ResourceProfile is 
> profile.ResourceProfile
>   ('pyspark.resource.profile', False)
>   >>> pickle.dumps(native.ResourceProfileBuilder().build())
>   TypeError: cannot pickle 'ResourceProfile' object
> With pyspark-client 4.3.0, avro has no __all__, streaming has no 
> StreamingQuery, and the
> import raises ImportError: cannot import name 'from_protobuf' from 
> 'pyspark.sql.connect.protobuf'.
> Expected
> The three packages are PySpark 4.3.0's files, byte for byte, and in the drift 
> gate, so they
> re-export nothing, as PySpark's. The extension has no resource profile 
> classes of its own.
> AGENTS.md and the parity ledger describe the vendored files.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to