[
https://issues.apache.org/jira/browse/SPARK-61967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61967:
---------------------------------
Fix Version/s: connect-rust-4.4.0
> Vendor PySpark's pyspark.sql.connect.avro, protobuf and streaming packages
> verbatim, and remove the native resource profile classes that nothing reaches
> --------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61967
> URL: https://issues.apache.org/jira/browse/SPARK-61967
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-4.4.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.4.0
>
>
> Problem
> 1. The __init__.py of pyspark.sql.connect.avro, pyspark.sql.connect.protobuf
> and
> pyspark.sql.connect.streaming are the drop-in's own. They re-export names
> that PySpark 4.3.0's
> packages do not have, each with an __all__:
> - from_avro and to_avro;
> - from_protobuf and to_protobuf;
> - StreamingQuery, StreamingQueryManager, DataStreamReader,
> DataStreamWriter,
> StreamingQueryListener, StatefulProcessor, StatefulProcessorHandle,
> TwsTester and
> StreamingQueryException.
> PySpark's files hold a docstring or the license header only. So code that
> imports these names
> from the packages works on the drop-in and fails on pyspark-client, as
> `from pyspark.sql.connect.resource import ResourceProfile` did before
> SPARK-61846 made that
> package PySpark's file. The three files are not in the drift gate.
> 2. Since SPARK-61846, pyspark.resource and pyspark.sql.connect.resource are
> PySpark's modules,
> vendored verbatim. The extension still registers its own ResourceProfile,
> ResourceProfileBuilder, ExecutorResourceRequests and TaskResourceRequests
> (crates/pyspark-rs/src/resource.rs), which no module exports any more:
> - their __module__ says pyspark.resource.profile, whose classes of those
> names are others;
> - so a native ResourceProfile does not pickle.
> AGENTS.md still classifies resource/profile.py and serializers.py as
> Rust-backed shims to
> keep, though both are PySpark's files in the drift gate now. The parity
> ledger still says
> that PyResourceProfile of resource.rs implements the connect
> ResourceProfile.
> Repro (master 81751efd, with the drop-in on the path)
> >>> import pyspark.sql.connect.avro as avro, pyspark.sql.connect.streaming
> as streaming
> >>> avro.__all__, hasattr(streaming, "StreamingQuery")
> (['from_avro', 'to_avro'], True)
> >>> from pyspark.sql.connect.protobuf import from_protobuf # no error
> >>> import pickle, pyspark._pyspark as native, pyspark.resource.profile as
> profile
> >>> native.ResourceProfile.__module__, native.ResourceProfile is
> profile.ResourceProfile
> ('pyspark.resource.profile', False)
> >>> pickle.dumps(native.ResourceProfileBuilder().build())
> TypeError: cannot pickle 'ResourceProfile' object
> With pyspark-client 4.3.0, avro has no __all__, streaming has no
> StreamingQuery, and the
> import raises ImportError: cannot import name 'from_protobuf' from
> 'pyspark.sql.connect.protobuf'.
> Expected
> The three packages are PySpark 4.3.0's files, byte for byte, and in the drift
> gate, so they
> re-export nothing, as PySpark's. The extension has no resource profile
> classes of its own.
> AGENTS.md and the parity ledger describe the vendored files.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]