Hyukjin Kwon created SPARK-61967:
------------------------------------
Summary: Vendor PySpark's pyspark.sql.connect.avro, protobuf and
streaming packages verbatim, and remove the native resource profile classes
that nothing reaches
Key: SPARK-61967
URL: https://issues.apache.org/jira/browse/SPARK-61967
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.4.0
Reporter: Hyukjin Kwon
Problem
1. The __init__.py of pyspark.sql.connect.avro, pyspark.sql.connect.protobuf and
pyspark.sql.connect.streaming are the drop-in's own. They re-export names
that PySpark 4.3.0's
packages do not have, each with an __all__:
- from_avro and to_avro;
- from_protobuf and to_protobuf;
- StreamingQuery, StreamingQueryManager, DataStreamReader, DataStreamWriter,
StreamingQueryListener, StatefulProcessor, StatefulProcessorHandle,
TwsTester and
StreamingQueryException.
PySpark's files hold a docstring or the license header only. So code that
imports these names
from the packages works on the drop-in and fails on pyspark-client, as
`from pyspark.sql.connect.resource import ResourceProfile` did before
SPARK-61846 made that
package PySpark's file. The three files are not in the drift gate.
2. Since SPARK-61846, pyspark.resource and pyspark.sql.connect.resource are
PySpark's modules,
vendored verbatim. The extension still registers its own ResourceProfile,
ResourceProfileBuilder, ExecutorResourceRequests and TaskResourceRequests
(crates/pyspark-rs/src/resource.rs), which no module exports any more:
- their __module__ says pyspark.resource.profile, whose classes of those
names are others;
- so a native ResourceProfile does not pickle.
AGENTS.md still classifies resource/profile.py and serializers.py as
Rust-backed shims to
keep, though both are PySpark's files in the drift gate now. The parity
ledger still says
that PyResourceProfile of resource.rs implements the connect ResourceProfile.
Repro (master 81751efd, with the drop-in on the path)
>>> import pyspark.sql.connect.avro as avro, pyspark.sql.connect.streaming as
streaming
>>> avro.__all__, hasattr(streaming, "StreamingQuery")
(['from_avro', 'to_avro'], True)
>>> from pyspark.sql.connect.protobuf import from_protobuf # no error
>>> import pickle, pyspark._pyspark as native, pyspark.resource.profile as
profile
>>> native.ResourceProfile.__module__, native.ResourceProfile is
profile.ResourceProfile
('pyspark.resource.profile', False)
>>> pickle.dumps(native.ResourceProfileBuilder().build())
TypeError: cannot pickle 'ResourceProfile' object
With pyspark-client 4.3.0, avro has no __all__, streaming has no
StreamingQuery, and the
import raises ImportError: cannot import name 'from_protobuf' from
'pyspark.sql.connect.protobuf'.
Expected
The three packages are PySpark 4.3.0's files, byte for byte, and in the drift
gate, so they
re-export nothing, as PySpark's. The extension has no resource profile classes
of its own.
AGENTS.md and the parity ledger describe the vendored files.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]