This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new c5d6aea [SPARK-61115][CONNECT][PYTHON] Add the module-level names and
the connect modules that the Python client lacks, as PySpark 4.3.0 has them
c5d6aea is described below
commit c5d6aea62aa0fe13e3caeab77675a47970de3f6c
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sat Nov 7 21:58:14 2026 +0900
[SPARK-61115][CONNECT][PYTHON] Add the module-level names and the connect
modules that the Python client lacks, as PySpark 4.3.0 has them
### What changes were proposed in this pull request?
[SPARK-61115](https://issues.apache.org/jira/browse/SPARK-61115): An audit
of the modules of the `pyspark-client` package of Apache Spark v4.3.0 found
module-level names and modules that the client lacks. This PR adds them, as
PySpark has them:
- `pyspark.util`: `fail_on_stopiteration`, `disable_gc`,
`spark_connect_mode`, `default_api_mode`, `FuncT` and its `__all__`, from
v4.3.0.
- The `__all__` of `pyspark.sql.udf`, `pyspark.sql.udtf`,
`pyspark.sql.avro` and `pyspark.sql.protobuf`. Star imports now export only the
listed names. The `__init__` of `avro` and `protobuf` are now identical to
v4.3.0.
- `pyspark.sql.functions.builtin` and
`pyspark.sql.connect.functions.builtin` now have every public function of
`pyspark.sql.functions`. That includes the deprecated ones that `__all__`
leaves out: `approxCountDistinct`, `bitwiseNOT`, `countDistinct`, `shiftLeft`,
`shiftRight`, `shiftRightUnsigned`, `sumDistinct`, `toDegrees`, `toRadians`,
`chr`, `random` and `uuid`. As in PySpark, they have no `__all__`.
- The modules of the connect import path:
- `functions.partitioning`, `avro.functions`, `protobuf.functions` and
`table_arg` re-export the functions and the `TableArg` of the client.
- `functions` imports `partitioning`, as PySpark's does.
- `datasource`: `DataSourceRegistration`, which registers a data source
with the session.
- `utils`: from v4.3.0, except that `check_dependencies` checks the
dependencies of this client. Its transport is the Rust extension, so it needs
no gRPC, `grpcio-status`, `googleapis-common-protos` or `zstandard`.
- `logging` and `_typing`, vendored byte for byte (and added to the drift
check).
`dev/parity/4.3.0-module-audit.md` records the audit:
- the method: 352 modules, with `pyspark.core` and `py4j` blocked
- each gap and the JIRA that closes it: SPARK-61111, SPARK-61113, this one,
SPARK-61116, SPARK-61117 and SPARK-61118
- why the rest are not gaps: server-side modules, RDD and classic
internals, the gRPC client, tools, and the native `pyspark.ml.connect`
### Why are the changes needed?
Code that imports these names from where PySpark has them failed:
- `from pyspark.sql.functions.builtin import countDistinct` raised
`ImportError`.
- `import pyspark.sql.connect.functions.partitioning` and the other connect
modules raised `ModuleNotFoundError`.
- `from pyspark.util import fail_on_stopiteration` raised `ImportError`.
Star imports also exported every public name of these modules.
### Does this PR introduce _any_ user-facing change?
Yes. These names and modules exist as in PySpark 4.3.0.
### How was this patch tested?
- New offline tests check:
- the names of `pyspark.util`, and the star imports of `pyspark.sql.udf`,
`udtf`, `avro` and `protobuf`. This test passes with official PySpark 4.3.0.
- the modules of the connect import path, the deprecated functions of the
builtin modules, and the registration of a data source, against a mock Connect
server
- `pyspark.sql.connect.utils`, in a process without gRPC
- The existing test of the exports of `pyspark.sql.functions` now expects a
`builtin` module without `__all__`, as PySpark's.
- The lib tests of both crates (590 passed), the build of all tests, also
with `wasm-udf`, the offline Python suites (2108 passed, 8 skipped), and the
live drop-in and Arrow suites (164 passed) and live cases (10 passed) against a
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #311 from HyukjinKwon/SPARK-61115-module-names.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
dev/parity/4.3.0-module-audit.md | 50 ++++++
python/.coveragerc | 3 +
python/pyspark/sql/avro/__init__.py | 2 +
python/pyspark/sql/connect/_typing.py | 87 ++++++++++
.../__init__.py => connect/avro/functions.py} | 5 +
python/pyspark/sql/connect/datasource.py | 46 +++++
python/pyspark/sql/connect/functions/__init__.py | 1 +
python/pyspark/sql/connect/functions/builtin.py | 4 +-
.../functions/partitioning.py} | 5 +
python/pyspark/sql/connect/logging.py | 67 ++++++++
.../__init__.py => connect/protobuf/functions.py} | 5 +
.../{protobuf/__init__.py => connect/table_arg.py} | 5 +
python/pyspark/sql/connect/utils.py | 124 +++++++++++++
python/pyspark/sql/functions.py | 8 +-
python/pyspark/sql/protobuf/__init__.py | 2 +
python/pyspark/sql/udf.py | 2 +
python/pyspark/sql/udtf.py | 10 ++
python/pyspark/util.py | 70 ++++++++
python/tests/test_dropin_offline.py | 191 ++++++++++++++++++++-
scripts/check_vendored_upstream.sh | 4 +
20 files changed, 681 insertions(+), 10 deletions(-)
diff --git a/dev/parity/4.3.0-module-audit.md b/dev/parity/4.3.0-module-audit.md
new file mode 100644
index 0000000..02e3853
--- /dev/null
+++ b/dev/parity/4.3.0-module-audit.md
@@ -0,0 +1,50 @@
+# PySpark 4.3.0 module audit (the `pyspark-client` packages)
+
+The class diff (`4.3.0-introspection-diff.md`) covers the connect classes.
This audit covers
+the modules and module-level names of the `pyspark-client` package: every
module of the
+packages of `connect_packages` (`python/packaging/client/setup.py` at
`v4.3.0`), 352 modules.
+
+Method: import each module from the v4.3.0 source with `pyspark.core` and
`py4j` blocked (they
+are not in `pyspark-client`, so `is_remote_only()` holds), and from this
client. Then diff the
+public names that the official module defines itself (functions and classes
whose
+`__module__` is the module), its submodules and its constants. Imported names
(typing
+aliases, re-exported helpers) are left out. 27 modules do not import in
`pyspark-client`
+either (`pyspark.mllib.*`, `pyspark.java_gateway`, `pyspark.shell`), and are
left out.
+
+## Gaps, and where they are closed
+
+| Gap | JIRA |
+|---|---|
+| `pyspark.ml` (the DataFrame-based ML API) and the ML internals of
`pyspark.ml.connect` (`io_utils`, ...), with the generated
`pyspark.sql.connect.proto` | SPARK-61111 |
+| `DataFrame.__arrow_c_stream__` and `pyspark.sql.interchange` | SPARK-61113 |
+| `pyspark.util`: `fail_on_stopiteration`, `disable_gc`, `spark_connect_mode`,
`default_api_mode`, `FuncT`, `__all__` | SPARK-61115 |
+| `__all__` of `pyspark.sql.udf`, `udtf`, `avro`, `protobuf` | SPARK-61115 |
+| the deprecated functions of `pyspark.sql.functions.builtin` and
`pyspark.sql.connect.functions.builtin` (`countDistinct`, ...), which have no
`__all__` | SPARK-61115 |
+| `pyspark.sql.connect`: `functions.partitioning`, `avro.functions`,
`protobuf.functions`, `table_arg`, `datasource`, `utils`, `logging`, `_typing`
| SPARK-61115 |
+| `pyspark.sql.connect.plan` `Read`/`Range`/`SQL` (placeholders),
`Column.to_plan`, `pyspark.sql.connect.types` helpers,
`pyspark.sql.connect.conversion` | SPARK-61116 |
+| `SparkSession.Hook`: the hooks of `_registerHook` are not called |
SPARK-61117 |
+| `pyspark.pipelines`: `cli`, `init_cli`, `add_pipeline_analysis_context`,
`block_session_mutations` | SPARK-61118 |
+
+## Not gaps
+
+- **Server-side modules**, which run on the server's Python worker (official
PySpark there):
+ - `pyspark.daemon`, `pyspark.worker`, `pyspark.worker_message`
+ - `pyspark.sql.streaming.python_streaming_source_runner`,
`pyspark.sql.streaming.proto.StateMessage_pb2`,
`stateful_processor_api_client.SCALAR_TYPES`
+ - `pyspark.sql.connect.streaming.worker.*`
+- **RDD and classic internals**, of which a Connect client has no use:
+ - modules: `pyspark.join`, `rddsampler`, `resultiterable`, `shuffle`,
`statcounter`, and the RDD-based `pyspark.mllib`, most of which does not import
in `pyspark-client` either
+ - the py4j converters of `pyspark.sql.types` (`DateConverter`,
`DatetimeConverter`, `DatetimeNTZConverter`, `DayTimeIntervalTypeConverter`,
`NumpyArrayConverter`, `NumpyScalarConverter`, `TimeConverter`)
+ - the classic dispatch helpers of `pyspark.sql.utils` (`dispatch_*_method`,
`try_remote_*`, `to_java_array`, `to_scala_map`, `get_active_spark_context`,
`ForeachBatchFunction`)
+ - `pyspark.sql.group.dfapi` and `df_varargs_api`, `PandasGroupedOpsMixin`,
`OptionUtils`, `JStreamingQueryListener`
+ - `StreamingCheckpointManager`, which PySpark documents as supported in
Spark Classic only
+- **The gRPC client**, which the Rust transport replaces:
+ - `pyspark.sql.connect.client.{core, artifact, reattach, retries}` and
`pyspark.sql.connect.proto.base_pb2_grpc`
+ - `convert_exception` and `convert_observation_errors` of
`pyspark.errors.exceptions.connect`
+ - the plan and expression classes of `pyspark.sql.connect.plan` and
`pyspark.sql.connect.expressions`, which the Rust core builds (except those
that SPARK-61111 and SPARK-61116 port), and `WindowFrame`,
`StreamingQueryListenerBus`
+- **Tools**, not part of the client API:
+ - `pyspark.errors_doc_gen`, `pyspark.install`, `pyspark.threaddump`
+ - `pyspark.sql.connect.local_server`, which the classic builder uses to
start a local server
+ - `pyspark.sql.transpile`, which the classic `UserDefinedFunction` uses
+- **Types of the Rust core**: `DataTypeSingleton`, the metaclass of the
singleton types of PySpark.
+- **`pyspark.ml.connect`**: this client's is the native ML API of the Rust
core (see `docs/compatibility.md`), not the deprecated torch-based one. So its
models are `MLModel`, without `LogisticRegressionModel`, `MaxAbsScalerModel`,
`StandardScalerModel` or `ArrayAssembler`.
+- **Submodule attributes** that differ only because of import order
(`pyspark.find_spark_home`, `pyspark.sql.conversion`,
`pyspark.sql.streaming.state`, ...). The modules import.
diff --git a/python/.coveragerc b/python/.coveragerc
index 9cb8189..ffd1a5a 100644
--- a/python/.coveragerc
+++ b/python/.coveragerc
@@ -58,6 +58,9 @@ omit =
*/pyspark/memory_profiler_ext.py
*/pyspark/worker_util.py
*/pyspark/instrumentation_utils.py
+ # The logger and the types of the connect import path, vendored.
+ */pyspark/sql/connect/logging.py
+ */pyspark/sql/connect/_typing.py
*/pyspark/_pyspark*
*/tests/*
diff --git a/python/pyspark/sql/avro/__init__.py
b/python/pyspark/sql/avro/__init__.py
index cce3aca..ac530a5 100644
--- a/python/pyspark/sql/avro/__init__.py
+++ b/python/pyspark/sql/avro/__init__.py
@@ -14,3 +14,5 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
+
+__all__ = ["functions"]
diff --git a/python/pyspark/sql/connect/_typing.py
b/python/pyspark/sql/connect/_typing.py
new file mode 100644
index 0000000..6465938
--- /dev/null
+++ b/python/pyspark/sql/connect/_typing.py
@@ -0,0 +1,87 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+from types import FunctionType
+from typing import Any, Callable, Iterable, Iterator, Union, Optional,
NewType, Protocol, Tuple
+import datetime
+import decimal
+
+import pyarrow
+from pandas.core.frame import DataFrame as PandasDataFrame
+
+from pyspark.sql.column import Column
+from pyspark.sql.connect.types import DataType
+from pyspark.sql.streaming.state import GroupState
+
+ColumnOrName = Union[Column, str]
+
+ColumnOrNameOrOrdinal = Union[Column, str, int]
+
+PrimitiveType = Union[bool, float, int, str]
+
+OptionalPrimitiveType = Optional[PrimitiveType]
+
+LiteralType = PrimitiveType
+
+DecimalLiteral = decimal.Decimal
+
+DateTimeLiteral = Union[datetime.date, datetime.time, datetime.datetime]
+
+DataTypeOrString = Union[DataType, str]
+
+DataFrameLike = PandasDataFrame
+
+PandasMapIterFunction = Callable[[Iterator[DataFrameLike]],
Iterator[DataFrameLike]]
+
+ArrowMapIterFunction = Callable[[Iterator[pyarrow.RecordBatch]],
Iterator[pyarrow.RecordBatch]]
+
+PandasGroupedMapFunction = Union[
+ Callable[[DataFrameLike], DataFrameLike],
+ Callable[[Any, DataFrameLike], DataFrameLike],
+]
+
+GroupedMapPandasUserDefinedFunction =
NewType("GroupedMapPandasUserDefinedFunction", FunctionType)
+
+PandasCogroupedMapFunction = Callable[[DataFrameLike, DataFrameLike],
DataFrameLike]
+
+PandasGroupedMapFunctionWithState = Callable[
+ [Any, Iterable[DataFrameLike], GroupState], Iterable[DataFrameLike]
+]
+ArrowGroupedMapFunction = Union[
+ Callable[[pyarrow.Table], pyarrow.Table],
+ Callable[[Tuple[pyarrow.Scalar, ...], pyarrow.Table], pyarrow.Table],
+]
+ArrowCogroupedMapFunction = Union[
+ Callable[[pyarrow.Table, pyarrow.Table], pyarrow.Table],
+ Callable[[Tuple[pyarrow.Scalar, ...], pyarrow.Table, pyarrow.Table],
pyarrow.Table],
+]
+
+
+class UserDefinedFunctionLike(Protocol):
+ func: Callable[..., Any]
+ evalType: int
+ deterministic: bool
+
+ @property
+ def returnType(self) -> DataType: ...
+
+ def __call__(self, *args: ColumnOrName) -> Column: ...
+
+ def asNondeterministic(self) -> "UserDefinedFunctionLike": ...
+
+
+class UserDefinedFunctionCallable(Protocol):
+ def __call__(self, *_: ColumnOrName) -> Column: ...
diff --git a/python/pyspark/sql/protobuf/__init__.py
b/python/pyspark/sql/connect/avro/functions.py
similarity index 75%
copy from python/pyspark/sql/protobuf/__init__.py
copy to python/pyspark/sql/connect/avro/functions.py
index cce3aca..6429b3e 100644
--- a/python/pyspark/sql/protobuf/__init__.py
+++ b/python/pyspark/sql/connect/avro/functions.py
@@ -14,3 +14,8 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
+"""The Avro functions under the connect import path, as PySpark Connect's
+``pyspark.sql.connect.avro.functions``: those of
``pyspark.sql.avro.functions``, which the Rust
+core builds."""
+
+from pyspark.sql.avro.functions import from_avro, to_avro # noqa: F401
diff --git a/python/pyspark/sql/connect/datasource.py
b/python/pyspark/sql/connect/datasource.py
new file mode 100644
index 0000000..c835a28
--- /dev/null
+++ b/python/pyspark/sql/connect/datasource.py
@@ -0,0 +1,46 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""The registration of data sources under the connect import path, as PySpark
Connect's
+``pyspark.sql.connect.datasource``: a wrapper of a session whose ``register``
registers a
+data source with the session, as ``spark.dataSource.register`` does."""
+
+from typing import Type, TYPE_CHECKING
+
+from pyspark.sql.datasource import DataSourceRegistration as
PySparkDataSourceRegistration
+
+if TYPE_CHECKING:
+ from pyspark.sql.datasource import DataSource
+ from pyspark.sql.session import SparkSession
+
+
+class DataSourceRegistration:
+ """
+ Wrapper for data source registration.
+
+ .. versionadded:: 4.0.0
+ """
+
+ def __init__(self, sparkSession: "SparkSession"):
+ self.sparkSession = sparkSession
+
+ def register(
+ self,
+ dataSource: Type["DataSource"],
+ ) -> None:
+ self.sparkSession.dataSource.register(dataSource)
+
+ register.__doc__ = PySparkDataSourceRegistration.register.__doc__
diff --git a/python/pyspark/sql/connect/functions/__init__.py
b/python/pyspark/sql/connect/functions/__init__.py
index 67aba57..896fb50 100644
--- a/python/pyspark/sql/connect/functions/__init__.py
+++ b/python/pyspark/sql/connect/functions/__init__.py
@@ -17,3 +17,4 @@
"""Re-export of ``pyspark.sql.functions`` under the Connect import path."""
from pyspark.sql.functions import * # noqa: F401,F403
from pyspark.sql.functions import col, expr # noqa: F401
+from pyspark.sql.connect.functions import partitioning # noqa: F401
diff --git a/python/pyspark/sql/connect/functions/builtin.py
b/python/pyspark/sql/connect/functions/builtin.py
index e0234da..4420628 100644
--- a/python/pyspark/sql/connect/functions/builtin.py
+++ b/python/pyspark/sql/connect/functions/builtin.py
@@ -17,7 +17,9 @@
"""Re-export of ``pyspark.sql.functions`` under the Connect import path."""
from typing import TYPE_CHECKING
-from pyspark.sql.functions import * # noqa: F401,F403
+# Every public name of pyspark.sql.functions.builtin, also the deprecated
functions that
+# ``pyspark.sql.functions.__all__`` leaves out, as PySpark Connect's builtin
has them.
+from pyspark.sql.functions.builtin import * # noqa: F401,F403
from pyspark.sql.functions import col, expr, _invoke_function, _to_col #
noqa: F401
if TYPE_CHECKING:
diff --git a/python/pyspark/sql/protobuf/__init__.py
b/python/pyspark/sql/connect/functions/partitioning.py
similarity index 71%
copy from python/pyspark/sql/protobuf/__init__.py
copy to python/pyspark/sql/connect/functions/partitioning.py
index cce3aca..1fb7d79 100644
--- a/python/pyspark/sql/protobuf/__init__.py
+++ b/python/pyspark/sql/connect/functions/partitioning.py
@@ -14,3 +14,8 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
+"""The partition transform functions under the connect import path, as PySpark
Connect's
+``pyspark.sql.connect.functions.partitioning``: those of
``pyspark.sql.functions.partitioning``,
+which the Rust core builds."""
+
+from pyspark.sql.functions.partitioning import bucket, days, hours, months,
years # noqa: F401
diff --git a/python/pyspark/sql/connect/logging.py
b/python/pyspark/sql/connect/logging.py
new file mode 100644
index 0000000..099193f
--- /dev/null
+++ b/python/pyspark/sql/connect/logging.py
@@ -0,0 +1,67 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+
+import logging
+from pyspark.logger import PySparkLogger
+import os
+from typing import Optional
+
+__all__ = ["configureLogging", "getLogLevel"]
+
+
+def configureLogging(level: Optional[str] = None) -> logging.Logger:
+ """
+ Configure log level for Spark Connect components.
+ When not specified as a parameter, log level will be configured based on
+ the SPARK_CONNECT_LOG_LEVEL environment variable.
+ When both are absent, logging is disabled.
+
+ .. versionadded:: 4.0.0
+ """
+ logger = PySparkLogger.getLogger(__name__)
+ handler = logging.StreamHandler()
+ handler.setFormatter(
+ logging.Formatter(fmt="%(asctime)s %(process)d %(levelname)s
%(funcName)s %(message)s")
+ )
+ logger.addHandler(handler)
+
+ if level is not None:
+ logger.setLevel(level.upper())
+ elif "SPARK_CONNECT_LOG_LEVEL" in os.environ:
+ logger.setLevel(os.environ["SPARK_CONNECT_LOG_LEVEL"].upper())
+ else:
+ logger.disabled = True
+ return logger
+
+
+# Instantiate the logger based on the environment configuration.
+logger = configureLogging()
+
+
+def getLogLevel() -> Optional[int]:
+ """
+ This returns this log level as integer, or none (if no logging is enabled).
+
+ Spark Connect logging can be configured with environment variable
'SPARK_CONNECT_LOG_LEVEL'
+
+ .. versionadded:: 3.5.0
+ """
+
+ if not logger.disabled:
+ return logger.level
+ return None
diff --git a/python/pyspark/sql/protobuf/__init__.py
b/python/pyspark/sql/connect/protobuf/functions.py
similarity index 73%
copy from python/pyspark/sql/protobuf/__init__.py
copy to python/pyspark/sql/connect/protobuf/functions.py
index cce3aca..33c618b 100644
--- a/python/pyspark/sql/protobuf/__init__.py
+++ b/python/pyspark/sql/connect/protobuf/functions.py
@@ -14,3 +14,8 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
+"""The Protobuf functions under the connect import path, as PySpark Connect's
+``pyspark.sql.connect.protobuf.functions``: those of
``pyspark.sql.protobuf.functions``, which the
+Rust core builds."""
+
+from pyspark.sql.protobuf.functions import from_protobuf, to_protobuf # noqa:
F401
diff --git a/python/pyspark/sql/protobuf/__init__.py
b/python/pyspark/sql/connect/table_arg.py
similarity index 76%
copy from python/pyspark/sql/protobuf/__init__.py
copy to python/pyspark/sql/connect/table_arg.py
index cce3aca..d3c1445 100644
--- a/python/pyspark/sql/protobuf/__init__.py
+++ b/python/pyspark/sql/connect/table_arg.py
@@ -14,3 +14,8 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
+"""The table argument under the connect import path, as PySpark Connect's
+``pyspark.sql.connect.table_arg``: the ``TableArg`` of
``pyspark.sql.table_arg``, of
+``DataFrame.asTable()``."""
+
+from pyspark.sql.table_arg import TableArg # noqa: F401
diff --git a/python/pyspark/sql/connect/utils.py
b/python/pyspark/sql/connect/utils.py
new file mode 100644
index 0000000..ce09bba
--- /dev/null
+++ b/python/pyspark/sql/connect/utils.py
@@ -0,0 +1,124 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""The utilities of the connect import path, as PySpark Connect's
``pyspark.sql.connect.utils``,
+from Apache Spark v4.3.0. ``check_dependencies`` checks the dependencies of
this client."""
+
+import sys
+
+from pyspark.loose_version import LooseVersion
+from pyspark.sql.pandas.utils import require_minimum_pandas_version,
require_minimum_pyarrow_version
+from pyspark.errors import PySparkImportError
+
+
+def check_dependencies() -> None:
+ main_module = sys.modules["__main__"]
+ if (
+ main_module.__spec__ is None
+ and not hasattr(main_module, "__file__")
+ and not hasattr(sys, "ps1")
+ ):
+ # The main module is not initialized at all at this point. We must be
running doctests.
+ from pyspark.testing.connectutils import should_test_connect,
connect_requirement_message
+
+ if not should_test_connect:
+ print(
+ f"Skipping doctests: {connect_requirement_message}",
+ file=sys.stderr,
+ )
+ sys.exit(0)
+
+ # The dependencies of this client: its transport is of the Rust extension,
so it needs no
+ # grpcio, grpcio-status, googleapis-common-protos or zstandard, which
PySpark checks.
+ require_minimum_pandas_version()
+ require_minimum_pyarrow_version()
+
+
+def require_minimum_grpc_version() -> None:
+ """Raise ImportError if minimum version of grpc is not installed"""
+ minimum_grpc_version = "1.48.1"
+
+ try:
+ import grpc
+ except ImportError as error:
+ raise PySparkImportError(
+ errorClass="PACKAGE_NOT_INSTALLED",
+ messageParameters={
+ "package_name": "grpcio",
+ "minimum_version": str(minimum_grpc_version),
+ },
+ ) from error
+ if LooseVersion(grpc.__version__) < LooseVersion(minimum_grpc_version):
+ raise PySparkImportError(
+ errorClass="UNSUPPORTED_PACKAGE_VERSION",
+ messageParameters={
+ "package_name": "grpcio",
+ "minimum_version": str(minimum_grpc_version),
+ "current_version": str(grpc.__version__),
+ },
+ )
+
+
+def require_minimum_grpcio_status_version() -> None:
+ """Raise ImportError if grpcio-status is not installed"""
+ minimum_grpc_version = "1.48.1"
+
+ try:
+ import grpc_status # noqa
+ except ImportError as error:
+ raise PySparkImportError(
+ errorClass="PACKAGE_NOT_INSTALLED",
+ messageParameters={
+ "package_name": "grpcio-status",
+ "minimum_version": str(minimum_grpc_version),
+ },
+ ) from error
+
+
+def require_minimum_googleapis_common_protos_version() -> None:
+ """Raise ImportError if googleapis-common-protos is not installed"""
+ minimum_common_protos_version = "1.56.4"
+
+ try:
+ import google.rpc # noqa
+ except ImportError as error:
+ raise PySparkImportError(
+ errorClass="PACKAGE_NOT_INSTALLED",
+ messageParameters={
+ "package_name": "googleapis-common-protos",
+ "minimum_version": str(minimum_common_protos_version),
+ },
+ ) from error
+
+
+def require_minimum_zstandard_version() -> None:
+ """Raise ImportError if zstandard is not installed"""
+ minimum_zstandard_version = "0.25.0"
+
+ try:
+ import zstandard # noqa
+ except ImportError as error:
+ raise PySparkImportError(
+ errorClass="PACKAGE_NOT_INSTALLED",
+ messageParameters={
+ "package_name": "zstandard",
+ "minimum_version": str(minimum_zstandard_version),
+ },
+ ) from error
+
+
+def get_python_ver() -> str:
+ return "%d.%d" % sys.version_info[:2]
diff --git a/python/pyspark/sql/functions.py b/python/pyspark/sql/functions.py
index 7e2034d..c6334b0 100644
--- a/python/pyspark/sql/functions.py
+++ b/python/pyspark/sql/functions.py
@@ -1325,11 +1325,11 @@ del name
# ``__init__`` re-exports them; here this module IS the implementation, so
register a synthetic
# ``builtin`` submodule mirroring it (so ``import
pyspark.sql.functions.builtin`` and
# ``from pyspark.sql.functions.builtin import col`` work like reference
pyspark).
+# As PySpark's, it has every public name of the functions, also the deprecated
ones that
+# ``__all__`` leaves out (such as ``countDistinct``), and no ``__all__``.
builtin = _types.ModuleType("pyspark.sql.functions.builtin")
builtin.__doc__ = "Built-in DataFrame functions (re-exported from
pyspark.sql.functions)."
-for _bname in list(__all__):
- _bobj = globals().get(_bname)
- if _bobj is not None:
+for _bname, _bobj in list(globals().items()):
+ if not _bname.startswith("_") and not isinstance(_bobj, _types.ModuleType):
setattr(builtin, _bname, _bobj)
-builtin.__all__ = list(__all__)
_sys.modules["pyspark.sql.functions.builtin"] = builtin
diff --git a/python/pyspark/sql/protobuf/__init__.py
b/python/pyspark/sql/protobuf/__init__.py
index cce3aca..ac530a5 100644
--- a/python/pyspark/sql/protobuf/__init__.py
+++ b/python/pyspark/sql/protobuf/__init__.py
@@ -14,3 +14,5 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
+
+__all__ = ["functions"]
diff --git a/python/pyspark/sql/udf.py b/python/pyspark/sql/udf.py
index 6c900e2..3759b26 100644
--- a/python/pyspark/sql/udf.py
+++ b/python/pyspark/sql/udf.py
@@ -31,6 +31,8 @@ from pyspark.sql.types import DataType, StringType, StructType
from pyspark.serializers import CloudPickleSerializer
from pyspark.util import PythonEvalType
+__all__ = ["UDFRegistration"]
+
def _as_concrete_datatype(returnType: Any) -> DataType:
"""Return a concrete, picklable pyspark ``DataType``.
diff --git a/python/pyspark/sql/udtf.py b/python/pyspark/sql/udtf.py
index 4c71345..b13f96e 100644
--- a/python/pyspark/sql/udtf.py
+++ b/python/pyspark/sql/udtf.py
@@ -32,6 +32,16 @@ from typing import Any, Optional, Type
from pyspark.errors import PySparkAttributeError, PySparkTypeError
from pyspark.serializers import CloudPickleSerializer
+__all__ = [
+ "AnalyzeArgument",
+ "AnalyzeResult",
+ "PartitioningColumn",
+ "OrderingColumn",
+ "SelectedColumn",
+ "SkipRestOfInputTableException",
+ "UDTFRegistration",
+]
+
# Eval types for Python UDTFs (mirrors pyspark.util.PythonEvalType).
SQL_TABLE_UDF = 300
SQL_ARROW_TABLE_UDF = 301
diff --git a/python/pyspark/util.py b/python/pyspark/util.py
index cf21945..5708a51 100644
--- a/python/pyspark/util.py
+++ b/python/pyspark/util.py
@@ -643,3 +643,73 @@ def _do_server_auth(conn: "io.IOBase", auth_secret: str)
-> None:
errorClass="UNEXPECTED_RESPONSE_FROM_SERVER",
messageParameters={},
)
+
+
+# The names of PySpark's pyspark.util that its other modules and tools use,
from Apache Spark
+# v4.3.0.
+import gc # noqa: E402
+from typing import Generator, List, TypeVar # noqa: E402
+
+__all__: List[str] = []
+
+FuncT = TypeVar("FuncT", bound=Callable[..., Any])
+
+
+def fail_on_stopiteration(f: Callable) -> Callable:
+ """
+ Wraps the input function to fail on 'StopIteration' by raising a
'RuntimeError'
+ prevents silent loss of data when 'f' is used in a for loop in Spark code
+ """
+
+ def wrapper(*args: Any, **kwargs: Any) -> Any:
+ try:
+ return f(*args, **kwargs)
+ except StopIteration as exc:
+ raise PySparkRuntimeError(
+ errorClass="STOP_ITERATION_OCCURRED",
+ messageParameters={
+ "exc": str(exc),
+ },
+ )
+
+ return wrapper
+
+
[email protected]
+def disable_gc() -> Generator[None, None, None]:
+ gc_enabled_originally = gc.isenabled()
+ if gc_enabled_originally:
+ gc.disable()
+ try:
+ yield
+ finally:
+ if gc_enabled_originally:
+ gc.enable()
+
+
+# This function will be called in `pyspark` script as well,
+# so this should be available without a running Spark.
+# If move or rename, please update the script too.
+def spark_connect_mode() -> str:
+ """
+ Return the env var SPARK_CONNECT_MODE; otherwise "1" if `pyspark_connect`
is available.
+ """
+ connect_by_default = os.environ.get("SPARK_CONNECT_MODE")
+ if connect_by_default is not None:
+ return connect_by_default
+ try:
+ import pyspark_connect # noqa: F401
+
+ return "1"
+ except ImportError:
+ return "0"
+
+
+def default_api_mode() -> str:
+ """
+ Return the default API mode.
+ """
+ if spark_connect_mode() == "1":
+ return "connect"
+ else:
+ return "classic"
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 7f3bf3f..9a7f7c7 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -912,13 +912,13 @@ def test_functions_exports_fresh_process():
assert all(namespace[name] is getattr(F, name) for name in
F.__all__)
builtin = importlib.import_module("pyspark.sql.functions.builtin")
assert builtin is F.builtin is sys.modules[builtin.__name__]
- assert type(builtin.__all__) is list
- assert builtin.__all__ == F.__all__
- assert builtin.__all__ is not F.__all__
+ # As PySpark's, the builtin module has no __all__, and has every
public function of
+ # pyspark.sql.functions, also the deprecated ones that __all__
leaves out.
+ assert not hasattr(builtin, "__all__")
assert all(getattr(builtin, name) is getattr(F, name) for name in
F.__all__)
builtin_namespace = {}
exec("from pyspark.sql.functions.builtin import *",
builtin_namespace)
- assert [name for name in builtin_namespace if name !=
"__builtins__"] == F.__all__
+ assert set(F.__all__) <= set(builtin_namespace)
assert all(builtin_namespace[name] is getattr(F, name) for name in
F.__all__)
part =
importlib.import_module("pyspark.sql.functions.partitioning")
assert part is F.partitioning is sys.modules[part.__name__]
@@ -935,9 +935,10 @@ def test_functions_exports_fresh_process():
expected = check_exports()
old_all = F.__all__
old_all.append("caller_added_name")
- F.builtin.__all__.append("builtin_only_name")
+ F.builtin.builtin_only_name = None
importlib.reload(F)
assert F.__all__ is not old_all
+ assert not hasattr(F.builtin, "builtin_only_name")
assert check_exports() == expected
''').replace("__OFFICIAL_FUNCTIONS_ALL__", repr(OFFICIAL_FUNCTIONS_ALL))],
capture_output=True, text=True)
assert result.returncode == 0, result.stdout + result.stderr
@@ -5150,6 +5151,186 @@ def test_connect_shim_imports():
assert repr(_invoke_function_over_columns("pandas_product", F.col("x")))
== "Column<'pandas_product(x)'>"
+def test_module_level_names_as_pyspark(monkeypatch):
+ # As official PySpark 4.3.0: the names of pyspark.util, and the __all__ of
the modules of
+ # pyspark.sql, of which a star import exports only those.
+ import gc
+ import typing
+
+ import pyspark.util as util
+ from pyspark.errors import PySparkRuntimeError
+
+ assert util.__all__ == []
+ assert isinstance(util.FuncT, typing.TypeVar)
+ monkeypatch.setenv("SPARK_CONNECT_MODE", "1")
+ assert util.spark_connect_mode() == "1" and util.default_api_mode() ==
"connect"
+ monkeypatch.setenv("SPARK_CONNECT_MODE", "0")
+ assert util.spark_connect_mode() == "0" and util.default_api_mode() ==
"classic"
+
+ def stops():
+ raise StopIteration("done")
+
+ assert util.fail_on_stopiteration(lambda x: x + 1)(1) == 2
+ with pytest.raises(PySparkRuntimeError) as error:
+ util.fail_on_stopiteration(stops)()
+ assert error.value.getCondition() == "STOP_ITERATION_OCCURRED"
+ assert error.value.getMessageParameters() == {"exc": "done"}
+ enabled = gc.isenabled()
+ with util.disable_gc():
+ assert not gc.isenabled()
+ assert gc.isenabled() == enabled
+
+ for module, expected in [
+ ("pyspark.sql.udf", ["UDFRegistration"]),
+ (
+ "pyspark.sql.udtf",
+ [
+ "AnalyzeArgument",
+ "AnalyzeResult",
+ "PartitioningColumn",
+ "OrderingColumn",
+ "SelectedColumn",
+ "SkipRestOfInputTableException",
+ "UDTFRegistration",
+ ],
+ ),
+ ("pyspark.sql.avro", ["functions"]),
+ ("pyspark.sql.protobuf", ["functions"]),
+ ]:
+ namespace = {}
+ exec(f"from {module} import *", namespace)
+ assert [name for name in namespace if name != "__builtins__"] ==
expected, module
+
+
+def test_connect_import_path_modules_as_pyspark(mock_connect):
+ # As official PySpark 4.3.0: the modules of the connect import path, of
the functions and
+ # the classes of this client, and the deprecated functions of the builtin
modules.
+ import importlib
+ import logging
+
+ from pyspark.sql import functions as F
+ from pyspark.sql.avro import functions as avro
+ from pyspark.sql.datasource import DataSource
+ from pyspark.sql.functions import partitioning
+ from pyspark.sql.protobuf import functions as protobuf
+ from pyspark.sql.table_arg import TableArg
+
+ spark, service = mock_connect
+ connect_partitioning =
importlib.import_module("pyspark.sql.connect.functions.partitioning")
+ assert
importlib.import_module("pyspark.sql.connect.functions").partitioning is (
+ connect_partitioning
+ )
+ for name in ["years", "months", "days", "hours", "bucket"]:
+ assert getattr(connect_partitioning, name) is getattr(partitioning,
name)
+ connect_avro =
importlib.import_module("pyspark.sql.connect.avro.functions")
+ assert connect_avro.from_avro is avro.from_avro and connect_avro.to_avro
is avro.to_avro
+ connect_protobuf =
importlib.import_module("pyspark.sql.connect.protobuf.functions")
+ assert connect_protobuf.from_protobuf is protobuf.from_protobuf
+ assert connect_protobuf.to_protobuf is protobuf.to_protobuf
+ assert importlib.import_module("pyspark.sql.connect.table_arg").TableArg
is TableArg
+ typing_module = importlib.import_module("pyspark.sql.connect._typing")
+ assert {"ColumnOrName", "DataTypeOrString", "LiteralType"} <=
set(dir(typing_module))
+
+ connect_logging = importlib.import_module("pyspark.sql.connect.logging")
+ assert connect_logging.__all__ == ["configureLogging", "getLogLevel"]
+ assert isinstance(connect_logging.logger, logging.Logger)
+ if "SPARK_CONNECT_LOG_LEVEL" not in os.environ:
+ assert connect_logging.getLogLevel() is None
+
+ connect_builtin =
importlib.import_module("pyspark.sql.connect.functions.builtin")
+ for name in [
+ "approxCountDistinct",
+ "bitwiseNOT",
+ "countDistinct",
+ "shiftLeft",
+ "shiftRight",
+ "shiftRightUnsigned",
+ "sumDistinct",
+ "toDegrees",
+ "toRadians",
+ "chr",
+ "random",
+ "uuid",
+ ]:
+ assert getattr(F.builtin, name) is getattr(F, name), name
+ assert getattr(connect_builtin, name) is getattr(F, name), name
+
+ # The registration of a data source registers it with the session.
+ from pyspark.sql.connect.datasource import DataSourceRegistration
+
+ class TestSource(DataSource):
+ @classmethod
+ def name(cls):
+ return "test_source"
+
+ DataSourceRegistration(spark).register(TestSource)
+ assert service.execute_calls == 1
+
+
+def test_connect_utils_check_the_dependencies_of_the_client(tmp_path):
+ # As PySpark Connect's pyspark.sql.connect.utils, but check_dependencies
checks the
+ # dependencies of this client, which do not include gRPC. The script runs
of a file: of
+ # a main module without one, check_dependencies takes it for doctests.
+ import subprocess
+ import sys
+ import textwrap
+
+ from pyspark.errors import PySparkImportError
+ from pyspark.sql.connect import utils
+
+ assert utils.get_python_ver() == "%d.%d" % sys.version_info[:2]
+ utils.check_dependencies()
+ for check in [
+ utils.require_minimum_grpc_version,
+ utils.require_minimum_grpcio_status_version,
+ utils.require_minimum_googleapis_common_protos_version,
+ utils.require_minimum_zstandard_version,
+ ]:
+ # Of the packages installed here, or not.
+ try:
+ check()
+ except PySparkImportError as error:
+ assert error.getCondition() in ("PACKAGE_NOT_INSTALLED",
"UNSUPPORTED_PACKAGE_VERSION")
+ script = textwrap.dedent(
+ """
+ import importlib.abc, sys
+
+ class Block(importlib.abc.MetaPathFinder):
+ def find_spec(self, name, path, target=None):
+ if name.split(".")[0] in ("grpc", "grpc_status", "zstandard")
or name.startswith(
+ "google.rpc"
+ ):
+ raise ImportError("not installed: " + name)
+
+ sys.meta_path.insert(0, Block())
+ from pyspark.errors import PySparkImportError
+ from pyspark.sql.connect import utils
+
+ utils.check_dependencies()
+ for check, package in [
+ (utils.require_minimum_grpc_version, "grpcio"),
+ (utils.require_minimum_grpcio_status_version, "grpcio-status"),
+ (utils.require_minimum_googleapis_common_protos_version,
"googleapis-common-protos"),
+ (utils.require_minimum_zstandard_version, "zstandard"),
+ ]:
+ try:
+ check()
+ except PySparkImportError as error:
+ assert error.getCondition() == "PACKAGE_NOT_INSTALLED"
+ assert error.getMessageParameters()["package_name"] == package
+ else:
+ raise AssertionError(package)
+ print("ok")
+ """
+ )
+ path = tmp_path / "check_dependencies.py"
+ path.write_text(script)
+ result = subprocess.run(
+ [sys.executable, str(path)], capture_output=True, text=True,
timeout=120
+ )
+ assert result.returncode == 0 and result.stdout.strip() == "ok",
result.stderr
+
+
def test_is_remote_flags():
# The Rust-backed drop-in is always Connect-only.
from pyspark.util import is_remote_only
diff --git a/scripts/check_vendored_upstream.sh
b/scripts/check_vendored_upstream.sh
index ecef653..bb31693 100644
--- a/scripts/check_vendored_upstream.sh
+++ b/scripts/check_vendored_upstream.sh
@@ -92,6 +92,10 @@ VENDORED_FILES=(
# ExecutePlanResponse is that of the message class of
pyspark/sql/connect/proto.py.
"sql/connect/shell/progress.py"
+ # The logger of Spark Connect, and the types of the connect import path.
+ "sql/connect/logging.py"
+ "sql/connect/_typing.py"
+
# The spatial reference systems of the spatial types, and the
SQLStringFormatter, of
# pyspark.sql.
"sql/geo_utils.py"
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]