This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new c5d6aea  [SPARK-61115][CONNECT][PYTHON] Add the module-level names and 
the connect modules that the Python client lacks, as PySpark 4.3.0 has them
c5d6aea is described below

commit c5d6aea62aa0fe13e3caeab77675a47970de3f6c
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sat Nov 7 21:58:14 2026 +0900

    [SPARK-61115][CONNECT][PYTHON] Add the module-level names and the connect 
modules that the Python client lacks, as PySpark 4.3.0 has them
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61115](https://issues.apache.org/jira/browse/SPARK-61115): An audit 
of the modules of the `pyspark-client` package of Apache Spark v4.3.0 found 
module-level names and modules that the client lacks. This PR adds them, as 
PySpark has them:
    - `pyspark.util`: `fail_on_stopiteration`, `disable_gc`, 
`spark_connect_mode`, `default_api_mode`, `FuncT` and its `__all__`, from 
v4.3.0.
    - The `__all__` of `pyspark.sql.udf`, `pyspark.sql.udtf`, 
`pyspark.sql.avro` and `pyspark.sql.protobuf`. Star imports now export only the 
listed names. The `__init__` of `avro` and `protobuf` are now identical to 
v4.3.0.
    - `pyspark.sql.functions.builtin` and 
`pyspark.sql.connect.functions.builtin` now have every public function of 
`pyspark.sql.functions`. That includes the deprecated ones that `__all__` 
leaves out: `approxCountDistinct`, `bitwiseNOT`, `countDistinct`, `shiftLeft`, 
`shiftRight`, `shiftRightUnsigned`, `sumDistinct`, `toDegrees`, `toRadians`, 
`chr`, `random` and `uuid`. As in PySpark, they have no `__all__`.
    - The modules of the connect import path:
      - `functions.partitioning`, `avro.functions`, `protobuf.functions` and 
`table_arg` re-export the functions and the `TableArg` of the client.
      - `functions` imports `partitioning`, as PySpark's does.
      - `datasource`: `DataSourceRegistration`, which registers a data source 
with the session.
      - `utils`: from v4.3.0, except that `check_dependencies` checks the 
dependencies of this client. Its transport is the Rust extension, so it needs 
no gRPC, `grpcio-status`, `googleapis-common-protos` or `zstandard`.
      - `logging` and `_typing`, vendored byte for byte (and added to the drift 
check).
    
    `dev/parity/4.3.0-module-audit.md` records the audit:
    - the method: 352 modules, with `pyspark.core` and `py4j` blocked
    - each gap and the JIRA that closes it: SPARK-61111, SPARK-61113, this one, 
SPARK-61116, SPARK-61117 and SPARK-61118
    - why the rest are not gaps: server-side modules, RDD and classic 
internals, the gRPC client, tools, and the native `pyspark.ml.connect`
    
    ### Why are the changes needed?
    
    Code that imports these names from where PySpark has them failed:
    - `from pyspark.sql.functions.builtin import countDistinct` raised 
`ImportError`.
    - `import pyspark.sql.connect.functions.partitioning` and the other connect 
modules raised `ModuleNotFoundError`.
    - `from pyspark.util import fail_on_stopiteration` raised `ImportError`.
    
    Star imports also exported every public name of these modules.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. These names and modules exist as in PySpark 4.3.0.
    
    ### How was this patch tested?
    
    - New offline tests check:
      - the names of `pyspark.util`, and the star imports of `pyspark.sql.udf`, 
`udtf`, `avro` and `protobuf`. This test passes with official PySpark 4.3.0.
      - the modules of the connect import path, the deprecated functions of the 
builtin modules, and the registration of a data source, against a mock Connect 
server
      - `pyspark.sql.connect.utils`, in a process without gRPC
    - The existing test of the exports of `pyspark.sql.functions` now expects a 
`builtin` module without `__all__`, as PySpark's.
    - The lib tests of both crates (590 passed), the build of all tests, also 
with `wasm-udf`, the offline Python suites (2108 passed, 8 skipped), and the 
live drop-in and Arrow suites (164 passed) and live cases (10 passed) against a 
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #311 from HyukjinKwon/SPARK-61115-module-names.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 dev/parity/4.3.0-module-audit.md                   |  50 ++++++
 python/.coveragerc                                 |   3 +
 python/pyspark/sql/avro/__init__.py                |   2 +
 python/pyspark/sql/connect/_typing.py              |  87 ++++++++++
 .../__init__.py => connect/avro/functions.py}      |   5 +
 python/pyspark/sql/connect/datasource.py           |  46 +++++
 python/pyspark/sql/connect/functions/__init__.py   |   1 +
 python/pyspark/sql/connect/functions/builtin.py    |   4 +-
 .../functions/partitioning.py}                     |   5 +
 python/pyspark/sql/connect/logging.py              |  67 ++++++++
 .../__init__.py => connect/protobuf/functions.py}  |   5 +
 .../{protobuf/__init__.py => connect/table_arg.py} |   5 +
 python/pyspark/sql/connect/utils.py                | 124 +++++++++++++
 python/pyspark/sql/functions.py                    |   8 +-
 python/pyspark/sql/protobuf/__init__.py            |   2 +
 python/pyspark/sql/udf.py                          |   2 +
 python/pyspark/sql/udtf.py                         |  10 ++
 python/pyspark/util.py                             |  70 ++++++++
 python/tests/test_dropin_offline.py                | 191 ++++++++++++++++++++-
 scripts/check_vendored_upstream.sh                 |   4 +
 20 files changed, 681 insertions(+), 10 deletions(-)

diff --git a/dev/parity/4.3.0-module-audit.md b/dev/parity/4.3.0-module-audit.md
new file mode 100644
index 0000000..02e3853
--- /dev/null
+++ b/dev/parity/4.3.0-module-audit.md
@@ -0,0 +1,50 @@
+# PySpark 4.3.0 module audit (the `pyspark-client` packages)
+
+The class diff (`4.3.0-introspection-diff.md`) covers the connect classes. 
This audit covers
+the modules and module-level names of the `pyspark-client` package: every 
module of the
+packages of `connect_packages` (`python/packaging/client/setup.py` at 
`v4.3.0`), 352 modules.
+
+Method: import each module from the v4.3.0 source with `pyspark.core` and 
`py4j` blocked (they
+are not in `pyspark-client`, so `is_remote_only()` holds), and from this 
client. Then diff the
+public names that the official module defines itself (functions and classes 
whose
+`__module__` is the module), its submodules and its constants. Imported names 
(typing
+aliases, re-exported helpers) are left out. 27 modules do not import in 
`pyspark-client`
+either (`pyspark.mllib.*`, `pyspark.java_gateway`, `pyspark.shell`), and are 
left out.
+
+## Gaps, and where they are closed
+
+| Gap | JIRA |
+|---|---|
+| `pyspark.ml` (the DataFrame-based ML API) and the ML internals of 
`pyspark.ml.connect` (`io_utils`, ...), with the generated 
`pyspark.sql.connect.proto` | SPARK-61111 |
+| `DataFrame.__arrow_c_stream__` and `pyspark.sql.interchange` | SPARK-61113 |
+| `pyspark.util`: `fail_on_stopiteration`, `disable_gc`, `spark_connect_mode`, 
`default_api_mode`, `FuncT`, `__all__` | SPARK-61115 |
+| `__all__` of `pyspark.sql.udf`, `udtf`, `avro`, `protobuf` | SPARK-61115 |
+| the deprecated functions of `pyspark.sql.functions.builtin` and 
`pyspark.sql.connect.functions.builtin` (`countDistinct`, ...), which have no 
`__all__` | SPARK-61115 |
+| `pyspark.sql.connect`: `functions.partitioning`, `avro.functions`, 
`protobuf.functions`, `table_arg`, `datasource`, `utils`, `logging`, `_typing` 
| SPARK-61115 |
+| `pyspark.sql.connect.plan` `Read`/`Range`/`SQL` (placeholders), 
`Column.to_plan`, `pyspark.sql.connect.types` helpers, 
`pyspark.sql.connect.conversion` | SPARK-61116 |
+| `SparkSession.Hook`: the hooks of `_registerHook` are not called | 
SPARK-61117 |
+| `pyspark.pipelines`: `cli`, `init_cli`, `add_pipeline_analysis_context`, 
`block_session_mutations` | SPARK-61118 |
+
+## Not gaps
+
+- **Server-side modules**, which run on the server's Python worker (official 
PySpark there):
+  - `pyspark.daemon`, `pyspark.worker`, `pyspark.worker_message`
+  - `pyspark.sql.streaming.python_streaming_source_runner`, 
`pyspark.sql.streaming.proto.StateMessage_pb2`, 
`stateful_processor_api_client.SCALAR_TYPES`
+  - `pyspark.sql.connect.streaming.worker.*`
+- **RDD and classic internals**, of which a Connect client has no use:
+  - modules: `pyspark.join`, `rddsampler`, `resultiterable`, `shuffle`, 
`statcounter`, and the RDD-based `pyspark.mllib`, most of which does not import 
in `pyspark-client` either
+  - the py4j converters of `pyspark.sql.types` (`DateConverter`, 
`DatetimeConverter`, `DatetimeNTZConverter`, `DayTimeIntervalTypeConverter`, 
`NumpyArrayConverter`, `NumpyScalarConverter`, `TimeConverter`)
+  - the classic dispatch helpers of `pyspark.sql.utils` (`dispatch_*_method`, 
`try_remote_*`, `to_java_array`, `to_scala_map`, `get_active_spark_context`, 
`ForeachBatchFunction`)
+  - `pyspark.sql.group.dfapi` and `df_varargs_api`, `PandasGroupedOpsMixin`, 
`OptionUtils`, `JStreamingQueryListener`
+  - `StreamingCheckpointManager`, which PySpark documents as supported in 
Spark Classic only
+- **The gRPC client**, which the Rust transport replaces:
+  - `pyspark.sql.connect.client.{core, artifact, reattach, retries}` and 
`pyspark.sql.connect.proto.base_pb2_grpc`
+  - `convert_exception` and `convert_observation_errors` of 
`pyspark.errors.exceptions.connect`
+  - the plan and expression classes of `pyspark.sql.connect.plan` and 
`pyspark.sql.connect.expressions`, which the Rust core builds (except those 
that SPARK-61111 and SPARK-61116 port), and `WindowFrame`, 
`StreamingQueryListenerBus`
+- **Tools**, not part of the client API:
+  - `pyspark.errors_doc_gen`, `pyspark.install`, `pyspark.threaddump`
+  - `pyspark.sql.connect.local_server`, which the classic builder uses to 
start a local server
+  - `pyspark.sql.transpile`, which the classic `UserDefinedFunction` uses
+- **Types of the Rust core**: `DataTypeSingleton`, the metaclass of the 
singleton types of PySpark.
+- **`pyspark.ml.connect`**: this client's is the native ML API of the Rust 
core (see `docs/compatibility.md`), not the deprecated torch-based one. So its 
models are `MLModel`, without `LogisticRegressionModel`, `MaxAbsScalerModel`, 
`StandardScalerModel` or `ArrayAssembler`.
+- **Submodule attributes** that differ only because of import order 
(`pyspark.find_spark_home`, `pyspark.sql.conversion`, 
`pyspark.sql.streaming.state`, ...). The modules import.
diff --git a/python/.coveragerc b/python/.coveragerc
index 9cb8189..ffd1a5a 100644
--- a/python/.coveragerc
+++ b/python/.coveragerc
@@ -58,6 +58,9 @@ omit =
     */pyspark/memory_profiler_ext.py
     */pyspark/worker_util.py
     */pyspark/instrumentation_utils.py
+    # The logger and the types of the connect import path, vendored.
+    */pyspark/sql/connect/logging.py
+    */pyspark/sql/connect/_typing.py
     */pyspark/_pyspark*
     */tests/*
 
diff --git a/python/pyspark/sql/avro/__init__.py 
b/python/pyspark/sql/avro/__init__.py
index cce3aca..ac530a5 100644
--- a/python/pyspark/sql/avro/__init__.py
+++ b/python/pyspark/sql/avro/__init__.py
@@ -14,3 +14,5 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #
+
+__all__ = ["functions"]
diff --git a/python/pyspark/sql/connect/_typing.py 
b/python/pyspark/sql/connect/_typing.py
new file mode 100644
index 0000000..6465938
--- /dev/null
+++ b/python/pyspark/sql/connect/_typing.py
@@ -0,0 +1,87 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+from types import FunctionType
+from typing import Any, Callable, Iterable, Iterator, Union, Optional, 
NewType, Protocol, Tuple
+import datetime
+import decimal
+
+import pyarrow
+from pandas.core.frame import DataFrame as PandasDataFrame
+
+from pyspark.sql.column import Column
+from pyspark.sql.connect.types import DataType
+from pyspark.sql.streaming.state import GroupState
+
+ColumnOrName = Union[Column, str]
+
+ColumnOrNameOrOrdinal = Union[Column, str, int]
+
+PrimitiveType = Union[bool, float, int, str]
+
+OptionalPrimitiveType = Optional[PrimitiveType]
+
+LiteralType = PrimitiveType
+
+DecimalLiteral = decimal.Decimal
+
+DateTimeLiteral = Union[datetime.date, datetime.time, datetime.datetime]
+
+DataTypeOrString = Union[DataType, str]
+
+DataFrameLike = PandasDataFrame
+
+PandasMapIterFunction = Callable[[Iterator[DataFrameLike]], 
Iterator[DataFrameLike]]
+
+ArrowMapIterFunction = Callable[[Iterator[pyarrow.RecordBatch]], 
Iterator[pyarrow.RecordBatch]]
+
+PandasGroupedMapFunction = Union[
+    Callable[[DataFrameLike], DataFrameLike],
+    Callable[[Any, DataFrameLike], DataFrameLike],
+]
+
+GroupedMapPandasUserDefinedFunction = 
NewType("GroupedMapPandasUserDefinedFunction", FunctionType)
+
+PandasCogroupedMapFunction = Callable[[DataFrameLike, DataFrameLike], 
DataFrameLike]
+
+PandasGroupedMapFunctionWithState = Callable[
+    [Any, Iterable[DataFrameLike], GroupState], Iterable[DataFrameLike]
+]
+ArrowGroupedMapFunction = Union[
+    Callable[[pyarrow.Table], pyarrow.Table],
+    Callable[[Tuple[pyarrow.Scalar, ...], pyarrow.Table], pyarrow.Table],
+]
+ArrowCogroupedMapFunction = Union[
+    Callable[[pyarrow.Table, pyarrow.Table], pyarrow.Table],
+    Callable[[Tuple[pyarrow.Scalar, ...], pyarrow.Table, pyarrow.Table], 
pyarrow.Table],
+]
+
+
+class UserDefinedFunctionLike(Protocol):
+    func: Callable[..., Any]
+    evalType: int
+    deterministic: bool
+
+    @property
+    def returnType(self) -> DataType: ...
+
+    def __call__(self, *args: ColumnOrName) -> Column: ...
+
+    def asNondeterministic(self) -> "UserDefinedFunctionLike": ...
+
+
+class UserDefinedFunctionCallable(Protocol):
+    def __call__(self, *_: ColumnOrName) -> Column: ...
diff --git a/python/pyspark/sql/protobuf/__init__.py 
b/python/pyspark/sql/connect/avro/functions.py
similarity index 75%
copy from python/pyspark/sql/protobuf/__init__.py
copy to python/pyspark/sql/connect/avro/functions.py
index cce3aca..6429b3e 100644
--- a/python/pyspark/sql/protobuf/__init__.py
+++ b/python/pyspark/sql/connect/avro/functions.py
@@ -14,3 +14,8 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #
+"""The Avro functions under the connect import path, as PySpark Connect's
+``pyspark.sql.connect.avro.functions``: those of 
``pyspark.sql.avro.functions``, which the Rust
+core builds."""
+
+from pyspark.sql.avro.functions import from_avro, to_avro  # noqa: F401
diff --git a/python/pyspark/sql/connect/datasource.py 
b/python/pyspark/sql/connect/datasource.py
new file mode 100644
index 0000000..c835a28
--- /dev/null
+++ b/python/pyspark/sql/connect/datasource.py
@@ -0,0 +1,46 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""The registration of data sources under the connect import path, as PySpark 
Connect's
+``pyspark.sql.connect.datasource``: a wrapper of a session whose ``register`` 
registers a
+data source with the session, as ``spark.dataSource.register`` does."""
+
+from typing import Type, TYPE_CHECKING
+
+from pyspark.sql.datasource import DataSourceRegistration as 
PySparkDataSourceRegistration
+
+if TYPE_CHECKING:
+    from pyspark.sql.datasource import DataSource
+    from pyspark.sql.session import SparkSession
+
+
+class DataSourceRegistration:
+    """
+    Wrapper for data source registration.
+
+    .. versionadded:: 4.0.0
+    """
+
+    def __init__(self, sparkSession: "SparkSession"):
+        self.sparkSession = sparkSession
+
+    def register(
+        self,
+        dataSource: Type["DataSource"],
+    ) -> None:
+        self.sparkSession.dataSource.register(dataSource)
+
+    register.__doc__ = PySparkDataSourceRegistration.register.__doc__
diff --git a/python/pyspark/sql/connect/functions/__init__.py 
b/python/pyspark/sql/connect/functions/__init__.py
index 67aba57..896fb50 100644
--- a/python/pyspark/sql/connect/functions/__init__.py
+++ b/python/pyspark/sql/connect/functions/__init__.py
@@ -17,3 +17,4 @@
 """Re-export of ``pyspark.sql.functions`` under the Connect import path."""
 from pyspark.sql.functions import *  # noqa: F401,F403
 from pyspark.sql.functions import col, expr  # noqa: F401
+from pyspark.sql.connect.functions import partitioning  # noqa: F401
diff --git a/python/pyspark/sql/connect/functions/builtin.py 
b/python/pyspark/sql/connect/functions/builtin.py
index e0234da..4420628 100644
--- a/python/pyspark/sql/connect/functions/builtin.py
+++ b/python/pyspark/sql/connect/functions/builtin.py
@@ -17,7 +17,9 @@
 """Re-export of ``pyspark.sql.functions`` under the Connect import path."""
 from typing import TYPE_CHECKING
 
-from pyspark.sql.functions import *  # noqa: F401,F403
+# Every public name of pyspark.sql.functions.builtin, also the deprecated 
functions that
+# ``pyspark.sql.functions.__all__`` leaves out, as PySpark Connect's builtin 
has them.
+from pyspark.sql.functions.builtin import *  # noqa: F401,F403
 from pyspark.sql.functions import col, expr, _invoke_function, _to_col  # 
noqa: F401
 
 if TYPE_CHECKING:
diff --git a/python/pyspark/sql/protobuf/__init__.py 
b/python/pyspark/sql/connect/functions/partitioning.py
similarity index 71%
copy from python/pyspark/sql/protobuf/__init__.py
copy to python/pyspark/sql/connect/functions/partitioning.py
index cce3aca..1fb7d79 100644
--- a/python/pyspark/sql/protobuf/__init__.py
+++ b/python/pyspark/sql/connect/functions/partitioning.py
@@ -14,3 +14,8 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #
+"""The partition transform functions under the connect import path, as PySpark 
Connect's
+``pyspark.sql.connect.functions.partitioning``: those of 
``pyspark.sql.functions.partitioning``,
+which the Rust core builds."""
+
+from pyspark.sql.functions.partitioning import bucket, days, hours, months, 
years  # noqa: F401
diff --git a/python/pyspark/sql/connect/logging.py 
b/python/pyspark/sql/connect/logging.py
new file mode 100644
index 0000000..099193f
--- /dev/null
+++ b/python/pyspark/sql/connect/logging.py
@@ -0,0 +1,67 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+
+import logging
+from pyspark.logger import PySparkLogger
+import os
+from typing import Optional
+
+__all__ = ["configureLogging", "getLogLevel"]
+
+
+def configureLogging(level: Optional[str] = None) -> logging.Logger:
+    """
+    Configure log level for Spark Connect components.
+    When not specified as a parameter, log level will be configured based on
+    the SPARK_CONNECT_LOG_LEVEL environment variable.
+    When both are absent, logging is disabled.
+
+    .. versionadded:: 4.0.0
+    """
+    logger = PySparkLogger.getLogger(__name__)
+    handler = logging.StreamHandler()
+    handler.setFormatter(
+        logging.Formatter(fmt="%(asctime)s %(process)d %(levelname)s 
%(funcName)s %(message)s")
+    )
+    logger.addHandler(handler)
+
+    if level is not None:
+        logger.setLevel(level.upper())
+    elif "SPARK_CONNECT_LOG_LEVEL" in os.environ:
+        logger.setLevel(os.environ["SPARK_CONNECT_LOG_LEVEL"].upper())
+    else:
+        logger.disabled = True
+    return logger
+
+
+# Instantiate the logger based on the environment configuration.
+logger = configureLogging()
+
+
+def getLogLevel() -> Optional[int]:
+    """
+    This returns this log level as integer, or none (if no logging is enabled).
+
+    Spark Connect logging can be configured with environment variable 
'SPARK_CONNECT_LOG_LEVEL'
+
+    .. versionadded:: 3.5.0
+    """
+
+    if not logger.disabled:
+        return logger.level
+    return None
diff --git a/python/pyspark/sql/protobuf/__init__.py 
b/python/pyspark/sql/connect/protobuf/functions.py
similarity index 73%
copy from python/pyspark/sql/protobuf/__init__.py
copy to python/pyspark/sql/connect/protobuf/functions.py
index cce3aca..33c618b 100644
--- a/python/pyspark/sql/protobuf/__init__.py
+++ b/python/pyspark/sql/connect/protobuf/functions.py
@@ -14,3 +14,8 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #
+"""The Protobuf functions under the connect import path, as PySpark Connect's
+``pyspark.sql.connect.protobuf.functions``: those of 
``pyspark.sql.protobuf.functions``, which the
+Rust core builds."""
+
+from pyspark.sql.protobuf.functions import from_protobuf, to_protobuf  # noqa: 
F401
diff --git a/python/pyspark/sql/protobuf/__init__.py 
b/python/pyspark/sql/connect/table_arg.py
similarity index 76%
copy from python/pyspark/sql/protobuf/__init__.py
copy to python/pyspark/sql/connect/table_arg.py
index cce3aca..d3c1445 100644
--- a/python/pyspark/sql/protobuf/__init__.py
+++ b/python/pyspark/sql/connect/table_arg.py
@@ -14,3 +14,8 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #
+"""The table argument under the connect import path, as PySpark Connect's
+``pyspark.sql.connect.table_arg``: the ``TableArg`` of 
``pyspark.sql.table_arg``, of
+``DataFrame.asTable()``."""
+
+from pyspark.sql.table_arg import TableArg  # noqa: F401
diff --git a/python/pyspark/sql/connect/utils.py 
b/python/pyspark/sql/connect/utils.py
new file mode 100644
index 0000000..ce09bba
--- /dev/null
+++ b/python/pyspark/sql/connect/utils.py
@@ -0,0 +1,124 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""The utilities of the connect import path, as PySpark Connect's 
``pyspark.sql.connect.utils``,
+from Apache Spark v4.3.0. ``check_dependencies`` checks the dependencies of 
this client."""
+
+import sys
+
+from pyspark.loose_version import LooseVersion
+from pyspark.sql.pandas.utils import require_minimum_pandas_version, 
require_minimum_pyarrow_version
+from pyspark.errors import PySparkImportError
+
+
+def check_dependencies() -> None:
+    main_module = sys.modules["__main__"]
+    if (
+        main_module.__spec__ is None
+        and not hasattr(main_module, "__file__")
+        and not hasattr(sys, "ps1")
+    ):
+        # The main module is not initialized at all at this point. We must be 
running doctests.
+        from pyspark.testing.connectutils import should_test_connect, 
connect_requirement_message
+
+        if not should_test_connect:
+            print(
+                f"Skipping doctests: {connect_requirement_message}",
+                file=sys.stderr,
+            )
+            sys.exit(0)
+
+    # The dependencies of this client: its transport is of the Rust extension, 
so it needs no
+    # grpcio, grpcio-status, googleapis-common-protos or zstandard, which 
PySpark checks.
+    require_minimum_pandas_version()
+    require_minimum_pyarrow_version()
+
+
+def require_minimum_grpc_version() -> None:
+    """Raise ImportError if minimum version of grpc is not installed"""
+    minimum_grpc_version = "1.48.1"
+
+    try:
+        import grpc
+    except ImportError as error:
+        raise PySparkImportError(
+            errorClass="PACKAGE_NOT_INSTALLED",
+            messageParameters={
+                "package_name": "grpcio",
+                "minimum_version": str(minimum_grpc_version),
+            },
+        ) from error
+    if LooseVersion(grpc.__version__) < LooseVersion(minimum_grpc_version):
+        raise PySparkImportError(
+            errorClass="UNSUPPORTED_PACKAGE_VERSION",
+            messageParameters={
+                "package_name": "grpcio",
+                "minimum_version": str(minimum_grpc_version),
+                "current_version": str(grpc.__version__),
+            },
+        )
+
+
+def require_minimum_grpcio_status_version() -> None:
+    """Raise ImportError if grpcio-status is not installed"""
+    minimum_grpc_version = "1.48.1"
+
+    try:
+        import grpc_status  # noqa
+    except ImportError as error:
+        raise PySparkImportError(
+            errorClass="PACKAGE_NOT_INSTALLED",
+            messageParameters={
+                "package_name": "grpcio-status",
+                "minimum_version": str(minimum_grpc_version),
+            },
+        ) from error
+
+
+def require_minimum_googleapis_common_protos_version() -> None:
+    """Raise ImportError if googleapis-common-protos is not installed"""
+    minimum_common_protos_version = "1.56.4"
+
+    try:
+        import google.rpc  # noqa
+    except ImportError as error:
+        raise PySparkImportError(
+            errorClass="PACKAGE_NOT_INSTALLED",
+            messageParameters={
+                "package_name": "googleapis-common-protos",
+                "minimum_version": str(minimum_common_protos_version),
+            },
+        ) from error
+
+
+def require_minimum_zstandard_version() -> None:
+    """Raise ImportError if zstandard is not installed"""
+    minimum_zstandard_version = "0.25.0"
+
+    try:
+        import zstandard  # noqa
+    except ImportError as error:
+        raise PySparkImportError(
+            errorClass="PACKAGE_NOT_INSTALLED",
+            messageParameters={
+                "package_name": "zstandard",
+                "minimum_version": str(minimum_zstandard_version),
+            },
+        ) from error
+
+
+def get_python_ver() -> str:
+    return "%d.%d" % sys.version_info[:2]
diff --git a/python/pyspark/sql/functions.py b/python/pyspark/sql/functions.py
index 7e2034d..c6334b0 100644
--- a/python/pyspark/sql/functions.py
+++ b/python/pyspark/sql/functions.py
@@ -1325,11 +1325,11 @@ del name
 # ``__init__`` re-exports them; here this module IS the implementation, so 
register a synthetic
 # ``builtin`` submodule mirroring it (so ``import 
pyspark.sql.functions.builtin`` and
 # ``from pyspark.sql.functions.builtin import col`` work like reference 
pyspark).
+# As PySpark's, it has every public name of the functions, also the deprecated 
ones that
+# ``__all__`` leaves out (such as ``countDistinct``), and no ``__all__``.
 builtin = _types.ModuleType("pyspark.sql.functions.builtin")
 builtin.__doc__ = "Built-in DataFrame functions (re-exported from 
pyspark.sql.functions)."
-for _bname in list(__all__):
-    _bobj = globals().get(_bname)
-    if _bobj is not None:
+for _bname, _bobj in list(globals().items()):
+    if not _bname.startswith("_") and not isinstance(_bobj, _types.ModuleType):
         setattr(builtin, _bname, _bobj)
-builtin.__all__ = list(__all__)
 _sys.modules["pyspark.sql.functions.builtin"] = builtin
diff --git a/python/pyspark/sql/protobuf/__init__.py 
b/python/pyspark/sql/protobuf/__init__.py
index cce3aca..ac530a5 100644
--- a/python/pyspark/sql/protobuf/__init__.py
+++ b/python/pyspark/sql/protobuf/__init__.py
@@ -14,3 +14,5 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #
+
+__all__ = ["functions"]
diff --git a/python/pyspark/sql/udf.py b/python/pyspark/sql/udf.py
index 6c900e2..3759b26 100644
--- a/python/pyspark/sql/udf.py
+++ b/python/pyspark/sql/udf.py
@@ -31,6 +31,8 @@ from pyspark.sql.types import DataType, StringType, StructType
 from pyspark.serializers import CloudPickleSerializer
 from pyspark.util import PythonEvalType
 
+__all__ = ["UDFRegistration"]
+
 
 def _as_concrete_datatype(returnType: Any) -> DataType:
     """Return a concrete, picklable pyspark ``DataType``.
diff --git a/python/pyspark/sql/udtf.py b/python/pyspark/sql/udtf.py
index 4c71345..b13f96e 100644
--- a/python/pyspark/sql/udtf.py
+++ b/python/pyspark/sql/udtf.py
@@ -32,6 +32,16 @@ from typing import Any, Optional, Type
 from pyspark.errors import PySparkAttributeError, PySparkTypeError
 from pyspark.serializers import CloudPickleSerializer
 
+__all__ = [
+    "AnalyzeArgument",
+    "AnalyzeResult",
+    "PartitioningColumn",
+    "OrderingColumn",
+    "SelectedColumn",
+    "SkipRestOfInputTableException",
+    "UDTFRegistration",
+]
+
 # Eval types for Python UDTFs (mirrors pyspark.util.PythonEvalType).
 SQL_TABLE_UDF = 300
 SQL_ARROW_TABLE_UDF = 301
diff --git a/python/pyspark/util.py b/python/pyspark/util.py
index cf21945..5708a51 100644
--- a/python/pyspark/util.py
+++ b/python/pyspark/util.py
@@ -643,3 +643,73 @@ def _do_server_auth(conn: "io.IOBase", auth_secret: str) 
-> None:
             errorClass="UNEXPECTED_RESPONSE_FROM_SERVER",
             messageParameters={},
         )
+
+
+# The names of PySpark's pyspark.util that its other modules and tools use, 
from Apache Spark
+# v4.3.0.
+import gc  # noqa: E402
+from typing import Generator, List, TypeVar  # noqa: E402
+
+__all__: List[str] = []
+
+FuncT = TypeVar("FuncT", bound=Callable[..., Any])
+
+
+def fail_on_stopiteration(f: Callable) -> Callable:
+    """
+    Wraps the input function to fail on 'StopIteration' by raising a 
'RuntimeError'
+    prevents silent loss of data when 'f' is used in a for loop in Spark code
+    """
+
+    def wrapper(*args: Any, **kwargs: Any) -> Any:
+        try:
+            return f(*args, **kwargs)
+        except StopIteration as exc:
+            raise PySparkRuntimeError(
+                errorClass="STOP_ITERATION_OCCURRED",
+                messageParameters={
+                    "exc": str(exc),
+                },
+            )
+
+    return wrapper
+
+
[email protected]
+def disable_gc() -> Generator[None, None, None]:
+    gc_enabled_originally = gc.isenabled()
+    if gc_enabled_originally:
+        gc.disable()
+    try:
+        yield
+    finally:
+        if gc_enabled_originally:
+            gc.enable()
+
+
+# This function will be called in `pyspark` script as well,
+# so this should be available without a running Spark.
+# If move or rename, please update the script too.
+def spark_connect_mode() -> str:
+    """
+    Return the env var SPARK_CONNECT_MODE; otherwise "1" if `pyspark_connect` 
is available.
+    """
+    connect_by_default = os.environ.get("SPARK_CONNECT_MODE")
+    if connect_by_default is not None:
+        return connect_by_default
+    try:
+        import pyspark_connect  # noqa: F401
+
+        return "1"
+    except ImportError:
+        return "0"
+
+
+def default_api_mode() -> str:
+    """
+    Return the default API mode.
+    """
+    if spark_connect_mode() == "1":
+        return "connect"
+    else:
+        return "classic"
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 7f3bf3f..9a7f7c7 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -912,13 +912,13 @@ def test_functions_exports_fresh_process():
             assert all(namespace[name] is getattr(F, name) for name in 
F.__all__)
             builtin = importlib.import_module("pyspark.sql.functions.builtin")
             assert builtin is F.builtin is sys.modules[builtin.__name__]
-            assert type(builtin.__all__) is list
-            assert builtin.__all__ == F.__all__
-            assert builtin.__all__ is not F.__all__
+            # As PySpark's, the builtin module has no __all__, and has every 
public function of
+            # pyspark.sql.functions, also the deprecated ones that __all__ 
leaves out.
+            assert not hasattr(builtin, "__all__")
             assert all(getattr(builtin, name) is getattr(F, name) for name in 
F.__all__)
             builtin_namespace = {}
             exec("from pyspark.sql.functions.builtin import *", 
builtin_namespace)
-            assert [name for name in builtin_namespace if name != 
"__builtins__"] == F.__all__
+            assert set(F.__all__) <= set(builtin_namespace)
             assert all(builtin_namespace[name] is getattr(F, name) for name in 
F.__all__)
             part = 
importlib.import_module("pyspark.sql.functions.partitioning")
             assert part is F.partitioning is sys.modules[part.__name__]
@@ -935,9 +935,10 @@ def test_functions_exports_fresh_process():
         expected = check_exports()
         old_all = F.__all__
         old_all.append("caller_added_name")
-        F.builtin.__all__.append("builtin_only_name")
+        F.builtin.builtin_only_name = None
         importlib.reload(F)
         assert F.__all__ is not old_all
+        assert not hasattr(F.builtin, "builtin_only_name")
         assert check_exports() == expected
     ''').replace("__OFFICIAL_FUNCTIONS_ALL__", repr(OFFICIAL_FUNCTIONS_ALL))], 
capture_output=True, text=True)
     assert result.returncode == 0, result.stdout + result.stderr
@@ -5150,6 +5151,186 @@ def test_connect_shim_imports():
     assert repr(_invoke_function_over_columns("pandas_product", F.col("x"))) 
== "Column<'pandas_product(x)'>"
 
 
+def test_module_level_names_as_pyspark(monkeypatch):
+    # As official PySpark 4.3.0: the names of pyspark.util, and the __all__ of 
the modules of
+    # pyspark.sql, of which a star import exports only those.
+    import gc
+    import typing
+
+    import pyspark.util as util
+    from pyspark.errors import PySparkRuntimeError
+
+    assert util.__all__ == []
+    assert isinstance(util.FuncT, typing.TypeVar)
+    monkeypatch.setenv("SPARK_CONNECT_MODE", "1")
+    assert util.spark_connect_mode() == "1" and util.default_api_mode() == 
"connect"
+    monkeypatch.setenv("SPARK_CONNECT_MODE", "0")
+    assert util.spark_connect_mode() == "0" and util.default_api_mode() == 
"classic"
+
+    def stops():
+        raise StopIteration("done")
+
+    assert util.fail_on_stopiteration(lambda x: x + 1)(1) == 2
+    with pytest.raises(PySparkRuntimeError) as error:
+        util.fail_on_stopiteration(stops)()
+    assert error.value.getCondition() == "STOP_ITERATION_OCCURRED"
+    assert error.value.getMessageParameters() == {"exc": "done"}
+    enabled = gc.isenabled()
+    with util.disable_gc():
+        assert not gc.isenabled()
+    assert gc.isenabled() == enabled
+
+    for module, expected in [
+        ("pyspark.sql.udf", ["UDFRegistration"]),
+        (
+            "pyspark.sql.udtf",
+            [
+                "AnalyzeArgument",
+                "AnalyzeResult",
+                "PartitioningColumn",
+                "OrderingColumn",
+                "SelectedColumn",
+                "SkipRestOfInputTableException",
+                "UDTFRegistration",
+            ],
+        ),
+        ("pyspark.sql.avro", ["functions"]),
+        ("pyspark.sql.protobuf", ["functions"]),
+    ]:
+        namespace = {}
+        exec(f"from {module} import *", namespace)
+        assert [name for name in namespace if name != "__builtins__"] == 
expected, module
+
+
+def test_connect_import_path_modules_as_pyspark(mock_connect):
+    # As official PySpark 4.3.0: the modules of the connect import path, of 
the functions and
+    # the classes of this client, and the deprecated functions of the builtin 
modules.
+    import importlib
+    import logging
+
+    from pyspark.sql import functions as F
+    from pyspark.sql.avro import functions as avro
+    from pyspark.sql.datasource import DataSource
+    from pyspark.sql.functions import partitioning
+    from pyspark.sql.protobuf import functions as protobuf
+    from pyspark.sql.table_arg import TableArg
+
+    spark, service = mock_connect
+    connect_partitioning = 
importlib.import_module("pyspark.sql.connect.functions.partitioning")
+    assert 
importlib.import_module("pyspark.sql.connect.functions").partitioning is (
+        connect_partitioning
+    )
+    for name in ["years", "months", "days", "hours", "bucket"]:
+        assert getattr(connect_partitioning, name) is getattr(partitioning, 
name)
+    connect_avro = 
importlib.import_module("pyspark.sql.connect.avro.functions")
+    assert connect_avro.from_avro is avro.from_avro and connect_avro.to_avro 
is avro.to_avro
+    connect_protobuf = 
importlib.import_module("pyspark.sql.connect.protobuf.functions")
+    assert connect_protobuf.from_protobuf is protobuf.from_protobuf
+    assert connect_protobuf.to_protobuf is protobuf.to_protobuf
+    assert importlib.import_module("pyspark.sql.connect.table_arg").TableArg 
is TableArg
+    typing_module = importlib.import_module("pyspark.sql.connect._typing")
+    assert {"ColumnOrName", "DataTypeOrString", "LiteralType"} <= 
set(dir(typing_module))
+
+    connect_logging = importlib.import_module("pyspark.sql.connect.logging")
+    assert connect_logging.__all__ == ["configureLogging", "getLogLevel"]
+    assert isinstance(connect_logging.logger, logging.Logger)
+    if "SPARK_CONNECT_LOG_LEVEL" not in os.environ:
+        assert connect_logging.getLogLevel() is None
+
+    connect_builtin = 
importlib.import_module("pyspark.sql.connect.functions.builtin")
+    for name in [
+        "approxCountDistinct",
+        "bitwiseNOT",
+        "countDistinct",
+        "shiftLeft",
+        "shiftRight",
+        "shiftRightUnsigned",
+        "sumDistinct",
+        "toDegrees",
+        "toRadians",
+        "chr",
+        "random",
+        "uuid",
+    ]:
+        assert getattr(F.builtin, name) is getattr(F, name), name
+        assert getattr(connect_builtin, name) is getattr(F, name), name
+
+    # The registration of a data source registers it with the session.
+    from pyspark.sql.connect.datasource import DataSourceRegistration
+
+    class TestSource(DataSource):
+        @classmethod
+        def name(cls):
+            return "test_source"
+
+    DataSourceRegistration(spark).register(TestSource)
+    assert service.execute_calls == 1
+
+
+def test_connect_utils_check_the_dependencies_of_the_client(tmp_path):
+    # As PySpark Connect's pyspark.sql.connect.utils, but check_dependencies 
checks the
+    # dependencies of this client, which do not include gRPC. The script runs 
of a file: of
+    # a main module without one, check_dependencies takes it for doctests.
+    import subprocess
+    import sys
+    import textwrap
+
+    from pyspark.errors import PySparkImportError
+    from pyspark.sql.connect import utils
+
+    assert utils.get_python_ver() == "%d.%d" % sys.version_info[:2]
+    utils.check_dependencies()
+    for check in [
+        utils.require_minimum_grpc_version,
+        utils.require_minimum_grpcio_status_version,
+        utils.require_minimum_googleapis_common_protos_version,
+        utils.require_minimum_zstandard_version,
+    ]:
+        # Of the packages installed here, or not.
+        try:
+            check()
+        except PySparkImportError as error:
+            assert error.getCondition() in ("PACKAGE_NOT_INSTALLED", 
"UNSUPPORTED_PACKAGE_VERSION")
+    script = textwrap.dedent(
+        """
+        import importlib.abc, sys
+
+        class Block(importlib.abc.MetaPathFinder):
+            def find_spec(self, name, path, target=None):
+                if name.split(".")[0] in ("grpc", "grpc_status", "zstandard") 
or name.startswith(
+                    "google.rpc"
+                ):
+                    raise ImportError("not installed: " + name)
+
+        sys.meta_path.insert(0, Block())
+        from pyspark.errors import PySparkImportError
+        from pyspark.sql.connect import utils
+
+        utils.check_dependencies()
+        for check, package in [
+            (utils.require_minimum_grpc_version, "grpcio"),
+            (utils.require_minimum_grpcio_status_version, "grpcio-status"),
+            (utils.require_minimum_googleapis_common_protos_version, 
"googleapis-common-protos"),
+            (utils.require_minimum_zstandard_version, "zstandard"),
+        ]:
+            try:
+                check()
+            except PySparkImportError as error:
+                assert error.getCondition() == "PACKAGE_NOT_INSTALLED"
+                assert error.getMessageParameters()["package_name"] == package
+            else:
+                raise AssertionError(package)
+        print("ok")
+        """
+    )
+    path = tmp_path / "check_dependencies.py"
+    path.write_text(script)
+    result = subprocess.run(
+        [sys.executable, str(path)], capture_output=True, text=True, 
timeout=120
+    )
+    assert result.returncode == 0 and result.stdout.strip() == "ok", 
result.stderr
+
+
 def test_is_remote_flags():
     # The Rust-backed drop-in is always Connect-only.
     from pyspark.util import is_remote_only
diff --git a/scripts/check_vendored_upstream.sh 
b/scripts/check_vendored_upstream.sh
index ecef653..bb31693 100644
--- a/scripts/check_vendored_upstream.sh
+++ b/scripts/check_vendored_upstream.sh
@@ -92,6 +92,10 @@ VENDORED_FILES=(
   # ExecutePlanResponse is that of the message class of 
pyspark/sql/connect/proto.py.
   "sql/connect/shell/progress.py"
 
+  # The logger of Spark Connect, and the types of the connect import path.
+  "sql/connect/logging.py"
+  "sql/connect/_typing.py"
+
   # The spatial reference systems of the spatial types, and the 
SQLStringFormatter, of
   # pyspark.sql.
   "sql/geo_utils.py"


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to