This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new a515753  [SPARK-60999][CONNECT][PYTHON] Create the UDTFs of udtf and 
arrow_udtf as PySpark does
a515753 is described below

commit a515753490f3294b4e1a432cd8dc0887459838d8
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Fri Nov 6 20:27:04 2026 +0900

    [SPARK-60999][CONNECT][PYTHON] Create the UDTFs of udtf and arrow_udtf as 
PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-60999](https://issues.apache.org/jira/browse/SPARK-60999): the 
Python client now creates the UDTFs of `udtf()` and `arrow_udtf()` as PySpark's 
`_create_py_udtf` and `_create_pyarrow_udtf` do.
    
    - `arrow_udtf()` makes a PyArrow-native UDTF (`SQL_ARROW_UDTF`), whose 
`eval()` takes `pyarrow` arrays and returns `pyarrow` tables. It made an 
Arrow-optimized UDTF (`SQL_ARROW_TABLE_UDF`) instead, which passes Python 
values.
    - `udtf()` without `useArrow` makes an Arrow-optimized UDTF when the active 
session enables `spark.sql.execution.pythonUDTF.arrow.enabled`, as the 4.3.0 
server does by default. Without a session, it makes a regular UDTF.
    - A UDTF is not deterministic unless `asDeterministic()` marks it so, in 
place, and the call sends that flag. It was deterministic, and 
`asDeterministic()` returned a copy.
    - `SparkSession.active()` raises `NO_ACTIVE_OR_DEFAULT_SESSION`, as 
PySpark's does, and so does a UDTF called without a session. It raised a plain 
`RuntimeError`.
    
    ### Why are the changes needed?
    
    Every `arrow_udtf` that used `pyarrow` failed, because its `eval()` was 
given Python values. For example, `pc.add(x, 1)` of a `PlusOne(lit(1))` returns 
`[Row(x=2, kind='Int32Array')]` in PySpark 4.3.0, and failed here. UDTFs also 
did not follow the Arrow configuration, so their eval type and determinism 
differed from PySpark's.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. `arrow_udtf()` and `udtf()` create the UDTFs of PySpark 4.3.0, and 
`SparkSession.active()` raises its error.
    
    ### How was this patch tested?
    
    - A new live test runs an `arrow_udtf` that uses `pyarrow`. It also passes 
with official PySpark 4.3.0, and fails without this change.
    - A new offline test checks the eval type and the determinism of the UDTFs 
that a mock Connect server receives:
      - `udtf()` follows the configuration, which the mock now serves with the 
server's default, and `useArrow` overrides it.
      - `arrow_udtf()` makes a `SQL_ARROW_UDTF`.
      - `asDeterministic()` marks the UDTF in place.
      - Only `udtf()` without `useArrow` reads the configuration.
    
      It fails without this change.
    - The test of a UDTF called without a session now runs in a new 
interpreter. It checks the eval type and the `NO_ACTIVE_OR_DEFAULT_SESSION` 
error.
    - Each offline test now starts without an active session, and the tests of 
the mock server have a session of it each, which is active while they run. A 
session that an earlier test left, of a server that does not listen, made 
`udtf()` wait on its configuration.
    - The tests of the eval types and the determinism of UDTFs now expect 
PySpark's.
    - The lib tests and every integration test, e2e included, against a local 
Spark 4.3.0 server (664 passed, and 125 of the transport crate), the build of 
all tests with `wasm-udf`, the offline Python suites (1158 passed, 8 skipped), 
and the live drop-in and Arrow suites (84 passed) and live cases (10 passed). 
Clippy reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #204 from HyukjinKwon/SPARK-60999-udtf-create.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/functions.rs   |   8 ++-
 crates/pyspark-rs/src/session.rs     |  12 +++-
 python/pyspark/sql/udtf.py           |  81 +++++++++++++++--------
 python/tests/_mock_connect_server.py |  20 ++++--
 python/tests/test_dropin_e2e.py      |  13 ++++
 python/tests/test_dropin_offline.py  | 123 ++++++++++++++++++++++++++++-------
 6 files changed, 195 insertions(+), 62 deletions(-)

diff --git a/crates/pyspark-rs/src/functions.rs 
b/crates/pyspark-rs/src/functions.rs
index 392357b..5a33a03 100644
--- a/crates/pyspark-rs/src/functions.rs
+++ b/crates/pyspark-rs/src/functions.rs
@@ -72,8 +72,11 @@ fn pyfunc_make_udf(
 /// - name/return_type/eval_type/command_bytes/python_ver: as for `make_udf`
 ///   (eval_type ∈ {300 SQL_TABLE_UDF, 301 SQL_ARROW_TABLE_UDF, 302 
SQL_ARROW_UDTF})
 /// - *args: Column arguments
+/// - deterministic: whether the UDTF is deterministic, which a PySpark UDTF 
is not unless
+///   `asDeterministic()` makes it so
 #[pyfunction]
-#[pyo3(signature = (session, name, return_type, eval_type, command_bytes, 
python_ver, *args))]
+#[pyo3(signature = (session, name, return_type, eval_type, command_bytes, 
python_ver, *args, deterministic=false))]
+#[allow(clippy::too_many_arguments)]
 fn pyfunc_make_udtf(
     session: PyRef<'_, PySparkSession>,
     name: String,
@@ -82,6 +85,7 @@ fn pyfunc_make_udtf(
     command_bytes: Vec<u8>,
     python_ver: String,
     args: Vec<Bound<'_, PyAny>>,
+    deterministic: bool,
 ) -> PyResult<PyDataFrame> {
     let mut col_args = Vec::new();
     for arg in args {
@@ -107,7 +111,7 @@ fn pyfunc_make_udtf(
         eval_type,
         command_bytes,
         python_ver,
-        true, // deterministic by default
+        deterministic,
     );
     Ok(PyDataFrame::new(df))
 }
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index 8dd873b..4e434a1 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -1089,11 +1089,17 @@ impl PySparkSession {
             .map(PySparkSession::new)
     }
 
-    /// The active session; errors if none. Mirrors `SparkSession.active`.
+    /// The active session; errors if none. Mirrors `SparkSession.active`, 
which raises
+    /// `NO_ACTIVE_OR_DEFAULT_SESSION`.
     #[staticmethod]
-    fn active() -> PyResult<PySparkSession> {
+    fn active(py: Python<'_>) -> PyResult<PySparkSession> {
         Self::get_active_session().ok_or_else(|| {
-            pyo3::exceptions::PyRuntimeError::new_err("No active SparkSession 
found")
+            crate::column::raise_pyspark(
+                py,
+                "PySparkRuntimeError",
+                "NO_ACTIVE_OR_DEFAULT_SESSION",
+                &[],
+            )
         })
     }
 
diff --git a/python/pyspark/sql/udtf.py b/python/pyspark/sql/udtf.py
index 47748f1..24f01a4 100644
--- a/python/pyspark/sql/udtf.py
+++ b/python/pyspark/sql/udtf.py
@@ -35,6 +35,7 @@ from pyspark.serializers import CloudPickleSerializer
 # Eval types for Python UDTFs (mirrors pyspark.util.PythonEvalType).
 SQL_TABLE_UDF = 300
 SQL_ARROW_TABLE_UDF = 301
+SQL_ARROW_UDTF = 302
 
 
 # Verbatim from PySpark (pyspark.sql.udtf): validates the handler class of a 
UDTF.
@@ -97,7 +98,7 @@ class UserDefinedTableFunction:
         returnType: Optional[Any] = None,
         name: Optional[str] = None,
         evalType: int = SQL_TABLE_UDF,
-        deterministic: bool = True,
+        deterministic: bool = False,
     ):
         # As PySpark's UserDefinedTableFunction, for udtf(), arrow_udtf() and 
register().
         _validate_udtf_handler(func, returnType)
@@ -113,39 +114,26 @@ class UserDefinedTableFunction:
         # worker command contains only the handler class, unlike scalar UDFs.
         self.command = CloudPickleSerializer().dumps(func)
 
-    def _active_session(self):
-        from pyspark.sql import SparkSession
-
-        session = SparkSession.getActiveSession()
-        if session is None:
-            raise RuntimeError(
-                "No active SparkSession; a UDTF must be called within an 
active session."
-            )
-        return session
-
     def __call__(self, *args: Any) -> Any:
         from pyspark import _pyspark
+        from pyspark.sql import SparkSession
 
         return _pyspark.functions.pyfunc_make_udtf(
-            self._active_session(),
+            SparkSession.active(),
             self.name,
             self.returnType,
             self.evalType,
             self.command,
             self.python_ver,
             *args,
+            deterministic=self.deterministic,
         )
 
     def asDeterministic(self) -> "UserDefinedTableFunction":
-        """Return a copy of this UDTF marked as deterministic. Mirrors
+        """Mark this UDTF as deterministic, and return it. Mirrors
         ``UserDefinedTableFunction.asDeterministic``."""
-        return UserDefinedTableFunction(
-            self.func,
-            returnType=self.returnType,
-            name=self.name,
-            evalType=self.evalType,
-            deterministic=True,
-        )
+        self.deterministic = True
+        return self
 
 
 def udtf(
@@ -157,11 +145,49 @@ def udtf(
     """Create a user-defined table function (UDTF).
 
     Use as ``@udtf(returnType=...)`` on a handler class, or call
-    ``udtf(MyClass, returnType=...)`` directly.
+    ``udtf(MyClass, returnType=...)`` directly. As PySpark's 
``_create_py_udtf``, the UDTF is
+    Arrow-optimized when ``useArrow`` is True, or is not given and
+    ``spark.sql.execution.pythonUDTF.arrow.enabled`` of the active session is 
true.
     """
-    eval_type = SQL_ARROW_TABLE_UDF if useArrow is True else SQL_TABLE_UDF
 
     def _udtf_decorator(handler: Type) -> UserDefinedTableFunction:
+        if useArrow is not None:
+            arrow_enabled = useArrow
+        else:
+            from pyspark.errors import PySparkRuntimeError
+            from pyspark.sql import SparkSession
+
+            arrow_enabled = False
+            try:
+                session = SparkSession.active()
+                arrow_enabled = (
+                    
str(session.conf.get("spark.sql.execution.pythonUDTF.arrow.enabled")).lower()
+                    == "true"
+                )
+            except PySparkRuntimeError as e:
+                if e.getCondition() == "NO_ACTIVE_OR_DEFAULT_SESSION":
+                    pass  # Just uses the default if no session found.
+                else:
+                    raise e
+        eval_type = SQL_TABLE_UDF
+        if arrow_enabled:
+            try:
+                from pyspark.sql.pandas.utils import (
+                    require_minimum_pandas_version,
+                    require_minimum_pyarrow_version,
+                )
+
+                require_minimum_pandas_version()
+                require_minimum_pyarrow_version()
+                eval_type = SQL_ARROW_TABLE_UDF
+            except ImportError as e:
+                import warnings
+
+                warnings.warn(
+                    f"Arrow optimization for Python UDTFs cannot be enabled: 
{str(e)}. "
+                    f"Falling back to using regular Python UDTFs.",
+                    UserWarning,
+                )
         return UserDefinedTableFunction(handler, returnType=returnType, 
evalType=eval_type)
 
     if cls is not None:
@@ -176,14 +202,15 @@ def arrow_udtf(
 ) -> Any:
     """Create an Arrow-based user-defined table function (UDTF).
 
-    Mirrors ``pyspark.sql.functions.arrow_udtf``: like :func:`udtf` but the 
handler
-    operates on Arrow data (``SQL_ARROW_TABLE_UDF``).
+    Mirrors ``pyspark.sql.functions.arrow_udtf``: a PyArrow-native UDTF 
(``SQL_ARROW_UDTF``),
+    whose ``eval`` takes ``pyarrow`` arrays and returns ``pyarrow`` tables.
     """
     def _arrow_udtf_decorator(handler: Type) -> UserDefinedTableFunction:
+        from pyspark.sql.pandas.utils import require_minimum_pyarrow_version
+
+        require_minimum_pyarrow_version()
         _validate_arrow_udtf_handler(handler, returnType)
-        return UserDefinedTableFunction(
-            handler, returnType=returnType, evalType=SQL_ARROW_TABLE_UDF
-        )
+        return UserDefinedTableFunction(handler, returnType=returnType, 
evalType=SQL_ARROW_UDTF)
 
     if cls is not None:
         return _arrow_udtf_decorator(cls)
diff --git a/python/tests/_mock_connect_server.py 
b/python/tests/_mock_connect_server.py
index 46278a0..96599bb 100644
--- a/python/tests/_mock_connect_server.py
+++ b/python/tests/_mock_connect_server.py
@@ -158,11 +158,13 @@ def serve(connection):
         # ErrorInfo of `reason` and `metadata` in grpc-status-details-bin.
         execute_error = None
         # FetchErrorDetails returns `error_details` (raw 
FetchErrorDetailsResponse fields) and
-        # records the error id asked for. Config Get returns the values in 
`confs`, and fails
-        # for a key without one, as the server does for an unset config; it 
records the keys.
+        # records the error id asked for. Config Get returns the values in 
`confs`, or else
+        # those of `default_confs`, the defaults of SQL configs of the server, 
and fails for a
+        # key without one, as the server does for an unset config; it records 
the keys.
         error_details = b""
         fetched_error_ids = []
         confs = {}
+        default_confs = {"spark.sql.execution.pythonUDTF.arrow.enabled": 
"true"}
         read_confs = []
 
         def execute(self, request, context):
@@ -219,11 +221,12 @@ def serve(connection):
                 context.abort(grpc.StatusCode.UNIMPLEMENTED, "only Get is 
served")
             keys = [key.decode() for key in 
parse_message(operation[2][0]).get(1, [])]
             self.read_confs.extend(keys)
-            if any(key not in self.confs for key in keys):
+            confs = {**self.default_confs, **self.confs}
+            if any(key not in confs for key in keys):
                 context.abort(grpc.StatusCode.INTERNAL, "[SQL_CONF_NOT_FOUND]")
             # The response names the session, as the server's does.
             return message_field(1, request[1][0]) + b"".join(
-                message_field(2, message_field(1, key) + message_field(2, 
self.confs[key]))
+                message_field(2, message_field(1, key) + message_field(2, 
confs[key]))
                 for key in keys
             )
 
@@ -303,9 +306,12 @@ def mock_connect_session():
     spark = None
     try:
         assert parent.poll(20), "Mock server did not start"
-        port = parent.recv()
-        spark = 
SparkSession.builder.remote(f"sc://127.0.0.1:{port}").getOrCreate()
-        yield spark, RemoteService(parent)
+        remote = f"sc://127.0.0.1:{parent.recv()}"
+        spark = SparkSession.builder.remote(remote).getOrCreate()
+        service = RemoteService(parent)
+        # The URL of the server, for more sessions of it.
+        object.__setattr__(service, "remote", remote)
+        yield spark, service
     finally:
         try:
             if spark is not None:
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 26a0d0c..5aec0ba 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -157,6 +157,19 @@ def 
test_udtf_takes_a_ddl_or_struct_return_type_as_pyspark(session):
     assert error.value.getCondition() == "PARSE_SYNTAX_ERROR"
 
 
+def test_arrow_udtf_takes_and_returns_arrow_data_as_pyspark(session):
+    # As PySpark's arrow_udtf(), a PyArrow-native UDTF: eval() takes a pyarrow 
array of the
+    # argument, and returns pyarrow tables.
+    import pyarrow.compute as pc
+
+    @F.arrow_udtf(returnType="x long, kind string")
+    class PlusOne:
+        def eval(self, x):
+            yield pa.table({"x": pc.add(x, 1), "kind": 
pa.array([type(x).__name__] * len(x))})
+
+    assert PlusOne(F.lit(1)).collect() == [Row(x=2, kind="Int32Array")]
+
+
 def test_map_functions_take_a_ddl_schema(session):
     df = session.createDataFrame([(1, 1.0), (1, 2.0), (2, 3.0)], ["k", "v"])
     schema = "k long, v double"
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 27efd02..565bd39 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -420,7 +420,7 @@ def test_functions_udtf_and_arrow_decorators():
         def eval(self, n):
             yield (n,)
 
-    assert AEcho.evalType == 301  # SQL_ARROW_TABLE_UDF
+    assert AEcho.evalType == 302  # SQL_ARROW_UDTF
 
     # functions.arrow_udf builds an Arrow scalar UDF.
     au = F.arrow_udf(lambda x: x, IntegerType())
@@ -712,6 +712,50 @@ def test_show_checks_its_arguments_as_pyspark():
         df._show_string(1, n=2)
 
 
+def test_udtfs_are_created_as_pyspark(mock_connect):
+    # As PySpark's _create_py_udtf and _create_pyarrow_udtf: a UDTF is not 
deterministic
+    # unless asDeterministic() marks it so, in place; arrow_udtf() makes a 
PyArrow-native
+    # UDTF (SQL_ARROW_UDTF); and udtf() makes an Arrow-optimized one 
(SQL_ARROW_TABLE_UDF)
+    # with useArrow=True, or without useArrow when the active session enables
+    # spark.sql.execution.pythonUDTF.arrow.enabled.
+    from pyspark.sql import functions as F
+
+    spark, service = mock_connect
+    arrow_conf = "spark.sql.execution.pythonUDTF.arrow.enabled"
+
+    class Echo:
+        def eval(self, n):
+            yield (n,)
+
+    def sent(fn):
+        # ExecutePlanRequest.plan=3, Plan.root=1, 
Relation.common_inline_user_defined_table_
+        # function=38; CommonInlineUserDefinedTableFunction.deterministic=2, 
python_udtf=4;
+        # PythonUDTF.eval_type=2.
+        fn(F.lit(1)).collect()
+        relation = 
parse_message(parse_message(service.execute_requests[-1])[3][0])[1][0]
+        udtf = parse_message(parse_message(relation)[38][0])
+        return udtf.get(2, [0]) == [1], parse_message(udtf[4][0])[2][0]
+
+    # The server enables the config by default.
+    arrow = F.udtf(Echo, returnType="a int")
+    assert arrow.deterministic is False
+    assert sent(arrow) == (False, 301)
+    assert arrow.asDeterministic() is arrow and arrow.deterministic is True
+    assert sent(arrow) == (True, 301)
+    assert sent(F.udtf(returnType="a int", useArrow=False)(Echo)) == (False, 
300)
+    assert sent(F.arrow_udtf(Echo, returnType="a int")) == (False, 302)
+    assert sent(F.arrow_udtf(returnType="a int")(Echo).asDeterministic()) == 
(True, 302)
+
+    service.confs = {arrow_conf: "false"}
+    try:
+        assert sent(F.udtf(returnType="a int")(Echo)) == (False, 300)
+        assert sent(F.udtf(Echo, returnType="a int", useArrow=True)) == 
(False, 301)
+    finally:
+        service.confs = {}
+    # Only udtf() without useArrow reads the configuration.
+    assert service.read_confs == [arrow_conf, arrow_conf]
+
+
 def 
test_decimal_literal_is_of_the_default_decimal_type_as_pyspark(mock_connect):
     # As PySpark's LiteralExpression of a Decimal, of DecimalType(): precision 
10 and scale
     # 0, which the server widens to those of the value, wherever the literal 
is.
@@ -1389,20 +1433,30 @@ def test_udtf_construction():
 
 
 def test_udtf_call_without_session_raises():
-    from pyspark.sql.udtf import UserDefinedTableFunction
-    from pyspark.sql import SparkSession
-    # Ensure no active session, then calling must raise a clear error.
-    prev = SparkSession.getActiveSession()
-    if prev is not None:
-        pytest.skip("an active session exists in this process")
+    # As PySpark's, udtf() without a session is not Arrow-optimized, and the 
call raises the
+    # error of SparkSession.active(). In a new interpreter, which has no 
active session.
+    import subprocess
+    import sys
 
-    @__import__("pyspark.sql.udtf", fromlist=["udtf"]).udtf(returnType="a int")
-    class T2:
-        def eval(self, n):
-            yield (n,)
+    subprocess.run([sys.executable, "-c", """
+from pyspark.errors import PySparkRuntimeError
+from pyspark.sql import SparkSession
+from pyspark.sql.functions import udtf
 
-    with pytest.raises(RuntimeError):
-        T2(1)
+@udtf(returnType="a int")
+class T2:
+    def eval(self, n):
+        yield (n,)
+
+assert T2.evalType == 300
+for call in (SparkSession.active, lambda: T2(1)):
+    try:
+        call()
+    except PySparkRuntimeError as error:
+        assert error.getCondition() == "NO_ACTIVE_OR_DEFAULT_SESSION", error
+    else:
+        raise AssertionError("no error")
+"""], check=True)
 
 
 def test_udtf_return_type_connect_validation_and_command():
@@ -1416,6 +1470,7 @@ def 
test_udtf_return_type_connect_validation_and_command():
 from pyspark.sql import SparkSession, DataFrame, functions as F, types as T
 from pyspark.serializers import CloudPickleSerializer
 
+# No server listens: with useArrow, udtf() does not read the Arrow config of 
the session.
 spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()
 class Echo:
     def eval(self, value):
@@ -1425,8 +1480,8 @@ schema = T.StructType([T.StructField("value", 
T.LongType())])
 for return_type in ["value BIGINT", "STRUCT<value: BIGINT>",
                     "value ARRAY<STRUCT<x: INT>>", "invalid DDL !!!", "INT", 
schema]:
     for decorate in [False, True]:
-        fn = (F.udtf(returnType=return_type)(Echo) if decorate else
-              F.udtf(Echo, returnType=return_type))
+        fn = (F.udtf(returnType=return_type, useArrow=False)(Echo) if decorate 
else
+              F.udtf(Echo, returnType=return_type, useArrow=False))
         assert isinstance(fn(F.lit(1)), DataFrame)
         handler = CloudPickleSerializer().loads(fn.command)
         assert isinstance(handler, type)
@@ -1438,7 +1493,7 @@ class Dynamic(Echo):
     @staticmethod
     def analyze(value):
         pass
-assert isinstance(F.udtf(Dynamic)(F.lit(1)), DataFrame)
+assert isinstance(F.udtf(Dynamic, useArrow=False)(F.lit(1)), DataFrame)
 """], check=True)
 
 
@@ -1508,7 +1563,8 @@ with mock_connect_session() as (spark, service):
                     assert parse_message(field[2][0]) == {7: [b""]}  # LongType
                     assert field.get(3, [0]) == [0]  # nullable=False
                     assert json.loads(field[4][0]) == {"comment": "kept"}
-            assert udtf[2] == [300]
+            # SQL_ARROW_TABLE_UDF, as the server enables Arrow-optimized UDTFs 
by default.
+            assert udtf[2] == [301]
             command_handler = CloudPickleSerializer().loads(udtf[3][0])
             assert isinstance(command_handler, type)
             assert command_handler.__name__ == handler.__name__
@@ -2154,8 +2210,10 @@ def test_udtf_variants():
     """Test UDTF builder methods."""
     from pyspark.sql.udtf import UserDefinedTableFunction
 
-    # Create a simple UDTF for testing
-    @__import__("pyspark.sql.functions", fromlist=["udtf"]).udtf(returnType="a 
int")
+    # Create a simple UDTF for testing; not Arrow-optimized whatever the 
active session.
+    @__import__("pyspark.sql.functions", fromlist=["udtf"]).udtf(
+        returnType="a int", useArrow=False
+    )
     class SimpleUDTF:
         def eval(self, n):
             yield (n,)
@@ -2165,8 +2223,8 @@ def test_udtf_variants():
     assert SimpleUDTF.name == "SimpleUDTF"
     assert SimpleUDTF.evalType == 300  # SQL_TABLE_UDF
 
-    # Test deterministic flag
-    assert SimpleUDTF.deterministic is True
+    # As PySpark's, a UDTF is not deterministic unless asDeterministic() marks 
it so.
+    assert SimpleUDTF.deterministic is False
 
 
 def test_column_with_alias_chain():
@@ -4645,9 +4703,28 @@ def _shared_mock_connect():
 
 @pytest.fixture
 def mock_connect(_shared_mock_connect):
-    spark, service = _shared_mock_connect
+    # Each test has a session of its own of the shared server, which is the 
active session
+    # while the test runs, as the tests of what reads the active session 
expect.
+    from pyspark.sql import SparkSession
+
+    _, service = _shared_mock_connect
     service.reset()
-    return spark, service
+    spark = SparkSession.builder.remote(service.remote).getOrCreate()
+    yield spark, service
+    spark.stop()
+
+
[email protected](autouse=True)
+def _no_session_left_active():
+    # Each test starts without an active session, as a new interpreter does. 
What reads the
+    # active session, such as udtf() of its Arrow configuration, would 
otherwise reach the
+    # server of a session that an earlier test left, which may not listen.
+    yield
+    from pyspark.sql import SparkSession
+
+    session = SparkSession.getActiveSession()
+    if session is not None:
+        session.stop()
 
 
 @pytest.mark.parametrize("first", ["schema", "columns", "dtypes"])


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to