This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new a515753 [SPARK-60999][CONNECT][PYTHON] Create the UDTFs of udtf and
arrow_udtf as PySpark does
a515753 is described below
commit a515753490f3294b4e1a432cd8dc0887459838d8
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Fri Nov 6 20:27:04 2026 +0900
[SPARK-60999][CONNECT][PYTHON] Create the UDTFs of udtf and arrow_udtf as
PySpark does
### What changes were proposed in this pull request?
[SPARK-60999](https://issues.apache.org/jira/browse/SPARK-60999): the
Python client now creates the UDTFs of `udtf()` and `arrow_udtf()` as PySpark's
`_create_py_udtf` and `_create_pyarrow_udtf` do.
- `arrow_udtf()` makes a PyArrow-native UDTF (`SQL_ARROW_UDTF`), whose
`eval()` takes `pyarrow` arrays and returns `pyarrow` tables. It made an
Arrow-optimized UDTF (`SQL_ARROW_TABLE_UDF`) instead, which passes Python
values.
- `udtf()` without `useArrow` makes an Arrow-optimized UDTF when the active
session enables `spark.sql.execution.pythonUDTF.arrow.enabled`, as the 4.3.0
server does by default. Without a session, it makes a regular UDTF.
- A UDTF is not deterministic unless `asDeterministic()` marks it so, in
place, and the call sends that flag. It was deterministic, and
`asDeterministic()` returned a copy.
- `SparkSession.active()` raises `NO_ACTIVE_OR_DEFAULT_SESSION`, as
PySpark's does, and so does a UDTF called without a session. It raised a plain
`RuntimeError`.
### Why are the changes needed?
Every `arrow_udtf` that used `pyarrow` failed, because its `eval()` was
given Python values. For example, `pc.add(x, 1)` of a `PlusOne(lit(1))` returns
`[Row(x=2, kind='Int32Array')]` in PySpark 4.3.0, and failed here. UDTFs also
did not follow the Arrow configuration, so their eval type and determinism
differed from PySpark's.
### Does this PR introduce _any_ user-facing change?
Yes. `arrow_udtf()` and `udtf()` create the UDTFs of PySpark 4.3.0, and
`SparkSession.active()` raises its error.
### How was this patch tested?
- A new live test runs an `arrow_udtf` that uses `pyarrow`. It also passes
with official PySpark 4.3.0, and fails without this change.
- A new offline test checks the eval type and the determinism of the UDTFs
that a mock Connect server receives:
- `udtf()` follows the configuration, which the mock now serves with the
server's default, and `useArrow` overrides it.
- `arrow_udtf()` makes a `SQL_ARROW_UDTF`.
- `asDeterministic()` marks the UDTF in place.
- Only `udtf()` without `useArrow` reads the configuration.
It fails without this change.
- The test of a UDTF called without a session now runs in a new
interpreter. It checks the eval type and the `NO_ACTIVE_OR_DEFAULT_SESSION`
error.
- Each offline test now starts without an active session, and the tests of
the mock server have a session of it each, which is active while they run. A
session that an earlier test left, of a server that does not listen, made
`udtf()` wait on its configuration.
- The tests of the eval types and the determinism of UDTFs now expect
PySpark's.
- The lib tests and every integration test, e2e included, against a local
Spark 4.3.0 server (664 passed, and 125 of the transport crate), the build of
all tests with `wasm-udf`, the offline Python suites (1158 passed, 8 skipped),
and the live drop-in and Arrow suites (84 passed) and live cases (10 passed).
Clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #204 from HyukjinKwon/SPARK-60999-udtf-create.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/functions.rs | 8 ++-
crates/pyspark-rs/src/session.rs | 12 +++-
python/pyspark/sql/udtf.py | 81 +++++++++++++++--------
python/tests/_mock_connect_server.py | 20 ++++--
python/tests/test_dropin_e2e.py | 13 ++++
python/tests/test_dropin_offline.py | 123 ++++++++++++++++++++++++++++-------
6 files changed, 195 insertions(+), 62 deletions(-)
diff --git a/crates/pyspark-rs/src/functions.rs
b/crates/pyspark-rs/src/functions.rs
index 392357b..5a33a03 100644
--- a/crates/pyspark-rs/src/functions.rs
+++ b/crates/pyspark-rs/src/functions.rs
@@ -72,8 +72,11 @@ fn pyfunc_make_udf(
/// - name/return_type/eval_type/command_bytes/python_ver: as for `make_udf`
/// (eval_type ∈ {300 SQL_TABLE_UDF, 301 SQL_ARROW_TABLE_UDF, 302
SQL_ARROW_UDTF})
/// - *args: Column arguments
+/// - deterministic: whether the UDTF is deterministic, which a PySpark UDTF
is not unless
+/// `asDeterministic()` makes it so
#[pyfunction]
-#[pyo3(signature = (session, name, return_type, eval_type, command_bytes,
python_ver, *args))]
+#[pyo3(signature = (session, name, return_type, eval_type, command_bytes,
python_ver, *args, deterministic=false))]
+#[allow(clippy::too_many_arguments)]
fn pyfunc_make_udtf(
session: PyRef<'_, PySparkSession>,
name: String,
@@ -82,6 +85,7 @@ fn pyfunc_make_udtf(
command_bytes: Vec<u8>,
python_ver: String,
args: Vec<Bound<'_, PyAny>>,
+ deterministic: bool,
) -> PyResult<PyDataFrame> {
let mut col_args = Vec::new();
for arg in args {
@@ -107,7 +111,7 @@ fn pyfunc_make_udtf(
eval_type,
command_bytes,
python_ver,
- true, // deterministic by default
+ deterministic,
);
Ok(PyDataFrame::new(df))
}
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index 8dd873b..4e434a1 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -1089,11 +1089,17 @@ impl PySparkSession {
.map(PySparkSession::new)
}
- /// The active session; errors if none. Mirrors `SparkSession.active`.
+ /// The active session; errors if none. Mirrors `SparkSession.active`,
which raises
+ /// `NO_ACTIVE_OR_DEFAULT_SESSION`.
#[staticmethod]
- fn active() -> PyResult<PySparkSession> {
+ fn active(py: Python<'_>) -> PyResult<PySparkSession> {
Self::get_active_session().ok_or_else(|| {
- pyo3::exceptions::PyRuntimeError::new_err("No active SparkSession
found")
+ crate::column::raise_pyspark(
+ py,
+ "PySparkRuntimeError",
+ "NO_ACTIVE_OR_DEFAULT_SESSION",
+ &[],
+ )
})
}
diff --git a/python/pyspark/sql/udtf.py b/python/pyspark/sql/udtf.py
index 47748f1..24f01a4 100644
--- a/python/pyspark/sql/udtf.py
+++ b/python/pyspark/sql/udtf.py
@@ -35,6 +35,7 @@ from pyspark.serializers import CloudPickleSerializer
# Eval types for Python UDTFs (mirrors pyspark.util.PythonEvalType).
SQL_TABLE_UDF = 300
SQL_ARROW_TABLE_UDF = 301
+SQL_ARROW_UDTF = 302
# Verbatim from PySpark (pyspark.sql.udtf): validates the handler class of a
UDTF.
@@ -97,7 +98,7 @@ class UserDefinedTableFunction:
returnType: Optional[Any] = None,
name: Optional[str] = None,
evalType: int = SQL_TABLE_UDF,
- deterministic: bool = True,
+ deterministic: bool = False,
):
# As PySpark's UserDefinedTableFunction, for udtf(), arrow_udtf() and
register().
_validate_udtf_handler(func, returnType)
@@ -113,39 +114,26 @@ class UserDefinedTableFunction:
# worker command contains only the handler class, unlike scalar UDFs.
self.command = CloudPickleSerializer().dumps(func)
- def _active_session(self):
- from pyspark.sql import SparkSession
-
- session = SparkSession.getActiveSession()
- if session is None:
- raise RuntimeError(
- "No active SparkSession; a UDTF must be called within an
active session."
- )
- return session
-
def __call__(self, *args: Any) -> Any:
from pyspark import _pyspark
+ from pyspark.sql import SparkSession
return _pyspark.functions.pyfunc_make_udtf(
- self._active_session(),
+ SparkSession.active(),
self.name,
self.returnType,
self.evalType,
self.command,
self.python_ver,
*args,
+ deterministic=self.deterministic,
)
def asDeterministic(self) -> "UserDefinedTableFunction":
- """Return a copy of this UDTF marked as deterministic. Mirrors
+ """Mark this UDTF as deterministic, and return it. Mirrors
``UserDefinedTableFunction.asDeterministic``."""
- return UserDefinedTableFunction(
- self.func,
- returnType=self.returnType,
- name=self.name,
- evalType=self.evalType,
- deterministic=True,
- )
+ self.deterministic = True
+ return self
def udtf(
@@ -157,11 +145,49 @@ def udtf(
"""Create a user-defined table function (UDTF).
Use as ``@udtf(returnType=...)`` on a handler class, or call
- ``udtf(MyClass, returnType=...)`` directly.
+ ``udtf(MyClass, returnType=...)`` directly. As PySpark's
``_create_py_udtf``, the UDTF is
+ Arrow-optimized when ``useArrow`` is True, or is not given and
+ ``spark.sql.execution.pythonUDTF.arrow.enabled`` of the active session is
true.
"""
- eval_type = SQL_ARROW_TABLE_UDF if useArrow is True else SQL_TABLE_UDF
def _udtf_decorator(handler: Type) -> UserDefinedTableFunction:
+ if useArrow is not None:
+ arrow_enabled = useArrow
+ else:
+ from pyspark.errors import PySparkRuntimeError
+ from pyspark.sql import SparkSession
+
+ arrow_enabled = False
+ try:
+ session = SparkSession.active()
+ arrow_enabled = (
+
str(session.conf.get("spark.sql.execution.pythonUDTF.arrow.enabled")).lower()
+ == "true"
+ )
+ except PySparkRuntimeError as e:
+ if e.getCondition() == "NO_ACTIVE_OR_DEFAULT_SESSION":
+ pass # Just uses the default if no session found.
+ else:
+ raise e
+ eval_type = SQL_TABLE_UDF
+ if arrow_enabled:
+ try:
+ from pyspark.sql.pandas.utils import (
+ require_minimum_pandas_version,
+ require_minimum_pyarrow_version,
+ )
+
+ require_minimum_pandas_version()
+ require_minimum_pyarrow_version()
+ eval_type = SQL_ARROW_TABLE_UDF
+ except ImportError as e:
+ import warnings
+
+ warnings.warn(
+ f"Arrow optimization for Python UDTFs cannot be enabled:
{str(e)}. "
+ f"Falling back to using regular Python UDTFs.",
+ UserWarning,
+ )
return UserDefinedTableFunction(handler, returnType=returnType,
evalType=eval_type)
if cls is not None:
@@ -176,14 +202,15 @@ def arrow_udtf(
) -> Any:
"""Create an Arrow-based user-defined table function (UDTF).
- Mirrors ``pyspark.sql.functions.arrow_udtf``: like :func:`udtf` but the
handler
- operates on Arrow data (``SQL_ARROW_TABLE_UDF``).
+ Mirrors ``pyspark.sql.functions.arrow_udtf``: a PyArrow-native UDTF
(``SQL_ARROW_UDTF``),
+ whose ``eval`` takes ``pyarrow`` arrays and returns ``pyarrow`` tables.
"""
def _arrow_udtf_decorator(handler: Type) -> UserDefinedTableFunction:
+ from pyspark.sql.pandas.utils import require_minimum_pyarrow_version
+
+ require_minimum_pyarrow_version()
_validate_arrow_udtf_handler(handler, returnType)
- return UserDefinedTableFunction(
- handler, returnType=returnType, evalType=SQL_ARROW_TABLE_UDF
- )
+ return UserDefinedTableFunction(handler, returnType=returnType,
evalType=SQL_ARROW_UDTF)
if cls is not None:
return _arrow_udtf_decorator(cls)
diff --git a/python/tests/_mock_connect_server.py
b/python/tests/_mock_connect_server.py
index 46278a0..96599bb 100644
--- a/python/tests/_mock_connect_server.py
+++ b/python/tests/_mock_connect_server.py
@@ -158,11 +158,13 @@ def serve(connection):
# ErrorInfo of `reason` and `metadata` in grpc-status-details-bin.
execute_error = None
# FetchErrorDetails returns `error_details` (raw
FetchErrorDetailsResponse fields) and
- # records the error id asked for. Config Get returns the values in
`confs`, and fails
- # for a key without one, as the server does for an unset config; it
records the keys.
+ # records the error id asked for. Config Get returns the values in
`confs`, or else
+ # those of `default_confs`, the defaults of SQL configs of the server,
and fails for a
+ # key without one, as the server does for an unset config; it records
the keys.
error_details = b""
fetched_error_ids = []
confs = {}
+ default_confs = {"spark.sql.execution.pythonUDTF.arrow.enabled":
"true"}
read_confs = []
def execute(self, request, context):
@@ -219,11 +221,12 @@ def serve(connection):
context.abort(grpc.StatusCode.UNIMPLEMENTED, "only Get is
served")
keys = [key.decode() for key in
parse_message(operation[2][0]).get(1, [])]
self.read_confs.extend(keys)
- if any(key not in self.confs for key in keys):
+ confs = {**self.default_confs, **self.confs}
+ if any(key not in confs for key in keys):
context.abort(grpc.StatusCode.INTERNAL, "[SQL_CONF_NOT_FOUND]")
# The response names the session, as the server's does.
return message_field(1, request[1][0]) + b"".join(
- message_field(2, message_field(1, key) + message_field(2,
self.confs[key]))
+ message_field(2, message_field(1, key) + message_field(2,
confs[key]))
for key in keys
)
@@ -303,9 +306,12 @@ def mock_connect_session():
spark = None
try:
assert parent.poll(20), "Mock server did not start"
- port = parent.recv()
- spark =
SparkSession.builder.remote(f"sc://127.0.0.1:{port}").getOrCreate()
- yield spark, RemoteService(parent)
+ remote = f"sc://127.0.0.1:{parent.recv()}"
+ spark = SparkSession.builder.remote(remote).getOrCreate()
+ service = RemoteService(parent)
+ # The URL of the server, for more sessions of it.
+ object.__setattr__(service, "remote", remote)
+ yield spark, service
finally:
try:
if spark is not None:
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 26a0d0c..5aec0ba 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -157,6 +157,19 @@ def
test_udtf_takes_a_ddl_or_struct_return_type_as_pyspark(session):
assert error.value.getCondition() == "PARSE_SYNTAX_ERROR"
+def test_arrow_udtf_takes_and_returns_arrow_data_as_pyspark(session):
+ # As PySpark's arrow_udtf(), a PyArrow-native UDTF: eval() takes a pyarrow
array of the
+ # argument, and returns pyarrow tables.
+ import pyarrow.compute as pc
+
+ @F.arrow_udtf(returnType="x long, kind string")
+ class PlusOne:
+ def eval(self, x):
+ yield pa.table({"x": pc.add(x, 1), "kind":
pa.array([type(x).__name__] * len(x))})
+
+ assert PlusOne(F.lit(1)).collect() == [Row(x=2, kind="Int32Array")]
+
+
def test_map_functions_take_a_ddl_schema(session):
df = session.createDataFrame([(1, 1.0), (1, 2.0), (2, 3.0)], ["k", "v"])
schema = "k long, v double"
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 27efd02..565bd39 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -420,7 +420,7 @@ def test_functions_udtf_and_arrow_decorators():
def eval(self, n):
yield (n,)
- assert AEcho.evalType == 301 # SQL_ARROW_TABLE_UDF
+ assert AEcho.evalType == 302 # SQL_ARROW_UDTF
# functions.arrow_udf builds an Arrow scalar UDF.
au = F.arrow_udf(lambda x: x, IntegerType())
@@ -712,6 +712,50 @@ def test_show_checks_its_arguments_as_pyspark():
df._show_string(1, n=2)
+def test_udtfs_are_created_as_pyspark(mock_connect):
+ # As PySpark's _create_py_udtf and _create_pyarrow_udtf: a UDTF is not
deterministic
+ # unless asDeterministic() marks it so, in place; arrow_udtf() makes a
PyArrow-native
+ # UDTF (SQL_ARROW_UDTF); and udtf() makes an Arrow-optimized one
(SQL_ARROW_TABLE_UDF)
+ # with useArrow=True, or without useArrow when the active session enables
+ # spark.sql.execution.pythonUDTF.arrow.enabled.
+ from pyspark.sql import functions as F
+
+ spark, service = mock_connect
+ arrow_conf = "spark.sql.execution.pythonUDTF.arrow.enabled"
+
+ class Echo:
+ def eval(self, n):
+ yield (n,)
+
+ def sent(fn):
+ # ExecutePlanRequest.plan=3, Plan.root=1,
Relation.common_inline_user_defined_table_
+ # function=38; CommonInlineUserDefinedTableFunction.deterministic=2,
python_udtf=4;
+ # PythonUDTF.eval_type=2.
+ fn(F.lit(1)).collect()
+ relation =
parse_message(parse_message(service.execute_requests[-1])[3][0])[1][0]
+ udtf = parse_message(parse_message(relation)[38][0])
+ return udtf.get(2, [0]) == [1], parse_message(udtf[4][0])[2][0]
+
+ # The server enables the config by default.
+ arrow = F.udtf(Echo, returnType="a int")
+ assert arrow.deterministic is False
+ assert sent(arrow) == (False, 301)
+ assert arrow.asDeterministic() is arrow and arrow.deterministic is True
+ assert sent(arrow) == (True, 301)
+ assert sent(F.udtf(returnType="a int", useArrow=False)(Echo)) == (False,
300)
+ assert sent(F.arrow_udtf(Echo, returnType="a int")) == (False, 302)
+ assert sent(F.arrow_udtf(returnType="a int")(Echo).asDeterministic()) ==
(True, 302)
+
+ service.confs = {arrow_conf: "false"}
+ try:
+ assert sent(F.udtf(returnType="a int")(Echo)) == (False, 300)
+ assert sent(F.udtf(Echo, returnType="a int", useArrow=True)) ==
(False, 301)
+ finally:
+ service.confs = {}
+ # Only udtf() without useArrow reads the configuration.
+ assert service.read_confs == [arrow_conf, arrow_conf]
+
+
def
test_decimal_literal_is_of_the_default_decimal_type_as_pyspark(mock_connect):
# As PySpark's LiteralExpression of a Decimal, of DecimalType(): precision
10 and scale
# 0, which the server widens to those of the value, wherever the literal
is.
@@ -1389,20 +1433,30 @@ def test_udtf_construction():
def test_udtf_call_without_session_raises():
- from pyspark.sql.udtf import UserDefinedTableFunction
- from pyspark.sql import SparkSession
- # Ensure no active session, then calling must raise a clear error.
- prev = SparkSession.getActiveSession()
- if prev is not None:
- pytest.skip("an active session exists in this process")
+ # As PySpark's, udtf() without a session is not Arrow-optimized, and the
call raises the
+ # error of SparkSession.active(). In a new interpreter, which has no
active session.
+ import subprocess
+ import sys
- @__import__("pyspark.sql.udtf", fromlist=["udtf"]).udtf(returnType="a int")
- class T2:
- def eval(self, n):
- yield (n,)
+ subprocess.run([sys.executable, "-c", """
+from pyspark.errors import PySparkRuntimeError
+from pyspark.sql import SparkSession
+from pyspark.sql.functions import udtf
- with pytest.raises(RuntimeError):
- T2(1)
+@udtf(returnType="a int")
+class T2:
+ def eval(self, n):
+ yield (n,)
+
+assert T2.evalType == 300
+for call in (SparkSession.active, lambda: T2(1)):
+ try:
+ call()
+ except PySparkRuntimeError as error:
+ assert error.getCondition() == "NO_ACTIVE_OR_DEFAULT_SESSION", error
+ else:
+ raise AssertionError("no error")
+"""], check=True)
def test_udtf_return_type_connect_validation_and_command():
@@ -1416,6 +1470,7 @@ def
test_udtf_return_type_connect_validation_and_command():
from pyspark.sql import SparkSession, DataFrame, functions as F, types as T
from pyspark.serializers import CloudPickleSerializer
+# No server listens: with useArrow, udtf() does not read the Arrow config of
the session.
spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()
class Echo:
def eval(self, value):
@@ -1425,8 +1480,8 @@ schema = T.StructType([T.StructField("value",
T.LongType())])
for return_type in ["value BIGINT", "STRUCT<value: BIGINT>",
"value ARRAY<STRUCT<x: INT>>", "invalid DDL !!!", "INT",
schema]:
for decorate in [False, True]:
- fn = (F.udtf(returnType=return_type)(Echo) if decorate else
- F.udtf(Echo, returnType=return_type))
+ fn = (F.udtf(returnType=return_type, useArrow=False)(Echo) if decorate
else
+ F.udtf(Echo, returnType=return_type, useArrow=False))
assert isinstance(fn(F.lit(1)), DataFrame)
handler = CloudPickleSerializer().loads(fn.command)
assert isinstance(handler, type)
@@ -1438,7 +1493,7 @@ class Dynamic(Echo):
@staticmethod
def analyze(value):
pass
-assert isinstance(F.udtf(Dynamic)(F.lit(1)), DataFrame)
+assert isinstance(F.udtf(Dynamic, useArrow=False)(F.lit(1)), DataFrame)
"""], check=True)
@@ -1508,7 +1563,8 @@ with mock_connect_session() as (spark, service):
assert parse_message(field[2][0]) == {7: [b""]} # LongType
assert field.get(3, [0]) == [0] # nullable=False
assert json.loads(field[4][0]) == {"comment": "kept"}
- assert udtf[2] == [300]
+ # SQL_ARROW_TABLE_UDF, as the server enables Arrow-optimized UDTFs
by default.
+ assert udtf[2] == [301]
command_handler = CloudPickleSerializer().loads(udtf[3][0])
assert isinstance(command_handler, type)
assert command_handler.__name__ == handler.__name__
@@ -2154,8 +2210,10 @@ def test_udtf_variants():
"""Test UDTF builder methods."""
from pyspark.sql.udtf import UserDefinedTableFunction
- # Create a simple UDTF for testing
- @__import__("pyspark.sql.functions", fromlist=["udtf"]).udtf(returnType="a
int")
+ # Create a simple UDTF for testing; not Arrow-optimized whatever the
active session.
+ @__import__("pyspark.sql.functions", fromlist=["udtf"]).udtf(
+ returnType="a int", useArrow=False
+ )
class SimpleUDTF:
def eval(self, n):
yield (n,)
@@ -2165,8 +2223,8 @@ def test_udtf_variants():
assert SimpleUDTF.name == "SimpleUDTF"
assert SimpleUDTF.evalType == 300 # SQL_TABLE_UDF
- # Test deterministic flag
- assert SimpleUDTF.deterministic is True
+ # As PySpark's, a UDTF is not deterministic unless asDeterministic() marks
it so.
+ assert SimpleUDTF.deterministic is False
def test_column_with_alias_chain():
@@ -4645,9 +4703,28 @@ def _shared_mock_connect():
@pytest.fixture
def mock_connect(_shared_mock_connect):
- spark, service = _shared_mock_connect
+ # Each test has a session of its own of the shared server, which is the
active session
+ # while the test runs, as the tests of what reads the active session
expect.
+ from pyspark.sql import SparkSession
+
+ _, service = _shared_mock_connect
service.reset()
- return spark, service
+ spark = SparkSession.builder.remote(service.remote).getOrCreate()
+ yield spark, service
+ spark.stop()
+
+
[email protected](autouse=True)
+def _no_session_left_active():
+ # Each test starts without an active session, as a new interpreter does.
What reads the
+ # active session, such as udtf() of its Arrow configuration, would
otherwise reach the
+ # server of a session that an earlier test left, which may not listen.
+ yield
+ from pyspark.sql import SparkSession
+
+ session = SparkSession.getActiveSession()
+ if session is not None:
+ session.stop()
@pytest.mark.parametrize("first", ["schema", "columns", "dtypes"])
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]