This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 281cecf [SPARK-61122][CONNECT][PYTHON] Re-export the exceptions of
pyspark.errors from pyspark.sql.utils, in the Python client as PySpark does
281cecf is described below
commit 281cecf25245350bdf3b6eb432fab5dd866e9844
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 06:05:23 2026 +0900
[SPARK-61122][CONNECT][PYTHON] Re-export the exceptions of pyspark.errors
from pyspark.sql.utils, in the Python client as PySpark does
### What changes were proposed in this pull request?
[SPARK-61122](https://issues.apache.org/jira/browse/SPARK-61122):
`pyspark.sql.utils` re-exports the exceptions of `pyspark.errors`, as PySpark's
does: `AnalysisException`, `ParseException`, `IllegalArgumentException`,
`StreamingQueryException`, `QueryExecutionException`, `PythonException`,
`UnknownException`, `SparkUpgradeException`, `PySparkImportError`,
`PySparkNotImplementedError`, `PySparkRuntimeError` and `CapturedException`.
### Why are the changes needed?
`pyspark.sql.utils` is the import path of these exceptions before
`pyspark.errors`, and user code and Apache Spark's tests still use it, as in
`from pyspark.sql.utils import AnalysisException`. The client re-exported none
of them, so that import raised `ImportError`.
### Does this PR introduce _any_ user-facing change?
Yes. `from pyspark.sql.utils import AnalysisException`, and the other
exceptions, work as with PySpark 4.3.0.
### How was this patch tested?
- A new offline test checks:
- that each exception of `pyspark.sql.utils` is that of `pyspark.errors`
- that the error of a mock Connect server is an instance of
`pyspark.sql.utils.AnalysisException`
- The lib tests of both crates (590 passed), the build of all tests, also
with `wasm-udf`, the offline Python suites (2166 passed, 8 skipped), and the
live drop-in and Arrow suites (173 passed) and live cases (10 passed) against a
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #317 from HyukjinKwon/SPARK-61122-sql-utils-exceptions.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
python/pyspark/sql/utils.py | 20 +++++++++++++++++++-
python/tests/test_dropin_offline.py | 36 ++++++++++++++++++++++++++++++++++++
2 files changed, 55 insertions(+), 1 deletion(-)
diff --git a/python/pyspark/sql/utils.py b/python/pyspark/sql/utils.py
index f8cc0bd..5e9b5b4 100644
--- a/python/pyspark/sql/utils.py
+++ b/python/pyspark/sql/utils.py
@@ -14,11 +14,29 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-"""Small subset of pyspark.sql.utils needed by the drop-in and vendored
modules."""
+"""Small subset of pyspark.sql.utils needed by the drop-in and vendored
modules, and the
+exceptions that PySpark's re-exports from ``pyspark.errors``, of their import
path before it."""
from enum import Enum
from typing import Any, Union, TYPE_CHECKING, Sequence
+# Re-exported, as PySpark's pyspark.sql.utils does: `from pyspark.sql.utils
import
+# AnalysisException` is their import path before pyspark.errors.
+from pyspark.errors import ( # noqa: F401
+ AnalysisException,
+ ParseException,
+ IllegalArgumentException,
+ StreamingQueryException,
+ QueryExecutionException,
+ PythonException,
+ UnknownException,
+ SparkUpgradeException,
+ PySparkImportError,
+ PySparkNotImplementedError,
+ PySparkRuntimeError,
+)
+from pyspark.errors.exceptions.captured import CapturedException # noqa: F401
+
if TYPE_CHECKING:
from pyspark.pandas._typing import IndexOpsLike, SeriesOrIndex
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index a8666e6..514a859 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -944,6 +944,42 @@ def test_functions_exports_fresh_process():
assert result.returncode == 0, result.stdout + result.stderr
+def test_sql_utils_reexports_the_exceptions_of_pyspark(mock_connect):
+ # As PySpark's pyspark.sql.utils: the exceptions of pyspark.errors, of
their import path
+ # before it, which the errors of the server are instances of.
+ import pyspark.errors as errors
+ import pyspark.sql.utils as utils
+ from pyspark.errors.exceptions.captured import CapturedException
+
+ for name in [
+ "AnalysisException",
+ "ParseException",
+ "IllegalArgumentException",
+ "StreamingQueryException",
+ "QueryExecutionException",
+ "PythonException",
+ "UnknownException",
+ "SparkUpgradeException",
+ "PySparkImportError",
+ "PySparkNotImplementedError",
+ "PySparkRuntimeError",
+ ]:
+ assert getattr(utils, name) is getattr(errors, name), name
+ assert utils.CapturedException is CapturedException
+
+ spark, service = mock_connect
+ classes = ["org.apache.spark.sql.AnalysisException", "java.lang.Exception"]
+ message = "[TABLE_OR_VIEW_NOT_FOUND] The table or view `t` cannot be
found."
+ service.execute_error = (message, classes[0], _error_info(classes))
+ service.confs = {
+ "spark.sql.connect.serverStacktrace.enabled": "false",
+ "spark.sql.pyspark.jvmStacktrace.enabled": "false",
+ }
+ with pytest.raises(utils.AnalysisException) as error:
+ spark.range(1).collect()
+ assert "TABLE_OR_VIEW_NOT_FOUND" in str(error.value)
+
+
def test_functions_build_columns():
from pyspark.sql import functions as F
c = F.col("id")
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]