This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 66c4a792 [SPARK-61157][CONNECT][PYTHON] Capture the call sites of the
origins of Column operations without importing a missing IPython module on
every call
66c4a792 is described below
commit 66c4a79277840c144de44b05cc696362bfeb3a74
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 13:18:54 2026 +0900
[SPARK-61157][CONNECT][PYTHON] Capture the call sites of the origins of
Column operations without importing a missing IPython module on every call
### What changes were proposed in this pull request?
[SPARK-61157](https://issues.apache.org/jira/browse/SPARK-61157): Capture
the call sites of the origins of Column operations without importing a missing
IPython module on every call.
Every Column operation of the Python client calls `_with_origin`, which in
PySpark captures the call site of the user code with
`pyspark.errors.utils._capture_call_site`. That function runs `import IPython`
and `import ipykernel` on every call. Where ipykernel is not installed, each
failing import scans `sys.path` again.
- `origin.rs` captures the call site in Rust, with the result of
`_capture_call_site(1)`: the frames outside the pyspark package, which are
"file:line", and within an IPython shell, without the frames of IPython and
ipykernel, "line N in cell [K]". The frames are joined by newlines, as before.
- IPython and ipykernel are read from `sys.modules`. When one of them is
not loaded, the import is tried once per process and a failure is not retried;
a module imported later by the user is still seen.
- The vendored `pyspark/errors/utils.py` is unchanged, and
`is_debugging_enabled` is unchanged.
- The call sites of errors are the same as official PySpark 4.3.0: a
failing select and a failing cast in a nested function, a failing division, and
the IPython cell form (checked with fake IPython and ipykernel modules). The
official query-context tests (`test_parity_dataframe_query_context.py`) pass on
the drop-in, 6 of 6.
### Why are the changes needed?
With ipykernel not installed, 90k Column operations (`F.col("a") + 1`, `*
2`, `.alias`) took 6.8 s, against 0.23 s with debugging off.
`spark.range(1).select(*2000 names)` five times took 0.66 s, against 0.004 s
with debugging off. Official PySpark pays the same per-call cost in this helper
(10.7 s for the same run on that machine); in the drop-in, whose own work per
Column operation is small, nearly all of the time went to the failing imports.
After this change, the same run takes 0.77 s with debugging on, and the
select 0.022 s.
### Does this PR introduce _any_ user-facing change?
No. The error messages and the query contexts are the same. Column
operations are faster when IPython or ipykernel is not installed.
### How was this patch tested?
- Offline
`test_column_origins_capture_their_call_site_without_importing_per_call`: 200
Column operations on the mock server import IPython and ipykernel at most once
(the counting `__import__` sees 800 imports without this change), and the
request of a Column built on the test line carries the string that
`_capture_call_site(1)` gives on that line.
- The official `test_parity_dataframe_query_context.py` on the drop-in
against a live server: 6 passed, as before.
- The tests of both crates, e2e included, against a local Spark 4.3.0
server (875 passed), the build of all tests, also with `wasm-udf`, the offline
Python suites (2202 passed, 8 skipped), and the live drop-in, Arrow and ML
suites (184 passed) and live cases (10 passed). rustfmt passes, and clippy
reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #346 from HyukjinKwon/SPARK-61157-origin-call-site-no-import.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/origin.rs | 104 ++++++++++++++++++++++++++++++++++--
python/tests/test_dropin_offline.py | 28 ++++++++++
2 files changed, 129 insertions(+), 3 deletions(-)
diff --git a/crates/pyspark-rs/src/origin.rs b/crates/pyspark-rs/src/origin.rs
index c73b78af..786ae809 100644
--- a/crates/pyspark-rs/src/origin.rs
+++ b/crates/pyspark-rs/src/origin.rs
@@ -1,18 +1,28 @@
//! As PySpark's `_with_origin`: the origin in user code of the expressions
that an API function
//! builds, which the server reports in the `DataFrameQueryContext` of an
error in them.
+use std::sync::atomic::{AtomicBool, Ordering};
+
use pyo3::prelude::*;
use pyo3::sync::PyOnceLock;
+use pyo3::types::PyDict;
use spark_connect::origin::{set_current, Origin, OriginGuard};
static UTILS: PyOnceLock<Py<PyModule>> = PyOnceLock::new();
+static SYS: PyOnceLock<Py<PyModule>> = PyOnceLock::new();
+/// The directory of the `pyspark` package, as
`os.path.dirname(pyspark.__file__)`.
+static PYSPARK_ROOT: PyOnceLock<String> = PyOnceLock::new();
+/// Whether the imports of IPython and ipykernel were tried, once per process.
A failed import
+/// is not retried, and a module imported later is still seen in `sys.modules`.
+static IPYTHON_IMPORTS_TRIED: AtomicBool = AtomicBool::new(false);
/// While the returned guard lives, the expressions that this thread builds
have the origin of
/// `fragment`, the name of the API function, at its call site in user code,
when DataFrame
/// debugging is enabled. As PySpark's `_with_origin`, with the functions of
/// `pyspark.errors.utils` that it calls: `is_debugging_enabled`, which reads
-/// `spark.python.sql.dataFrameDebugging.enabled` once, and
`_capture_call_site`. An origin that
-/// cannot be read is not set, as an expression is the same without one.
+/// `spark.python.sql.dataFrameDebugging.enabled` once, and
`_capture_call_site`, which
+/// `call_site` gives without importing a module per call. An origin that
cannot be read is
+/// not set, as an expression is the same without one.
pub(crate) fn with_origin(fragment: &str) -> Option<OriginGuard> {
Python::attach(|py| -> PyResult<Option<OriginGuard>> {
let utils = UTILS
@@ -21,7 +31,7 @@ pub(crate) fn with_origin(fragment: &str) ->
Option<OriginGuard> {
if !utils.call_method0("is_debugging_enabled")?.is_truthy()? {
return Ok(None);
}
- let call_site: String = utils.call_method1("_capture_call_site",
(1,))?.extract()?;
+ let call_site = call_site(py, 1)?;
Ok(Some(set_current(Origin {
fragment: fragment.to_string(),
call_site,
@@ -29,3 +39,91 @@ pub(crate) fn with_origin(fragment: &str) ->
Option<OriginGuard> {
})
.unwrap_or(None)
}
+
+/// The call site of the first `depth` frames of user code, as
`_capture_call_site(depth)` of
+/// `pyspark.errors.utils` gives it: the frames outside the pyspark package,
and within an
+/// IPython shell (when IPython and ipykernel are loaded) without the frames
of IPython and
+/// ipykernel, each "line N in cell [K]", else "file:line", joined by newlines.
+fn call_site(py: Python<'_>, depth: usize) -> PyResult<String> {
+ let sys = SYS
+ .get_or_try_init(py, || py.import("sys").map(Bound::unbind))?
+ .bind(py);
+ let pyspark_root = PYSPARK_ROOT.get_or_try_init(py, || {
+ let file = py.import("pyspark")?.getattr("__file__")?;
+ py.import("os")?
+ .getattr("path")?
+ .call_method1("dirname", (file,))?
+ .extract::<String>()
+ })?;
+
+ // The frames as `inspect` walks them from the innermost:
`sys._getframe(0)` is the Python
+ // frame that called the API function, as the native code of it adds no
frame.
+ let mut selected = Vec::with_capacity(depth);
+ let mut frame = sys.call_method1("_getframe", (0,))?;
+ while selected.len() < depth && !frame.is_none() {
+ let file: String =
frame.getattr("f_code")?.getattr("co_filename")?.extract()?;
+ if !file.contains(pyspark_root.as_str()) {
+ selected.push((file, frame.getattr("f_lineno")?));
+ }
+ frame = frame.getattr("f_back")?;
+ }
+
+ let Some((ipython, ipykernel)) = loaded_ipython(py, sys)? else {
+ return join_frames(&selected);
+ };
+ // The frames of IPython and ipykernel are dropped after the first `depth`
frames, as
+ // `_capture_call_site` does.
+ let os_path = py.import("os")?.getattr("path")?;
+ let ipy_root: String = os_path
+ .call_method1("dirname", (ipython.getattr("__file__")?,))?
+ .extract()?;
+ let kernel_root: String = os_path
+ .call_method1("dirname", (ipykernel.getattr("__file__")?,))?
+ .extract()?;
+ let frames: Vec<_> = selected
+ .into_iter()
+ .filter(|(file, _)| {
+ !file.contains(ipy_root.as_str()) &&
!file.contains(kernel_root.as_str())
+ })
+ .collect();
+ let shell = ipython.call_method0("get_ipython")?;
+ if !shell.is_truthy()? {
+ return join_frames(&frames);
+ }
+ // Identifying the cell is useful when the error is generated from IPython
Notebook.
+ let count = shell.getattr("execution_count")?;
+ let cells: Vec<String> = frames
+ .iter()
+ .map(|(_, lineno)| Ok(format!("line {lineno} in cell [{count}]")))
+ .collect::<PyResult<_>>()?;
+ Ok(cells.join("\n"))
+}
+
+/// The "file:line" of each frame, joined by newlines.
+fn join_frames(frames: &[(String, Bound<'_, PyAny>)]) -> PyResult<String> {
+ let lines: Vec<String> = frames
+ .iter()
+ .map(|(file, lineno)| Ok(format!("{file}:{lineno}")))
+ .collect::<PyResult<_>>()?;
+ Ok(lines.join("\n"))
+}
+
+/// The modules IPython and ipykernel, when both are loaded. They are read
from `sys.modules`,
+/// and imported once per process when one is missing, as `_capture_call_site`
imports them.
+fn loaded_ipython<'py>(
+ py: Python<'py>,
+ sys: &Bound<'py, PyModule>,
+) -> PyResult<Option<(Bound<'py, PyAny>, Bound<'py, PyAny>)>> {
+ let modules = sys.getattr("modules")?;
+ let modules = modules.cast::<PyDict>()?;
+ let find = |name: &str| modules.get_item(name);
+ let (mut ipython, mut ipykernel) = (find("IPython")?, find("ipykernel")?);
+ if (ipython.is_none() || ipykernel.is_none())
+ && !IPYTHON_IMPORTS_TRIED.swap(true, Ordering::Relaxed)
+ {
+ // `import IPython`, then `import ipykernel`, as `_capture_call_site`
does.
+ let _ = py.import("IPython").and_then(|_| py.import("ipykernel"));
+ (ipython, ipykernel) = (find("IPython")?, find("ipykernel")?);
+ }
+ Ok(ipython.zip(ipykernel))
+}
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 35a45aff..9a099a57 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -12055,6 +12055,34 @@ def
test_add_artifacts_refuses_the_paths_pyspark_refuses(tmp_path):
assert error.value.errno in (errno.ENOENT, errno.EISDIR), path
+def
test_column_origins_capture_their_call_site_without_importing_per_call(mock_connect,
monkeypatch):
+ # As PySpark's `_with_origin`: a Column operation carries the call site of
its user code,
+ # which `_capture_call_site` gives, but the capture does not import
IPython and ipykernel
+ # on every call, where ipykernel is missing.
+ import builtins
+
+ import pyspark.errors.utils as errors_utils
+ from pyspark.sql import functions as F
+
+ spark, service = mock_connect
+ monkeypatch.setattr(errors_utils, "_enable_debugging_cache", True)
+ column = F.col("id") + 1; expected = errors_utils._capture_call_site(1)
+ imports = []
+ real_import = builtins.__import__
+
+ def counting_import(name, *args, **kwargs):
+ if name in ("IPython", "ipykernel"):
+ imports.append(name)
+ return real_import(name, *args, **kwargs)
+
+ monkeypatch.setattr(builtins, "__import__", counting_import)
+ for _ in range(200):
+ F.col("id") + 1
+ assert len(imports) <= 1
+ spark.range(1).select(column).collect()
+ assert expected.encode() in service.execute_requests[-1]
+
+
def
test_add_artifacts_puts_a_pyfile_zip_on_the_client_path_before_opening_it(tmp_path,
monkeypatch):
# As PySpark's _parse_artifacts: a zip, egg or jar of pyfile is put on
sys.path before its
# file is opened, and a Python file is not.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]