This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 66c4a792 [SPARK-61157][CONNECT][PYTHON] Capture the call sites of the 
origins of Column operations without importing a missing IPython module on 
every call
66c4a792 is described below

commit 66c4a79277840c144de44b05cc696362bfeb3a74
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 13:18:54 2026 +0900

    [SPARK-61157][CONNECT][PYTHON] Capture the call sites of the origins of 
Column operations without importing a missing IPython module on every call
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61157](https://issues.apache.org/jira/browse/SPARK-61157): Capture 
the call sites of the origins of Column operations without importing a missing 
IPython module on every call.
    
    Every Column operation of the Python client calls `_with_origin`, which in 
PySpark captures the call site of the user code with 
`pyspark.errors.utils._capture_call_site`. That function runs `import IPython` 
and `import ipykernel` on every call. Where ipykernel is not installed, each 
failing import scans `sys.path` again.
    
    - `origin.rs` captures the call site in Rust, with the result of 
`_capture_call_site(1)`: the frames outside the pyspark package, which are 
"file:line", and within an IPython shell, without the frames of IPython and 
ipykernel, "line N in cell [K]". The frames are joined by newlines, as before.
    - IPython and ipykernel are read from `sys.modules`. When one of them is 
not loaded, the import is tried once per process and a failure is not retried; 
a module imported later by the user is still seen.
    - The vendored `pyspark/errors/utils.py` is unchanged, and 
`is_debugging_enabled` is unchanged.
    - The call sites of errors are the same as official PySpark 4.3.0: a 
failing select and a failing cast in a nested function, a failing division, and 
the IPython cell form (checked with fake IPython and ipykernel modules). The 
official query-context tests (`test_parity_dataframe_query_context.py`) pass on 
the drop-in, 6 of 6.
    
    ### Why are the changes needed?
    
    With ipykernel not installed, 90k Column operations (`F.col("a") + 1`, `* 
2`, `.alias`) took 6.8 s, against 0.23 s with debugging off. 
`spark.range(1).select(*2000 names)` five times took 0.66 s, against 0.004 s 
with debugging off. Official PySpark pays the same per-call cost in this helper 
(10.7 s for the same run on that machine); in the drop-in, whose own work per 
Column operation is small, nearly all of the time went to the failing imports.
    
    After this change, the same run takes 0.77 s with debugging on, and the 
select 0.022 s.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No. The error messages and the query contexts are the same. Column 
operations are faster when IPython or ipykernel is not installed.
    
    ### How was this patch tested?
    
    - Offline 
`test_column_origins_capture_their_call_site_without_importing_per_call`: 200 
Column operations on the mock server import IPython and ipykernel at most once 
(the counting `__import__` sees 800 imports without this change), and the 
request of a Column built on the test line carries the string that 
`_capture_call_site(1)` gives on that line.
    - The official `test_parity_dataframe_query_context.py` on the drop-in 
against a live server: 6 passed, as before.
    - The tests of both crates, e2e included, against a local Spark 4.3.0 
server (875 passed), the build of all tests, also with `wasm-udf`, the offline 
Python suites (2202 passed, 8 skipped), and the live drop-in, Arrow and ML 
suites (184 passed) and live cases (10 passed). rustfmt passes, and clippy 
reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #346 from HyukjinKwon/SPARK-61157-origin-call-site-no-import.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/origin.rs     | 104 ++++++++++++++++++++++++++++++++++--
 python/tests/test_dropin_offline.py |  28 ++++++++++
 2 files changed, 129 insertions(+), 3 deletions(-)

diff --git a/crates/pyspark-rs/src/origin.rs b/crates/pyspark-rs/src/origin.rs
index c73b78af..786ae809 100644
--- a/crates/pyspark-rs/src/origin.rs
+++ b/crates/pyspark-rs/src/origin.rs
@@ -1,18 +1,28 @@
 //! As PySpark's `_with_origin`: the origin in user code of the expressions 
that an API function
 //! builds, which the server reports in the `DataFrameQueryContext` of an 
error in them.
 
+use std::sync::atomic::{AtomicBool, Ordering};
+
 use pyo3::prelude::*;
 use pyo3::sync::PyOnceLock;
+use pyo3::types::PyDict;
 use spark_connect::origin::{set_current, Origin, OriginGuard};
 
 static UTILS: PyOnceLock<Py<PyModule>> = PyOnceLock::new();
+static SYS: PyOnceLock<Py<PyModule>> = PyOnceLock::new();
+/// The directory of the `pyspark` package, as 
`os.path.dirname(pyspark.__file__)`.
+static PYSPARK_ROOT: PyOnceLock<String> = PyOnceLock::new();
+/// Whether the imports of IPython and ipykernel were tried, once per process. 
A failed import
+/// is not retried, and a module imported later is still seen in `sys.modules`.
+static IPYTHON_IMPORTS_TRIED: AtomicBool = AtomicBool::new(false);
 
 /// While the returned guard lives, the expressions that this thread builds 
have the origin of
 /// `fragment`, the name of the API function, at its call site in user code, 
when DataFrame
 /// debugging is enabled. As PySpark's `_with_origin`, with the functions of
 /// `pyspark.errors.utils` that it calls: `is_debugging_enabled`, which reads
-/// `spark.python.sql.dataFrameDebugging.enabled` once, and 
`_capture_call_site`. An origin that
-/// cannot be read is not set, as an expression is the same without one.
+/// `spark.python.sql.dataFrameDebugging.enabled` once, and 
`_capture_call_site`, which
+/// `call_site` gives without importing a module per call. An origin that 
cannot be read is
+/// not set, as an expression is the same without one.
 pub(crate) fn with_origin(fragment: &str) -> Option<OriginGuard> {
     Python::attach(|py| -> PyResult<Option<OriginGuard>> {
         let utils = UTILS
@@ -21,7 +31,7 @@ pub(crate) fn with_origin(fragment: &str) -> 
Option<OriginGuard> {
         if !utils.call_method0("is_debugging_enabled")?.is_truthy()? {
             return Ok(None);
         }
-        let call_site: String = utils.call_method1("_capture_call_site", 
(1,))?.extract()?;
+        let call_site = call_site(py, 1)?;
         Ok(Some(set_current(Origin {
             fragment: fragment.to_string(),
             call_site,
@@ -29,3 +39,91 @@ pub(crate) fn with_origin(fragment: &str) -> 
Option<OriginGuard> {
     })
     .unwrap_or(None)
 }
+
+/// The call site of the first `depth` frames of user code, as 
`_capture_call_site(depth)` of
+/// `pyspark.errors.utils` gives it: the frames outside the pyspark package, 
and within an
+/// IPython shell (when IPython and ipykernel are loaded) without the frames 
of IPython and
+/// ipykernel, each "line N in cell [K]", else "file:line", joined by newlines.
+fn call_site(py: Python<'_>, depth: usize) -> PyResult<String> {
+    let sys = SYS
+        .get_or_try_init(py, || py.import("sys").map(Bound::unbind))?
+        .bind(py);
+    let pyspark_root = PYSPARK_ROOT.get_or_try_init(py, || {
+        let file = py.import("pyspark")?.getattr("__file__")?;
+        py.import("os")?
+            .getattr("path")?
+            .call_method1("dirname", (file,))?
+            .extract::<String>()
+    })?;
+
+    // The frames as `inspect` walks them from the innermost: 
`sys._getframe(0)` is the Python
+    // frame that called the API function, as the native code of it adds no 
frame.
+    let mut selected = Vec::with_capacity(depth);
+    let mut frame = sys.call_method1("_getframe", (0,))?;
+    while selected.len() < depth && !frame.is_none() {
+        let file: String = 
frame.getattr("f_code")?.getattr("co_filename")?.extract()?;
+        if !file.contains(pyspark_root.as_str()) {
+            selected.push((file, frame.getattr("f_lineno")?));
+        }
+        frame = frame.getattr("f_back")?;
+    }
+
+    let Some((ipython, ipykernel)) = loaded_ipython(py, sys)? else {
+        return join_frames(&selected);
+    };
+    // The frames of IPython and ipykernel are dropped after the first `depth` 
frames, as
+    // `_capture_call_site` does.
+    let os_path = py.import("os")?.getattr("path")?;
+    let ipy_root: String = os_path
+        .call_method1("dirname", (ipython.getattr("__file__")?,))?
+        .extract()?;
+    let kernel_root: String = os_path
+        .call_method1("dirname", (ipykernel.getattr("__file__")?,))?
+        .extract()?;
+    let frames: Vec<_> = selected
+        .into_iter()
+        .filter(|(file, _)| {
+            !file.contains(ipy_root.as_str()) && 
!file.contains(kernel_root.as_str())
+        })
+        .collect();
+    let shell = ipython.call_method0("get_ipython")?;
+    if !shell.is_truthy()? {
+        return join_frames(&frames);
+    }
+    // Identifying the cell is useful when the error is generated from IPython 
Notebook.
+    let count = shell.getattr("execution_count")?;
+    let cells: Vec<String> = frames
+        .iter()
+        .map(|(_, lineno)| Ok(format!("line {lineno} in cell [{count}]")))
+        .collect::<PyResult<_>>()?;
+    Ok(cells.join("\n"))
+}
+
+/// The "file:line" of each frame, joined by newlines.
+fn join_frames(frames: &[(String, Bound<'_, PyAny>)]) -> PyResult<String> {
+    let lines: Vec<String> = frames
+        .iter()
+        .map(|(file, lineno)| Ok(format!("{file}:{lineno}")))
+        .collect::<PyResult<_>>()?;
+    Ok(lines.join("\n"))
+}
+
+/// The modules IPython and ipykernel, when both are loaded. They are read 
from `sys.modules`,
+/// and imported once per process when one is missing, as `_capture_call_site` 
imports them.
+fn loaded_ipython<'py>(
+    py: Python<'py>,
+    sys: &Bound<'py, PyModule>,
+) -> PyResult<Option<(Bound<'py, PyAny>, Bound<'py, PyAny>)>> {
+    let modules = sys.getattr("modules")?;
+    let modules = modules.cast::<PyDict>()?;
+    let find = |name: &str| modules.get_item(name);
+    let (mut ipython, mut ipykernel) = (find("IPython")?, find("ipykernel")?);
+    if (ipython.is_none() || ipykernel.is_none())
+        && !IPYTHON_IMPORTS_TRIED.swap(true, Ordering::Relaxed)
+    {
+        // `import IPython`, then `import ipykernel`, as `_capture_call_site` 
does.
+        let _ = py.import("IPython").and_then(|_| py.import("ipykernel"));
+        (ipython, ipykernel) = (find("IPython")?, find("ipykernel")?);
+    }
+    Ok(ipython.zip(ipykernel))
+}
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 35a45aff..9a099a57 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -12055,6 +12055,34 @@ def 
test_add_artifacts_refuses_the_paths_pyspark_refuses(tmp_path):
         assert error.value.errno in (errno.ENOENT, errno.EISDIR), path
 
 
+def 
test_column_origins_capture_their_call_site_without_importing_per_call(mock_connect,
 monkeypatch):
+    # As PySpark's `_with_origin`: a Column operation carries the call site of 
its user code,
+    # which `_capture_call_site` gives, but the capture does not import 
IPython and ipykernel
+    # on every call, where ipykernel is missing.
+    import builtins
+
+    import pyspark.errors.utils as errors_utils
+    from pyspark.sql import functions as F
+
+    spark, service = mock_connect
+    monkeypatch.setattr(errors_utils, "_enable_debugging_cache", True)
+    column = F.col("id") + 1; expected = errors_utils._capture_call_site(1)
+    imports = []
+    real_import = builtins.__import__
+
+    def counting_import(name, *args, **kwargs):
+        if name in ("IPython", "ipykernel"):
+            imports.append(name)
+        return real_import(name, *args, **kwargs)
+
+    monkeypatch.setattr(builtins, "__import__", counting_import)
+    for _ in range(200):
+        F.col("id") + 1
+    assert len(imports) <= 1
+    spark.range(1).select(column).collect()
+    assert expected.encode() in service.execute_requests[-1]
+
+
 def 
test_add_artifacts_puts_a_pyfile_zip_on_the_client_path_before_opening_it(tmp_path,
 monkeypatch):
     # As PySpark's _parse_artifacts: a zip, egg or jar of pyfile is put on 
sys.path before its
     # file is opened, and a Python file is not.


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to