This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new daae9e4 [SPARK-60915][CONNECT][PYTHON] Send the origins of DataFrame
API calls, so that errors carry the DataFrame query contexts of PySpark
daae9e4 is described below
commit daae9e47bfb23e3a75294f4e7b857b60a97a5bb7
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sat Nov 7 03:48:08 2026 +0900
[SPARK-60915][CONNECT][PYTHON] Send the origins of DataFrame API calls, so
that errors carry the DataFrame query contexts of PySpark
### What changes were proposed in this pull request?
[SPARK-60915](https://issues.apache.org/jira/browse/SPARK-60915): the
Python client now sends the origins of DataFrame API calls, as PySpark Connect
does, so that errors carry the DataFrame query contexts of PySpark.
As PySpark's `_with_origin` and `with_origin_to_class`, `col()` and the
`Column` methods give the expressions that they build an origin. It has the
name of the function, such as `col` or `__add__`, and its call site in user
code. The client sends it as the `python_origin` of the expressions' common
fields. So does a column name that a DataFrame method takes, as PySpark's
`F._to_col` makes it with `col()`.
`spark.python.sql.dataFrameDebugging.enabled`, which PySpark reads once, turns
t [...]
For an error in such an expression, the server reports a
`DataFrameQueryContext` whose `fragment()` is the function and whose
`callSite()` is the `file:line` of the user code.
`repartitionById`, `sampleBy` and `unpivot` make their columns of column
names with `col()`, as PySpark's `F._to_col` does. A column name of a UDF or
UDTF call is a `ColumnReference` of it without an origin, as PySpark's
`to_expr` makes it, rather than the column of `col()`.
The client reads the configuration and the call site with
`pyspark.errors.utils.is_debugging_enabled` and `_capture_call_site`, as
PySpark's `_with_origin` does. The core expressions take the origin of the
thread when they are made, as PySpark's `Expression.__init__` does.
### Why are the changes needed?
Errors carried no DataFrame query context. In PySpark 4.3.0,
`spark.range(1).select("no_such_col").collect()` and `select(F.col("nope") +
1)` raise errors whose `getQueryContext()` is a `DataFrameQueryContext` of
`col` at the call site. An ANSI `F.lit(1) / F.lit(0)` reports `__truediv__`.
The client returned none.
### Does this PR introduce _any_ user-facing change?
Yes. Errors carry the DataFrame query contexts of PySpark 4.3.0.
### How was this patch tested?
- A new live test checks the `DataFrameQueryContext` of each of these
errors, with the call site in the test file, of `repartitionById`, `sampleBy`
and `unpivot` by a name, and none for a UDF called with a name. It also passes
with official PySpark 4.3.0, and fails without this change.
- The offline tests read `spark.python.sql.dataFrameDebugging.enabled`
once, without a session, before they run, so that a test whose session has no
server does not wait to read it.
- A new offline test checks, in new interpreters against a mock Connect
server, the origins of the expressions of `F.col("a") + 1`, and none when
DataFrame debugging is disabled. It fails without this change. The mock now
serves Config `GetWithDefault` requests.
- A new lib test checks the origins of core expressions.
- The lib tests and every integration test, e2e included, against a local
Spark 4.3.0 server (673 passed, and 129 of the transport crate), the build of
all tests with `wasm-udf`, the offline Python suites (1788 passed, 8 skipped),
and the live drop-in and Arrow suites (96 passed) and live cases (10 passed).
Clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #225 from HyukjinKwon/SPARK-60915-origins.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/column.rs | 64 +++++++++++++++++++++++++++
crates/pyspark-rs/src/dataframe.rs | 6 +++
crates/pyspark-rs/src/functions.rs | 7 +++
crates/pyspark-rs/src/group.rs | 2 +
crates/pyspark-rs/src/lib.rs | 1 +
crates/pyspark-rs/src/origin.rs | 31 +++++++++++++
crates/pyspark-rs/src/stat.rs | 2 +
crates/spark-connect/src/expression.rs | 41 +++++++++++++++---
crates/spark-connect/src/lib.rs | 1 +
crates/spark-connect/src/origin.rs | 79 ++++++++++++++++++++++++++++++++++
python/pyspark/sql/udf.py | 14 +++---
python/pyspark/sql/udtf.py | 10 ++---
python/tests/test_dropin_e2e.py | 45 +++++++++++++++++++
python/tests/test_dropin_offline.py | 68 +++++++++++++++++++++++++++++
14 files changed, 352 insertions(+), 19 deletions(-)
diff --git a/crates/pyspark-rs/src/column.rs b/crates/pyspark-rs/src/column.rs
index e78334a..00bf4e2 100644
--- a/crates/pyspark-rs/src/column.rs
+++ b/crates/pyspark-rs/src/column.rs
@@ -74,6 +74,7 @@ impl PyColumn {
alias: Vec<String>,
kwargs: Option<&Bound<'_, PyDict>>,
) -> PyResult<PyColumn> {
+ let _origin = crate::origin::with_origin("alias");
let mut metadata = None;
if let Some(kwargs) = kwargs {
let rest = kwargs.copy()?;
@@ -127,12 +128,14 @@ impl PyColumn {
alias: Vec<String>,
kwargs: Option<&Bound<'_, PyDict>>,
) -> PyResult<PyColumn> {
+ let _origin = crate::origin::with_origin("name");
self.alias(py, alias, kwargs)
}
/// Apply a transformation function to this column. Mirrors
`Column.transform(f)`,
/// which is simply `f(self)`.
fn transform<'py>(slf: Bound<'py, Self>, f: Bound<'py, PyAny>) ->
PyResult<Bound<'py, PyAny>> {
+ let _origin = crate::origin::with_origin("transform");
f.call1((slf,))
}
@@ -140,12 +143,14 @@ impl PyColumn {
/// joins). Mirrors `Column.outer()`, which returns the same expression —
outer
/// resolution is performed server-side via the plan id.
fn outer(&self) -> PyColumn {
+ let _origin = crate::origin::with_origin("outer");
PyColumn::new(self.column.clone())
}
/// Cast to a different type. Accepts a `DataType` or a DDL type string,
/// matching `pyspark.sql.Column.cast`.
fn cast(&self, dataType: &Bound<'_, PyAny>) -> PyResult<PyColumn> {
+ let _origin = crate::origin::with_origin("cast");
// A DDL string keeps the unparsed type_str form (reference
cast("int"));
// any DataType object is converted to a typed cast.
if let Ok(s) = dataType.extract::<String>() {
@@ -158,6 +163,7 @@ impl PyColumn {
/// Alias for `cast` (pyspark `Column.astype`).
fn astype(&self, dataType: &Bound<'_, PyAny>) -> PyResult<PyColumn> {
+ let _origin = crate::origin::with_origin("astype");
self.cast(dataType)
}
@@ -165,6 +171,7 @@ impl PyColumn {
/// Mirrors `pyspark.sql.Column.try_cast`.
#[pyo3(name = "try_cast")]
fn try_cast(&self, dataType: &Bound<'_, PyAny>) -> PyResult<PyColumn> {
+ let _origin = crate::origin::with_origin("try_cast");
if let Ok(s) = dataType.extract::<String>() {
Ok(PyColumn::new(self.column.clone().try_cast_str(&s)))
} else {
@@ -176,6 +183,7 @@ impl PyColumn {
/// Get an item from a list/map by key (pyspark `Column.getItem`).
#[pyo3(name = "getItem")]
fn get_item(&self, key: &Bound<'_, PyAny>) -> PyResult<PyColumn> {
+ let _origin = crate::origin::with_origin("getItem");
let key_col = py_obj_to_column(key)?;
Ok(PyColumn::new(self.column.clone().get_item(key_col)))
}
@@ -183,17 +191,20 @@ impl PyColumn {
/// Check if NULL.
#[pyo3(name = "isNull")]
fn is_null(&self) -> PyColumn {
+ let _origin = crate::origin::with_origin("isNull");
PyColumn::new(self.column.clone().is_null())
}
/// Check if NOT NULL.
#[pyo3(name = "isNotNull")]
fn is_not_null(&self) -> PyColumn {
+ let _origin = crate::origin::with_origin("isNotNull");
PyColumn::new(self.column.clone().is_not_null())
}
/// Substring extraction (start and length as columns).
fn substr(&self, startPos: &Bound<'_, PyAny>, length: &Bound<'_, PyAny>)
-> PyResult<PyColumn> {
+ let _origin = crate::origin::with_origin("substr");
// As PySpark: the arguments, as values of an Enum, must be of one
type, a Column or
// an int.
let py = startPos.py();
@@ -239,65 +250,77 @@ impl PyColumn {
/// Pattern matching with LIKE.
fn like(&self, other: &str) -> PyColumn {
+ let _origin = crate::origin::with_origin("like");
PyColumn::new(self.column.clone().like(other))
}
/// Pattern matching with RLIKE (regex).
fn rlike(&self, other: &str) -> PyColumn {
+ let _origin = crate::origin::with_origin("rlike");
PyColumn::new(self.column.clone().rlike(other))
}
/// Check if contains another column value.
fn contains(&self, other: &Bound<'_, PyAny>) -> PyResult<PyColumn> {
+ let _origin = crate::origin::with_origin("contains");
let other_col = py_obj_to_column(other)?;
Ok(PyColumn::new(self.column.clone().contains(other_col)))
}
/// Check if starts with another column value.
fn startswith(&self, other: &Bound<'_, PyAny>) -> PyResult<PyColumn> {
+ let _origin = crate::origin::with_origin("startswith");
let other_col = py_obj_to_column(other)?;
Ok(PyColumn::new(self.column.clone().startswith(other_col)))
}
/// Check if ends with another column value.
fn endswith(&self, other: &Bound<'_, PyAny>) -> PyResult<PyColumn> {
+ let _origin = crate::origin::with_origin("endswith");
let other_col = py_obj_to_column(other)?;
Ok(PyColumn::new(self.column.clone().endswith(other_col)))
}
/// Ascending sort.
fn asc(&self) -> PyColumn {
+ let _origin = crate::origin::with_origin("asc");
PyColumn::new(self.column.clone().asc())
}
/// Ascending sort with nulls first.
fn asc_nulls_first(&self) -> PyColumn {
+ let _origin = crate::origin::with_origin("asc_nulls_first");
PyColumn::new(self.column.clone().asc_nulls_first())
}
/// Ascending sort with nulls last.
fn asc_nulls_last(&self) -> PyColumn {
+ let _origin = crate::origin::with_origin("asc_nulls_last");
PyColumn::new(self.column.clone().asc_nulls_last())
}
/// Descending sort.
fn desc(&self) -> PyColumn {
+ let _origin = crate::origin::with_origin("desc");
PyColumn::new(self.column.clone().desc())
}
/// Descending sort with nulls first.
fn desc_nulls_first(&self) -> PyColumn {
+ let _origin = crate::origin::with_origin("desc_nulls_first");
PyColumn::new(self.column.clone().desc_nulls_first())
}
/// Descending sort with nulls last.
fn desc_nulls_last(&self) -> PyColumn {
+ let _origin = crate::origin::with_origin("desc_nulls_last");
PyColumn::new(self.column.clone().desc_nulls_last())
}
/// A branch more of a CASE WHEN. Mirrors `Column.when`: the condition is
a Column, and
/// the column a `when()` without `otherwise()` (`INVALID_WHEN_USAGE`).
fn when(&self, condition: &Bound<'_, PyAny>, value: &Bound<'_, PyAny>) ->
PyResult<PyColumn> {
+ let _origin = crate::origin::with_origin("when");
let py = condition.py();
let Ok(condition) = condition.extract::<PyRef<'_, PyColumn>>() else {
let arg_type = condition.get_type().name()?.to_string();
@@ -332,6 +355,7 @@ impl PyColumn {
/// The value of a CASE WHEN when no branch holds. Mirrors
`Column.otherwise`, which
/// raises PySparkTypeError for a column that is not a `when()`, or one
that has one.
fn otherwise(&self, value: &Bound<'_, PyAny>) -> PyResult<PyColumn> {
+ let _origin = crate::origin::with_origin("otherwise");
let message = match self.column.expression().node() {
ExpressionNode::CaseWhen(case_when) if
case_when.else_expr.is_none() => None,
ExpressionNode::CaseWhen(_) => Some(
@@ -354,6 +378,7 @@ impl PyColumn {
/// Get item by key/index, or a slice as `substr(start, stop)`. Mirrors
/// `Column.__getitem__`.
fn __getitem__(&self, key: &Bound<'_, PyAny>) -> PyResult<PyColumn> {
+ let _origin = crate::origin::with_origin("__getitem__");
if key.is_instance_of::<pyo3::types::PySlice>() {
if !key.getattr("step")?.is_none() {
return Err(raise_pyspark(
@@ -373,6 +398,7 @@ impl PyColumn {
/// `col["name"]`. Mirrors `Column.__getattr__`, which raises for `_jc`
and for a name
/// that starts with `__`.
fn __getattr__(&self, py: Python<'_>, item: &str) -> PyResult<PyColumn> {
+ let _origin = crate::origin::with_origin("__getattr__");
if item == "_jc" {
return Err(raise_pyspark(
py,
@@ -394,36 +420,42 @@ impl PyColumn {
/// Equality (==) - returns a Column condition
fn __eq__(&self, py: Python<'_>, other: Py<PyAny>) -> PyResult<PyColumn> {
+ let _origin = crate::origin::with_origin("__eq__");
let other_col = to_column(&other.bind(py))?;
Ok(PyColumn::new(self.column.clone().eq(other_col)))
}
/// Inequality (!=) - returns a Column condition
fn __ne__(&self, py: Python<'_>, other: Py<PyAny>) -> PyResult<PyColumn> {
+ let _origin = crate::origin::with_origin("__ne__");
let other_col = to_column(&other.bind(py))?;
Ok(PyColumn::new(self.column.clone().ne(other_col)))
}
/// Greater than (>) - returns a Column condition
fn __gt__(&self, py: Python<'_>, other: Py<PyAny>) -> PyResult<PyColumn> {
+ let _origin = crate::origin::with_origin("__gt__");
let other_col = to_column(&other.bind(py))?;
Ok(PyColumn::new(self.column.clone().gt(other_col)))
}
/// Less than (<) - returns a Column condition
fn __lt__(&self, py: Python<'_>, other: Py<PyAny>) -> PyResult<PyColumn> {
+ let _origin = crate::origin::with_origin("__lt__");
let other_col = to_column(&other.bind(py))?;
Ok(PyColumn::new(self.column.clone().lt(other_col)))
}
/// Greater than or equal (>=) - returns a Column condition
fn __ge__(&self, py: Python<'_>, other: Py<PyAny>) -> PyResult<PyColumn> {
+ let _origin = crate::origin::with_origin("__ge__");
let other_col = to_column(&other.bind(py))?;
Ok(PyColumn::new(self.column.clone().ge(other_col)))
}
/// Less than or equal (<=) - returns a Column condition
fn __le__(&self, py: Python<'_>, other: Py<PyAny>) -> PyResult<PyColumn> {
+ let _origin = crate::origin::with_origin("__le__");
let other_col = to_column(&other.bind(py))?;
Ok(PyColumn::new(self.column.clone().le(other_col)))
}
@@ -431,48 +463,56 @@ impl PyColumn {
// Arithmetic operators
/// Addition (+)
fn __add__(&self, py: Python<'_>, other: Py<PyAny>) -> PyResult<PyColumn> {
+ let _origin = crate::origin::with_origin("__add__");
let other_col = to_column(&other.bind(py))?;
Ok(PyColumn::new(self.column.clone().add(other_col)))
}
/// Right addition (for reverse operations)
fn __radd__(&self, py: Python<'_>, other: Py<PyAny>) -> PyResult<PyColumn>
{
+ let _origin = crate::origin::with_origin("__radd__");
let other_col = to_column(&other.bind(py))?;
Ok(PyColumn::new(other_col.add(self.column.clone())))
}
/// Subtraction (-)
fn __sub__(&self, py: Python<'_>, other: Py<PyAny>) -> PyResult<PyColumn> {
+ let _origin = crate::origin::with_origin("__sub__");
let other_col = to_column(&other.bind(py))?;
Ok(PyColumn::new(self.column.clone().sub(other_col)))
}
/// Right subtraction (for reverse operations)
fn __rsub__(&self, py: Python<'_>, other: Py<PyAny>) -> PyResult<PyColumn>
{
+ let _origin = crate::origin::with_origin("__rsub__");
let other_col = to_column(&other.bind(py))?;
Ok(PyColumn::new(other_col.sub(self.column.clone())))
}
/// Multiplication (*)
fn __mul__(&self, py: Python<'_>, other: Py<PyAny>) -> PyResult<PyColumn> {
+ let _origin = crate::origin::with_origin("__mul__");
let other_col = to_column(&other.bind(py))?;
Ok(PyColumn::new(self.column.clone().mul(other_col)))
}
/// Right multiplication (for reverse operations)
fn __rmul__(&self, py: Python<'_>, other: Py<PyAny>) -> PyResult<PyColumn>
{
+ let _origin = crate::origin::with_origin("__rmul__");
let other_col = to_column(&other.bind(py))?;
Ok(PyColumn::new(other_col.mul(self.column.clone())))
}
/// Division (/)
fn __truediv__(&self, py: Python<'_>, other: Py<PyAny>) ->
PyResult<PyColumn> {
+ let _origin = crate::origin::with_origin("__truediv__");
let other_col = to_column(&other.bind(py))?;
Ok(PyColumn::new(self.column.clone().div(other_col)))
}
/// Modulo (%)
fn __mod__(&self, py: Python<'_>, other: Py<PyAny>) -> PyResult<PyColumn> {
+ let _origin = crate::origin::with_origin("__mod__");
let other_col = to_column(&other.bind(py))?;
Ok(PyColumn::new(self.column.clone().modulo(other_col)))
}
@@ -480,34 +520,40 @@ impl PyColumn {
// Bitwise operators (logical in Spark context)
/// Bitwise AND (&) - logical AND in Spark
fn __and__(&self, py: Python<'_>, other: Py<PyAny>) -> PyResult<PyColumn> {
+ let _origin = crate::origin::with_origin("__and__");
let other_col = to_column(&other.bind(py))?;
Ok(PyColumn::new(self.column.clone().and(other_col)))
}
/// Bitwise OR (|) - logical OR in Spark
fn __or__(&self, py: Python<'_>, other: Py<PyAny>) -> PyResult<PyColumn> {
+ let _origin = crate::origin::with_origin("__or__");
let other_col = to_column(&other.bind(py))?;
Ok(PyColumn::new(self.column.clone().or(other_col)))
}
/// Bitwise NOT (~) - logical NOT in Spark
fn __invert__(&self) -> PyColumn {
+ let _origin = crate::origin::with_origin("__invert__");
PyColumn::new(self.column.clone().not())
}
/// Negation (-)
fn __neg__(&self) -> PyColumn {
+ let _origin = crate::origin::with_origin("__neg__");
PyColumn::new(self.column.clone().neg())
}
/// Get field by name (for struct/nested types).
#[pyo3(name = "getField")]
fn get_field(&self, name: &str) -> PyColumn {
+ let _origin = crate::origin::with_origin("getField");
PyColumn::new(self.column.clone().get_field(name))
}
/// Apply a window function over a window specification.
fn over(&self, window: &crate::window::PyWindowSpec) -> PyColumn {
+ let _origin = crate::origin::with_origin("over");
PyColumn::new(self.column.clone().over(window.spec.clone()))
}
@@ -518,23 +564,28 @@ impl PyColumn {
}
fn __str__(&self) -> String {
+ let _origin = crate::origin::with_origin("__str__");
self.__repr__()
}
// --- reverse arithmetic/logical dunders (operand order swapped) ---
fn __rtruediv__(&self, py: Python<'_>, other: Py<PyAny>) ->
PyResult<PyColumn> {
+ let _origin = crate::origin::with_origin("__rtruediv__");
let o = to_column(&other.bind(py))?;
Ok(PyColumn::new(o.div(self.column.clone())))
}
fn __rmod__(&self, py: Python<'_>, other: Py<PyAny>) -> PyResult<PyColumn>
{
+ let _origin = crate::origin::with_origin("__rmod__");
let o = to_column(&other.bind(py))?;
Ok(PyColumn::new(o.modulo(self.column.clone())))
}
fn __rand__(&self, py: Python<'_>, other: Py<PyAny>) -> PyResult<PyColumn>
{
+ let _origin = crate::origin::with_origin("__rand__");
let o = to_column(&other.bind(py))?;
Ok(PyColumn::new(o.and(self.column.clone())))
}
fn __ror__(&self, py: Python<'_>, other: Py<PyAny>) -> PyResult<PyColumn> {
+ let _origin = crate::origin::with_origin("__ror__");
let o = to_column(&other.bind(py))?;
Ok(PyColumn::new(o.or(self.column.clone())))
}
@@ -542,6 +593,7 @@ impl PyColumn {
/// `col ** other`. Mirrors `Column.__pow__` = `power(self, other)`.
/// (`modulo` is part of Python's ternary pow protocol; Spark has no 3-arg
pow.)
fn __pow__(&self, py: Python<'_>, other: Py<PyAny>, _modulo: Py<PyAny>) ->
PyResult<PyColumn> {
+ let _origin = crate::origin::with_origin("__pow__");
let o = to_column(&other.bind(py))?;
Ok(PyColumn::new(spark_connect::functions::pow(
self.column.clone(),
@@ -551,6 +603,7 @@ impl PyColumn {
/// `other ** col`. Mirrors `Column.__rpow__` = `power(other, self)`.
fn __rpow__(&self, py: Python<'_>, other: Py<PyAny>, _modulo: Py<PyAny>)
-> PyResult<PyColumn> {
+ let _origin = crate::origin::with_origin("__rpow__");
let o = to_column(&other.bind(py))?;
Ok(PyColumn::new(spark_connect::functions::pow(
o,
@@ -563,6 +616,7 @@ impl PyColumn {
/// `x in col` is not supported. Mirrors `Column.__contains__`.
fn __contains__(&self, py: Python<'_>, _item: Py<PyAny>) -> PyResult<()> {
+ let _origin = crate::origin::with_origin("__contains__");
Err(raise_pyspark(
py,
"PySparkValueError",
@@ -598,39 +652,47 @@ impl PyColumn {
lowerBound: Py<PyAny>,
upperBound: Py<PyAny>,
) -> PyResult<PyColumn> {
+ let _origin = crate::origin::with_origin("between");
let l = to_column(&lowerBound.bind(py))?;
let u = to_column(&upperBound.bind(py))?;
Ok(PyColumn::new(self.column.clone().between(l, u)))
}
#[pyo3(name = "bitwiseAND")]
fn bitwise_and(&self, py: Python<'_>, other: Py<PyAny>) ->
PyResult<PyColumn> {
+ let _origin = crate::origin::with_origin("bitwiseAND");
let o = to_column(&other.bind(py))?;
Ok(PyColumn::new(self.column.clone().bitwise_and(o)))
}
#[pyo3(name = "bitwiseOR")]
fn bitwise_or(&self, py: Python<'_>, other: Py<PyAny>) ->
PyResult<PyColumn> {
+ let _origin = crate::origin::with_origin("bitwiseOR");
let o = to_column(&other.bind(py))?;
Ok(PyColumn::new(self.column.clone().bitwise_or(o)))
}
#[pyo3(name = "bitwiseXOR")]
fn bitwise_xor(&self, py: Python<'_>, other: Py<PyAny>) ->
PyResult<PyColumn> {
+ let _origin = crate::origin::with_origin("bitwiseXOR");
let o = to_column(&other.bind(py))?;
Ok(PyColumn::new(self.column.clone().bitwise_xor(o)))
}
fn ilike(&self, other: &str) -> PyColumn {
+ let _origin = crate::origin::with_origin("ilike");
PyColumn::new(self.column.clone().ilike(other))
}
#[pyo3(name = "isNaN")]
fn is_nan(&self) -> PyColumn {
+ let _origin = crate::origin::with_origin("isNaN");
PyColumn::new(self.column.clone().is_nan())
}
#[pyo3(name = "eqNullSafe")]
fn eq_null_safe(&self, py: Python<'_>, other: Py<PyAny>) ->
PyResult<PyColumn> {
+ let _origin = crate::origin::with_origin("eqNullSafe");
let o = to_column(&other.bind(py))?;
Ok(PyColumn::new(self.column.clone().eq_null_safe(o)))
}
#[pyo3(signature = (*cols))]
fn isin(&self, py: Python<'_>, cols: Vec<Py<PyAny>>) -> PyResult<PyColumn>
{
+ let _origin = crate::origin::with_origin("isin");
// As pyspark's, a single DataFrame makes an IN subquery of it.
if let [only] = cols.as_slice() {
if let Ok(df) = only
@@ -666,11 +728,13 @@ impl PyColumn {
}
#[pyo3(name = "withField")]
fn with_field(&self, py: Python<'_>, fieldName: &str, col: Py<PyAny>) ->
PyResult<PyColumn> {
+ let _origin = crate::origin::with_origin("withField");
let v = to_column(&col.bind(py))?;
Ok(PyColumn::new(self.column.clone().with_field(fieldName, v)))
}
#[pyo3(name = "dropFields", signature = (*fieldNames))]
fn drop_fields(&self, fieldNames: Vec<String>) -> PyColumn {
+ let _origin = crate::origin::with_origin("dropFields");
let refs: Vec<&str> = fieldNames.iter().map(|s| s.as_str()).collect();
PyColumn::new(self.column.clone().drop_fields(refs))
}
diff --git a/crates/pyspark-rs/src/dataframe.rs
b/crates/pyspark-rs/src/dataframe.rs
index 52e8516..a1e0517 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -183,6 +183,8 @@ pub(crate) fn to_column_list(
} else {
// Try as string (column name)
if let Ok(name) = arg.extract::<String>() {
+ // As PySpark's `F._to_col`, by `col`, of its origin.
+ let _origin = crate::origin::with_origin("col");
cols.push(spark_connect::functions::col(&name));
} else {
return Err(PyErr::new::<pyo3::exceptions::PyTypeError, _>(
@@ -510,6 +512,8 @@ impl PyDataFrame {
let col = if let Ok(c) = partitionIdCol.extract::<Bound<'_,
PyColumn>>() {
c.borrow().column.clone()
} else if let Ok(name) = partitionIdCol.extract::<String>() {
+ // As PySpark's `F._to_col`, by `col`, of its origin.
+ let _origin = crate::origin::with_origin("col");
spark_connect::functions::col(&name)
} else {
return Err(PyErr::new::<pyo3::exceptions::PyTypeError, _>(
@@ -2014,6 +2018,8 @@ fn column_or_name(col: &Bound<'_, PyAny>) ->
PyResult<spark_connect::column::Col
return Ok(column.column.clone());
}
if col.is_instance_of::<pyo3::types::PyString>() {
+ // As PySpark's `F._to_col`, by `col`, of its origin.
+ let _origin = crate::origin::with_origin("col");
return Ok(spark_connect::functions::col(&col.extract::<String>()?));
}
let type_name = col
diff --git a/crates/pyspark-rs/src/functions.rs
b/crates/pyspark-rs/src/functions.rs
index 763812c..a806563 100644
--- a/crates/pyspark-rs/src/functions.rs
+++ b/crates/pyspark-rs/src/functions.rs
@@ -76,6 +76,11 @@ fn udf_arguments(
return Ok(table_arg.inner.to_column());
}
}
+ // As PySpark's `to_expr` of a UDF call: a column name is a
ColumnReference of it,
+ // without the origin of `col`.
+ if arg.is_instance_of::<pyo3::types::PyString>() {
+ return
Ok(Column::new(Expression::column(arg.extract::<String>()?)));
+ }
to_column(arg)
};
let mut columns = Vec::with_capacity(args.len() + named_args.len());
@@ -350,6 +355,8 @@ fn scalar_literal(obj: &Bound<'_, PyAny>) ->
PyResult<Option<LiteralExpression>>
#[pyfunction]
fn pyfunc_col(name: &str) -> PyColumn {
+ // As PySpark's `col`, of `_with_origin`.
+ let _origin = crate::origin::with_origin("col");
PyColumn::new(spark_funcs::col(name))
}
diff --git a/crates/pyspark-rs/src/group.rs b/crates/pyspark-rs/src/group.rs
index 107d84f..8b7d4ad 100644
--- a/crates/pyspark-rs/src/group.rs
+++ b/crates/pyspark-rs/src/group.rs
@@ -35,6 +35,8 @@ impl PyGroupedData {
for (k, v) in dict.iter() {
let col_name: String = k.extract()?;
let func_name: String = v.extract()?;
+ // As PySpark's `agg` of a dict, by `col`, of its origin.
+ let _origin = crate::origin::with_origin("col");
let c = spark_connect::functions::col(&col_name);
// Build func(col) as an unresolved function resolved
server-side, so
// aggregate names (max/min/sum/avg/count/...) all work
regardless of
diff --git a/crates/pyspark-rs/src/lib.rs b/crates/pyspark-rs/src/lib.rs
index 96c9f43..d9cbbad 100644
--- a/crates/pyspark-rs/src/lib.rs
+++ b/crates/pyspark-rs/src/lib.rs
@@ -36,6 +36,7 @@ mod group;
mod infer;
mod ml;
mod observation;
+mod origin;
mod pandas_input;
mod pipelines;
mod profiler;
diff --git a/crates/pyspark-rs/src/origin.rs b/crates/pyspark-rs/src/origin.rs
new file mode 100644
index 0000000..c73b78a
--- /dev/null
+++ b/crates/pyspark-rs/src/origin.rs
@@ -0,0 +1,31 @@
+//! As PySpark's `_with_origin`: the origin in user code of the expressions
that an API function
+//! builds, which the server reports in the `DataFrameQueryContext` of an
error in them.
+
+use pyo3::prelude::*;
+use pyo3::sync::PyOnceLock;
+use spark_connect::origin::{set_current, Origin, OriginGuard};
+
+static UTILS: PyOnceLock<Py<PyModule>> = PyOnceLock::new();
+
+/// While the returned guard lives, the expressions that this thread builds
have the origin of
+/// `fragment`, the name of the API function, at its call site in user code,
when DataFrame
+/// debugging is enabled. As PySpark's `_with_origin`, with the functions of
+/// `pyspark.errors.utils` that it calls: `is_debugging_enabled`, which reads
+/// `spark.python.sql.dataFrameDebugging.enabled` once, and
`_capture_call_site`. An origin that
+/// cannot be read is not set, as an expression is the same without one.
+pub(crate) fn with_origin(fragment: &str) -> Option<OriginGuard> {
+ Python::attach(|py| -> PyResult<Option<OriginGuard>> {
+ let utils = UTILS
+ .get_or_try_init(py, ||
py.import("pyspark.errors.utils").map(Bound::unbind))?
+ .bind(py);
+ if !utils.call_method0("is_debugging_enabled")?.is_truthy()? {
+ return Ok(None);
+ }
+ let call_site: String = utils.call_method1("_capture_call_site",
(1,))?.extract()?;
+ Ok(Some(set_current(Origin {
+ fragment: fragment.to_string(),
+ call_site,
+ })))
+ })
+ .unwrap_or(None)
+}
diff --git a/crates/pyspark-rs/src/stat.rs b/crates/pyspark-rs/src/stat.rs
index bb45d83..cc7ced3 100644
--- a/crates/pyspark-rs/src/stat.rs
+++ b/crates/pyspark-rs/src/stat.rs
@@ -105,6 +105,8 @@ pub(crate) fn sample_by_arguments(
let type_name =
|obj: &Bound<'_, PyAny>| -> PyResult<String> {
Ok(obj.get_type().name()?.to_string()) };
let col = if let Ok(name) = col.cast::<PyString>() {
+ // As PySpark's `F._to_col`, by `col`, of its origin.
+ let _origin = crate::origin::with_origin("col");
spark_connect::functions::col(&name.to_cow()?)
} else if let Ok(column) = col.extract::<PyRef<'_, PyColumn>>() {
column.column.clone()
diff --git a/crates/spark-connect/src/expression.rs
b/crates/spark-connect/src/expression.rs
index b80fbfc..c96787b 100644
--- a/crates/spark-connect/src/expression.rs
+++ b/crates/spark-connect/src/expression.rs
@@ -24,9 +24,13 @@ pub fn next_lambda_var_index() -> u32 {
/// Cloning a handle shares its node rather than copying the expression
subtree.
/// Build common expressions with [`Expression::literal`],
[`Expression::column`],
/// and [`Expression::function`], or use [`Expression::new`] for other node
types.
-/// Inspect expressions with [`Expression::node`].
+/// Inspect expressions with [`Expression::node`]. An expression keeps the
origin in user code
+/// of its construction, if any, as PySpark's does (see [`crate::origin`]).
#[derive(Clone)]
-pub struct Expression(Arc<ExpressionNode>);
+pub struct Expression {
+ node: Arc<ExpressionNode>,
+ origin: Option<Arc<crate::origin::Origin>>,
+}
impl std::fmt::Debug for Expression {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
@@ -287,9 +291,18 @@ impl NamedArgument {
}
impl Expression {
- /// Allocate an immutable expression node.
+ /// Allocate an immutable expression node, of the current origin, as
PySpark's
+ /// `Expression.__init__` takes it.
pub fn new(node: ExpressionNode) -> Self {
- Self(Arc::new(node))
+ Self {
+ node: Arc::new(node),
+ origin: crate::origin::current(),
+ }
+ }
+
+ /// The origin in user code of this expression, if any.
+ pub fn origin(&self) -> Option<&crate::origin::Origin> {
+ self.origin.as_deref()
}
/// Create a literal expression.
@@ -311,7 +324,7 @@ impl Expression {
/// Borrow this expression's node without exposing mutable shared state.
pub fn node(&self) -> &ExpressionNode {
- self.0.as_ref()
+ self.node.as_ref()
}
/// Render this expression to a human-readable string, mirroring the
@@ -394,8 +407,24 @@ impl Expression {
}
}
- /// Converts the expression to a Spark Connect protobuf expression.
+ /// Converts the expression to a Spark Connect protobuf expression, with
its origin, as
+ /// PySpark's `Expression._create_proto_expression` sends it.
pub fn to_proto(&self) -> proto::Expression {
+ let mut expr = self.node_to_proto();
+ if let Some(origin) = &self.origin {
+ expr.common = Some(proto::ExpressionCommon {
+ origin: Some(proto::Origin {
+ function:
Some(proto::origin::Function::PythonOrigin(proto::PythonOrigin {
+ fragment: origin.fragment.clone(),
+ call_site: origin.call_site.clone(),
+ })),
+ }),
+ });
+ }
+ expr
+ }
+
+ fn node_to_proto(&self) -> proto::Expression {
match self.node() {
ExpressionNode::Literal(lit) => lit.to_proto(),
ExpressionNode::ColumnReference(col_ref) => col_ref.to_proto(),
diff --git a/crates/spark-connect/src/lib.rs b/crates/spark-connect/src/lib.rs
index e12c0a4..38fb6a9 100644
--- a/crates/spark-connect/src/lib.rs
+++ b/crates/spark-connect/src/lib.rs
@@ -17,6 +17,7 @@ pub mod group;
pub mod merge;
pub mod ml;
pub mod observation;
+pub mod origin;
pub mod pipelines;
pub mod plan;
pub mod profiler;
diff --git a/crates/spark-connect/src/origin.rs
b/crates/spark-connect/src/origin.rs
new file mode 100644
index 0000000..d5df917
--- /dev/null
+++ b/crates/spark-connect/src/origin.rs
@@ -0,0 +1,79 @@
+//! The origins of expressions in user code, as PySpark's
`pyspark.errors.utils.current_origin`:
+//! the API function that builds an expression, and its call site, which the
server reports in
+//! the `DataFrameQueryContext` of an error in the expression.
+
+use std::cell::RefCell;
+use std::sync::Arc;
+
+/// The origin of an expression: the API function that built it, and its call
site in user
+/// code. Mirrors `pyspark.sql.connect.proto.PythonOrigin`.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct Origin {
+ /// The name of the function, such as `col` or `__add__`.
+ pub fragment: String,
+ /// Where the user code called it, such as `script.py:12`.
+ pub call_site: String,
+}
+
+thread_local! {
+ /// The origin of the expressions that this thread builds now, as
PySpark's thread-local
+ /// `_current_origin`.
+ static CURRENT: RefCell<Option<Arc<Origin>>> = const { RefCell::new(None)
};
+}
+
+/// The origin of the expressions that this thread builds now, if any.
+pub(crate) fn current() -> Option<Arc<Origin>> {
+ CURRENT.with(|current| current.borrow().clone())
+}
+
+/// Sets the origin of the expressions that this thread builds while the
returned guard lives,
+/// as PySpark's `_with_origin` does for a call of an API function. As
PySpark's, the guard then
+/// unsets the origin.
+pub fn set_current(origin: Origin) -> OriginGuard {
+ CURRENT.with(|current| *current.borrow_mut() = Some(Arc::new(origin)));
+ OriginGuard { _private: () }
+}
+
+/// Unsets the current origin when dropped. See [`set_current`].
+#[must_use]
+pub struct OriginGuard {
+ _private: (),
+}
+
+impl Drop for OriginGuard {
+ fn drop(&mut self) {
+ CURRENT.with(|current| *current.borrow_mut() = None);
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+ use crate::expression::Expression;
+ use spark_connect_proto as proto;
+
+ #[test]
+ fn expressions_carry_the_origin_of_their_construction() {
+ // As PySpark's Expression, which takes the current origin when it is
made, and sends
+ // it as the python_origin of its common fields.
+ assert!(Expression::column("before").to_proto().common.is_none());
+ let built = {
+ let _origin = set_current(Origin {
+ fragment: "col".to_string(),
+ call_site: "script.py:3".to_string(),
+ });
+ Expression::column("a")
+ };
+ let after = Expression::column("after");
+ assert!(after.to_proto().common.is_none());
+ assert_eq!(built, Expression::column("a"));
+ let common = built.to_proto().common.expect("the origin");
+ let Some(proto::origin::Function::PythonOrigin(origin)) =
+ common.origin.and_then(|origin| origin.function)
+ else {
+ panic!("expected a PythonOrigin");
+ };
+ assert_eq!(origin.fragment, "col");
+ assert_eq!(origin.call_site, "script.py:3");
+ }
+}
diff --git a/python/pyspark/sql/udf.py b/python/pyspark/sql/udf.py
index 67d79e9..65d846c 100644
--- a/python/pyspark/sql/udf.py
+++ b/python/pyspark/sql/udf.py
@@ -305,14 +305,12 @@ class UserDefinedFunction:
Returns a Column representing the UDF call.
"""
from pyspark import _pyspark
- from pyspark.sql.functions import col
-
- # A UDF argument is ColumnOrName: a bare string is a COLUMN NAME (like
PySpark),
- # not a string literal. Coerce str -> col(str) before handing to the
Rust layer
- # (whose to_column would otherwise treat a str as a literal). As
PySpark's, each
- # keyword argument is a NamedArgumentExpression, after the positional
ones.
- cols = [col(a) if isinstance(a, str) else a for a in args]
- named = [(k, col(v) if isinstance(v, str) else v) for k, v in
kwargs.items()]
+
+ # A UDF argument is ColumnOrName: as PySpark's, a bare string is a
column name, of
+ # which pyfunc_make_udf makes a ColumnReference, and each keyword
argument is a
+ # NamedArgumentExpression, after the positional ones.
+ cols = list(args)
+ named = list(kwargs.items())
# As PySpark's PythonUDF, the command is the pickled (func,
returnType) for the
# server-side Python worker, and pyfunc_make_udf builds the proto
output type.
diff --git a/python/pyspark/sql/udtf.py b/python/pyspark/sql/udtf.py
index 7b7d165..79ea32b 100644
--- a/python/pyspark/sql/udtf.py
+++ b/python/pyspark/sql/udtf.py
@@ -117,12 +117,12 @@ class UserDefinedTableFunction:
def __call__(self, *args: Any, **kwargs: Any) -> Any:
from pyspark import _pyspark
from pyspark.sql import SparkSession
- from pyspark.sql.functions import col
- # As PySpark's: an argument is ColumnOrName, so a bare string is a
column name, and
- # each keyword argument is a NamedArgumentExpression, after the
positional ones.
- cols = [col(a) if isinstance(a, str) else a for a in args]
- named = [(k, col(v) if isinstance(v, str) else v) for k, v in
kwargs.items()]
+ # As PySpark's: an argument is ColumnOrName, so a bare string is a
column name, of
+ # which pyfunc_make_udtf makes a ColumnReference, and each keyword
argument is a
+ # NamedArgumentExpression, after the positional ones.
+ cols = list(args)
+ named = list(kwargs.items())
return _pyspark.functions.pyfunc_make_udtf(
SparkSession.active(),
self.name,
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 61042fa..e11b270 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -783,6 +783,51 @@ def
test_timedelta_is_a_day_time_interval_as_pyspark(session):
session.createDataFrame([("x",)], "d interval day to second")
+def test_errors_carry_the_dataframe_query_contexts_of_pyspark(session):
+ # As PySpark's: an error in an expression that col() or a Column method
built reports its
+ # DataFrameQueryContext, of the function and of its call site in this file.
+ import inspect
+
+ from pyspark.errors import ArithmeticException, QueryContextType
+
+ def contexts(df):
+ with pytest.raises((AnalysisException, ArithmeticException)) as error:
+ df.collect()
+ return [
+ (c.contextType(), c.fragment(), c.callSite().rsplit("/", 1)[-1])
+ for c in error.value.getQueryContext()
+ ]
+
+ here = f"{os.path.basename(__file__)}:{{}}"
+ line = inspect.currentframe().f_lineno
+ assert contexts(session.range(1).select("no_such_col")) == [
+ (QueryContextType.DataFrame, "col", here.format(line + 1))
+ ]
+ assert contexts(session.range(1).select(F.col("nope") + 1)) == [
+ (QueryContextType.DataFrame, "col", here.format(line + 4))
+ ]
+ session.conf.set("spark.sql.ansi.enabled", "true")
+ assert contexts(session.range(1).select(F.lit(1) / F.lit(0))) == [
+ (QueryContextType.DataFrame, "__truediv__", here.format(line + 8))
+ ]
+ session.conf.unset("spark.sql.ansi.enabled")
+ # repartitionById makes a column of a name with col(), while a UDF call
refers to a
+ # column by its name without an origin.
+ assert contexts(session.range(1).repartitionById(2, "nope")) == [
+ (QueryContextType.DataFrame, "col", here.format(line + 14))
+ ]
+ udf = F.udf(lambda x: x, "long")
+ assert contexts(session.range(1).select(udf("nope"), udf(x="nope2"))) == []
+ # sampleBy takes a column of a name with col(), as PySpark's F._to_col.
+ assert contexts(session.range(1).sampleBy("nope", {0: 1.0})) == [
+ (QueryContextType.DataFrame, "col", here.format(line + 20))
+ ]
+ # unpivot takes its columns of names with col(), as PySpark's F._to_col.
+ assert contexts(session.range(1).unpivot("nope", ["id"], "v", "w")) == [
+ (QueryContextType.DataFrame, "col", here.format(line + 24))
+ ]
+
+
def test_server_errors_carry_their_condition_and_details(session):
from pyspark.errors import AnalysisException, ArithmeticException,
ParseException
from pyspark.errors.exceptions.connect import SparkConnectGrpcException,
SQLQueryContext
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 404731a..9852f80 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -5636,6 +5636,20 @@ def mock_connect(_shared_mock_connect):
spark.stop()
[email protected](autouse=True, scope="session")
+def _debugging_read_without_a_session():
+ # PySpark's is_debugging_enabled reads
spark.python.sql.dataFrameDebugging.enabled of the
+ # active session at the first expression of a Column method, and keeps it
for the
+ # interpreter. It is read without a session, before any test, so that a
test whose session
+ # has no server, such as one of _offline_session(), does not wait to read
it, whatever
+ # the tests that run. The tests of origins run in their own interpreters.
+ from pyspark.errors.utils import is_debugging_enabled
+ from pyspark.sql import SparkSession
+
+ assert SparkSession.getActiveSession() is None
+ is_debugging_enabled()
+
+
@pytest.fixture(autouse=True)
def _no_session_left_active():
# Each test starts without an active session, as a new interpreter does.
What reads the
@@ -6550,6 +6564,60 @@ def test_window_and_lambda_reprs_as_pyspark():
repr(F.transform("a", lambda x: x + 1)))
+def test_expressions_carry_their_origins_as_pyspark():
+ # As PySpark's _with_origin: col() and the Column methods give the
expressions that they
+ # build the origin of their name and their call site in user code, as the
python_origin of
+ # their common fields, unless spark.python.sql.dataFrameDebugging.enabled,
which PySpark
+ # reads once, is false. In new interpreters, whose first column of a
session reads it.
+ import subprocess
+ import sys
+ from pathlib import Path
+
+ pytest.importorskip("grpc")
+ code = """
+import sys
+sys.path.insert(0, sys.argv[1])
+from _mock_connect_server import mock_connect_session, parse_message
+from pyspark.sql import functions as F
+
+def origins(expression):
+ # Expression.common=18 -> ExpressionCommon.origin=1 ->
Origin.python_origin=1:
+ # fragment=1, call_site=2; .unresolved_function=3, whose arguments=2.
+ expression = parse_message(expression)
+ found = []
+ if 18 in expression:
+ origin =
parse_message(parse_message(parse_message(expression[18][0])[1][0])[1][0])
+ found.append((origin[1][0].decode(), origin[2][0].decode()))
+ if 3 in expression:
+ for argument in parse_message(expression[3][0]).get(2, []):
+ found += origins(argument)
+ return found
+
+if __name__ == "__main__":
+ with mock_connect_session() as (spark, service):
+ service.confs = dict(arg.split("=") for arg in sys.argv[2:])
+ column = F.col("a") + 1
+ spark.range(1).select(column).collect()
+ # ExecutePlanRequest.plan=3, Plan.root=1, Relation.project=3,
Project.expressions=3.
+ plan =
parse_message(parse_message(service.execute_requests[-1])[3][0])[1][0]
+ project = parse_message(parse_message(plan)[3][0])
+ print(origins(project[3][0]))
+"""
+ tests = str(Path(__file__).resolve().parent)
+
+ def run(*confs):
+ result = subprocess.run(
+ [sys.executable, "-c", code, tests, *confs], check=True,
capture_output=True,
+ text=True,
+ )
+ return result.stdout.strip().splitlines()[-1]
+
+ assert run() == (
+ "[('__add__', '<string>:23'), ('col', '<string>:23'), ('__add__',
'<string>:23')]"
+ )
+ assert run("spark.python.sql.dataFrameDebugging.enabled=false") == "[]"
+
+
@pytest.mark.parametrize("column,expected", [
# instr and regexp_replace are plain functions, as other functions are.
(lambda F: F.instr("s", "b"), "instr(s, b)"),
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]