This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 3135a04 [SPARK-60953][CONNECT] Render instr, regexp_replace and
call_function columns as PySpark does
3135a04 is described below
commit 3135a04ed7a7ace0e401365f45eaa2caacd6aa48
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Fri Nov 6 12:07:28 2026 +0900
[SPARK-60953][CONNECT] Render instr, regexp_replace and call_function
columns as PySpark does
### What changes were proposed in this pull request?
[SPARK-60953](https://issues.apache.org/jira/browse/SPARK-60953): `instr`,
`regexp_replace` and `call_function` columns are now built and rendered as in
PySpark Connect.
- **`instr` and `regexp_replace`** are plain function calls
(`UnresolvedFunction`), as `_invoke_function_over_columns` builds them, with
their optional 4.3.0 arguments when given. Before, they were `CallFunction`
expressions so that the optional arguments could be passed.
- **`call_function`** still builds a `CallFunction`. It now renders as
PySpark's `CallFunction.__repr__`, `CallFunction('<name>', <arg>, ...)`,
instead of the Rust debug form of the expression.
- **Rust API:** for the optional arguments, `spark_connect::functions`
gains `instr_with_start`, `instr_with_start_and_occurrence` and
`regexp_replace_with_position`.
| Column | PySpark 4.3.0 and now | Before |
| --- | --- | --- |
| `F.instr("s", "b", occurrence=2)` | `Column<'instr(s, b, 1, 2)'>` |
`Column<'CallFunction(CallFunctionWrapper { function_name: "instr", arguments:
[ColumnReference(...` |
| `F.regexp_replace("s", "a", "b")` | `Column<'regexp_replace(s, a, b)'>` |
the same debug form |
| `F.call_function("abs", "x")` | `Column<'CallFunction('abs', x)'>` | the
same debug form |
### Why are the changes needed?
The repr of each of these columns was the Rust debug dump of the
expression. That dump also appeared in the reprs of the columns built from them
and in error messages. `instr` and `regexp_replace` also sent a different
expression from PySpark's.
### Does this PR introduce _any_ user-facing change?
Yes, these columns render as in PySpark 4.3.0, and `instr` and
`regexp_replace` send the same expressions as PySpark. Their results are
unchanged.
### How was this patch tested?
- A probe of 14 columns gives the same reprs as official PySpark 4.3.0: the
forms of `instr` and `regexp_replace`, and `call_function` with no argument,
nested, aliased and with a null literal. Before this change, all 14 differed.
- Against a local Spark 4.3.0 server, the column names, values and error of
a probe of these functions are the same as with official PySpark 4.3.0.
- New unit tests check the builders and the render of `CallFunction`. The
expression-sharing test now pins the new render; the debug form is unchanged.
- A new offline test checks the reprs, and a new live test checks the
column names and values. Both also pass with official PySpark 4.3.0.
- The lib tests and every integration test, e2e included, against the local
server (644 passed), the build of all tests with `wasm-udf`, the offline Python
suites (624 passed, 8 live cases skipped), and the live drop-in and Arrow
suites (58 passed) and live cases (10 passed). Clippy reports no diagnostics in
the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #178 from HyukjinKwon/SPARK-60953-callfunction-repr.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/dispatch_generated.rs | 10 ++-
crates/spark-connect/src/expression.rs | 24 +++++-
crates/spark-connect/src/functions.rs | 101 +++++++++++++++++++++++
crates/spark-connect/tests/expression_sharing.rs | 3 +-
python/tests/test_dropin_e2e.py | 19 +++++
python/tests/test_dropin_offline.py | 22 +++++
6 files changed, 170 insertions(+), 9 deletions(-)
diff --git a/crates/pyspark-rs/src/dispatch_generated.rs
b/crates/pyspark-rs/src/dispatch_generated.rs
index 2d4352f..e6356c1 100644
--- a/crates/pyspark-rs/src/dispatch_generated.rs
+++ b/crates/pyspark-rs/src/dispatch_generated.rs
@@ -1379,8 +1379,11 @@ pub fn call_builtin(name: &str, args:
Vec<spark_connect::column::Column>) -> PyR
},
"instr" => {
if args.len() < 2 { return
Err(PyErr::new::<pyo3::exceptions::PyTypeError, _>(format!("Missing required
arguments for instr: expected at least 2, got {}", args.len()))); }
- // Forward the optional 4.3.0 start/occurrence args too (not
just the first 2).
- Ok(spark_funcs::call_function("instr", args.clone()))
+ match args.len() {
+ 2 => Ok(spark_funcs::instr(args[0].clone(),
args[1].clone())),
+ 3 => Ok(spark_funcs::instr_with_start(args[0].clone(),
args[1].clone(), args[2].clone())),
+ _ =>
Ok(spark_funcs::instr_with_start_and_occurrence(args[0].clone(),
args[1].clone(), args[2].clone(), args[3].clone())),
+ }
},
"json_tuple" => Ok(spark_funcs::json_tuple(args.clone())),
"least" => {
@@ -1729,8 +1732,7 @@ pub fn call_builtin(name: &str, args:
Vec<spark_connect::column::Column>) -> PyR
},
"regexp_replace" => {
if args.len() < 3 { return
Err(PyErr::new::<pyo3::exceptions::PyTypeError, _>(format!("Missing required
arguments for regexp_replace: expected at least 3, got {}", args.len()))); }
- // Forward the optional 4.3.0 position arg too (not just the
first 3).
- Ok(spark_funcs::call_function("regexp_replace", args.clone()))
+ if args.len() >= 4 {
Ok(spark_funcs::regexp_replace_with_position(args[0].clone(), args[1].clone(),
args[2].clone(), args[3].clone())) } else {
Ok(spark_funcs::regexp_replace(args[0].clone(), args[1].clone(),
args[2].clone())) }
},
"rpad" => {
if args.len() < 3 { return
Err(PyErr::new::<pyo3::exceptions::PyTypeError, _>(format!("Missing required
arguments for rpad: expected at least 3, got {}", args.len()))); }
diff --git a/crates/spark-connect/src/expression.rs
b/crates/spark-connect/src/expression.rs
index 7331052..01ca28e 100644
--- a/crates/spark-connect/src/expression.rs
+++ b/crates/spark-connect/src/expression.rs
@@ -174,7 +174,7 @@ impl Expression {
),
},
ExpressionNode::SQLExpression(sql) => sql.clone(),
- ExpressionNode::CallFunction(_) => format!("{self:?}"),
+ ExpressionNode::CallFunction(cf) => cf.render(),
ExpressionNode::WindowExpression(_) => format!("{self:?}"),
ExpressionNode::LambdaFunction(lf) => format!("{lf:?}"),
ExpressionNode::UnresolvedNamedLambdaVariable(var) =>
format!("{var:?}"),
@@ -833,6 +833,20 @@ impl CallFunctionWrapper {
}
}
+ /// Mirrors `CallFunction.__repr__`: `CallFunction('<name>', <arg>, ...)`.
+ pub fn render(&self) -> String {
+ if self.arguments.is_empty() {
+ format!("CallFunction('{}')", self.function_name)
+ } else {
+ let args: Vec<String> =
self.arguments.iter().map(Expression::render).collect();
+ format!(
+ "CallFunction('{}', {})",
+ self.function_name,
+ args.join(", ")
+ )
+ }
+ }
+
/// Convert to protobuf.
pub fn to_proto(&self) -> proto::Expression {
let mut expr = proto::Expression::default();
@@ -1519,9 +1533,11 @@ mod tests {
fn test_call_function_wrapper_render() {
let cf = CallFunctionWrapper::new("my_func", vec![col("a"), col("b")]);
let expr: Expression =
ExpressionNode::CallFunction(Box::new(cf)).into();
- // CallFunction renders as debug format
- let rendered = expr.render();
- assert!(rendered.contains("CallFunctionWrapper"));
+ // As `CallFunction.__repr__` in PySpark Connect.
+ assert_eq!(expr.render(), "CallFunction('my_func', a, b)");
+ let no_args = CallFunctionWrapper::new("current_date", vec![]);
+ let expr: Expression =
ExpressionNode::CallFunction(Box::new(no_args)).into();
+ assert_eq!(expr.render(), "CallFunction('current_date')");
}
#[test]
diff --git a/crates/spark-connect/src/functions.rs
b/crates/spark-connect/src/functions.rs
index bdd049c..3941ed5 100644
--- a/crates/spark-connect/src/functions.rs
+++ b/crates/spark-connect/src/functions.rs
@@ -2238,6 +2238,38 @@ pub fn instr(col1: Column, col2: Column) -> Column {
)
}
+/// Mirrors `pyspark.sql.functions.instr(str, substr, start)`: searches from
the
+/// 1-based position `start`.
+pub fn instr_with_start(col1: Column, col2: Column, start: Column) -> Column {
+ func(
+ "instr",
+ vec![
+ col1.expression().clone(),
+ col2.expression().clone(),
+ start.expression().clone(),
+ ],
+ )
+}
+
+/// Mirrors `pyspark.sql.functions.instr(str, substr, start, occurrence)`: the
+/// position of the `occurrence`-th match, searching from the 1-based position
`start`.
+pub fn instr_with_start_and_occurrence(
+ col1: Column,
+ col2: Column,
+ start: Column,
+ occurrence: Column,
+) -> Column {
+ func(
+ "instr",
+ vec![
+ col1.expression().clone(),
+ col2.expression().clone(),
+ start.expression().clone(),
+ occurrence.expression().clone(),
+ ],
+ )
+}
+
/// Mirrors `pyspark.sql.functions.json_tuple`.
pub fn json_tuple<C: Into<Column>>(cols: impl IntoIterator<Item = C>) ->
Column {
let cols: Vec<Column> = cols.into_iter().map(Into::into).collect();
@@ -3022,6 +3054,25 @@ pub fn regexp_replace(col1: Column, col2: Column, col3:
Column) -> Column {
)
}
+/// Mirrors `pyspark.sql.functions.regexp_replace(string, pattern, replacement,
+/// position)`: replaces the matches from the 1-based position `position`.
+pub fn regexp_replace_with_position(
+ col1: Column,
+ col2: Column,
+ col3: Column,
+ position: Column,
+) -> Column {
+ func(
+ "regexp_replace",
+ vec![
+ col1.expression().clone(),
+ col2.expression().clone(),
+ col3.expression().clone(),
+ position.expression().clone(),
+ ],
+ )
+}
+
/// Mirrors `pyspark.sql.functions.rpad`.
pub fn rpad(col1: Column, col2: Column, col3: Column) -> Column {
func(
@@ -4544,6 +4595,56 @@ mod tests {
let _ = map_res.to_proto();
}
+ #[test]
+ fn test_instr_and_regexp_replace_optional_args() {
+ // Each form is a plain function call, as
`_invoke_function_over_columns` builds.
+ use spark_connect_proto::expression::ExprType;
+ let unresolved = |c: Column| match c.to_proto().expr_type {
+ Some(ExprType::UnresolvedFunction(f)) => (f.function_name,
f.arguments.len()),
+ other => panic!("expected an UnresolvedFunction, got {other:?}"),
+ };
+ let s = || col("s");
+ assert_eq!(
+ unresolved(instr(s(), Column::new(lit_str("b")))),
+ ("instr".to_string(), 2)
+ );
+ assert_eq!(
+ unresolved(instr_with_start(
+ s(),
+ Column::new(lit_str("b")),
+ Column::new(lit_int(2))
+ )),
+ ("instr".to_string(), 3)
+ );
+ let four = instr_with_start_and_occurrence(
+ s(),
+ Column::new(lit_str("b")),
+ Column::new(lit_int(1)),
+ Column::new(lit_int(2)),
+ );
+ assert_eq!(four.expression().render(), "instr(s, b, 1, 2)");
+ assert_eq!(unresolved(four), ("instr".to_string(), 4));
+ assert_eq!(
+ unresolved(regexp_replace(
+ s(),
+ Column::new(lit_str("a")),
+ Column::new(lit_str("b"))
+ )),
+ ("regexp_replace".to_string(), 3)
+ );
+ let positioned = regexp_replace_with_position(
+ s(),
+ Column::new(lit_str("a")),
+ Column::new(lit_str("b")),
+ Column::new(lit_int(2)),
+ );
+ assert_eq!(
+ positioned.expression().render(),
+ "regexp_replace(s, a, b, 2)"
+ );
+ assert_eq!(unresolved(positioned), ("regexp_replace".to_string(), 4));
+ }
+
#[test]
fn test_sort_functions_serialize() {
let test_col = col("test");
diff --git a/crates/spark-connect/tests/expression_sharing.rs
b/crates/spark-connect/tests/expression_sharing.rs
index 17cf2a5..a4865f4 100644
--- a/crates/spark-connect/tests/expression_sharing.rs
+++ b/crates/spark-connect/tests/expression_sharing.rs
@@ -164,7 +164,8 @@ fn
debug_render_and_proto_keep_the_existing_representation() {
"arguments: [Literal(Integer(7))] })"
);
assert_eq!(format!("{call:?}"), expected_debug);
- assert_eq!(call.render(), expected_debug);
+ // The render is PySpark's `CallFunction.__repr__`.
+ assert_eq!(call.render(), "CallFunction('f', 7)");
assert_eq!(call.to_proto(), expected_proto);
let source = col("value") + lit(1);
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index fc5c388..15c5714 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -62,6 +62,25 @@ def test_conf_get_with_a_none_default(session):
session.conf.get(key)
+def test_instr_and_regexp_replace_apply_their_optional_arguments(session):
+ # Column b holds "c", so the literal "b" and the column b find different
positions.
+ df = session.createDataFrame([("abcabc", "c", "a", "z")], ["s", "b", "p",
"r"])
+ out = df.select(
+ F.instr("s", "b"),
+ F.instr("s", "b", 3),
+ F.instr("s", "b", occurrence=2),
+ F.instr(F.col("s"), F.col("b"), F.lit(4)),
+ F.regexp_replace("s", "a", "z", 2),
+ F.regexp_replace("s", F.col("p"), F.col("r")),
+ F.call_function("instr", "s", F.lit("b")),
+ )
+ assert out.columns == [
+ "instr(s, b)", "instr(s, b, 3, 1)", "instr(s, b, 1, 2)", "instr(s, b,
4, 1)",
+ "regexp_replace(s, a, z, 2)", "regexp_replace(s, p, r, 1)", "instr(s,
b)",
+ ]
+ assert tuple(out.first()) == (2, 5, 5, 6, "abczbc", "zbczbc", 2)
+
+
@pytest.mark.parametrize("by_name", [True, False])
def test_repartition_by_id_takes_a_column_name(session, by_name):
df = session.range(12).withColumn("p", (F.col("id") % 3).cast("int"))
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 39e0638..7a3f10f 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -4153,6 +4153,28 @@ def test_instr_regexp_replace_optional_args():
F._call_function = orig
[email protected]("column,expected", [
+ # instr and regexp_replace are plain functions, as other functions are.
+ (lambda F: F.instr("s", "b"), "instr(s, b)"),
+ (lambda F: F.instr("s", "b", 2), "instr(s, b, 2)"),
+ (lambda F: F.instr("s", "b", occurrence=2), "instr(s, b, 1, 2)"),
+ (lambda F: F.instr("s", "b", 2, 3), "instr(s, b, 2, 3)"),
+ (lambda F: F.regexp_replace("s", "a", "b"), "regexp_replace(s, a, b)"),
+ (lambda F: F.regexp_replace("s", "a", "b", 2), "regexp_replace(s, a, b,
2)"),
+ (lambda F: F.regexp_replace("s", F.col("p"), F.col("r")),
"regexp_replace(s, p, r)"),
+ # call_function builds a CallFunction, which renders as in PySpark.
+ (lambda F: F.call_function("abs", "x"), "CallFunction('abs', x)"),
+ (lambda F: F.call_function("current_date"),
"CallFunction('current_date')"),
+ (lambda F: F.upper(F.call_function("abs", "x")),
"upper(CallFunction('abs', x))"),
+ (lambda F: F.call_function("coalesce", "x", F.lit(None)).alias("y"),
+ "CallFunction('coalesce', x, NULL) AS y"),
+])
+def test_function_calls_render_as_pyspark(column, expected):
+ from pyspark.sql import functions as F
+
+ assert repr(column(F)) == f"Column<'{expected}'>"
+
+
def test_variant_delete_requires_paths():
from pyspark.errors import PySparkValueError
from pyspark.sql import functions as F
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]