This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 5e12e05 [SPARK-60948][CONNECT][PYTHON] Render Column reprs as PySpark
Connect does
5e12e05 is described below
commit 5e12e05d814de00036bb29474cd42f7ca8af2613
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Fri Nov 6 14:14:15 2026 +0900
[SPARK-60948][CONNECT][PYTHON] Render Column reprs as PySpark Connect does
### What changes were proposed in this pull request?
[SPARK-60948](https://issues.apache.org/jira/browse/SPARK-60948): Column
reprs now follow the `Expression` reprs of PySpark Connect 4.3.0.
| Expression | PySpark 4.3.0 and now | Before |
| --- | --- | --- |
| `F.col("id") + 1` | `+(id, 1)` | `(id + 1)` |
| `-F.col("a")`, `~F.col("a")` | `negative(a)`, `not(a)` | `(- a)`, `(NOT
a)` |
| `F.col("a").between(1, 2)` | `and(>=(a, 1), <=(a, 2))` | `((a >= 1) and
(a <= 2))` |
| `F.countDistinct("a")` | `count(distinct a)` | `count(a)` |
| `F.col("a").cast("int")`, `try_cast` | `CAST(a AS INT)`, `TRY_CAST(a AS
INT)` | `CAST(a AS int)` for both |
| `F.col("*")`, `F.col("t.*")` | `unresolvedstar()`, `unresolvedstar(t.*)`
| `*`, `t.*` |
| `F.col("a")[0]` | `a['0']` | `a[0]` |
| `F.lit(True)`, `F.lit(b"xy")` | `True`, `b'xy'` | `true`, `[120, 121]` |
| `F.lit(date(2020, 1, 2))` | `2020-01-02` | `18263` |
| `F.lit(datetime(2020, 1, 2, 3, 4, 5))` | `2020-01-02 03:04:05.000000`, in
the local time zone | microseconds since the epoch |
| `F.lit(1e20)` | `1e+20`, Python's float repr | `100000000000000000000` |
| a window function | `WindowExpression(sum(a), (WindowSpec(PartitionBy(g),
OrderBy(o DESC NULLS LAST), WindowFrame(ROW_FRAME, -1, 1))))` | the Rust debug
form |
| `F.transform("a", lambda x: x + 1)` | `transform(a, LambdaFunction(+(x_0,
1), x_0))` | the Rust debug form |
Also:
- a day-time interval literal renders as pandas' ISO duration,
`P1DT0H0M2.000003S`;
- a time literal renders as `01:02:00.000000`;
- a regex column renders as `UnresolvedRegex(...)`, and a shuffle partition
id as `DirectShufflePartitionID(child=...)`;
- a user-defined function renders as its name and arguments;
- `WindowSpec.__repr__` shows the partitioning, ordering and frame, as in
PySpark, instead of `WindowSpec()`.
The crate now depends on `chrono` directly, already in the lockfile through
Arrow, for the local time of timestamp literals, as Python's
`datetime.fromtimestamp` gives it. AGENTS.md now describes the render this way.
### Why are the changes needed?
The repr differed for 50 of 77 common expressions. It also appears in error
messages, such as `SparkSession.sql`'s for a keyword argument that is not a
column reference. Window, lambda and UDF columns rendered the Rust debug form
of the expression.
### Does this PR introduce _any_ user-facing change?
Yes, Column reprs are those of PySpark Connect 4.3.0. Plans are unchanged.
### How was this patch tested?
- A probe of 77 expressions gives the same reprs as official PySpark 4.3.0.
One is left: a window ordered by a plain column renders `ASC NULLS LAST` here.
That is a sort-order bug rather than a repr bug, which #191 (SPARK-60990) fixes.
- A probe of 22 more expressions gives the same reprs in the UTC,
`America/Los_Angeles` and `Asia/Seoul` time zones: timestamps, dates, bytes,
special floats, decimals, window frames and specs, and casts to nested types.
The exception is `timedelta` literals, which #189 (SPARK-60986) adds, and whose
repr this change formats.
- New offline tests check 30 reprs, also in a non-UTC time zone. They also
pass with official PySpark 4.3.0.
- The unit tests and the expression-sharing tests now pin the new reprs.
- The lib tests and every integration test, e2e included, against a local
Spark 4.3.0 server (647 passed), the build of all tests with `wasm-udf`, the
offline Python suites (717 passed, 9 skipped), and the live drop-in and Arrow
suites (70 passed) and live cases (10 passed). Clippy reports no diagnostics in
the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #192 from HyukjinKwon/SPARK-60948-column-repr.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
AGENTS.md | 2 +-
Cargo.lock | 1 +
crates/pyspark-rs/src/window.rs | 3 +-
crates/spark-connect/Cargo.toml | 2 +
crates/spark-connect/src/expression.rs | 414 +++++++++++++++++------
crates/spark-connect/src/window.rs | 20 ++
crates/spark-connect/tests/expression_sharing.rs | 8 +-
python/tests/test_dropin_offline.py | 60 ++++
8 files changed, 409 insertions(+), 101 deletions(-)
diff --git a/AGENTS.md b/AGENTS.md
index 289b62c..119b4b4 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -443,7 +443,7 @@ For every method matching a pyspark name:
on-Spark's `spark_column_equals` decides column identity by comparing repr
strings (it does
`repr(left).replace(backtick, "") == repr(right)...`); a generic
`"Column()"` makes every column
look equal and collapses frames to `(n, 0)`. `Expression::render()`
mirrors the pyspark-connect
- expression `__repr__` (infix binary ops, `AS`, `CAST`, star, ...).
`pyspark.sql.connect.column`
+ expression `__repr__` (operators as functions, `+(a, 1)`, `AS`, `CAST`,
`unresolvedstar()`, ...). `pyspark.sql.connect.column`
re-exports the Rust `Column`, so its `isinstance(x, ConnectColumn)` checks
already pass.
- **Internal pandas functions dispatch by NAME, not by client
reimplementation.**
`pyspark.sql.internal.InternalFunction` (`distributed_sequence_id`,
`pandas_product`,
diff --git a/Cargo.lock b/Cargo.lock
index 3b65c41..8fba36c 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -61,6 +61,7 @@ dependencies = [
"arrow",
"base64 0.21.7",
"bytes",
+ "chrono",
"datafusion",
"futures",
"polars",
diff --git a/crates/pyspark-rs/src/window.rs b/crates/pyspark-rs/src/window.rs
index 09b2b70..9d96729 100644
--- a/crates/pyspark-rs/src/window.rs
+++ b/crates/pyspark-rs/src/window.rs
@@ -133,8 +133,9 @@ impl PyWindowSpec {
Ok(PyWindowSpec::new(new_spec))
}
+ /// As `WindowSpec.__repr__` of PySpark Connect.
fn __repr__(&self) -> String {
- "WindowSpec()".to_string()
+ self.spec.repr()
}
}
diff --git a/crates/spark-connect/Cargo.toml b/crates/spark-connect/Cargo.toml
index aea3ff3..422e671 100644
--- a/crates/spark-connect/Cargo.toml
+++ b/crates/spark-connect/Cargo.toml
@@ -21,6 +21,8 @@ thiserror.workspace = true
tokio.workspace = true
futures.workspace = true
serde_json = "1.0"
+# Local time for the reprs of timestamp literals, as Python's
`datetime.fromtimestamp`.
+chrono = { version = "0.4", default-features = false, features = ["clock"] }
uuid.workspace = true
# Optional: only pulled in by the `wasm-udf` feature.
diff --git a/crates/spark-connect/src/expression.rs
b/crates/spark-connect/src/expression.rs
index 54e25c5..a818ef3 100644
--- a/crates/spark-connect/src/expression.rs
+++ b/crates/spark-connect/src/expression.rs
@@ -133,28 +133,35 @@ impl Expression {
ExpressionNode::ColumnReference(col_ref) => col_ref.name.clone(),
ExpressionNode::UnresolvedFunction(func) => func.render(),
ExpressionNode::UnresolvedStar(target) => match target {
- Some(t) => t.clone(),
- None => "*".to_string(),
+ Some(t) => format!("unresolvedstar({t})"),
+ None => "unresolvedstar()".to_string(),
},
// As `ColumnAlias.__repr__`: the names joined by commas.
ExpressionNode::Alias(alias) => {
format!("{} AS {}", alias.child.render(),
alias.names.join(","))
}
+ // As `CastExpression.__repr__`: the type in upper case.
ExpressionNode::Cast(cast) => {
let type_str = match &cast.target {
CastTarget::Type(dt) => dt.simple_string(),
CastTarget::TypeStr(s) => s.clone(),
+ }
+ .to_uppercase();
+ let name = if cast.eval_mode == Some(CastEvalMode::Try) {
+ "TRY_CAST"
+ } else {
+ "CAST"
};
- format!("CAST({} AS {})", cast.child.render(), type_str)
+ format!("{name}({} AS {type_str})", cast.child.render())
}
ExpressionNode::DirectShufflePartitionId(child) => {
- format!("DIRECT_SHUFFLE_PARTITION_ID({})", child.render())
+ format!("DirectShufflePartitionID(child={})", child.render())
}
- ExpressionNode::UnresolvedRegex(col_name) => col_name.clone(),
+ ExpressionNode::UnresolvedRegex(col_name) =>
format!("UnresolvedRegex({col_name})"),
ExpressionNode::SortOrder(sort) => sort.render(),
ExpressionNode::CaseWhen(case_when) => case_when.render(),
ExpressionNode::UnresolvedExtractValue(ev) => {
- format!("{}[{}]", ev.child.render(), ev.extraction.render())
+ format!("{}['{}']", ev.child.render(), ev.extraction.render())
}
ExpressionNode::UpdateFields(uf) => match &uf.value_expression {
Some(v) => format!(
@@ -171,10 +178,22 @@ impl Expression {
},
ExpressionNode::SQLExpression(sql) => sql.clone(),
ExpressionNode::CallFunction(cf) => cf.render(),
- ExpressionNode::WindowExpression(_) => format!("{self:?}"),
- ExpressionNode::LambdaFunction(lf) => format!("{lf:?}"),
- ExpressionNode::UnresolvedNamedLambdaVariable(var) =>
format!("{var:?}"),
- ExpressionNode::CommonInlineUserDefinedFunction(_) =>
format!("{self:?}"),
+ ExpressionNode::WindowExpression(we) => we.render(),
+ // As `LambdaFunction.__repr__` and
`UnresolvedNamedLambdaVariable.__repr__`.
+ ExpressionNode::LambdaFunction(lf) => {
+ let args: Vec<&str> = lf.arguments.iter().map(|a|
a.name_parts.as_str()).collect();
+ format!(
+ "LambdaFunction({}, {})",
+ lf.function.render(),
+ args.join(", ")
+ )
+ }
+ ExpressionNode::UnresolvedNamedLambdaVariable(var) =>
var.name_parts.clone(),
+ // As `CommonInlineUserDefinedFunction.__repr__`.
+ ExpressionNode::CommonInlineUserDefinedFunction(udf) => {
+ let args: Vec<String> =
udf.arguments.iter().map(Expression::render).collect();
+ format!("{}({})", udf.function_name, args.join(", "))
+ }
}
}
@@ -344,37 +363,45 @@ impl LiteralExpression {
expr
}
- /// Render the literal value, mirroring `LiteralExpression.__repr__`.
+ /// Render the literal value, mirroring `LiteralExpression.__repr__`: the
Python value,
+ /// as `f"{value}"` shows it, and a date, time, timestamp or day-time
interval in the
+ /// formats that it gives them.
pub fn render(&self) -> String {
match self {
LiteralExpression::Null(_) => "NULL".to_string(),
- LiteralExpression::Boolean(b) => {
- if *b {
- "true".to_string()
- } else {
- "false".to_string()
- }
- }
+ LiteralExpression::Boolean(b) => if *b { "True" } else { "False"
}.to_string(),
LiteralExpression::Byte(v)
| LiteralExpression::Short(v)
| LiteralExpression::Integer(v) => v.to_string(),
LiteralExpression::Long(v) => v.to_string(),
- LiteralExpression::Float(v) => v.to_string(),
- LiteralExpression::Double(v) => v.to_string(),
+ LiteralExpression::Float(v) => python_float_repr(*v as f64),
+ LiteralExpression::Double(v) => python_float_repr(*v),
LiteralExpression::Decimal { value, .. } => value.clone(),
LiteralExpression::String(v) => v.clone(),
- LiteralExpression::Binary(v) => format!("{v:?}"),
- LiteralExpression::Date(v) => v.to_string(),
- LiteralExpression::Timestamp(v) |
LiteralExpression::TimestampNtz(v) => v.to_string(),
- LiteralExpression::DayTimeInterval(v) => v.to_string(),
- LiteralExpression::Time { nano, .. } => nano.to_string(),
+ LiteralExpression::Binary(v) => python_bytes_repr(v),
+ LiteralExpression::Date(v) => date_repr(*v),
+ LiteralExpression::Timestamp(v) => timestamp_repr(*v, true),
+ LiteralExpression::TimestampNtz(v) => timestamp_repr(*v, false),
+ LiteralExpression::DayTimeInterval(v) => iso_duration(*v),
+ LiteralExpression::Time { nano, .. } => time_repr(*nano),
+ // A Python list, as `str(list)` shows its elements.
LiteralExpression::Array { elements, .. } => {
- let inner: Vec<String> = elements.iter().map(|e|
e.render()).collect();
+ let inner: Vec<String> = elements.iter().map(|e|
e.render_in_container()).collect();
format!("[{}]", inner.join(", "))
}
}
}
+ /// The literal as Python's `repr` shows it within a container, such as a
list: a string
+ /// quoted, and None for a null.
+ fn render_in_container(&self) -> String {
+ match self {
+ LiteralExpression::Null(_) => "None".to_string(),
+ LiteralExpression::String(v) => python_str_repr(v),
+ other => other.render(),
+ }
+ }
+
/// Construct a null literal of a specific type.
pub fn null(data_type: DataType) -> Self {
LiteralExpression::Null(data_type)
@@ -552,27 +579,13 @@ impl UnresolvedFunction {
/// binary operators render infix as `(a op b)`, the unary `not`/negate
render
/// prefixed, everything else as `name(arg, arg, ...)`.
pub fn render(&self) -> String {
- const INFIX_OPS: &[&str] = &[
- "+", "-", "*", "/", "%", "==", "!=", "<", "<=", ">", ">=", "and",
"or", "&", "|", "^",
- "<=>",
- ];
- if self.args.len() == 2 && INFIX_OPS.contains(&self.name.as_str()) {
- return format!(
- "({} {} {})",
- self.args[0].render(),
- self.name,
- self.args[1].render()
- );
- }
- if self.args.len() == 1 {
- match self.name.as_str() {
- "not" => return format!("(NOT {})", self.args[0].render()),
- "negative" | "negate" => return format!("(- {})",
self.args[0].render()),
- _ => {}
- }
- }
+ // As `UnresolvedFunction.__repr__`, which renders an operator as a
function too.
let inner: Vec<String> = self.args.iter().map(|a|
a.render()).collect();
- format!("{}({})", self.name, inner.join(", "))
+ if self.is_distinct {
+ format!("{}(distinct {})", self.name, inner.join(", "))
+ } else {
+ format!("{}({})", self.name, inner.join(", "))
+ }
}
pub fn to_proto(&self) -> proto::Expression {
@@ -888,6 +901,51 @@ pub enum FrameBoundary {
UnboundedFollowing,
}
+impl FrameBoundary {
+ /// The boundary as the int that `rowsBetween` and `rangeBetween` take, as
+ /// `WindowFrame.__repr__` shows it.
+ pub fn value(&self) -> i64 {
+ match self {
+ FrameBoundary::UnboundedPreceding => i64::MIN,
+ FrameBoundary::Preceding(n) => -n,
+ FrameBoundary::CurrentRow => 0,
+ FrameBoundary::Following(n) => *n,
+ FrameBoundary::UnboundedFollowing => i64::MAX,
+ }
+ }
+}
+
+/// As `WindowSpec.__repr__` of PySpark Connect: its partitioning, ordering
and frame. A
+/// frame type of 1 is a row frame, and another a range frame, as in the proto.
+pub fn window_spec_repr(
+ partition_spec: &[Expression],
+ order_spec: &[SortOrder],
+ frame_spec: Option<&(u32, FrameBoundary, FrameBoundary)>,
+) -> String {
+ let mut parts = Vec::new();
+ if !partition_spec.is_empty() {
+ let partitions: Vec<String> =
partition_spec.iter().map(Expression::render).collect();
+ parts.push(format!("PartitionBy({})", partitions.join(", ")));
+ }
+ if !order_spec.is_empty() {
+ let orders: Vec<String> =
order_spec.iter().map(SortOrder::render).collect();
+ parts.push(format!("OrderBy({})", orders.join(", ")));
+ }
+ if let Some((frame_type, lower, upper)) = frame_spec {
+ let kind = if *frame_type == 1 {
+ "ROW_FRAME"
+ } else {
+ "RANGE_FRAME"
+ };
+ parts.push(format!(
+ "WindowFrame({kind}, {}, {})",
+ lower.value(),
+ upper.value()
+ ));
+ }
+ format!("WindowSpec({})", parts.join(", "))
+}
+
impl WindowExpressionWrapper {
/// Create a new WindowExpressionWrapper.
pub fn new(
@@ -905,6 +963,19 @@ impl WindowExpressionWrapper {
}
/// Convert to protobuf.
+ /// As `WindowExpression.__repr__`.
+ pub fn render(&self) -> String {
+ let spec = window_spec_repr(
+ &self.partition_spec,
+ &self.order_spec,
+ self.frame_spec.as_ref(),
+ );
+ format!(
+ "WindowExpression({}, ({spec}))",
+ self.window_function.render()
+ )
+ }
+
pub fn to_proto(&self) -> proto::Expression {
let mut expr = proto::Expression::default();
let mut window = proto::expression::Window::default();
@@ -1038,6 +1109,158 @@ impl UnresolvedNamedLambdaVariable {
}
}
+/// Python's `repr` of a float, as PySpark's reprs show one: the shortest
digits that
+/// round-trip, in fixed notation for a decimal exponent from -4 to 15,
otherwise in
+/// scientific notation with a signed exponent of at least two digits.
+fn python_float_repr(x: f64) -> String {
+ if x.is_nan() {
+ return "nan".to_string();
+ }
+ if x.is_infinite() {
+ return if x > 0.0 { "inf" } else { "-inf" }.to_string();
+ }
+ if x == 0.0 {
+ return if x.is_sign_negative() { "-0.0" } else { "0.0" }.to_string();
+ }
+ let scientific = format!("{x:e}");
+ let (mantissa, exponent) =
scientific.split_once('e').unwrap_or((&scientific, "0"));
+ let exponent: i32 = exponent.parse().unwrap_or(0);
+ let (sign, mantissa) = match mantissa.strip_prefix('-') {
+ Some(mantissa) => ("-", mantissa),
+ None => ("", mantissa),
+ };
+ let digits = mantissa.replace('.', "");
+ if (-4..16).contains(&exponent) {
+ let point = exponent + 1;
+ let fixed = if point <= 0 {
+ format!("0.{}{digits}", "0".repeat(point.unsigned_abs() as usize))
+ } else if point as usize >= digits.len() {
+ format!("{digits}{}.0", "0".repeat(point as usize - digits.len()))
+ } else {
+ format!(
+ "{}.{}",
+ &digits[..point as usize],
+ &digits[point as usize..]
+ )
+ };
+ format!("{sign}{fixed}")
+ } else {
+ let mantissa = if digits.len() == 1 {
+ digits
+ } else {
+ format!("{}.{}", &digits[..1], &digits[1..])
+ };
+ let exponent_sign = if exponent < 0 { '-' } else { '+' };
+ format!("{sign}{mantissa}e{exponent_sign}{:02}", exponent.abs())
+ }
+}
+
+/// The quote that Python's `repr` puts around a string or bytes: a double
quote if it has
+/// a single quote and no double quote, otherwise a single quote.
+fn python_quote(has_single: bool, has_double: bool) -> char {
+ if has_single && !has_double {
+ '"'
+ } else {
+ '\''
+ }
+}
+
+/// Python's `repr` of `bytes`, which `f"{value}"` of bytes also is.
+fn python_bytes_repr(bytes: &[u8]) -> String {
+ let quote = python_quote(bytes.contains(&b'\''), bytes.contains(&b'"'));
+ let mut out = String::from("b");
+ out.push(quote);
+ for &byte in bytes {
+ match byte {
+ b'\\' => out.push_str("\\\\"),
+ b'\t' => out.push_str("\\t"),
+ b'\n' => out.push_str("\\n"),
+ b'\r' => out.push_str("\\r"),
+ byte if byte as char == quote => {
+ out.push('\\');
+ out.push(quote);
+ }
+ 0x20..=0x7e => out.push(byte as char),
+ byte => out.push_str(&format!("\\x{byte:02x}")),
+ }
+ }
+ out.push(quote);
+ out
+}
+
+/// Python's `repr` of a string, as a list shows its strings.
+fn python_str_repr(value: &str) -> String {
+ let quote = python_quote(value.contains('\''), value.contains('"'));
+ let mut out = String::new();
+ out.push(quote);
+ for c in value.chars() {
+ match c {
+ '\\' => out.push_str("\\\\"),
+ '\t' => out.push_str("\\t"),
+ '\n' => out.push_str("\\n"),
+ '\r' => out.push_str("\\r"),
+ c if c == quote => {
+ out.push('\\');
+ out.push(c);
+ }
+ c if c.is_control() => {
+ let code = c as u32;
+ if code < 0x100 {
+ out.push_str(&format!("\\x{code:02x}"));
+ } else if code < 0x10000 {
+ out.push_str(&format!("\\u{code:04x}"));
+ } else {
+ out.push_str(&format!("\\U{code:08x}"));
+ }
+ }
+ c => out.push(c),
+ }
+ }
+ out.push(quote);
+ out
+}
+
+/// A date of `days` since the epoch, as `DateType.fromInternal` and
`%Y-%m-%d` give it.
+fn date_repr(days: i32) -> String {
+ chrono::NaiveDate::from_num_days_from_ce_opt(days.saturating_add(719_163))
+ .map(|date| date.format("%Y-%m-%d").to_string())
+ .unwrap_or_else(|| days.to_string())
+}
+
+/// A timestamp of `micros` since the epoch, as `%Y-%m-%d %H:%M:%S.%f` gives
it: in the local
+/// time zone, as `TimestampType.fromInternal` does, or in UTC, as
`TimestampNTZType`'s does.
+fn timestamp_repr(micros: i64, local: bool) -> String {
+ const FORMAT: &str = "%Y-%m-%d %H:%M:%S%.6f";
+ match chrono::DateTime::from_timestamp_micros(micros) {
+ Some(utc) if local =>
utc.with_timezone(&chrono::Local).format(FORMAT).to_string(),
+ Some(utc) => utc.naive_utc().format(FORMAT).to_string(),
+ None => micros.to_string(),
+ }
+}
+
+/// A time of `nano` since midnight, as `TimeType.fromInternal` and
`%H:%M:%S.%f` give it.
+fn time_repr(nano: i64) -> String {
+ let (hours, rest) = (nano / 3_600_000_000_000, nano % 3_600_000_000_000);
+ let (minutes, rest) = (rest / 60_000_000_000, rest % 60_000_000_000);
+ let (seconds, rest) = (rest / 1_000_000_000, rest % 1_000_000_000);
+ format!("{hours:02}:{minutes:02}:{seconds:02}.{:06}", rest / 1_000)
+}
+
+/// A day-time interval of `micros`, as pandas' `Timedelta.isoformat` gives
it: the days of
+/// the interval, which are negative for a negative one, and the time of day
after them, with
+/// the seconds of the fraction without trailing zeros.
+fn iso_duration(micros: i64) -> String {
+ const DAY: i64 = 86_400_000_000;
+ let days = micros.div_euclid(DAY);
+ let rest = micros.rem_euclid(DAY);
+ let (hours, rest) = (rest / 3_600_000_000, rest % 3_600_000_000);
+ let (minutes, rest) = (rest / 60_000_000, rest % 60_000_000);
+ let (seconds, micro) = (rest / 1_000_000, rest % 1_000_000);
+ let fraction = format!("{seconds}.{micro:06}000");
+ let seconds = fraction.trim_end_matches('0').trim_end_matches('.');
+ format!("P{days}DT{hours}H{minutes}M{seconds}S")
+}
+
#[cfg(test)]
mod tests {
use super::*;
@@ -1059,19 +1282,19 @@ mod tests {
"NULL"
);
- // Binary operators render infix, mirroring
UnresolvedFunction.__repr__.
+ // An operator renders as a function, as UnresolvedFunction.__repr__.
let add = Expression::function(
"+",
vec![col("x"), Expression::literal(LiteralExpression::Integer(1))],
);
- assert_eq!(add.render(), "(x + 1)");
+ assert_eq!(add.render(), "+(x, 1)");
let eq = Expression::function("==", vec![col("a"), col("b")]);
- assert_eq!(eq.render(), "(a == b)");
+ assert_eq!(eq.render(), "==(a, b)");
// Unary not.
let neq = Expression::function("not", vec![eq.clone()]);
- assert_eq!(neq.render(), "(NOT (a == b))");
+ assert_eq!(neq.render(), "not(==(a, b))");
// Non-operator function renders as name(args).
let f = Expression::function("coalesce", vec![col("a"), col("b")]);
@@ -1089,11 +1312,11 @@ mod tests {
eval_mode: None,
})))
.render(),
- "CAST(x AS int)"
+ "CAST(x AS INT)"
);
assert_eq!(
Expression::from(ExpressionNode::UnresolvedStar(None)).render(),
- "*"
+ "unresolvedstar()"
);
// Equal expressions render identically; different ones differ
@@ -1197,7 +1420,7 @@ mod tests {
#[test]
fn test_literal_render_float() {
let lit = LiteralExpression::Float(3.14);
- assert_eq!(lit.render(), "3.14");
+ assert_eq!(lit.render(), "3.140000104904175");
}
#[test]
@@ -1214,21 +1437,21 @@ mod tests {
#[test]
fn test_literal_render_binary() {
- let lit = LiteralExpression::Binary(vec![1, 2, 3]);
- let rendered = lit.render();
- assert!(rendered.contains("[1, 2, 3]"));
+ let lit = LiteralExpression::Binary(vec![1, 2, 3, b'a', b'\'']);
+ // As Python's repr of bytes.
+ assert_eq!(lit.render(), "b\"\\x01\\x02\\x03a'\"");
}
#[test]
fn test_literal_render_date() {
let lit = LiteralExpression::Date(18993);
- assert_eq!(lit.render(), "18993");
+ assert_eq!(lit.render(), "2022-01-01");
}
#[test]
fn test_literal_render_timestamp_ntz() {
let lit = LiteralExpression::TimestampNtz(1693526400000000);
- assert_eq!(lit.render(), "1693526400000000");
+ assert_eq!(lit.render(), "2023-09-01 00:00:00.000000");
}
#[test]
@@ -1237,7 +1460,7 @@ mod tests {
nano: 3600000000000i64,
precision: 9,
};
- assert_eq!(lit.render(), "3600000000000");
+ assert_eq!(lit.render(), "01:00:00.000000");
}
#[test]
@@ -1275,20 +1498,20 @@ mod tests {
#[test]
fn test_expression_render_unresolved_star_with_target() {
let expr: Expression =
ExpressionNode::UnresolvedStar(Some("table.*".to_string())).into();
- assert_eq!(expr.render(), "table.*");
+ assert_eq!(expr.render(), "unresolvedstar(table.*)");
}
#[test]
fn test_expression_render_unresolved_regex() {
let expr: Expression =
ExpressionNode::UnresolvedRegex("`col_.*`".to_string()).into();
- assert_eq!(expr.render(), "`col_.*`");
+ assert_eq!(expr.render(), "UnresolvedRegex(`col_.*`)");
}
#[test]
fn test_expression_render_direct_shuffle_partition_id() {
let child = col("x");
let expr: Expression =
ExpressionNode::DirectShufflePartitionId(child).into();
- assert_eq!(expr.render(), "DIRECT_SHUFFLE_PARTITION_ID(x)");
+ assert_eq!(expr.render(), "DirectShufflePartitionID(child=x)");
}
#[test]
@@ -1297,7 +1520,7 @@ mod tests {
let extraction =
Expression::literal(LiteralExpression::string("field"));
let ev = ExtractValue::new(child, extraction);
let expr: Expression =
ExpressionNode::UnresolvedExtractValue(Box::new(ev)).into();
- assert_eq!(expr.render(), "struct_col[field]");
+ assert_eq!(expr.render(), "struct_col['field']");
}
#[test]
@@ -1354,7 +1577,7 @@ mod tests {
Expression::literal(LiteralExpression::boolean(true)),
Expression::literal(LiteralExpression::int(1)),
)]);
- assert_eq!(cw.render(), "CASE WHEN true THEN 1 END");
+ assert_eq!(cw.render(), "CASE WHEN True THEN 1 END");
}
#[test]
@@ -1369,7 +1592,7 @@ mod tests {
Expression::literal(LiteralExpression::int(2)),
),
]);
- assert_eq!(cw.render(), "CASE WHEN true THEN 1 WHEN false THEN 2 END");
+ assert_eq!(cw.render(), "CASE WHEN True THEN 1 WHEN False THEN 2 END");
}
#[test]
@@ -1379,7 +1602,7 @@ mod tests {
Expression::literal(LiteralExpression::int(1)),
)])
.with_else(Expression::literal(LiteralExpression::int(99)));
- assert_eq!(cw.render(), "CASE WHEN true THEN 1 ELSE 99 END");
+ assert_eq!(cw.render(), "CASE WHEN True THEN 1 ELSE 99 END");
}
#[test]
@@ -1395,13 +1618,13 @@ mod tests {
#[test]
fn test_unresolved_function_render_negate() {
let func = UnresolvedFunction::new("negate", vec![col("x")]);
- assert_eq!(func.render(), "(- x)");
+ assert_eq!(func.render(), "negate(x)");
}
#[test]
fn test_unresolved_function_render_negative() {
let func = UnresolvedFunction::new("negative", vec![col("x")]);
- assert_eq!(func.render(), "(- x)");
+ assert_eq!(func.render(), "negative(x)");
}
#[test]
@@ -1442,7 +1665,7 @@ mod tests {
},
);
let expr: Expression = ExpressionNode::Cast(Box::new(cast)).into();
- assert_eq!(expr.render(), "CAST(x AS string)");
+ assert_eq!(expr.render(), "CAST(x AS STRING)");
}
#[test]
@@ -1455,13 +1678,13 @@ mod tests {
#[test]
fn test_literal_boolean_true_render() {
let lit = LiteralExpression::boolean(true);
- assert_eq!(lit.render(), "true");
+ assert_eq!(lit.render(), "True");
}
#[test]
fn test_literal_boolean_false_render() {
let lit = LiteralExpression::boolean(false);
- assert_eq!(lit.render(), "false");
+ assert_eq!(lit.render(), "False");
}
#[test]
@@ -1470,27 +1693,27 @@ mod tests {
"not",
vec![Expression::literal(LiteralExpression::boolean(true))],
);
- assert_eq!(func.render(), "(NOT true)");
+ assert_eq!(func.render(), "not(True)");
}
#[test]
fn test_infix_operators_render() {
let ops = vec![
- ("+", "(1 + 2)"),
- ("-", "(1 - 2)"),
- ("*", "(1 * 2)"),
- ("/", "(1 / 2)"),
- ("%", "(1 % 2)"),
- ("==", "(1 == 2)"),
- ("!=", "(1 != 2)"),
- ("<", "(1 < 2)"),
- ("<=", "(1 <= 2)"),
- (">", "(1 > 2)"),
- (">=", "(1 >= 2)"),
- ("&", "(1 & 2)"),
- ("|", "(1 | 2)"),
- ("^", "(1 ^ 2)"),
- ("<=>", "(1 <=> 2)"),
+ ("+", "+(1, 2)"),
+ ("-", "-(1, 2)"),
+ ("*", "*(1, 2)"),
+ ("/", "/(1, 2)"),
+ ("%", "%(1, 2)"),
+ ("==", "==(1, 2)"),
+ ("!=", "!=(1, 2)"),
+ ("<", "<(1, 2)"),
+ ("<=", "<=(1, 2)"),
+ (">", ">(1, 2)"),
+ (">=", ">=(1, 2)"),
+ ("&", "&(1, 2)"),
+ ("|", "|(1, 2)"),
+ ("^", "^(1, 2)"),
+ ("<=>", "<=>(1, 2)"),
];
for (op_name, expected_result) in ops.iter().take(15) {
@@ -1517,7 +1740,7 @@ mod tests {
Expression::literal(LiteralExpression::boolean(false)),
],
);
- assert_eq!(and_func.render(), "(true and false)");
+ assert_eq!(and_func.render(), "and(True, False)");
let or_func = UnresolvedFunction::new(
"or",
@@ -1526,7 +1749,7 @@ mod tests {
Expression::literal(LiteralExpression::boolean(false)),
],
);
- assert_eq!(or_func.render(), "(true or false)");
+ assert_eq!(or_func.render(), "or(True, False)");
}
#[test]
@@ -1563,24 +1786,25 @@ mod tests {
None,
);
let expr: Expression =
ExpressionNode::WindowExpression(Box::new(we)).into();
- let rendered = expr.render();
- assert!(rendered.contains("WindowExpressionWrapper"));
+ // As WindowExpression.__repr__.
+ assert_eq!(
+ expr.render(),
+ "WindowExpression(sum(x), (WindowSpec(PartitionBy(group_col))))"
+ );
}
#[test]
fn test_lambda_function_render() {
let lf = LambdaFunction::new(col("x"),
vec![UnresolvedNamedLambdaVariable::new("x")]);
let expr: Expression =
ExpressionNode::LambdaFunction(Box::new(lf)).into();
- let rendered = expr.render();
- assert!(rendered.contains("LambdaFunction"));
+ assert_eq!(expr.render(), "LambdaFunction(x, x)");
}
#[test]
fn test_unresolved_named_lambda_variable_render() {
let var = UnresolvedNamedLambdaVariable::new("x");
let expr: Expression =
ExpressionNode::UnresolvedNamedLambdaVariable(var).into();
- let rendered = expr.render();
- assert!(rendered.contains("UnresolvedNamedLambdaVariable"));
+ assert_eq!(expr.render(), "x");
}
#[test]
diff --git a/crates/spark-connect/src/window.rs
b/crates/spark-connect/src/window.rs
index 6513b06..9672068 100644
--- a/crates/spark-connect/src/window.rs
+++ b/crates/spark-connect/src/window.rs
@@ -44,6 +44,26 @@ pub struct WindowSpec {
}
impl WindowSpec {
+ /// As `WindowSpec.__repr__` of PySpark Connect, such as
+ /// `WindowSpec(PartitionBy(g), OrderBy(o ASC NULLS FIRST),
WindowFrame(ROW_FRAME, -1, 1))`.
+ pub fn repr(&self) -> String {
+ let boundary = |bound: &FrameBound| match bound {
+ FrameBound::UnboundedPreceding =>
crate::expression::FrameBoundary::UnboundedPreceding,
+ FrameBound::Preceding(n) =>
crate::expression::FrameBoundary::Preceding(*n),
+ FrameBound::CurrentRow =>
crate::expression::FrameBoundary::CurrentRow,
+ FrameBound::Following(n) =>
crate::expression::FrameBoundary::Following(*n),
+ FrameBound::UnboundedFollowing =>
crate::expression::FrameBoundary::UnboundedFollowing,
+ };
+ let frame = self.frame_spec.as_ref().map(|(frame_type, lower, upper)| {
+ let frame_type = match frame_type {
+ FrameType::Row => 1u32,
+ FrameType::Range => 2u32,
+ };
+ (frame_type, boundary(lower), boundary(upper))
+ });
+ crate::expression::window_spec_repr(&self.partition_spec,
&self.order_spec, frame.as_ref())
+ }
+
/// Create a new empty WindowSpec.
pub fn new() -> Self {
Self {
diff --git a/crates/spark-connect/tests/expression_sharing.rs
b/crates/spark-connect/tests/expression_sharing.rs
index a4865f4..ae7becf 100644
--- a/crates/spark-connect/tests/expression_sharing.rs
+++ b/crates/spark-connect/tests/expression_sharing.rs
@@ -66,8 +66,8 @@ fn aliases_and_function_arguments_share_reused_subtrees() {
assert_shared(argument, source.expression());
}
assert_eq!(source.to_proto(), before);
- assert_eq!(left.expression().render(), "(value + 1) AS left");
- assert_eq!(right.expression().render(), "(value + 1) AS right");
+ assert_eq!(left.expression().render(), "+(value, 1) AS left");
+ assert_eq!(right.expression().render(), "+(value, 1) AS right");
}
#[test]
@@ -100,7 +100,7 @@ fn
case_branches_share_children_without_mutating_the_original() {
assert_eq!(original.to_proto(), before);
assert_eq!(
completed.expression().render(),
- "CASE WHEN (value > 0) THEN (value + 1) ELSE 0 END"
+ "CASE WHEN >(value, 0) THEN +(value, 1) ELSE 0 END"
);
}
@@ -170,7 +170,7 @@ fn
debug_render_and_proto_keep_the_existing_representation() {
let source = col("value") + lit(1);
let alias = source.clone().alias("total");
- assert_eq!(alias.expression().render(), "(value + 1) AS total");
+ assert_eq!(alias.expression().render(), "+(value, 1) AS total");
assert_eq!(
alias.to_proto(),
Alias::new(source.expression().clone(), "total").to_proto()
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 05ed52f..b38873a 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -4436,6 +4436,66 @@ def test_instr_regexp_replace_optional_args():
F._call_function = orig
[email protected]("column,expected", [
+ # As PySpark Connect's Expression reprs: an operator renders as a function.
+ (lambda F: F.col("a") + 1, "+(a, 1)"),
+ (lambda F: 1 - F.col("a"), "-(1, a)"),
+ (lambda F: (F.col("a") + 1) * (F.col("b") - 2), "*(+(a, 1), -(b, 2))"),
+ (lambda F: -F.col("a"), "negative(a)"),
+ (lambda F: ~F.col("a"), "not(a)"),
+ (lambda F: F.col("a") != 1, "not(==(a, 1))"),
+ (lambda F: F.col("a") & F.col("b"), "and(a, b)"),
+ (lambda F: F.col("a").between(1, 2), "and(>=(a, 1), <=(a, 2))"),
+ (lambda F: F.when(F.col("a") > 1, 1).otherwise(0), "CASE WHEN >(a, 1) THEN
1 ELSE 0 END"),
+ (lambda F: F.countDistinct("a"), "count(distinct a)"),
+ (lambda F: F.col("a").cast("int"), "CAST(a AS INT)"),
+ (lambda F: F.col("a").cast("struct<x:int,y:string>"), "CAST(a AS
STRUCT<X:INT,Y:STRING>)"),
+ (lambda F: F.col("a").try_cast("int"), "TRY_CAST(a AS INT)"),
+ (lambda F: F.col("*"), "unresolvedstar()"),
+ (lambda F: F.count("*"), "count(unresolvedstar())"),
+ (lambda F: F.col("t.*"), "unresolvedstar(t.*)"),
+ (lambda F: F.col("a")[0], "a['0']"),
+ (lambda F: F.col("a")["k"], "a['k']"),
+ # Literals, as LiteralExpression.__repr__ shows their Python values.
+ (lambda F: F.lit(True), "True"),
+ (lambda F: F.lit(None), "NULL"),
+ (lambda F: F.lit(1.5), "1.5"),
+ (lambda F: F.lit(1e20), "1e+20"),
+ (lambda F: F.lit(1e-07), "1e-07"),
+ (lambda F: F.lit(float("nan")), "nan"),
+ (lambda F: F.lit(b"a'b"), 'b"a\'b"'),
+ (lambda F: F.lit(bytes([0, 9, 92, 200])), "b'\\x00\\t\\\\\\xc8'"),
+ (lambda F: F.lit(__import__("datetime").date(1999, 12, 31)), "1999-12-31"),
+ # A naive datetime is a local time, rendered in the local time zone again.
+ (lambda F: F.lit(__import__("datetime").datetime(2020, 1, 2, 3, 4, 5, 6)),
+ "2020-01-02 03:04:05.000006"),
+ (lambda F: F.lit(__import__("decimal").Decimal("-1.50")), "-1.50"),
+])
+def test_column_reprs_as_pyspark(column, expected):
+ from pyspark.sql import functions as F
+
+ assert repr(column(F)) == f"Column<'{expected}'>"
+
+
+def test_window_and_lambda_reprs_as_pyspark():
+ import re
+
+ from pyspark.sql import Window
+ from pyspark.sql import functions as F
+
+ spec = Window.partitionBy("g").orderBy(F.col("o").desc()).rowsBetween(-1,
1)
+ assert repr(spec) == "WindowSpec(PartitionBy(g), OrderBy(o DESC NULLS
LAST), WindowFrame(ROW_FRAME, -1, 1))"
+ window = F.sum("a").over(
+
Window.orderBy(F.col("o").asc()).rangeBetween(Window.unboundedPreceding,
Window.currentRow))
+ assert repr(window) == (
+ "Column<'WindowExpression(sum(a), (WindowSpec(OrderBy(o ASC NULLS
FIRST), "
+ "WindowFrame(RANGE_FRAME, -9223372036854775808, 0))))'>")
+ # A lambda variable is named by a counter, as in PySpark.
+ assert re.fullmatch(
+ r"Column<'transform\(a, LambdaFunction\(\+\(x_(\d+), 1\), x_\1\)\)'>",
+ repr(F.transform("a", lambda x: x + 1)))
+
+
@pytest.mark.parametrize("column,expected", [
# instr and regexp_replace are plain functions, as other functions are.
(lambda F: F.instr("s", "b"), "instr(s, b)"),
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]