This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 5e12e05  [SPARK-60948][CONNECT][PYTHON] Render Column reprs as PySpark 
Connect does
5e12e05 is described below

commit 5e12e05d814de00036bb29474cd42f7ca8af2613
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Fri Nov 6 14:14:15 2026 +0900

    [SPARK-60948][CONNECT][PYTHON] Render Column reprs as PySpark Connect does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-60948](https://issues.apache.org/jira/browse/SPARK-60948): Column 
reprs now follow the `Expression` reprs of PySpark Connect 4.3.0.
    
    | Expression | PySpark 4.3.0 and now | Before |
    | --- | --- | --- |
    | `F.col("id") + 1` | `+(id, 1)` | `(id + 1)` |
    | `-F.col("a")`, `~F.col("a")` | `negative(a)`, `not(a)` | `(- a)`, `(NOT 
a)` |
    | `F.col("a").between(1, 2)` | `and(>=(a, 1), <=(a, 2))` | `((a >= 1) and 
(a <= 2))` |
    | `F.countDistinct("a")` | `count(distinct a)` | `count(a)` |
    | `F.col("a").cast("int")`, `try_cast` | `CAST(a AS INT)`, `TRY_CAST(a AS 
INT)` | `CAST(a AS int)` for both |
    | `F.col("*")`, `F.col("t.*")` | `unresolvedstar()`, `unresolvedstar(t.*)` 
| `*`, `t.*` |
    | `F.col("a")[0]` | `a['0']` | `a[0]` |
    | `F.lit(True)`, `F.lit(b"xy")` | `True`, `b'xy'` | `true`, `[120, 121]` |
    | `F.lit(date(2020, 1, 2))` | `2020-01-02` | `18263` |
    | `F.lit(datetime(2020, 1, 2, 3, 4, 5))` | `2020-01-02 03:04:05.000000`, in 
the local time zone | microseconds since the epoch |
    | `F.lit(1e20)` | `1e+20`, Python's float repr | `100000000000000000000` |
    | a window function | `WindowExpression(sum(a), (WindowSpec(PartitionBy(g), 
OrderBy(o DESC NULLS LAST), WindowFrame(ROW_FRAME, -1, 1))))` | the Rust debug 
form |
    | `F.transform("a", lambda x: x + 1)` | `transform(a, LambdaFunction(+(x_0, 
1), x_0))` | the Rust debug form |
    
    Also:
    - a day-time interval literal renders as pandas' ISO duration, 
`P1DT0H0M2.000003S`;
    - a time literal renders as `01:02:00.000000`;
    - a regex column renders as `UnresolvedRegex(...)`, and a shuffle partition 
id as `DirectShufflePartitionID(child=...)`;
    - a user-defined function renders as its name and arguments;
    - `WindowSpec.__repr__` shows the partitioning, ordering and frame, as in 
PySpark, instead of `WindowSpec()`.
    
    The crate now depends on `chrono` directly, already in the lockfile through 
Arrow, for the local time of timestamp literals, as Python's 
`datetime.fromtimestamp` gives it. AGENTS.md now describes the render this way.
    
    ### Why are the changes needed?
    
    The repr differed for 50 of 77 common expressions. It also appears in error 
messages, such as `SparkSession.sql`'s for a keyword argument that is not a 
column reference. Window, lambda and UDF columns rendered the Rust debug form 
of the expression.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes, Column reprs are those of PySpark Connect 4.3.0. Plans are unchanged.
    
    ### How was this patch tested?
    
    - A probe of 77 expressions gives the same reprs as official PySpark 4.3.0. 
One is left: a window ordered by a plain column renders `ASC NULLS LAST` here. 
That is a sort-order bug rather than a repr bug, which #191 (SPARK-60990) fixes.
    - A probe of 22 more expressions gives the same reprs in the UTC, 
`America/Los_Angeles` and `Asia/Seoul` time zones: timestamps, dates, bytes, 
special floats, decimals, window frames and specs, and casts to nested types. 
The exception is `timedelta` literals, which #189 (SPARK-60986) adds, and whose 
repr this change formats.
    - New offline tests check 30 reprs, also in a non-UTC time zone. They also 
pass with official PySpark 4.3.0.
    - The unit tests and the expression-sharing tests now pin the new reprs.
    - The lib tests and every integration test, e2e included, against a local 
Spark 4.3.0 server (647 passed), the build of all tests with `wasm-udf`, the 
offline Python suites (717 passed, 9 skipped), and the live drop-in and Arrow 
suites (70 passed) and live cases (10 passed). Clippy reports no diagnostics in 
the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #192 from HyukjinKwon/SPARK-60948-column-repr.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 AGENTS.md                                        |   2 +-
 Cargo.lock                                       |   1 +
 crates/pyspark-rs/src/window.rs                  |   3 +-
 crates/spark-connect/Cargo.toml                  |   2 +
 crates/spark-connect/src/expression.rs           | 414 +++++++++++++++++------
 crates/spark-connect/src/window.rs               |  20 ++
 crates/spark-connect/tests/expression_sharing.rs |   8 +-
 python/tests/test_dropin_offline.py              |  60 ++++
 8 files changed, 409 insertions(+), 101 deletions(-)

diff --git a/AGENTS.md b/AGENTS.md
index 289b62c..119b4b4 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -443,7 +443,7 @@ For every method matching a pyspark name:
     on-Spark's `spark_column_equals` decides column identity by comparing repr 
strings (it does
     `repr(left).replace(backtick, "") == repr(right)...`); a generic 
`"Column()"` makes every column
     look equal and collapses frames to `(n, 0)`. `Expression::render()` 
mirrors the pyspark-connect
-    expression `__repr__` (infix binary ops, `AS`, `CAST`, star, ...). 
`pyspark.sql.connect.column`
+    expression `__repr__` (operators as functions, `+(a, 1)`, `AS`, `CAST`, 
`unresolvedstar()`, ...). `pyspark.sql.connect.column`
     re-exports the Rust `Column`, so its `isinstance(x, ConnectColumn)` checks 
already pass.
   - **Internal pandas functions dispatch by NAME, not by client 
reimplementation.**
     `pyspark.sql.internal.InternalFunction` (`distributed_sequence_id`, 
`pandas_product`,
diff --git a/Cargo.lock b/Cargo.lock
index 3b65c41..8fba36c 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -61,6 +61,7 @@ dependencies = [
  "arrow",
  "base64 0.21.7",
  "bytes",
+ "chrono",
  "datafusion",
  "futures",
  "polars",
diff --git a/crates/pyspark-rs/src/window.rs b/crates/pyspark-rs/src/window.rs
index 09b2b70..9d96729 100644
--- a/crates/pyspark-rs/src/window.rs
+++ b/crates/pyspark-rs/src/window.rs
@@ -133,8 +133,9 @@ impl PyWindowSpec {
         Ok(PyWindowSpec::new(new_spec))
     }
 
+    /// As `WindowSpec.__repr__` of PySpark Connect.
     fn __repr__(&self) -> String {
-        "WindowSpec()".to_string()
+        self.spec.repr()
     }
 }
 
diff --git a/crates/spark-connect/Cargo.toml b/crates/spark-connect/Cargo.toml
index aea3ff3..422e671 100644
--- a/crates/spark-connect/Cargo.toml
+++ b/crates/spark-connect/Cargo.toml
@@ -21,6 +21,8 @@ thiserror.workspace = true
 tokio.workspace = true
 futures.workspace = true
 serde_json = "1.0"
+# Local time for the reprs of timestamp literals, as Python's 
`datetime.fromtimestamp`.
+chrono = { version = "0.4", default-features = false, features = ["clock"] }
 uuid.workspace = true
 
 # Optional: only pulled in by the `wasm-udf` feature.
diff --git a/crates/spark-connect/src/expression.rs 
b/crates/spark-connect/src/expression.rs
index 54e25c5..a818ef3 100644
--- a/crates/spark-connect/src/expression.rs
+++ b/crates/spark-connect/src/expression.rs
@@ -133,28 +133,35 @@ impl Expression {
             ExpressionNode::ColumnReference(col_ref) => col_ref.name.clone(),
             ExpressionNode::UnresolvedFunction(func) => func.render(),
             ExpressionNode::UnresolvedStar(target) => match target {
-                Some(t) => t.clone(),
-                None => "*".to_string(),
+                Some(t) => format!("unresolvedstar({t})"),
+                None => "unresolvedstar()".to_string(),
             },
             // As `ColumnAlias.__repr__`: the names joined by commas.
             ExpressionNode::Alias(alias) => {
                 format!("{} AS {}", alias.child.render(), 
alias.names.join(","))
             }
+            // As `CastExpression.__repr__`: the type in upper case.
             ExpressionNode::Cast(cast) => {
                 let type_str = match &cast.target {
                     CastTarget::Type(dt) => dt.simple_string(),
                     CastTarget::TypeStr(s) => s.clone(),
+                }
+                .to_uppercase();
+                let name = if cast.eval_mode == Some(CastEvalMode::Try) {
+                    "TRY_CAST"
+                } else {
+                    "CAST"
                 };
-                format!("CAST({} AS {})", cast.child.render(), type_str)
+                format!("{name}({} AS {type_str})", cast.child.render())
             }
             ExpressionNode::DirectShufflePartitionId(child) => {
-                format!("DIRECT_SHUFFLE_PARTITION_ID({})", child.render())
+                format!("DirectShufflePartitionID(child={})", child.render())
             }
-            ExpressionNode::UnresolvedRegex(col_name) => col_name.clone(),
+            ExpressionNode::UnresolvedRegex(col_name) => 
format!("UnresolvedRegex({col_name})"),
             ExpressionNode::SortOrder(sort) => sort.render(),
             ExpressionNode::CaseWhen(case_when) => case_when.render(),
             ExpressionNode::UnresolvedExtractValue(ev) => {
-                format!("{}[{}]", ev.child.render(), ev.extraction.render())
+                format!("{}['{}']", ev.child.render(), ev.extraction.render())
             }
             ExpressionNode::UpdateFields(uf) => match &uf.value_expression {
                 Some(v) => format!(
@@ -171,10 +178,22 @@ impl Expression {
             },
             ExpressionNode::SQLExpression(sql) => sql.clone(),
             ExpressionNode::CallFunction(cf) => cf.render(),
-            ExpressionNode::WindowExpression(_) => format!("{self:?}"),
-            ExpressionNode::LambdaFunction(lf) => format!("{lf:?}"),
-            ExpressionNode::UnresolvedNamedLambdaVariable(var) => 
format!("{var:?}"),
-            ExpressionNode::CommonInlineUserDefinedFunction(_) => 
format!("{self:?}"),
+            ExpressionNode::WindowExpression(we) => we.render(),
+            // As `LambdaFunction.__repr__` and 
`UnresolvedNamedLambdaVariable.__repr__`.
+            ExpressionNode::LambdaFunction(lf) => {
+                let args: Vec<&str> = lf.arguments.iter().map(|a| 
a.name_parts.as_str()).collect();
+                format!(
+                    "LambdaFunction({}, {})",
+                    lf.function.render(),
+                    args.join(", ")
+                )
+            }
+            ExpressionNode::UnresolvedNamedLambdaVariable(var) => 
var.name_parts.clone(),
+            // As `CommonInlineUserDefinedFunction.__repr__`.
+            ExpressionNode::CommonInlineUserDefinedFunction(udf) => {
+                let args: Vec<String> = 
udf.arguments.iter().map(Expression::render).collect();
+                format!("{}({})", udf.function_name, args.join(", "))
+            }
         }
     }
 
@@ -344,37 +363,45 @@ impl LiteralExpression {
         expr
     }
 
-    /// Render the literal value, mirroring `LiteralExpression.__repr__`.
+    /// Render the literal value, mirroring `LiteralExpression.__repr__`: the 
Python value,
+    /// as `f"{value}"` shows it, and a date, time, timestamp or day-time 
interval in the
+    /// formats that it gives them.
     pub fn render(&self) -> String {
         match self {
             LiteralExpression::Null(_) => "NULL".to_string(),
-            LiteralExpression::Boolean(b) => {
-                if *b {
-                    "true".to_string()
-                } else {
-                    "false".to_string()
-                }
-            }
+            LiteralExpression::Boolean(b) => if *b { "True" } else { "False" 
}.to_string(),
             LiteralExpression::Byte(v)
             | LiteralExpression::Short(v)
             | LiteralExpression::Integer(v) => v.to_string(),
             LiteralExpression::Long(v) => v.to_string(),
-            LiteralExpression::Float(v) => v.to_string(),
-            LiteralExpression::Double(v) => v.to_string(),
+            LiteralExpression::Float(v) => python_float_repr(*v as f64),
+            LiteralExpression::Double(v) => python_float_repr(*v),
             LiteralExpression::Decimal { value, .. } => value.clone(),
             LiteralExpression::String(v) => v.clone(),
-            LiteralExpression::Binary(v) => format!("{v:?}"),
-            LiteralExpression::Date(v) => v.to_string(),
-            LiteralExpression::Timestamp(v) | 
LiteralExpression::TimestampNtz(v) => v.to_string(),
-            LiteralExpression::DayTimeInterval(v) => v.to_string(),
-            LiteralExpression::Time { nano, .. } => nano.to_string(),
+            LiteralExpression::Binary(v) => python_bytes_repr(v),
+            LiteralExpression::Date(v) => date_repr(*v),
+            LiteralExpression::Timestamp(v) => timestamp_repr(*v, true),
+            LiteralExpression::TimestampNtz(v) => timestamp_repr(*v, false),
+            LiteralExpression::DayTimeInterval(v) => iso_duration(*v),
+            LiteralExpression::Time { nano, .. } => time_repr(*nano),
+            // A Python list, as `str(list)` shows its elements.
             LiteralExpression::Array { elements, .. } => {
-                let inner: Vec<String> = elements.iter().map(|e| 
e.render()).collect();
+                let inner: Vec<String> = elements.iter().map(|e| 
e.render_in_container()).collect();
                 format!("[{}]", inner.join(", "))
             }
         }
     }
 
+    /// The literal as Python's `repr` shows it within a container, such as a 
list: a string
+    /// quoted, and None for a null.
+    fn render_in_container(&self) -> String {
+        match self {
+            LiteralExpression::Null(_) => "None".to_string(),
+            LiteralExpression::String(v) => python_str_repr(v),
+            other => other.render(),
+        }
+    }
+
     /// Construct a null literal of a specific type.
     pub fn null(data_type: DataType) -> Self {
         LiteralExpression::Null(data_type)
@@ -552,27 +579,13 @@ impl UnresolvedFunction {
     /// binary operators render infix as `(a op b)`, the unary `not`/negate 
render
     /// prefixed, everything else as `name(arg, arg, ...)`.
     pub fn render(&self) -> String {
-        const INFIX_OPS: &[&str] = &[
-            "+", "-", "*", "/", "%", "==", "!=", "<", "<=", ">", ">=", "and", 
"or", "&", "|", "^",
-            "<=>",
-        ];
-        if self.args.len() == 2 && INFIX_OPS.contains(&self.name.as_str()) {
-            return format!(
-                "({} {} {})",
-                self.args[0].render(),
-                self.name,
-                self.args[1].render()
-            );
-        }
-        if self.args.len() == 1 {
-            match self.name.as_str() {
-                "not" => return format!("(NOT {})", self.args[0].render()),
-                "negative" | "negate" => return format!("(- {})", 
self.args[0].render()),
-                _ => {}
-            }
-        }
+        // As `UnresolvedFunction.__repr__`, which renders an operator as a 
function too.
         let inner: Vec<String> = self.args.iter().map(|a| 
a.render()).collect();
-        format!("{}({})", self.name, inner.join(", "))
+        if self.is_distinct {
+            format!("{}(distinct {})", self.name, inner.join(", "))
+        } else {
+            format!("{}({})", self.name, inner.join(", "))
+        }
     }
 
     pub fn to_proto(&self) -> proto::Expression {
@@ -888,6 +901,51 @@ pub enum FrameBoundary {
     UnboundedFollowing,
 }
 
+impl FrameBoundary {
+    /// The boundary as the int that `rowsBetween` and `rangeBetween` take, as
+    /// `WindowFrame.__repr__` shows it.
+    pub fn value(&self) -> i64 {
+        match self {
+            FrameBoundary::UnboundedPreceding => i64::MIN,
+            FrameBoundary::Preceding(n) => -n,
+            FrameBoundary::CurrentRow => 0,
+            FrameBoundary::Following(n) => *n,
+            FrameBoundary::UnboundedFollowing => i64::MAX,
+        }
+    }
+}
+
+/// As `WindowSpec.__repr__` of PySpark Connect: its partitioning, ordering 
and frame. A
+/// frame type of 1 is a row frame, and another a range frame, as in the proto.
+pub fn window_spec_repr(
+    partition_spec: &[Expression],
+    order_spec: &[SortOrder],
+    frame_spec: Option<&(u32, FrameBoundary, FrameBoundary)>,
+) -> String {
+    let mut parts = Vec::new();
+    if !partition_spec.is_empty() {
+        let partitions: Vec<String> = 
partition_spec.iter().map(Expression::render).collect();
+        parts.push(format!("PartitionBy({})", partitions.join(", ")));
+    }
+    if !order_spec.is_empty() {
+        let orders: Vec<String> = 
order_spec.iter().map(SortOrder::render).collect();
+        parts.push(format!("OrderBy({})", orders.join(", ")));
+    }
+    if let Some((frame_type, lower, upper)) = frame_spec {
+        let kind = if *frame_type == 1 {
+            "ROW_FRAME"
+        } else {
+            "RANGE_FRAME"
+        };
+        parts.push(format!(
+            "WindowFrame({kind}, {}, {})",
+            lower.value(),
+            upper.value()
+        ));
+    }
+    format!("WindowSpec({})", parts.join(", "))
+}
+
 impl WindowExpressionWrapper {
     /// Create a new WindowExpressionWrapper.
     pub fn new(
@@ -905,6 +963,19 @@ impl WindowExpressionWrapper {
     }
 
     /// Convert to protobuf.
+    /// As `WindowExpression.__repr__`.
+    pub fn render(&self) -> String {
+        let spec = window_spec_repr(
+            &self.partition_spec,
+            &self.order_spec,
+            self.frame_spec.as_ref(),
+        );
+        format!(
+            "WindowExpression({}, ({spec}))",
+            self.window_function.render()
+        )
+    }
+
     pub fn to_proto(&self) -> proto::Expression {
         let mut expr = proto::Expression::default();
         let mut window = proto::expression::Window::default();
@@ -1038,6 +1109,158 @@ impl UnresolvedNamedLambdaVariable {
     }
 }
 
+/// Python's `repr` of a float, as PySpark's reprs show one: the shortest 
digits that
+/// round-trip, in fixed notation for a decimal exponent from -4 to 15, 
otherwise in
+/// scientific notation with a signed exponent of at least two digits.
+fn python_float_repr(x: f64) -> String {
+    if x.is_nan() {
+        return "nan".to_string();
+    }
+    if x.is_infinite() {
+        return if x > 0.0 { "inf" } else { "-inf" }.to_string();
+    }
+    if x == 0.0 {
+        return if x.is_sign_negative() { "-0.0" } else { "0.0" }.to_string();
+    }
+    let scientific = format!("{x:e}");
+    let (mantissa, exponent) = 
scientific.split_once('e').unwrap_or((&scientific, "0"));
+    let exponent: i32 = exponent.parse().unwrap_or(0);
+    let (sign, mantissa) = match mantissa.strip_prefix('-') {
+        Some(mantissa) => ("-", mantissa),
+        None => ("", mantissa),
+    };
+    let digits = mantissa.replace('.', "");
+    if (-4..16).contains(&exponent) {
+        let point = exponent + 1;
+        let fixed = if point <= 0 {
+            format!("0.{}{digits}", "0".repeat(point.unsigned_abs() as usize))
+        } else if point as usize >= digits.len() {
+            format!("{digits}{}.0", "0".repeat(point as usize - digits.len()))
+        } else {
+            format!(
+                "{}.{}",
+                &digits[..point as usize],
+                &digits[point as usize..]
+            )
+        };
+        format!("{sign}{fixed}")
+    } else {
+        let mantissa = if digits.len() == 1 {
+            digits
+        } else {
+            format!("{}.{}", &digits[..1], &digits[1..])
+        };
+        let exponent_sign = if exponent < 0 { '-' } else { '+' };
+        format!("{sign}{mantissa}e{exponent_sign}{:02}", exponent.abs())
+    }
+}
+
+/// The quote that Python's `repr` puts around a string or bytes: a double 
quote if it has
+/// a single quote and no double quote, otherwise a single quote.
+fn python_quote(has_single: bool, has_double: bool) -> char {
+    if has_single && !has_double {
+        '"'
+    } else {
+        '\''
+    }
+}
+
+/// Python's `repr` of `bytes`, which `f"{value}"` of bytes also is.
+fn python_bytes_repr(bytes: &[u8]) -> String {
+    let quote = python_quote(bytes.contains(&b'\''), bytes.contains(&b'"'));
+    let mut out = String::from("b");
+    out.push(quote);
+    for &byte in bytes {
+        match byte {
+            b'\\' => out.push_str("\\\\"),
+            b'\t' => out.push_str("\\t"),
+            b'\n' => out.push_str("\\n"),
+            b'\r' => out.push_str("\\r"),
+            byte if byte as char == quote => {
+                out.push('\\');
+                out.push(quote);
+            }
+            0x20..=0x7e => out.push(byte as char),
+            byte => out.push_str(&format!("\\x{byte:02x}")),
+        }
+    }
+    out.push(quote);
+    out
+}
+
+/// Python's `repr` of a string, as a list shows its strings.
+fn python_str_repr(value: &str) -> String {
+    let quote = python_quote(value.contains('\''), value.contains('"'));
+    let mut out = String::new();
+    out.push(quote);
+    for c in value.chars() {
+        match c {
+            '\\' => out.push_str("\\\\"),
+            '\t' => out.push_str("\\t"),
+            '\n' => out.push_str("\\n"),
+            '\r' => out.push_str("\\r"),
+            c if c == quote => {
+                out.push('\\');
+                out.push(c);
+            }
+            c if c.is_control() => {
+                let code = c as u32;
+                if code < 0x100 {
+                    out.push_str(&format!("\\x{code:02x}"));
+                } else if code < 0x10000 {
+                    out.push_str(&format!("\\u{code:04x}"));
+                } else {
+                    out.push_str(&format!("\\U{code:08x}"));
+                }
+            }
+            c => out.push(c),
+        }
+    }
+    out.push(quote);
+    out
+}
+
+/// A date of `days` since the epoch, as `DateType.fromInternal` and 
`%Y-%m-%d` give it.
+fn date_repr(days: i32) -> String {
+    chrono::NaiveDate::from_num_days_from_ce_opt(days.saturating_add(719_163))
+        .map(|date| date.format("%Y-%m-%d").to_string())
+        .unwrap_or_else(|| days.to_string())
+}
+
+/// A timestamp of `micros` since the epoch, as `%Y-%m-%d %H:%M:%S.%f` gives 
it: in the local
+/// time zone, as `TimestampType.fromInternal` does, or in UTC, as 
`TimestampNTZType`'s does.
+fn timestamp_repr(micros: i64, local: bool) -> String {
+    const FORMAT: &str = "%Y-%m-%d %H:%M:%S%.6f";
+    match chrono::DateTime::from_timestamp_micros(micros) {
+        Some(utc) if local => 
utc.with_timezone(&chrono::Local).format(FORMAT).to_string(),
+        Some(utc) => utc.naive_utc().format(FORMAT).to_string(),
+        None => micros.to_string(),
+    }
+}
+
+/// A time of `nano` since midnight, as `TimeType.fromInternal` and 
`%H:%M:%S.%f` give it.
+fn time_repr(nano: i64) -> String {
+    let (hours, rest) = (nano / 3_600_000_000_000, nano % 3_600_000_000_000);
+    let (minutes, rest) = (rest / 60_000_000_000, rest % 60_000_000_000);
+    let (seconds, rest) = (rest / 1_000_000_000, rest % 1_000_000_000);
+    format!("{hours:02}:{minutes:02}:{seconds:02}.{:06}", rest / 1_000)
+}
+
+/// A day-time interval of `micros`, as pandas' `Timedelta.isoformat` gives 
it: the days of
+/// the interval, which are negative for a negative one, and the time of day 
after them, with
+/// the seconds of the fraction without trailing zeros.
+fn iso_duration(micros: i64) -> String {
+    const DAY: i64 = 86_400_000_000;
+    let days = micros.div_euclid(DAY);
+    let rest = micros.rem_euclid(DAY);
+    let (hours, rest) = (rest / 3_600_000_000, rest % 3_600_000_000);
+    let (minutes, rest) = (rest / 60_000_000, rest % 60_000_000);
+    let (seconds, micro) = (rest / 1_000_000, rest % 1_000_000);
+    let fraction = format!("{seconds}.{micro:06}000");
+    let seconds = fraction.trim_end_matches('0').trim_end_matches('.');
+    format!("P{days}DT{hours}H{minutes}M{seconds}S")
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
@@ -1059,19 +1282,19 @@ mod tests {
             "NULL"
         );
 
-        // Binary operators render infix, mirroring 
UnresolvedFunction.__repr__.
+        // An operator renders as a function, as UnresolvedFunction.__repr__.
         let add = Expression::function(
             "+",
             vec![col("x"), Expression::literal(LiteralExpression::Integer(1))],
         );
-        assert_eq!(add.render(), "(x + 1)");
+        assert_eq!(add.render(), "+(x, 1)");
 
         let eq = Expression::function("==", vec![col("a"), col("b")]);
-        assert_eq!(eq.render(), "(a == b)");
+        assert_eq!(eq.render(), "==(a, b)");
 
         // Unary not.
         let neq = Expression::function("not", vec![eq.clone()]);
-        assert_eq!(neq.render(), "(NOT (a == b))");
+        assert_eq!(neq.render(), "not(==(a, b))");
 
         // Non-operator function renders as name(args).
         let f = Expression::function("coalesce", vec![col("a"), col("b")]);
@@ -1089,11 +1312,11 @@ mod tests {
                 eval_mode: None,
             })))
             .render(),
-            "CAST(x AS int)"
+            "CAST(x AS INT)"
         );
         assert_eq!(
             Expression::from(ExpressionNode::UnresolvedStar(None)).render(),
-            "*"
+            "unresolvedstar()"
         );
 
         // Equal expressions render identically; different ones differ
@@ -1197,7 +1420,7 @@ mod tests {
     #[test]
     fn test_literal_render_float() {
         let lit = LiteralExpression::Float(3.14);
-        assert_eq!(lit.render(), "3.14");
+        assert_eq!(lit.render(), "3.140000104904175");
     }
 
     #[test]
@@ -1214,21 +1437,21 @@ mod tests {
 
     #[test]
     fn test_literal_render_binary() {
-        let lit = LiteralExpression::Binary(vec![1, 2, 3]);
-        let rendered = lit.render();
-        assert!(rendered.contains("[1, 2, 3]"));
+        let lit = LiteralExpression::Binary(vec![1, 2, 3, b'a', b'\'']);
+        // As Python's repr of bytes.
+        assert_eq!(lit.render(), "b\"\\x01\\x02\\x03a'\"");
     }
 
     #[test]
     fn test_literal_render_date() {
         let lit = LiteralExpression::Date(18993);
-        assert_eq!(lit.render(), "18993");
+        assert_eq!(lit.render(), "2022-01-01");
     }
 
     #[test]
     fn test_literal_render_timestamp_ntz() {
         let lit = LiteralExpression::TimestampNtz(1693526400000000);
-        assert_eq!(lit.render(), "1693526400000000");
+        assert_eq!(lit.render(), "2023-09-01 00:00:00.000000");
     }
 
     #[test]
@@ -1237,7 +1460,7 @@ mod tests {
             nano: 3600000000000i64,
             precision: 9,
         };
-        assert_eq!(lit.render(), "3600000000000");
+        assert_eq!(lit.render(), "01:00:00.000000");
     }
 
     #[test]
@@ -1275,20 +1498,20 @@ mod tests {
     #[test]
     fn test_expression_render_unresolved_star_with_target() {
         let expr: Expression = 
ExpressionNode::UnresolvedStar(Some("table.*".to_string())).into();
-        assert_eq!(expr.render(), "table.*");
+        assert_eq!(expr.render(), "unresolvedstar(table.*)");
     }
 
     #[test]
     fn test_expression_render_unresolved_regex() {
         let expr: Expression = 
ExpressionNode::UnresolvedRegex("`col_.*`".to_string()).into();
-        assert_eq!(expr.render(), "`col_.*`");
+        assert_eq!(expr.render(), "UnresolvedRegex(`col_.*`)");
     }
 
     #[test]
     fn test_expression_render_direct_shuffle_partition_id() {
         let child = col("x");
         let expr: Expression = 
ExpressionNode::DirectShufflePartitionId(child).into();
-        assert_eq!(expr.render(), "DIRECT_SHUFFLE_PARTITION_ID(x)");
+        assert_eq!(expr.render(), "DirectShufflePartitionID(child=x)");
     }
 
     #[test]
@@ -1297,7 +1520,7 @@ mod tests {
         let extraction = 
Expression::literal(LiteralExpression::string("field"));
         let ev = ExtractValue::new(child, extraction);
         let expr: Expression = 
ExpressionNode::UnresolvedExtractValue(Box::new(ev)).into();
-        assert_eq!(expr.render(), "struct_col[field]");
+        assert_eq!(expr.render(), "struct_col['field']");
     }
 
     #[test]
@@ -1354,7 +1577,7 @@ mod tests {
             Expression::literal(LiteralExpression::boolean(true)),
             Expression::literal(LiteralExpression::int(1)),
         )]);
-        assert_eq!(cw.render(), "CASE WHEN true THEN 1 END");
+        assert_eq!(cw.render(), "CASE WHEN True THEN 1 END");
     }
 
     #[test]
@@ -1369,7 +1592,7 @@ mod tests {
                 Expression::literal(LiteralExpression::int(2)),
             ),
         ]);
-        assert_eq!(cw.render(), "CASE WHEN true THEN 1 WHEN false THEN 2 END");
+        assert_eq!(cw.render(), "CASE WHEN True THEN 1 WHEN False THEN 2 END");
     }
 
     #[test]
@@ -1379,7 +1602,7 @@ mod tests {
             Expression::literal(LiteralExpression::int(1)),
         )])
         .with_else(Expression::literal(LiteralExpression::int(99)));
-        assert_eq!(cw.render(), "CASE WHEN true THEN 1 ELSE 99 END");
+        assert_eq!(cw.render(), "CASE WHEN True THEN 1 ELSE 99 END");
     }
 
     #[test]
@@ -1395,13 +1618,13 @@ mod tests {
     #[test]
     fn test_unresolved_function_render_negate() {
         let func = UnresolvedFunction::new("negate", vec![col("x")]);
-        assert_eq!(func.render(), "(- x)");
+        assert_eq!(func.render(), "negate(x)");
     }
 
     #[test]
     fn test_unresolved_function_render_negative() {
         let func = UnresolvedFunction::new("negative", vec![col("x")]);
-        assert_eq!(func.render(), "(- x)");
+        assert_eq!(func.render(), "negative(x)");
     }
 
     #[test]
@@ -1442,7 +1665,7 @@ mod tests {
             },
         );
         let expr: Expression = ExpressionNode::Cast(Box::new(cast)).into();
-        assert_eq!(expr.render(), "CAST(x AS string)");
+        assert_eq!(expr.render(), "CAST(x AS STRING)");
     }
 
     #[test]
@@ -1455,13 +1678,13 @@ mod tests {
     #[test]
     fn test_literal_boolean_true_render() {
         let lit = LiteralExpression::boolean(true);
-        assert_eq!(lit.render(), "true");
+        assert_eq!(lit.render(), "True");
     }
 
     #[test]
     fn test_literal_boolean_false_render() {
         let lit = LiteralExpression::boolean(false);
-        assert_eq!(lit.render(), "false");
+        assert_eq!(lit.render(), "False");
     }
 
     #[test]
@@ -1470,27 +1693,27 @@ mod tests {
             "not",
             vec![Expression::literal(LiteralExpression::boolean(true))],
         );
-        assert_eq!(func.render(), "(NOT true)");
+        assert_eq!(func.render(), "not(True)");
     }
 
     #[test]
     fn test_infix_operators_render() {
         let ops = vec![
-            ("+", "(1 + 2)"),
-            ("-", "(1 - 2)"),
-            ("*", "(1 * 2)"),
-            ("/", "(1 / 2)"),
-            ("%", "(1 % 2)"),
-            ("==", "(1 == 2)"),
-            ("!=", "(1 != 2)"),
-            ("<", "(1 < 2)"),
-            ("<=", "(1 <= 2)"),
-            (">", "(1 > 2)"),
-            (">=", "(1 >= 2)"),
-            ("&", "(1 & 2)"),
-            ("|", "(1 | 2)"),
-            ("^", "(1 ^ 2)"),
-            ("<=>", "(1 <=> 2)"),
+            ("+", "+(1, 2)"),
+            ("-", "-(1, 2)"),
+            ("*", "*(1, 2)"),
+            ("/", "/(1, 2)"),
+            ("%", "%(1, 2)"),
+            ("==", "==(1, 2)"),
+            ("!=", "!=(1, 2)"),
+            ("<", "<(1, 2)"),
+            ("<=", "<=(1, 2)"),
+            (">", ">(1, 2)"),
+            (">=", ">=(1, 2)"),
+            ("&", "&(1, 2)"),
+            ("|", "|(1, 2)"),
+            ("^", "^(1, 2)"),
+            ("<=>", "<=>(1, 2)"),
         ];
 
         for (op_name, expected_result) in ops.iter().take(15) {
@@ -1517,7 +1740,7 @@ mod tests {
                 Expression::literal(LiteralExpression::boolean(false)),
             ],
         );
-        assert_eq!(and_func.render(), "(true and false)");
+        assert_eq!(and_func.render(), "and(True, False)");
 
         let or_func = UnresolvedFunction::new(
             "or",
@@ -1526,7 +1749,7 @@ mod tests {
                 Expression::literal(LiteralExpression::boolean(false)),
             ],
         );
-        assert_eq!(or_func.render(), "(true or false)");
+        assert_eq!(or_func.render(), "or(True, False)");
     }
 
     #[test]
@@ -1563,24 +1786,25 @@ mod tests {
             None,
         );
         let expr: Expression = 
ExpressionNode::WindowExpression(Box::new(we)).into();
-        let rendered = expr.render();
-        assert!(rendered.contains("WindowExpressionWrapper"));
+        // As WindowExpression.__repr__.
+        assert_eq!(
+            expr.render(),
+            "WindowExpression(sum(x), (WindowSpec(PartitionBy(group_col))))"
+        );
     }
 
     #[test]
     fn test_lambda_function_render() {
         let lf = LambdaFunction::new(col("x"), 
vec![UnresolvedNamedLambdaVariable::new("x")]);
         let expr: Expression = 
ExpressionNode::LambdaFunction(Box::new(lf)).into();
-        let rendered = expr.render();
-        assert!(rendered.contains("LambdaFunction"));
+        assert_eq!(expr.render(), "LambdaFunction(x, x)");
     }
 
     #[test]
     fn test_unresolved_named_lambda_variable_render() {
         let var = UnresolvedNamedLambdaVariable::new("x");
         let expr: Expression = 
ExpressionNode::UnresolvedNamedLambdaVariable(var).into();
-        let rendered = expr.render();
-        assert!(rendered.contains("UnresolvedNamedLambdaVariable"));
+        assert_eq!(expr.render(), "x");
     }
 
     #[test]
diff --git a/crates/spark-connect/src/window.rs 
b/crates/spark-connect/src/window.rs
index 6513b06..9672068 100644
--- a/crates/spark-connect/src/window.rs
+++ b/crates/spark-connect/src/window.rs
@@ -44,6 +44,26 @@ pub struct WindowSpec {
 }
 
 impl WindowSpec {
+    /// As `WindowSpec.__repr__` of PySpark Connect, such as
+    /// `WindowSpec(PartitionBy(g), OrderBy(o ASC NULLS FIRST), 
WindowFrame(ROW_FRAME, -1, 1))`.
+    pub fn repr(&self) -> String {
+        let boundary = |bound: &FrameBound| match bound {
+            FrameBound::UnboundedPreceding => 
crate::expression::FrameBoundary::UnboundedPreceding,
+            FrameBound::Preceding(n) => 
crate::expression::FrameBoundary::Preceding(*n),
+            FrameBound::CurrentRow => 
crate::expression::FrameBoundary::CurrentRow,
+            FrameBound::Following(n) => 
crate::expression::FrameBoundary::Following(*n),
+            FrameBound::UnboundedFollowing => 
crate::expression::FrameBoundary::UnboundedFollowing,
+        };
+        let frame = self.frame_spec.as_ref().map(|(frame_type, lower, upper)| {
+            let frame_type = match frame_type {
+                FrameType::Row => 1u32,
+                FrameType::Range => 2u32,
+            };
+            (frame_type, boundary(lower), boundary(upper))
+        });
+        crate::expression::window_spec_repr(&self.partition_spec, 
&self.order_spec, frame.as_ref())
+    }
+
     /// Create a new empty WindowSpec.
     pub fn new() -> Self {
         Self {
diff --git a/crates/spark-connect/tests/expression_sharing.rs 
b/crates/spark-connect/tests/expression_sharing.rs
index a4865f4..ae7becf 100644
--- a/crates/spark-connect/tests/expression_sharing.rs
+++ b/crates/spark-connect/tests/expression_sharing.rs
@@ -66,8 +66,8 @@ fn aliases_and_function_arguments_share_reused_subtrees() {
         assert_shared(argument, source.expression());
     }
     assert_eq!(source.to_proto(), before);
-    assert_eq!(left.expression().render(), "(value + 1) AS left");
-    assert_eq!(right.expression().render(), "(value + 1) AS right");
+    assert_eq!(left.expression().render(), "+(value, 1) AS left");
+    assert_eq!(right.expression().render(), "+(value, 1) AS right");
 }
 
 #[test]
@@ -100,7 +100,7 @@ fn 
case_branches_share_children_without_mutating_the_original() {
     assert_eq!(original.to_proto(), before);
     assert_eq!(
         completed.expression().render(),
-        "CASE WHEN (value > 0) THEN (value + 1) ELSE 0 END"
+        "CASE WHEN >(value, 0) THEN +(value, 1) ELSE 0 END"
     );
 }
 
@@ -170,7 +170,7 @@ fn 
debug_render_and_proto_keep_the_existing_representation() {
 
     let source = col("value") + lit(1);
     let alias = source.clone().alias("total");
-    assert_eq!(alias.expression().render(), "(value + 1) AS total");
+    assert_eq!(alias.expression().render(), "+(value, 1) AS total");
     assert_eq!(
         alias.to_proto(),
         Alias::new(source.expression().clone(), "total").to_proto()
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 05ed52f..b38873a 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -4436,6 +4436,66 @@ def test_instr_regexp_replace_optional_args():
         F._call_function = orig
 
 
[email protected]("column,expected", [
+    # As PySpark Connect's Expression reprs: an operator renders as a function.
+    (lambda F: F.col("a") + 1, "+(a, 1)"),
+    (lambda F: 1 - F.col("a"), "-(1, a)"),
+    (lambda F: (F.col("a") + 1) * (F.col("b") - 2), "*(+(a, 1), -(b, 2))"),
+    (lambda F: -F.col("a"), "negative(a)"),
+    (lambda F: ~F.col("a"), "not(a)"),
+    (lambda F: F.col("a") != 1, "not(==(a, 1))"),
+    (lambda F: F.col("a") & F.col("b"), "and(a, b)"),
+    (lambda F: F.col("a").between(1, 2), "and(>=(a, 1), <=(a, 2))"),
+    (lambda F: F.when(F.col("a") > 1, 1).otherwise(0), "CASE WHEN >(a, 1) THEN 
1 ELSE 0 END"),
+    (lambda F: F.countDistinct("a"), "count(distinct a)"),
+    (lambda F: F.col("a").cast("int"), "CAST(a AS INT)"),
+    (lambda F: F.col("a").cast("struct<x:int,y:string>"), "CAST(a AS 
STRUCT<X:INT,Y:STRING>)"),
+    (lambda F: F.col("a").try_cast("int"), "TRY_CAST(a AS INT)"),
+    (lambda F: F.col("*"), "unresolvedstar()"),
+    (lambda F: F.count("*"), "count(unresolvedstar())"),
+    (lambda F: F.col("t.*"), "unresolvedstar(t.*)"),
+    (lambda F: F.col("a")[0], "a['0']"),
+    (lambda F: F.col("a")["k"], "a['k']"),
+    # Literals, as LiteralExpression.__repr__ shows their Python values.
+    (lambda F: F.lit(True), "True"),
+    (lambda F: F.lit(None), "NULL"),
+    (lambda F: F.lit(1.5), "1.5"),
+    (lambda F: F.lit(1e20), "1e+20"),
+    (lambda F: F.lit(1e-07), "1e-07"),
+    (lambda F: F.lit(float("nan")), "nan"),
+    (lambda F: F.lit(b"a'b"), 'b"a\'b"'),
+    (lambda F: F.lit(bytes([0, 9, 92, 200])), "b'\\x00\\t\\\\\\xc8'"),
+    (lambda F: F.lit(__import__("datetime").date(1999, 12, 31)), "1999-12-31"),
+    # A naive datetime is a local time, rendered in the local time zone again.
+    (lambda F: F.lit(__import__("datetime").datetime(2020, 1, 2, 3, 4, 5, 6)),
+     "2020-01-02 03:04:05.000006"),
+    (lambda F: F.lit(__import__("decimal").Decimal("-1.50")), "-1.50"),
+])
+def test_column_reprs_as_pyspark(column, expected):
+    from pyspark.sql import functions as F
+
+    assert repr(column(F)) == f"Column<'{expected}'>"
+
+
+def test_window_and_lambda_reprs_as_pyspark():
+    import re
+
+    from pyspark.sql import Window
+    from pyspark.sql import functions as F
+
+    spec = Window.partitionBy("g").orderBy(F.col("o").desc()).rowsBetween(-1, 
1)
+    assert repr(spec) == "WindowSpec(PartitionBy(g), OrderBy(o DESC NULLS 
LAST), WindowFrame(ROW_FRAME, -1, 1))"
+    window = F.sum("a").over(
+        
Window.orderBy(F.col("o").asc()).rangeBetween(Window.unboundedPreceding, 
Window.currentRow))
+    assert repr(window) == (
+        "Column<'WindowExpression(sum(a), (WindowSpec(OrderBy(o ASC NULLS 
FIRST), "
+        "WindowFrame(RANGE_FRAME, -9223372036854775808, 0))))'>")
+    # A lambda variable is named by a counter, as in PySpark.
+    assert re.fullmatch(
+        r"Column<'transform\(a, LambdaFunction\(\+\(x_(\d+), 1\), x_\1\)\)'>",
+        repr(F.transform("a", lambda x: x + 1)))
+
+
 @pytest.mark.parametrize("column,expected", [
     # instr and regexp_replace are plain functions, as other functions are.
     (lambda F: F.instr("s", "b"), "instr(s, b)"),


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to