This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 0e0a95d7 [SPARK-61176][CONNECT] Keep the sign of a negative year-month
interval of less than a year in the collected values of the Rust client
0e0a95d7 is described below
commit 0e0a95d785a08025393e6e9e482e2321e465ac0a
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 20:29:14 2026 +0900
[SPARK-61176][CONNECT] Keep the sign of a negative year-month interval of
less than a year in the collected values of the Rust client
### What changes were proposed in this pull request?
[SPARK-61176](https://issues.apache.org/jira/browse/SPARK-61176): Spark
renders a negative year-month interval with the sign on the years, and the
absolute value of the months: `-1` month is `-0-1`, `-12` months is `-1-0`, and
`-13` months is `-1-1`. The Spark source is
`SparkIntervalUtils.toYearMonthIntervalString`.
The Rust client now takes the sign from the total number of months and
formats the absolute years and months. The old code took the years and the
months with truncating `/` and `%`, so the sign of an interval shorter than a
year was lost.
The Rust client has one such formatter, the `IntervalYearMonth` arm of
`ArrowValueDecoder` in `crates/spark-connect/src/dataframe.rs`. The Python
drop-in does not reach it, because it rejects collecting a year-month interval
with `NOT_IMPLEMENTED`.
### Why are the changes needed?
Collected values of negative year-month intervals had the wrong sign when
the interval was shorter than a year. With the old code, the Rust client gave:
```
months = -1 -> "0-1" (expected "-0-1")
months = -11 -> "0-11" (expected "-0-11")
months = -12 -> "-1-0" (correct before and after the change)
months = -13 -> "-1-1" (correct before and after the change)
```
The output for `-1` and `-11` is the output for `+1` and `+11`, so a user
could not tell the two intervals apart.
### Does this PR introduce _any_ user-facing change?
Yes. The Rust client's collected string value of a negative year-month
interval of less than a year now has the sign, for example `-0-1` where it used
to give `0-1`. Values of one year or more and non-negative values are unchanged.
### How was this patch tested?
- `year_month_interval_keeps_sign` in
`crates/spark-connect/src/dataframe.rs` checks the rendering of 0, 1, -1, 11,
-11, 12, -12, 13, -13, 24, -24, `i32::MAX` and `i32::MIN` months. It fails
without this change (`months = -1`: left `"0-1"`, right `"-0-1"`), and passes
with it. `i32::MIN` renders as `-178956970-8`, so the absolute value is taken
in `i64`.
- `interval_types_render_as_string` is unchanged and passes.
- Official PySpark 4.3.0 does not give a value to compare against for the
collected form. Its Connect `collect()` raises
`UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION` for `month_interval`, and `show()`
and `CAST(... AS STRING)` print the SQL literal form (`INTERVAL '-1' MONTH`).
So the `-0-1` form is taken from the Spark source, not from a server output.
- The lib tests of both crates (600 passed), the build of all tests, also
with `wasm-udf`, the offline Python suites (2227 passed, 8 skipped), and the
live drop-in and Arrow suites (209 passed) and live cases (10 passed) against a
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #372 from HyukjinKwon/SPARK-61176-ym-interval-sign.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/spark-connect/src/dataframe.rs | 53 +++++++++++++++++++++++++++++++----
1 file changed, 47 insertions(+), 6 deletions(-)
diff --git a/crates/spark-connect/src/dataframe.rs
b/crates/spark-connect/src/dataframe.rs
index bafdebba..6e6cb56f 100644
--- a/crates/spark-connect/src/dataframe.rs
+++ b/crates/spark-connect/src/dataframe.rs
@@ -2898,12 +2898,12 @@ impl<'a> ArrowValueLayout<'a> {
}
// Intervals likewise have no dedicated Value variant, so render
as strings.
ArrowValueDecoder::IntervalYearMonth(arr) => {
- let months = arr.value(index);
- Ok(Value::String(format!(
- "{}-{}",
- months / 12,
- (months % 12).abs()
- )))
+ // The sign goes on the years, as Spark's
`toYearMonthIntervalString`
+ // does, so -1 month is "-0-1" and not "0-1".
+ let months = arr.value(index) as i64;
+ let sign = if months < 0 { "-" } else { "" };
+ let abs = months.abs();
+ Ok(Value::String(format!("{sign}{}-{}", abs / 12, abs % 12)))
}
ArrowValueDecoder::IntervalDayTime(arr) => {
let v = arr.value(index);
@@ -5223,6 +5223,47 @@ mod arrow_value_tests {
assert!(matches!(arrow_value_at(&mdn, 0).unwrap(), Value::String(_)));
}
+ #[test]
+ fn year_month_interval_keeps_sign() {
+ let months = [
+ 0,
+ 1,
+ -1,
+ 11,
+ -11,
+ 12,
+ -12,
+ 13,
+ -13,
+ 24,
+ -24,
+ i32::MAX,
+ i32::MIN,
+ ];
+ let expected = [
+ "0-0",
+ "0-1",
+ "-0-1",
+ "0-11",
+ "-0-11",
+ "1-0",
+ "-1-0",
+ "1-1",
+ "-1-1",
+ "2-0",
+ "-2-0",
+ "178956970-7",
+ "-178956970-8",
+ ];
+ for (m, want) in months.into_iter().zip(expected) {
+ assert_eq!(
+ arrow_value_at(&IntervalYearMonthArray::from(vec![m]),
0).unwrap(),
+ Value::String(want.to_string()),
+ "months = {m}"
+ );
+ }
+ }
+
#[test]
fn fixed_size_binary() {
let arr = FixedSizeBinaryArray::try_from_iter(vec![vec![1u8,
2u8]].into_iter()).unwrap();
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]