This is an automated email from the ASF dual-hosted git repository.
dbtsai pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 6bdf6e8 [SPARK-60607][CONNECT][PYTHON] Decode Arrow day-time
intervals as timedelta
6bdf6e8 is described below
commit 6bdf6e8033ce5721a7a5c42f1b40afe9baf32e8f
Author: DB Tsai <[email protected]>
AuthorDate: Sat Oct 31 15:19:44 2026 -0700
[SPARK-60607][CONNECT][PYTHON] Decode Arrow day-time intervals as timedelta
### What changes were proposed in this pull request?
Decode Arrow `Duration(Microsecond)` into Rust
`Value::DayTimeInterval(i64)` and exact Python `datetime.timedelta` values
using integer arithmetic. This covers scalar and nested results, nulls, and the
full signed 64-bit microsecond range.
Encode explicit Rust pivot interval values as typed proto literals rather
than Debug strings. Keep typed NA-fill encoding too: Spark rejects interval
fill values, so this prevents silently filling string columns with
`"DayTimeInterval(5)"`. Rust display and stringified interval map keys use raw
microseconds, consistent with the existing Date/Timestamp convention.
### Why are the changes needed?
[SPARK-60607](https://issues.apache.org/jira/browse/SPARK-60607):
collecting a non-null day-time interval currently fails with `Unsupported Arrow
type Duration(Microsecond) - cannot convert to Value`, including intervals
inside structs, lists and maps.
### Does this PR introduce _any_ user-facing change?
Yes. Python `collect()` and `toLocalIterator()` return `datetime.timedelta`
for interval results while preserving the logical schema. Other non-null Arrow
duration units remain unsupported; existing Arrow interval physical types keep
their conversions.
The public Rust `Value` and `LiteralExpression` enums add
`DayTimeInterval(i64)`; downstream exhaustive matches must handle the new
variant. The migration documentation records this change. Python interval map
keys still use strings rather than `timedelta` keys.
Interval input support remains out of scope and is tracked by
[SPARK-60659](https://issues.apache.org/jira/browse/SPARK-60659).
`value_to_datatype` stays infallible and infers the interval type; the input
encoder still rejects it. This PR does not promise a uniform decode-only error
across inferred, explicit-schema, null-first or raw `timedelta` inputs. The
follow-up covers these paths and collected-row round trips, coordinating with
#109 and #113. The Duration decoder arm and tests a [...]
### How was this patch tested?
- Rust library tests: 389 passed. Plan coverage tests: 6 passed. The pivot
regression inspects the aggregate proto for exact typed interval literals at
`i64::MIN`, -1, 0 and `i64::MAX`.
- Offline Python suite: 274 passed, 1 unrelated skip. Both interval tests
ran against the compiled extension with real Arrow IPC and envelopes generated
from the vendored protobuf definitions. They cover all three collection paths,
signed boundaries, nested values/nulls, slices, empty/multiple batches and
schema preservation.
- New `SPARK_REMOTE`-gated Rust integration test passed against Spark
4.2.0: SQL interval literal, `make_dt_interval`, negative intervals, nested
struct/list/map and both iterator modes. A separate Python check against the
same real server verified exact `timedelta` results in all three collection
paths.
- All main-crate test targets compile with default features and `wasm-udf`;
Python extension build, rustfmt and diff whitespace checks passed.
- Full workspace test compilation is locally limited by the missing
`wasm32-unknown-unknown` target for the WASM example. Strict Clippy stops on
existing warnings in unchanged core files; the repository's Clippy job is
advisory.
Latest-head CI is pending. The coverage job still collects the interval
test class through the existing offline-suite import; switching that job to
discover the entire tests directory is separate CI housekeeping.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-6)
Closes #110 from dbtsai/codex/spark-60607-day-time-interval.
Authored-by: DB Tsai <[email protected]>
Signed-off-by: DB Tsai <[email protected]>
---
crates/pyspark-rs/src/row.rs | 12 ++
crates/pyspark-rs/src/session.rs | 8 +-
crates/spark-connect/src/dataframe.rs | 94 +++++++-
crates/spark-connect/src/expression.rs | 6 +
crates/spark-connect/src/group.rs | 31 +++
crates/spark-connect/src/plan.rs | 3 +
crates/spark-connect/src/row.rs | 8 +
crates/spark-connect/tests/coverage_plan_extra.rs | 1 +
crates/spark-connect/tests/e2e_integration.rs | 51 +++++
docs/architecture.md | 4 +
docs/compatibility.md | 5 +
python/tests/day_time_interval_cases.py | 247 ++++++++++++++++++++++
python/tests/test_dropin_offline.py | 3 +
13 files changed, 470 insertions(+), 3 deletions(-)
diff --git a/crates/pyspark-rs/src/row.rs b/crates/pyspark-rs/src/row.rs
index dfdf506..d3bb100 100644
--- a/crates/pyspark-rs/src/row.rs
+++ b/crates/pyspark-rs/src/row.rs
@@ -50,6 +50,18 @@ pub fn value_to_py<'py>(py: Python<'py>, v: &Value) ->
PyResult<Bound<'py, PyAny
.call_method1("fromtimestamp", (seconds,))?
.into_any()
}
+ Value::DayTimeInterval(micros) => {
+ // Normalize using integer arithmetic: neither floats nor an
intermediate
+ // nanosecond duration can preserve the full signed i64
microsecond range.
+ // The resulting +/-107 million days fit timedelta's +/-999999999
days.
+ let days = micros.div_euclid(86_400_000_000);
+ let remainder = micros.rem_euclid(86_400_000_000);
+ py.import("datetime")?.getattr("timedelta")?.call1((
+ days,
+ remainder / 1_000_000,
+ remainder % 1_000_000,
+ ))?
+ }
Value::Decimal { value, .. } => {
// Convert to Python decimal.Decimal by instantiating the class
let decimal_mod = py.import("decimal")?;
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index 5f6302e..cb7a316 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -881,8 +881,8 @@ impl PyConnectClientStub {
}
/// Infer the Spark `DataType` for a `createDataFrame` column from its (first)
value.
-/// Scalars, Date/Timestamp, and Decimal map precisely; Null and nested
collections
-/// fall back to String (createDataFrame here does not deep-infer nested
schemas).
+/// Nested collections are inferred recursively; nulls and empty collections
use
+/// String as a fallback. Interval input conversion is tracked in SPARK-60659.
fn value_to_datatype(v: &Value) -> DataType {
let utf8 = || DataType::String {
collation: "UTF8_BINARY".to_string(),
@@ -899,6 +899,10 @@ fn value_to_datatype(v: &Value) -> DataType {
Value::Binary(_) => DataType::Binary,
Value::Date(_) => DataType::Date,
Value::Timestamp(_) => DataType::Timestamp,
+ Value::DayTimeInterval(_) => DataType::DayTimeInterval {
+ start_field: 0,
+ end_field: 3,
+ },
Value::Decimal {
precision, scale, ..
} => DataType::Decimal {
diff --git a/crates/spark-connect/src/dataframe.rs
b/crates/spark-connect/src/dataframe.rs
index 4ce3898..8e0a8ca 100644
--- a/crates/spark-connect/src/dataframe.rs
+++ b/crates/spark-connect/src/dataframe.rs
@@ -2318,6 +2318,7 @@ fn map_key_to_string(v: Value) -> String {
Value::Double(x) => x.to_string(),
Value::Date(d) => d.to_string(),
Value::Timestamp(t) => t.to_string(),
+ interval @ Value::DayTimeInterval(_) => interval.to_string(),
Value::Decimal { value, .. } => value,
other => format!("{other:?}"),
}
@@ -2474,6 +2475,12 @@ fn arrow_value_at_with_layout(
if let Some(arr) = array.as_any().downcast_ref::<Date64Array>() {
return Ok(Value::Date((arr.value(index) / 86_400_000) as i32));
}
+ // Spark DayTimeIntervalType uses Duration(Microsecond), not an Arrow
+ // Interval physical type. Keep signed micros exactly, including i64::MIN.
+ // Other duration units remain unsupported rather than being rounded.
+ if let Some(arr) =
array.as_any().downcast_ref::<DurationMicrosecondArray>() {
+ return Ok(Value::DayTimeInterval(arr.value(index)));
+ }
// Nested: list, struct, map (recurse).
if let Some(arr) = array.as_any().downcast_ref::<ListArray>() {
let ArrowValueLayout::List(child_layout) = value_layout else {
@@ -3708,6 +3715,90 @@ mod arrow_value_tests {
};
use std::sync::Arc;
+ #[test]
+ fn day_time_interval_microseconds_are_exact() {
+ let values = vec![
+ Some(i64::MIN),
+ Some(i64::MIN + 1),
+ Some(-172_800_000_001),
+ Some(-1),
+ Some(0),
+ Some(1),
+ Some(172_800_000_001),
+ Some(i64::MAX - 1),
+ Some(i64::MAX),
+ None,
+ ];
+ let array = DurationMicrosecondArray::from(values.clone());
+ for (i, value) in values.iter().enumerate() {
+ assert_eq!(
+ arrow_value_at(&array, i).unwrap(),
+ value.map(Value::DayTimeInterval).unwrap_or(Value::Null)
+ );
+ }
+ let sliced = array.slice(2, 5);
+ for i in 0..sliced.len() {
+ assert_eq!(
+ arrow_value_at(&sliced, i).unwrap(),
+ Value::DayTimeInterval(values[i + 2].unwrap())
+ );
+ }
+ let nulls = DurationMicrosecondArray::from(vec![None, None]);
+ assert_eq!(arrow_value_at(&nulls, 1).unwrap(), Value::Null);
+ for array in [
+ Arc::new(DurationSecondArray::from(vec![1])) as ArrayRef,
+ Arc::new(DurationMillisecondArray::from(vec![1])),
+ Arc::new(DurationNanosecondArray::from(vec![1])),
+ ] {
+ assert!(arrow_value_at(array.as_ref(), 0).is_err());
+ }
+ }
+
+ #[test]
+ fn day_time_interval_ipc_empty_and_multiple_batches() {
+ use arrow::datatypes::{Schema, TimeUnit};
+ use arrow::ipc::writer::StreamWriter;
+ use arrow::record_batch::RecordBatch;
+ let schema = Arc::new(Schema::new(vec![Field::new(
+ "interval",
+ ArrowDataType::Duration(TimeUnit::Microsecond),
+ true,
+ )]));
+ let mut data = Vec::new();
+ let mut writer = StreamWriter::try_new(&mut data, &schema).unwrap();
+ for values in [
+ vec![],
+ vec![Some(i64::MIN), None],
+ vec![],
+ vec![Some(i64::MAX)],
+ ] {
+ let batch = RecordBatch::try_new(
+ schema.clone(),
+ vec![Arc::new(DurationMicrosecondArray::from(values))],
+ )
+ .unwrap();
+ writer.write(&batch).unwrap();
+ }
+ writer.finish().unwrap();
+ drop(writer);
+ let rows =
decode_arrow_batch(&proto::execute_plan_response::ArrowBatch {
+ data: data.into(),
+ row_count: 3,
+ ..Default::default()
+ })
+ .unwrap();
+ assert_eq!(
+ rows.iter()
+ .map(|r| r.get(0).unwrap().clone())
+ .collect::<Vec<_>>(),
+ vec![
+ Value::DayTimeInterval(i64::MIN),
+ Value::Null,
+ Value::DayTimeInterval(i64::MAX)
+ ]
+ );
+ }
+
#[test]
fn primitives_and_signed_ints() {
assert!(matches!(
@@ -3898,7 +3989,7 @@ mod arrow_value_tests {
arrow_value_at(&with_null, 0).unwrap(),
Value::Null
));
- // Duration has no dedicated Value arm -> the final unsupported-type
Err.
+ // Only microsecond durations are supported (Spark's physical
encoding).
let dur = DurationSecondArray::from(vec![1i64]);
assert!(arrow_value_at(&dur, 0).is_err());
}
@@ -3915,6 +4006,7 @@ mod arrow_value_tests {
assert_eq!(map_key_to_string(Value::Double(2.5)), "2.5");
assert_eq!(map_key_to_string(Value::Date(5)), "5");
assert_eq!(map_key_to_string(Value::Timestamp(6)), "6");
+ assert_eq!(map_key_to_string(Value::DayTimeInterval(-1)), "-1");
assert_eq!(
map_key_to_string(Value::Decimal {
value: "7.5".to_string(),
diff --git a/crates/spark-connect/src/expression.rs
b/crates/spark-connect/src/expression.rs
index be1e57a..7331052 100644
--- a/crates/spark-connect/src/expression.rs
+++ b/crates/spark-connect/src/expression.rs
@@ -268,6 +268,8 @@ pub enum LiteralExpression {
Date(i32),
Timestamp(i64),
TimestampNtz(i64),
+ /// Signed microseconds in a Spark day-time interval.
+ DayTimeInterval(i64),
Time {
nano: i64,
precision: i32,
@@ -316,6 +318,9 @@ impl LiteralExpression {
LiteralExpression::TimestampNtz(v) => {
proto::expression::literal::LiteralType::TimestampNtz(*v)
}
+ LiteralExpression::DayTimeInterval(v) => {
+ proto::expression::literal::LiteralType::DayTimeInterval(*v)
+ }
LiteralExpression::Time { nano, precision } => {
let mut time = proto::expression::literal::Time::default();
time.nano = *nano;
@@ -365,6 +370,7 @@ impl LiteralExpression {
LiteralExpression::Binary(v) => format!("{v:?}"),
LiteralExpression::Date(v) => v.to_string(),
LiteralExpression::Timestamp(v) |
LiteralExpression::TimestampNtz(v) => v.to_string(),
+ LiteralExpression::DayTimeInterval(v) => v.to_string(),
LiteralExpression::Time { nano, .. } => nano.to_string(),
LiteralExpression::Array { elements, .. } => {
let inner: Vec<String> = elements.iter().map(|e|
e.render()).collect();
diff --git a/crates/spark-connect/src/group.rs
b/crates/spark-connect/src/group.rs
index 54c9eea..dd0431c 100644
--- a/crates/spark-connect/src/group.rs
+++ b/crates/spark-connect/src/group.rs
@@ -26,6 +26,7 @@ fn value_to_lit_expr(v: crate::row::Value) -> Expression {
Value::Float(x) => L::double(x as f64),
Value::Double(x) => L::double(x),
Value::String(s) => L::string(s),
+ Value::DayTimeInterval(v) => L::DayTimeInterval(v),
other => L::string(format!("{:?}", other)),
};
Expression::literal(lit)
@@ -676,6 +677,36 @@ mod tests {
assert!(pivot_grouped.pivot_col.is_some());
}
+ #[test]
+ fn pivot_day_time_intervals_are_exact_literals() {
+ use spark_connect_proto::{expression::literal::LiteralType,
relation::RelType};
+ let values = [i64::MIN, -1, 0, i64::MAX];
+ let frame = session().range(1).unwrap();
+ let pivot = frame
+ .group_by(vec![] as Vec<Column>)
+ .pivot(
+ crate::column::col("interval"),
+ Some(
+ values
+ .into_iter()
+ .map(crate::row::Value::DayTimeInterval)
+ .collect(),
+ ),
+ )
+ .count();
+ let Some(RelType::Aggregate(aggregate)) =
pivot.plan.to_proto().rel_type else {
+ panic!("expected Aggregate");
+ };
+ let literals = aggregate.pivot.unwrap().values;
+ assert_eq!(literals.len(), values.len());
+ for (literal, micros) in literals.iter().zip(values) {
+ assert_eq!(
+ literal.literal_type,
+ Some(LiteralType::DayTimeInterval(micros))
+ );
+ }
+ }
+
#[test]
fn stat_crosstab_plan() {
let spark = session();
diff --git a/crates/spark-connect/src/plan.rs b/crates/spark-connect/src/plan.rs
index b24309b..7d5274a 100644
--- a/crates/spark-connect/src/plan.rs
+++ b/crates/spark-connect/src/plan.rs
@@ -1646,6 +1646,9 @@ fn value_to_proto_literal(v: &crate::row::Value) ->
proto::expression::Literal {
Value::String(s) => LiteralType::String(s.clone()),
Value::Date(d) => LiteralType::Date(*d),
Value::Timestamp(t) => LiteralType::Timestamp(*t),
+ // Spark rejects interval NA fills. Preserve the type so it errors
instead
+ // of silently filling string columns with the Value's Debug text.
+ Value::DayTimeInterval(t) => LiteralType::DayTimeInterval(*t),
Value::Decimal {
value,
precision,
diff --git a/crates/spark-connect/src/row.rs b/crates/spark-connect/src/row.rs
index 8c5c979..a817b04 100644
--- a/crates/spark-connect/src/row.rs
+++ b/crates/spark-connect/src/row.rs
@@ -34,6 +34,10 @@ pub enum Value {
Date(i32),
/// Timestamp (microseconds since epoch)
Timestamp(i64),
+ /// Spark day-time interval, in signed microseconds. The full i64 range is
+ /// representable by Python's datetime.timedelta without nanosecond
conversion.
+ /// Display uses raw microseconds, like Date and Timestamp use their raw
units.
+ DayTimeInterval(i64),
/// Decimal (string value with optional precision and scale)
Decimal {
value: String,
@@ -68,6 +72,7 @@ impl fmt::Debug for Value {
Self::Binary(v) => f.debug_tuple("Binary").field(v).finish(),
Self::Date(v) => f.debug_tuple("Date").field(v).finish(),
Self::Timestamp(v) => f.debug_tuple("Timestamp").field(v).finish(),
+ Self::DayTimeInterval(v) =>
f.debug_tuple("DayTimeInterval").field(v).finish(),
Self::Decimal {
value,
precision,
@@ -169,6 +174,7 @@ impl fmt::Display for Value {
Value::Binary(b) => write!(f, "{:?}", b),
Value::Date(d) => write!(f, "{}", d),
Value::Timestamp(t) => write!(f, "{}", t),
+ Value::DayTimeInterval(t) => write!(f, "{}", t),
Value::Decimal {
value,
precision,
@@ -744,6 +750,8 @@ mod tests {
assert_eq!(Value::Binary(vec![1, 2]).to_string(), "[1, 2]");
assert_eq!(Value::Date(19_000).to_string(), "19000");
assert_eq!(Value::Timestamp(123).to_string(), "123");
+ assert_eq!(Value::DayTimeInterval(-1).to_string(), "-1");
+ assert_eq!(Value::DayTimeInterval(1).as_i64(), None);
assert_eq!(
Value::List(vec![Value::Integer(1),
Value::Integer(2)]).to_string(),
"[1, 2]"
diff --git a/crates/spark-connect/tests/coverage_plan_extra.rs
b/crates/spark-connect/tests/coverage_plan_extra.rs
index 9a581b8..81784f9 100644
--- a/crates/spark-connect/tests/coverage_plan_extra.rs
+++ b/crates/spark-connect/tests/coverage_plan_extra.rs
@@ -66,6 +66,7 @@ fn nafill_covers_all_value_literal_arms() {
Value::String("s".to_string()),
Value::Date(5),
Value::Timestamp(6),
+ Value::DayTimeInterval(5),
Value::Decimal {
value: "1.23".to_string(),
precision: Some(5),
diff --git a/crates/spark-connect/tests/e2e_integration.rs
b/crates/spark-connect/tests/e2e_integration.rs
index f6a14d5..2c137cc 100644
--- a/crates/spark-connect/tests/e2e_integration.rs
+++ b/crates/spark-connect/tests/e2e_integration.rs
@@ -478,3 +478,54 @@ fn test_collect_batch_larger_than_grpc_default_cap() {
"Expected the full {n}-byte string to round-trip through collect()"
);
}
+
+#[test]
+fn test_day_time_interval_real_server() {
+ if !should_run() {
+ return;
+ }
+ let remote = std::env::var("SPARK_REMOTE").unwrap();
+ let spark = SparkSession::builder()
+ .remote(&remote)
+ .get_or_create()
+ .unwrap();
+ let micros = 93_784_000_005;
+ let frame = spark.sql(
+ "WITH t AS (SELECT INTERVAL '1 02:03:04.000005' DAY TO SECOND AS d) \
+ SELECT d, -d AS negative, make_dt_interval(1, 2, 3, 4.000005) AS
made, \
+ named_struct('leaf', d) AS nested, array(d, -d, CAST(NULL AS INTERVAL
DAY TO SECOND)) AS items, \
+ map('k', d) AS mapping FROM t"
+ ).unwrap();
+ let rows = frame.collect().unwrap();
+ assert_eq!(rows.len(), 1);
+ let row = &rows[0];
+ assert_eq!(row.get(0), Some(&Value::DayTimeInterval(micros)));
+ assert_eq!(row.get(1), Some(&Value::DayTimeInterval(-micros)));
+ assert_eq!(row.get(2), Some(&Value::DayTimeInterval(micros)));
+ let Some(Value::Struct(nested)) = row.get(3) else {
+ panic!("expected struct")
+ };
+ assert_eq!(nested.values(), &[Value::DayTimeInterval(micros)]);
+ assert_eq!(
+ row.get(4),
+ Some(&Value::List(vec![
+ Value::DayTimeInterval(micros),
+ Value::DayTimeInterval(-micros),
+ Value::Null,
+ ]))
+ );
+ assert_eq!(
+ row.get(5),
+ Some(&Value::Map(
+ [("k".to_string(), Value::DayTimeInterval(micros))].into()
+ ))
+ );
+ for prefetch in [false, true] {
+ let streamed: Vec<_> = frame
+ .to_local_iterator(prefetch)
+ .unwrap()
+ .collect::<Result<_, _>>()
+ .unwrap();
+ assert_eq!(streamed, rows);
+ }
+}
diff --git a/docs/architecture.md b/docs/architecture.md
index 17559ed..f017d9d 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -79,6 +79,10 @@ flowchart LR
When you call an action such as `count()` or `show()`, the lazily-built plan is
serialized and sent to the server; results stream back as Arrow batches.
+Day-time intervals are returned as Python `datetime.timedelta` values,
including
+nested values and nulls. Creating DataFrames from `timedelta` inputs or
collected
+interval rows is not yet supported; see
[SPARK-60659](https://issues.apache.org/jira/browse/SPARK-60659).
+
```mermaid
sequenceDiagram
participant App as Your code (Rust or pyspark)
diff --git a/docs/compatibility.md b/docs/compatibility.md
index cc5fb6a..ace65f6 100644
--- a/docs/compatibility.md
+++ b/docs/compatibility.md
@@ -15,6 +15,11 @@ Spark it speaks.
## Rust migration for the 4.3 line
+`Value` and `LiteralExpression` add a `DayTimeInterval(i64)` variant for signed
+microseconds. Downstream exhaustive matches on these public enums must handle
+the new variant. Rust interval map keys are stringified as raw microseconds;
+Python `timedelta` map keys are not yet supported.
+
The 4.3 line changes the public Rust representation of expressions and logical
plans to share immutable nodes. This is a breaking change for code that
directly
constructs or matches these representations. The `Column` and `DataFrame` APIs,
diff --git a/python/tests/day_time_interval_cases.py
b/python/tests/day_time_interval_cases.py
new file mode 100644
index 0000000..11fb422
--- /dev/null
+++ b/python/tests/day_time_interval_cases.py
@@ -0,0 +1,247 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""SPARK-60607: exact day-time interval results, without a JVM.
+
+The small gRPC fixture sends real Arrow IPC and just the protobuf envelopes
+needed by these tests. It does not depend on the reference PySpark package
+(which occupies the same import namespace as this client).
+
+Envelopes and logical types are generated from the vendored Spark Connect
protos
+using protoc, which is also required to build the Rust client.
+"""
+
+from concurrent.futures import ThreadPoolExecutor
+from datetime import timedelta
+import unittest
+import os
+from pathlib import Path
+import shutil
+import subprocess
+import tempfile
+
+try:
+ import grpc
+ import pyarrow as pa
+ from google.protobuf import descriptor_pb2, descriptor_pool,
message_factory
+except ImportError:
+ grpc = pa = None
+
+from pyspark.sql import Row, SparkSession, types as T
+
+
+def _messages():
+ proto_root = Path(__file__).resolve().parents[2] /
"crates/spark-connect-proto/proto"
+ compiler = shutil.which(os.environ.get("PROTOC") or "protoc")
+ if not compiler:
+ raise unittest.SkipTest("protoc is required for the interval fixture")
+ with tempfile.TemporaryDirectory() as directory:
+ output = Path(directory) / "connect.pb"
+ command = [compiler, f"-I{proto_root}", "--include_imports",
+ f"--descriptor_set_out={output}"]
+ if os.environ.get("PROTOC_INCLUDE"):
+ command.append(f"-I{os.environ['PROTOC_INCLUDE']}")
+ subprocess.run(command + ["spark/connect/base.proto"], check=True)
+ descriptors =
descriptor_pb2.FileDescriptorSet.FromString(output.read_bytes())
+ pool = descriptor_pool.DescriptorPool()
+ for file in descriptors.file:
+ pool.Add(file)
+ names = ["DataType", "ExecutePlanRequest", "ExecutePlanResponse",
+ "AnalyzePlanRequest", "AnalyzePlanResponse",
"ReleaseExecuteRequest",
+ "ReleaseExecuteResponse", "ReleaseSessionRequest",
"ReleaseSessionResponse"]
+ return {name: message_factory.GetMessageClass(
+ pool.FindMessageTypeByName(f"spark.connect.{name}")) for name in names}
+
+
+def _type(data_type, message):
+ result = message()
+ if pa.types.is_duration(data_type):
+ result.day_time_interval.start_field = 0
+ result.day_time_interval.end_field = 3
+ elif pa.types.is_string(data_type):
+ result.string.SetInParent()
+ elif pa.types.is_struct(data_type):
+ for field in data_type:
+ result.struct.fields.add(name=field.name, nullable=field.nullable,
+ data_type=_type(field.type, message))
+ elif pa.types.is_list(data_type):
+ result.array.element_type.CopyFrom(_type(data_type.value_type,
message))
+ result.array.contains_null = True
+ elif pa.types.is_map(data_type):
+ result.map.key_type.CopyFrom(_type(data_type.key_type, message))
+ result.map.value_type.CopyFrom(_type(data_type.item_type, message))
+ result.map.value_contains_null = True
+ else:
+ raise AssertionError(data_type)
+ return result
+
+
[email protected](grpc is None or pa is None, "grpcio, pyarrow and protobuf are
required")
+class DayTimeIntervalTests(unittest.TestCase):
+ @classmethod
+ def setUpClass(cls):
+ cls.messages = _messages()
+
+ def _normalize(self, value):
+ if isinstance(value, Row):
+ value = value.asDict()
+ if isinstance(value, dict):
+ return {k: self._normalize(v) for k, v in value.items()}
+ if isinstance(value, list):
+ return [self._normalize(v) for v in value]
+ return value
+
+ def _check_schema(self, actual, expected):
+ if pa.types.is_duration(expected):
+ self.assertIsInstance(actual, T.DayTimeIntervalType)
+ self.assertEqual(actual.json(), T.DayTimeIntervalType(0, 3).json())
+ elif pa.types.is_struct(expected):
+ self.assertIsInstance(actual, T.StructType)
+ for field in expected:
+ self._check_schema(actual[field.name].dataType, field.type)
+ elif pa.types.is_list(expected):
+ self._check_schema(actual.elementType, expected.value_type)
+ elif pa.types.is_map(expected):
+ self._check_schema(actual.valueType, expected.item_type)
+
+ def _check(self, batches, expected):
+ schema = batches[0].schema
+ logical_schema = _type(pa.struct(list(schema)),
self.messages["DataType"])
+ payloads = []
+ # Each response contains multiple IPC record batches, including empty
ones.
+ for group in (batches[:2], batches[2:]):
+ if not group:
+ continue
+ sink = pa.BufferOutputStream()
+ with pa.ipc.new_stream(sink, schema) as writer:
+ for batch in group:
+ writer.write_batch(batch)
+ payloads.append((sum(b.num_rows for b in group),
sink.getvalue().to_pybytes()))
+
+ def execute(raw, context):
+ request = self.messages["ExecutePlanRequest"].FromString(raw)
+ for i, (count, data) in enumerate(payloads):
+ response = self.messages["ExecutePlanResponse"](
+ session_id=request.session_id,
server_side_session_id="interval-fixture",
+ operation_id=request.operation_id, response_id=str(i),
schema=logical_schema)
+ response.arrow_batch.row_count = count
+ response.arrow_batch.data = data
+ yield response.SerializeToString()
+ done = self.messages["ExecutePlanResponse"](
+ session_id=request.session_id,
server_side_session_id="interval-fixture",
+ operation_id=request.operation_id, response_id="complete")
+ done.result_complete.SetInParent()
+ yield done.SerializeToString()
+
+ def analyze(raw, context):
+ request = self.messages["AnalyzePlanRequest"].FromString(raw)
+ response = self.messages["AnalyzePlanResponse"](
+ session_id=request.session_id,
server_side_session_id="interval-fixture")
+ response.schema.schema.CopyFrom(logical_schema)
+ return response.SerializeToString()
+
+ def release_execute(raw, context):
+ request = self.messages["ReleaseExecuteRequest"].FromString(raw)
+ return self.messages["ReleaseExecuteResponse"](
+ session_id=request.session_id).SerializeToString()
+
+ def release_session(raw, context):
+ request = self.messages["ReleaseSessionRequest"].FromString(raw)
+ return self.messages["ReleaseSessionResponse"](
+ session_id=request.session_id).SerializeToString()
+
+ with ThreadPoolExecutor(max_workers=2) as pool:
+ server = grpc.server(pool)
+
server.add_generic_rpc_handlers([grpc.method_handlers_generic_handler(
+ "spark.connect.SparkConnectService", {
+ "ExecutePlan":
grpc.unary_stream_rpc_method_handler(execute),
+ "AnalyzePlan":
grpc.unary_unary_rpc_method_handler(analyze),
+ "ReleaseExecute":
grpc.unary_unary_rpc_method_handler(release_execute),
+ "ReleaseSession":
grpc.unary_unary_rpc_method_handler(release_session),
+ }
+ )])
+ port = server.add_insecure_port("127.0.0.1:0")
+ server.start()
+ spark = None
+ try:
+ spark =
SparkSession.builder.remote(f"sc://127.0.0.1:{port}").create()
+ frame = spark.range(1)
+ self._check_schema(frame.schema, pa.struct(list(schema)))
+ before = frame.schema.json()
+ for rows in (frame.collect(), list(frame.toLocalIterator()),
+
list(frame.toLocalIterator(prefetchPartitions=True))):
+ self.assertEqual(len(rows), len(expected))
+ for row, wanted in zip(rows, expected):
+ actual = self._normalize(row)
+ self.assertEqual(actual, wanted)
+ self._assert_timedeltas(actual, wanted)
+ self.assertEqual(frame.schema.json(), before)
+ self.assertIn("interval", before)
+ finally:
+ if spark is not None:
+ spark.stop()
+ server.stop(0).wait()
+
+ def _assert_timedeltas(self, actual, expected):
+ if isinstance(expected, timedelta):
+ self.assertIs(type(actual), timedelta)
+ elif isinstance(expected, dict):
+ for key, value in expected.items():
+ self._assert_timedeltas(actual[key], value)
+ elif isinstance(expected, list):
+ for a, e in zip(actual, expected):
+ self._assert_timedeltas(a, e)
+
+ def test_scalars_boundaries_nulls_slices_and_batches(self):
+ values = [-(2**63), -(2**63) + 1, -172800000001, -1000001, -1,
+ 0, 1, 1000001, 172800000001, 2**63 - 2, 2**63 - 1, None]
+ # Slice away non-null sentinels, checking both value and validity
offsets.
+ array = pa.array([99] + values + [99],
type=pa.duration("us")).slice(1, len(values))
+ schema = pa.schema([pa.field("interval", pa.duration("us"))])
+ batch = pa.RecordBatch.from_arrays([array], schema=schema)
+ empty = batch.slice(0, 0)
+ self._check([empty, batch.slice(0, 5), empty, batch.slice(5)], [
+ {"interval": None if v is None else timedelta(microseconds=v)} for
v in values
+ ])
+
+ def test_nested_struct_list_map_and_parent_nulls(self):
+ duration = pa.duration("us")
+ typ = pa.struct([
+ pa.field("leaf", duration),
+ pa.field("items", pa.list_(duration)),
+ pa.field("mapping", pa.map_(pa.string(), pa.list_(duration))),
+ ])
+ values = [None, {"leaf": -1, "items": [None, -(2**63), 2**63 - 1],
+ "mapping": [("k", [-172800000001, None])]},
+ {"leaf": 0, "items": [], "mapping": []},
+ {"leaf": None, "items": None, "mapping": None}]
+ array = pa.array([None] + values + [None], type=typ).slice(1,
len(values))
+ batch = pa.RecordBatch.from_arrays([array], names=["nested"])
+ self._check([batch.slice(0, 0), batch.slice(0, 2), batch.slice(2)], [
+ {"nested": None},
+ {"nested": {"leaf": timedelta(microseconds=-1),
+ "items": [None, timedelta(microseconds=-(2**63)),
+ timedelta(microseconds=2**63 - 1)],
+ "mapping": {"k":
[timedelta(microseconds=-172800000001), None]}}},
+ {"nested": {"leaf": timedelta(0), "items": [], "mapping": {}}},
+ {"nested": {"leaf": None, "items": None, "mapping": None}},
+ ])
+
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 9988f65..0c7c068 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -16,6 +16,9 @@ import pytest
from _mock_connect_server import message_field, mock_connect_session,
struct_schema
from pyspark.sql import SparkSession, types as T
+# Collected here so the existing offline CI suite exercises real Arrow results.
+from day_time_interval_cases import DayTimeIntervalTests # noqa: F401
+
# ---------------------------------------------------------------------------
types
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]