This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new e5b3b8c [SPARK-60659][CONNECT][TESTS] Run interval input cases in the
live suite
e5b3b8c is described below
commit e5b3b8cf20e1f36c8f565f61780402300969a177
Author: DB Tsai <[email protected]>
AuthorDate: Sun Nov 8 05:26:02 2026 +0900
[SPARK-60659][CONNECT][TESTS] Run interval input cases in the live suite
### What changes were proposed in this pull request?
The three day-time interval live tests added by #116 are absent from the
live CI test collection. Import `IntervalInputLiveTests` into the existing live
suite, so the current CI command covers inferred and explicit inputs, nested
values, nulls and collected-row round trips. Also assert the decoded row count
before comparing interval IPC rows, so truncation cannot pass the unit test.
Clean up the three offline sessions so the cases are self-contained when run
directly with unittest, an [...]
### Why are the changes needed?
Follow-up to
[SPARK-60659](https://issues.apache.org/jira/browse/SPARK-60659) / #116. The
interval cases module is not named for pytest discovery, and only its
conversion class was registered in the offline suite.
### Does this PR introduce any user-facing change?
No. Test coverage only.
### How was this patch tested?
- Test collection on campaign base
`62080cbe5ac168a112dea656134c580b9ae4df03`: 105 tests and no interval live
cases; with this patch on that base: 108 tests including all three interval
live cases. The tests skip when no server is configured. This check used
installed PySpark 4.2 only to import/collect the tests, not to claim Spark 4.3
behavior.
- Python syntax checks, `git diff --check`, and the no-stub audit pass.
- The patch applies cleanly to newer upstream master
`2b9559113aa073e9204e11cfb0b10e2f904c8e13`. The interval cases, session stop
method and timedelta conversion are unchanged there; this is a static
integration check, not runtime validation.
- Genuine local Claude Code reviewed the landed patch and the complete
final four-file follow-up. No actionable source findings remain; every finding
and nit was assessed.
- Draft: required Rust checks, native-extension tests, live Spark 4.3 tests
and the official parity gate remain pending. The local host has no Rust
toolchain. A campaign-lock-wrapped Cargo test stopped before compilation
because `cargo` is unavailable.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-6).
This pull request and its description were written by Isaac.
Closes #292 from dbtsai/codex/postmerge-pr-116-interval-test-coverage.
Authored-by: DB Tsai <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/session.rs | 2 +-
crates/spark-connect/src/session.rs | 1 +
python/tests/interval_input_cases.py | 3 +++
python/tests/test_dropin_e2e.py | 3 +++
4 files changed, 8 insertions(+), 1 deletion(-)
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index cafbceb..5e59672 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -2463,7 +2463,7 @@ fn pandas_decimal_type(data: &Bound<'_, PyList>, rows:
&[Row], i: usize) -> PyRe
/// Infer the Spark `DataType` for a `createDataFrame` column from its (first)
value.
/// Nested collections are inferred recursively; nulls and empty collections
use
-/// String as a fallback. Interval input conversion is tracked in SPARK-60659.
+/// String as a fallback.
fn value_to_datatype(v: &Value) -> DataType {
let utf8 = || DataType::String {
collation: "UTF8_BINARY".to_string(),
diff --git a/crates/spark-connect/src/session.rs
b/crates/spark-connect/src/session.rs
index fa7e096..0f279aa 100644
--- a/crates/spark-connect/src/session.rs
+++ b/crates/spark-connect/src/session.rs
@@ -2042,6 +2042,7 @@ mod tests {
..Default::default()
})
.unwrap();
+ assert_eq!(decoded.len(), rows.len());
for (actual, expected) in decoded.iter().zip(&rows) {
assert_eq!(actual.values(), expected.values());
}
diff --git a/python/tests/interval_input_cases.py
b/python/tests/interval_input_cases.py
index c38d7c2..6d80e5c 100644
--- a/python/tests/interval_input_cases.py
+++ b/python/tests/interval_input_cases.py
@@ -47,6 +47,7 @@ INFERENCE_CONFS = {
class IntervalInputConversionTests(unittest.TestCase):
def test_overflow_is_checked_recursively_before_rpc(self):
spark = SparkSession.builder.remote("sc://127.0.0.1:1").create()
+ self.addCleanup(spark.stop)
interval = T.DayTimeIntervalType()
for micros in (-(2**63) - 1, 2**63, -(2**63) - 86400000000):
value = timedelta(microseconds=micros)
@@ -73,6 +74,7 @@ class IntervalInputConversionTests(unittest.TestCase):
def test_inference_rejects_mixed_interval_and_numeric_values(self):
spark = SparkSession.builder.remote("sc://127.0.0.1:1").create()
+ self.addCleanup(spark.stop)
delta = timedelta(microseconds=-1)
with self.assertRaises(PySparkTypeError) as error:
spark.createDataFrame([(delta,), (1,)])
@@ -101,6 +103,7 @@ class IntervalInputConversionTests(unittest.TestCase):
def test_bounds_and_null_first_inputs_build_without_rpc(self):
spark = SparkSession.builder.remote("sc://127.0.0.1:1").create()
+ self.addCleanup(spark.stop)
for schema in (None, ["interval"], T.StructType([
T.StructField("interval", T.DayTimeIntervalType())])):
spark.createDataFrame([(None,), (timedelta(microseconds=LOWEST),),
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 4d4b4e9..1c3e4b4 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -42,6 +42,9 @@ from pyspark.sql.types import (
TimestampType,
)
+# Keep interval round trips in the live suite selected by CI.
+from interval_input_cases import IntervalInputLiveTests # noqa: F401
+
@pytest.fixture
def session():
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]