This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new e5b3b8c  [SPARK-60659][CONNECT][TESTS] Run interval input cases in the 
live suite
e5b3b8c is described below

commit e5b3b8cf20e1f36c8f565f61780402300969a177
Author: DB Tsai <[email protected]>
AuthorDate: Sun Nov 8 05:26:02 2026 +0900

    [SPARK-60659][CONNECT][TESTS] Run interval input cases in the live suite
    
    ### What changes were proposed in this pull request?
    
    The three day-time interval live tests added by #116 are absent from the 
live CI test collection. Import `IntervalInputLiveTests` into the existing live 
suite, so the current CI command covers inferred and explicit inputs, nested 
values, nulls and collected-row round trips. Also assert the decoded row count 
before comparing interval IPC rows, so truncation cannot pass the unit test. 
Clean up the three offline sessions so the cases are self-contained when run 
directly with unittest, an [...]
    
    ### Why are the changes needed?
    
    Follow-up to 
[SPARK-60659](https://issues.apache.org/jira/browse/SPARK-60659) / #116. The 
interval cases module is not named for pytest discovery, and only its 
conversion class was registered in the offline suite.
    
    ### Does this PR introduce any user-facing change?
    
    No. Test coverage only.
    
    ### How was this patch tested?
    
    - Test collection on campaign base 
`62080cbe5ac168a112dea656134c580b9ae4df03`: 105 tests and no interval live 
cases; with this patch on that base: 108 tests including all three interval 
live cases. The tests skip when no server is configured. This check used 
installed PySpark 4.2 only to import/collect the tests, not to claim Spark 4.3 
behavior.
    - Python syntax checks, `git diff --check`, and the no-stub audit pass.
    - The patch applies cleanly to newer upstream master 
`2b9559113aa073e9204e11cfb0b10e2f904c8e13`. The interval cases, session stop 
method and timedelta conversion are unchanged there; this is a static 
integration check, not runtime validation.
    - Genuine local Claude Code reviewed the landed patch and the complete 
final four-file follow-up. No actionable source findings remain; every finding 
and nit was assessed.
    - Draft: required Rust checks, native-extension tests, live Spark 4.3 tests 
and the official parity gate remain pending. The local host has no Rust 
toolchain. A campaign-lock-wrapped Cargo test stopped before compilation 
because `cargo` is unavailable.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: OpenAI Codex (GPT-6).
    
    This pull request and its description were written by Isaac.
    
    Closes #292 from dbtsai/codex/postmerge-pr-116-interval-test-coverage.
    
    Authored-by: DB Tsai <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/session.rs     | 2 +-
 crates/spark-connect/src/session.rs  | 1 +
 python/tests/interval_input_cases.py | 3 +++
 python/tests/test_dropin_e2e.py      | 3 +++
 4 files changed, 8 insertions(+), 1 deletion(-)

diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index cafbceb..5e59672 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -2463,7 +2463,7 @@ fn pandas_decimal_type(data: &Bound<'_, PyList>, rows: 
&[Row], i: usize) -> PyRe
 
 /// Infer the Spark `DataType` for a `createDataFrame` column from its (first) 
value.
 /// Nested collections are inferred recursively; nulls and empty collections 
use
-/// String as a fallback. Interval input conversion is tracked in SPARK-60659.
+/// String as a fallback.
 fn value_to_datatype(v: &Value) -> DataType {
     let utf8 = || DataType::String {
         collation: "UTF8_BINARY".to_string(),
diff --git a/crates/spark-connect/src/session.rs 
b/crates/spark-connect/src/session.rs
index fa7e096..0f279aa 100644
--- a/crates/spark-connect/src/session.rs
+++ b/crates/spark-connect/src/session.rs
@@ -2042,6 +2042,7 @@ mod tests {
                 ..Default::default()
             })
             .unwrap();
+        assert_eq!(decoded.len(), rows.len());
         for (actual, expected) in decoded.iter().zip(&rows) {
             assert_eq!(actual.values(), expected.values());
         }
diff --git a/python/tests/interval_input_cases.py 
b/python/tests/interval_input_cases.py
index c38d7c2..6d80e5c 100644
--- a/python/tests/interval_input_cases.py
+++ b/python/tests/interval_input_cases.py
@@ -47,6 +47,7 @@ INFERENCE_CONFS = {
 class IntervalInputConversionTests(unittest.TestCase):
     def test_overflow_is_checked_recursively_before_rpc(self):
         spark = SparkSession.builder.remote("sc://127.0.0.1:1").create()
+        self.addCleanup(spark.stop)
         interval = T.DayTimeIntervalType()
         for micros in (-(2**63) - 1, 2**63, -(2**63) - 86400000000):
             value = timedelta(microseconds=micros)
@@ -73,6 +74,7 @@ class IntervalInputConversionTests(unittest.TestCase):
 
     def test_inference_rejects_mixed_interval_and_numeric_values(self):
         spark = SparkSession.builder.remote("sc://127.0.0.1:1").create()
+        self.addCleanup(spark.stop)
         delta = timedelta(microseconds=-1)
         with self.assertRaises(PySparkTypeError) as error:
             spark.createDataFrame([(delta,), (1,)])
@@ -101,6 +103,7 @@ class IntervalInputConversionTests(unittest.TestCase):
 
     def test_bounds_and_null_first_inputs_build_without_rpc(self):
         spark = SparkSession.builder.remote("sc://127.0.0.1:1").create()
+        self.addCleanup(spark.stop)
         for schema in (None, ["interval"], T.StructType([
                 T.StructField("interval", T.DayTimeIntervalType())])):
             spark.createDataFrame([(None,), (timedelta(microseconds=LOWEST),),
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 4d4b4e9..1c3e4b4 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -42,6 +42,9 @@ from pyspark.sql.types import (
     TimestampType,
 )
 
+# Keep interval round trips in the live suite selected by CI.
+from interval_input_cases import IntervalInputLiveTests  # noqa: F401
+
 
 @pytest.fixture
 def session():


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to