This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 774f939  [SPARK-60947][CONNECT][PYTHON] Send and collect TIME values, 
and parse the TIME type of DDL strings, as PySpark does
774f939 is described below

commit 774f939fce453df1d0fbe1872bdb191edff24f96
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Fri Nov 6 23:24:14 2026 +0900

    [SPARK-60947][CONNECT][PYTHON] Send and collect TIME values, and parse the 
TIME type of DDL strings, as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-60947](https://issues.apache.org/jira/browse/SPARK-60947): the 
Python client now sends and collects TIME values, and parses the TIME type of 
DDL strings, as PySpark Connect does.
    
    - `createDataFrame` infers `TimeType()` of a `datetime.time`, and sends a 
TIME value as PyArrow's `time64[ns]` of it, as `to_arrow_type` declares. That 
type also takes the nanoseconds of an int.
    - `lit()` of a `datetime.time` is a time literal of its nanoseconds since 
midnight.
    - Collected TIME values are `datetime.time`, as `TimeType.fromInternal` 
reads them.
    - A bare `time`, in a DDL string or in the JSON of a type, is `TimeType(6)`.
    
    So, as for PySpark, the server raises `UNSUPPORTED_TIME_TYPE` unless the 
session enables `spark.sql.timeType.enabled`, and with it TIME values 
round-trip. The core values have `Time`, in nanoseconds since midnight, for 
these.
    
    ### Why are the changes needed?
    
    TIME values were unsupported, and differed from PySpark 4.3.0:
    
    - `createDataFrame([(datetime.time(1, 2, 3),)])` raised `TypeError: 
unsupported value type for createDataFrame: time`, rather than the server's 
`UNSUPPORTED_TIME_TYPE`.
    - `createDataFrame(..., "t time")` raised `ValueError: invalid type string 
't time'`.
    - `lit(datetime.time(...))` raised `UNSUPPORTED_DATA_TYPE`.
    - Collected TIME values, such as of `SELECT TIME'12:34:56.789'`, were 
strings.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. TIME values work as in PySpark 4.3.0.
    
    ### How was this patch tested?
    
    - A new live test round-trips TIME values of `createDataFrame`, inferred 
and of a DDL schema, of `lit()` and of SQL, with the TIME type enabled. It also 
checks `UNSUPPORTED_TIME_TYPE` without the TIME type. It passes with official 
PySpark 4.3.0, and fails without this change.
    - A new offline test checks, against a mock Connect server, the 
`time64[ns]` values and the `time(6)` schema that `createDataFrame` sends, and 
the time literal of `lit()`. Another checks the parsing of `time` and 
`time(p)`. They fail without this change.
    - New lib tests check the decoding of Arrow times, the encoding of TIME 
values, and the parsing of `time`.
    - The lib tests and every integration test, e2e included, against a local 
Spark 4.3.0 server (664 passed, and 125 of the transport crate), the build of 
all tests with `wasm-udf`, the offline Python suites (1330 passed, 8 skipped), 
and the live drop-in and Arrow suites (88 passed) and live cases (10 passed). 
Clippy reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #214 from HyukjinKwon/SPARK-60947-time.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/functions.rs    |  8 ++++
 crates/pyspark-rs/src/infer.rs        | 26 +++++++++++++
 crates/pyspark-rs/src/row.rs          | 15 ++++++++
 crates/pyspark-rs/src/session.rs      | 11 ++++++
 crates/spark-connect/src/dataframe.rs | 71 ++++++++++++-----------------------
 crates/spark-connect/src/row.rs       |  4 ++
 crates/spark-connect/src/session.rs   | 31 ++++++++++++++-
 crates/spark-connect/src/types.rs     | 14 ++++++-
 python/tests/test_dropin_e2e.py       | 24 ++++++++++++
 python/tests/test_dropin_offline.py   | 52 +++++++++++++++++++++++++
 10 files changed, 208 insertions(+), 48 deletions(-)

diff --git a/crates/pyspark-rs/src/functions.rs 
b/crates/pyspark-rs/src/functions.rs
index 32f2bc8..2952a07 100644
--- a/crates/pyspark-rs/src/functions.rs
+++ b/crates/pyspark-rs/src/functions.rs
@@ -305,6 +305,14 @@ fn scalar_literal(obj: &Bound<'_, PyAny>) -> 
PyResult<Option<LiteralExpression>>
             let micros = crate::session::datetime_to_micros(obj)?;
             return Ok(Some(LiteralExpression::Timestamp(micros)));
         }
+        // datetime.time -> TimeType(), as PySpark's literal of one: its 
nanoseconds since
+        // midnight, of the default precision.
+        if obj.is_instance(&dt_mod.getattr("time")?)? {
+            return Ok(Some(LiteralExpression::Time {
+                nano: crate::infer::time_nanos(obj)?,
+                precision: 6,
+            }));
+        }
         // datetime.timedelta -> DayTimeIntervalType, as PySpark's literal of 
one.
         if obj.is_instance(&dt_mod.getattr("timedelta")?)? {
             let micros = crate::infer::timedelta_micros(obj)?;
diff --git a/crates/pyspark-rs/src/infer.rs b/crates/pyspark-rs/src/infer.rs
index 0119d54..7b3d0fe 100644
--- a/crates/pyspark-rs/src/infer.rs
+++ b/crates/pyspark-rs/src/infer.rs
@@ -1051,10 +1051,36 @@ pub(crate) fn convert(obj: &Bound<'_, PyAny>, dt: 
&DataType, nullable: bool) ->
             let ordinal: i64 = obj.call_method0(intern!(py, 
"toordinal"))?.extract()?;
             Ok(Value::Date((ordinal - 719_163) as i32))
         }
+        // As PyArrow's `time64[ns]`, to which PySpark passes the value as it 
is: a time, and
+        // the nanoseconds of an int, or of a float, truncated.
+        DataType::Time { .. } => {
+            if obj.is_instance(TIME.import(py, "datetime", "time")?.as_any())? 
{
+                Ok(Value::Time(time_nanos(obj)?))
+            } else if obj.is_instance_of::<PyInt>() {
+                Ok(Value::Time(obj.extract()?))
+            } else if obj.is_instance_of::<PyFloat>() {
+                Ok(Value::Time(obj.extract::<f64>()? as i64))
+            } else {
+                Err(PyTypeError::new_err(format!(
+                    "object of type {} cannot be converted to int",
+                    obj.get_type().repr()?
+                )))
+            }
+        }
         _ => Err(unsupported(obj)?),
     }
 }
 
+/// The nanoseconds since midnight of a `datetime.time`, as PySpark's 
`TimeType.toInternal`.
+pub(crate) fn time_nanos(obj: &Bound<'_, PyAny>) -> PyResult<i64> {
+    let py = obj.py();
+    let field = |name: &Bound<'_, PyString>| -> PyResult<i64> { 
obj.getattr(name)?.extract() };
+    let seconds = field(intern!(py, "hour"))? * 3600
+        + field(intern!(py, "minute"))? * 60
+        + field(intern!(py, "second"))?;
+    Ok(seconds * 1_000_000_000 + field(intern!(py, "microsecond"))? * 1_000)
+}
+
 /// A value of a day-time interval column, as PyArrow converts one to 
`duration[us]`: a
 /// timedelta to its microseconds, as `DayTimeIntervalType.toInternal`, and an 
int, or a
 /// float truncated, as microseconds. Another value raises PyArrow's error.
diff --git a/crates/pyspark-rs/src/row.rs b/crates/pyspark-rs/src/row.rs
index 66b2956..669a073 100644
--- a/crates/pyspark-rs/src/row.rs
+++ b/crates/pyspark-rs/src/row.rs
@@ -211,6 +211,21 @@ pub fn value_to_py<'py>(py: Python<'py>, v: &Value) -> 
PyResult<Bound<'py, PyAny
                 .call_method1("fromtimestamp", (seconds,))?
                 .into_any()
         }
+        Value::Time(nanos) => {
+            // As PySpark's `TimeType.fromInternal`: a datetime.time of the 
microseconds.
+            let (hours, remainder) = (
+                nanos.div_euclid(3_600_000_000_000),
+                nanos.rem_euclid(3_600_000_000_000),
+            );
+            let (minutes, remainder) = (remainder / 60_000_000_000, remainder 
% 60_000_000_000);
+            let (seconds, remainder) = (remainder / 1_000_000_000, remainder % 
1_000_000_000);
+            py.import("datetime")?.getattr("time")?.call1((
+                hours,
+                minutes,
+                seconds,
+                remainder / 1_000,
+            ))?
+        }
         Value::DayTimeInterval(micros) => {
             // Normalize using integer arithmetic: neither floats nor an 
intermediate
             // nanosecond duration can preserve the full signed i64 
microsecond range.
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index 96a092a..baf3515 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -1668,6 +1668,8 @@ fn value_to_datatype(v: &Value) -> DataType {
         Value::Binary(_) => DataType::Binary,
         Value::Date(_) => DataType::Date,
         Value::Timestamp(_) => DataType::Timestamp,
+        // As PySpark infers TimeType() of a datetime.time.
+        Value::Time(_) => DataType::Time { precision: 6 },
         Value::DayTimeInterval(_) => DataType::DayTimeInterval {
             start_field: 0,
             end_field: 3,
@@ -1912,6 +1914,15 @@ pub(crate) fn py_to_value(obj: &Bound<'_, PyAny>) -> 
PyResult<Value> {
         }
     }
 
+    // datetime.time -> nanoseconds since midnight, as TimeType.toInternal.
+    if let Some(dt_mod) = &datetime_mod {
+        if let Ok(time_cls) = dt_mod.getattr("time") {
+            if obj.is_instance(&time_cls)? {
+                return Ok(Value::Time(crate::infer::time_nanos(obj)?));
+            }
+        }
+    }
+
     // datetime.timedelta -> a day-time interval, as 
DayTimeIntervalType.toInternal.
     if let Some(dt_mod) = &datetime_mod {
         if let Ok(timedelta_cls) = dt_mod.getattr("timedelta") {
diff --git a/crates/spark-connect/src/dataframe.rs 
b/crates/spark-connect/src/dataframe.rs
index 6eb2b14..1fd7d8e 100644
--- a/crates/spark-connect/src/dataframe.rs
+++ b/crates/spark-connect/src/dataframe.rs
@@ -2478,19 +2478,15 @@ impl<'a> ArrowValueLayout<'a> {
                 scale: Some(arr.scale() as i32),
             }),
             ArrowValueDecoder::FixedSizeBinary(arr) => 
Ok(Value::Binary(arr.value(index).to_vec())),
-            // TimeType has no dedicated Value variant, so render as an ISO 
time string.
-            ArrowValueDecoder::Time64Microsecond(arr) => {
-                Ok(Value::String(micros_to_time_string(arr.value(index))))
+            // A TIME, in nanoseconds since midnight, as PySpark's 
`TimeType.fromInternal` reads.
+            ArrowValueDecoder::Time64Microsecond(arr) => 
Ok(Value::Time(arr.value(index) * 1_000)),
+            ArrowValueDecoder::Time64Nanosecond(arr) => 
Ok(Value::Time(arr.value(index))),
+            ArrowValueDecoder::Time32Millisecond(arr) => {
+                Ok(Value::Time(arr.value(index) as i64 * 1_000_000))
+            }
+            ArrowValueDecoder::Time32Second(arr) => {
+                Ok(Value::Time(arr.value(index) as i64 * 1_000_000_000))
             }
-            ArrowValueDecoder::Time64Nanosecond(arr) => 
Ok(Value::String(micros_to_time_string(
-                arr.value(index) / 1_000,
-            ))),
-            ArrowValueDecoder::Time32Millisecond(arr) => 
Ok(Value::String(micros_to_time_string(
-                arr.value(index) as i64 * 1_000,
-            ))),
-            ArrowValueDecoder::Time32Second(arr) => 
Ok(Value::String(micros_to_time_string(
-                arr.value(index) as i64 * 1_000_000,
-            ))),
             // Intervals likewise have no dedicated Value variant, so render 
as strings.
             ArrowValueDecoder::IntervalYearMonth(arr) => {
                 let months = arr.value(index);
@@ -3128,18 +3124,6 @@ mod nested_decode_tests {
     }
 }
 
-/// Render microseconds-since-midnight as an ISO time string 
`HH:MM:SS[.ffffff]`.
-fn micros_to_time_string(micros: i64) -> String {
-    let total_secs = micros.div_euclid(1_000_000);
-    let us = micros.rem_euclid(1_000_000);
-    let (h, m, s) = (total_secs / 3600, (total_secs % 3600) / 60, total_secs % 
60);
-    if us == 0 {
-        format!("{h:02}:{m:02}:{s:02}")
-    } else {
-        format!("{h:02}:{m:02}:{s:02}.{us:06}")
-    }
-}
-
 #[cfg(test)]
 mod cache_tests {
     use super::*;
@@ -4187,11 +4171,11 @@ mod arrow_value_tests {
             ),
             (
                 Arc::new(Time64NanosecondArray::from(vec![1_234_567])),
-                Value::String("00:00:00.001234".into()),
+                Value::Time(1_234_567),
             ),
             (
                 Arc::new(Time32MillisecondArray::from(vec![1])),
-                Value::String("00:00:00.001000".into()),
+                Value::Time(1_000_000),
             ),
         ];
         for (array, expected) in cases {
@@ -4341,23 +4325,24 @@ mod arrow_value_tests {
     }
 
     #[test]
-    fn time_types_render_as_string() {
-        assert!(matches!(
+    fn time_types_decode_as_nanoseconds_since_midnight() {
+        // As PySpark's internal value of TimeType, whatever the unit of the 
Arrow time.
+        assert_eq!(
             arrow_value_at(&Time64MicrosecondArray::from(vec![1i64]), 
0).unwrap(),
-            Value::String(_)
-        ));
-        assert!(matches!(
-            arrow_value_at(&Time64NanosecondArray::from(vec![1000i64]), 
0).unwrap(),
-            Value::String(_)
-        ));
-        assert!(matches!(
+            Value::Time(1_000)
+        );
+        assert_eq!(
+            arrow_value_at(&Time64NanosecondArray::from(vec![1001i64]), 
0).unwrap(),
+            Value::Time(1_001)
+        );
+        assert_eq!(
             arrow_value_at(&Time32MillisecondArray::from(vec![1i32]), 
0).unwrap(),
-            Value::String(_)
-        ));
-        assert!(matches!(
+            Value::Time(1_000_000)
+        );
+        assert_eq!(
             arrow_value_at(&Time32SecondArray::from(vec![1i32]), 0).unwrap(),
-            Value::String(_)
-        ));
+            Value::Time(1_000_000_000)
+        );
     }
 
     #[test]
@@ -4444,12 +4429,6 @@ mod arrow_value_tests {
         assert_eq!(i128_to_decimal_string(5, 4), "0.0005");
         assert_eq!(i128_to_decimal_string(-5, 4), "-0.0005");
     }
-
-    #[test]
-    fn micros_to_time_string_branches() {
-        assert_eq!(micros_to_time_string(0), "00:00:00");
-        assert!(micros_to_time_string(1).contains('.'));
-    }
 }
 
 #[cfg(test)]
diff --git a/crates/spark-connect/src/row.rs b/crates/spark-connect/src/row.rs
index 38dab85..219f916 100644
--- a/crates/spark-connect/src/row.rs
+++ b/crates/spark-connect/src/row.rs
@@ -33,6 +33,8 @@ pub enum Value {
     Date(i32),
     /// Timestamp (microseconds since epoch)
     Timestamp(i64),
+    /// TIME, in nanoseconds since midnight, as PySpark's internal value of 
`TimeType`.
+    Time(i64),
     /// Spark day-time interval, in signed microseconds. The full i64 range is
     /// representable by Python's datetime.timedelta without nanosecond 
conversion.
     /// Display uses raw microseconds, like Date and Timestamp use their raw 
units.
@@ -72,6 +74,7 @@ impl fmt::Debug for Value {
             Self::Binary(v) => f.debug_tuple("Binary").field(v).finish(),
             Self::Date(v) => f.debug_tuple("Date").field(v).finish(),
             Self::Timestamp(v) => f.debug_tuple("Timestamp").field(v).finish(),
+            Self::Time(v) => f.debug_tuple("Time").field(v).finish(),
             Self::DayTimeInterval(v) => 
f.debug_tuple("DayTimeInterval").field(v).finish(),
             Self::Decimal {
                 value,
@@ -174,6 +177,7 @@ impl fmt::Display for Value {
             Value::Binary(b) => write!(f, "{:?}", b),
             Value::Date(d) => write!(f, "{}", d),
             Value::Timestamp(t) => write!(f, "{}", t),
+            Value::Time(t) => write!(f, "{}", t),
             Value::DayTimeInterval(t) => write!(f, "{}", t),
             Value::Decimal {
                 value,
diff --git a/crates/spark-connect/src/session.rs 
b/crates/spark-connect/src/session.rs
index 5727e19..3bee47d 100644
--- a/crates/spark-connect/src/session.rs
+++ b/crates/spark-connect/src/session.rs
@@ -964,7 +964,8 @@ fn datatype_to_arrow(dt: &DataType) -> 
Result<arrow::datatypes::DataType> {
             A::Timestamp(TimeUnit::Nanosecond, Some("UTC".into()))
         }
         DataType::TimestampNtzNanos { .. } => 
A::Timestamp(TimeUnit::Nanosecond, None),
-        DataType::Time { .. } => A::Time64(TimeUnit::Microsecond),
+        // As PySpark's `to_arrow_type` of TimeType.
+        DataType::Time { .. } => A::Time64(TimeUnit::Nanosecond),
         // As Spark's Arrow conversion of DayTimeIntervalType.
         DataType::DayTimeInterval { .. } => A::Duration(TimeUnit::Microsecond),
         DataType::Decimal { precision, scale } => A::Decimal128(*precision as 
u8, *scale as i8),
@@ -1109,6 +1110,8 @@ fn values_to_arrow(values: &[&Value], dt: &DataType) -> 
Result<Arc<dyn arrow::ar
             Ok(Arc::new(BinaryArray::from(vals?)))
         }
         DataType::Date => prim!(Date32Array, Value::Date),
+        // As PySpark's `to_arrow_type`: nanoseconds since midnight, as 
`TimeType.toInternal`.
+        DataType::Time { .. } => prim!(Time64NanosecondArray, Value::Time),
         DataType::DayTimeInterval { .. } => prim!(DurationMicrosecondArray, 
Value::DayTimeInterval),
         DataType::Timestamp | DataType::TimestampNtz => {
             let vals: Result<Vec<Option<i64>>> = values
@@ -2202,6 +2205,32 @@ mod tests {
         );
     }
 
+    #[test]
+    fn input_ipc_sends_times_as_nanoseconds_as_pyspark() {
+        // As PySpark's `to_arrow_type` of TimeType: time64[ns] of the 
nanoseconds since
+        // midnight, whatever the precision.
+        use arrow::array::Time64NanosecondArray;
+        use arrow::datatypes::{DataType as A, TimeUnit};
+        let schema = schema(vec![DataType::Time { precision: 3 }]);
+        let rows = vec![
+            row(vec![Value::Time(3_723_456_789_000)]),
+            row(vec![Value::Null]),
+        ];
+        let batch = read_batch(&rows, &schema);
+        assert_eq!(
+            batch.schema().field(0).data_type(),
+            &A::Time64(TimeUnit::Nanosecond)
+        );
+        assert_eq!(
+            batch
+                .column(0)
+                .as_any()
+                .downcast_ref::<Time64NanosecondArray>()
+                .unwrap(),
+            &Time64NanosecondArray::from(vec![Some(3_723_456_789_000), None])
+        );
+    }
+
     #[test]
     fn input_ipc_scales_nanos_timestamps_and_keeps_zone() {
         use arrow::array::{ListArray, TimestampNanosecondArray};
diff --git a/crates/spark-connect/src/types.rs 
b/crates/spark-connect/src/types.rs
index c357273..043b951 100644
--- a/crates/spark-connect/src/types.rs
+++ b/crates/spark-connect/src/types.rs
@@ -1198,6 +1198,8 @@ fn parse_json_type_string(s: &str) -> Result<DataType> {
         "date" => Ok(DataType::Date),
         "timestamp" => Ok(DataType::Timestamp),
         "timestamp_ntz" => Ok(DataType::TimestampNtz),
+        // A bare `time`, as `TimeType().typeName()` names it, is the default 
time(6).
+        "time" => Ok(DataType::Time { precision: 6 }),
         "interval" | "calendarinterval" => Ok(DataType::CalendarInterval),
         "variant" => Ok(DataType::Variant),
         // A bare `decimal`, as `DecimalType().typeName()` names it, is the 
default decimal(10,0).
@@ -1741,6 +1743,8 @@ fn parse_single_type(input: &str) -> Result<DataType> {
         "date" => Ok(DataType::Date),
         "timestamp" => Ok(DataType::Timestamp),
         "timestamp_ntz" => Ok(DataType::TimestampNtz),
+        // As Spark's `TIME`, of the default precision.
+        "time" => Ok(DataType::Time { precision: 6 }),
         "interval" => Ok(DataType::CalendarInterval),
         "variant" => Ok(DataType::Variant),
         _ => {
@@ -2574,11 +2578,19 @@ mod tests {
             DataType::Varchar { length: 100 }
         );
 
-        // Test time
+        // Test time; a bare time is of the default precision, as Spark's TIME.
         assert_eq!(
             DataType::from_ddl("time(6)").unwrap(),
             DataType::Time { precision: 6 }
         );
+        assert_eq!(
+            DataType::from_ddl("time").unwrap(),
+            DataType::Time { precision: 6 }
+        );
+        assert_eq!(
+            DataType::from_json_str("\"time\"").unwrap(),
+            DataType::Time { precision: 6 }
+        );
     }
 
     #[test]
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 64217f5..bdca686 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -634,6 +634,30 @@ def 
test_collect_reads_a_result_across_stream_ends(session):
     assert session.range(n).toArrow().num_rows == n
 
 
+def test_time_values_round_trip_as_pyspark(session):
+    # As PySpark's: TIME values are sent, and collected, as datetime.time, 
when the session
+    # enables the TIME type.
+    import datetime
+
+    t = datetime.time(1, 2, 3, 456789)
+    session.conf.set("spark.sql.timeType.enabled", "true")
+    try:
+        assert session.createDataFrame([(t,)]).collect() == [Row(_1=t)]
+        assert session.createDataFrame([(t,)], "t time").collect() == 
[Row(t=t)]
+        assert session.range(1).select(F.lit(t).alias("t")).collect() == 
[Row(t=t)]
+        assert session.sql("SELECT TIME'12:34:56.789' AS t").collect() == [
+            Row(t=datetime.time(12, 34, 56, 789000))
+        ]
+        # Without it, the server does not support the type, as for PySpark. It 
is set to
+        # false, as the server enables it by default when SPARK_TESTING is set.
+        session.conf.set("spark.sql.timeType.enabled", "false")
+        with pytest.raises(AnalysisException) as error:
+            session.createDataFrame([(t,)], "t time").collect()
+        assert error.value.getCondition() == "UNSUPPORTED_TIME_TYPE"
+    finally:
+        session.conf.unset("spark.sql.timeType.enabled")
+
+
 def test_timedelta_is_a_day_time_interval_as_pyspark(session):
     import datetime
 
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 4e12e66..16798f0 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -930,6 +930,58 @@ def test_udtfs_are_created_as_pyspark(mock_connect):
     assert service.read_confs == [arrow_conf, arrow_conf]
 
 
+def test_time_values_are_sent_as_pyspark(mock_connect):
+    # As PySpark: a datetime.time is of TimeType(), and createDataFrame sends 
it as PyArrow's
+    # time64[ns] of the value, which takes the nanoseconds of an int too, and 
lit() of it is
+    # a time literal of its nanoseconds since midnight.
+    import datetime
+
+    import pyarrow as pa
+    from pyspark.sql import functions as F
+
+    spark, service = mock_connect
+    t = datetime.time(1, 2, 3, 456789)
+    nanos = 3_723_456_789_000
+
+    def local_relation(df):
+        df.toArrow()
+        request = parse_message(service.execute_requests[-1])
+        relation = parse_message(parse_message(request[3][0])[1][0])
+        while 18 in relation:  # Relation.to_df -> ToDF.input.
+            relation = parse_message(parse_message(relation[18][0])[1][0])
+        # Relation.local_relation=11 -> LocalRelation.data=1, schema=2.
+        local = parse_message(relation[11][0])
+        table = pa.ipc.open_stream(local[1][0]).read_all()
+        return table, [field["type"] for field in 
json.loads(local[2][0])["fields"]]
+
+    for schema in (None, "t time"):
+        table, types = local_relation(spark.createDataFrame([(t,), (None,)], 
schema))
+        assert types == ["time(6)"]
+        assert table.schema.field(0).type == pa.time64("ns")
+        assert table.column(0).cast(pa.int64()).to_pylist() == [nanos, None]
+    table, _ = local_relation(spark.createDataFrame([(5,)], "t time"))
+    assert table.column(0).cast(pa.int64()).to_pylist() == [5]
+    with pytest.raises(TypeError, match="cannot be converted to int"):
+        spark.createDataFrame([("01:02:03",)], "t time")
+
+    literal = F.lit(t)
+    assert repr(literal) == "Column<'01:02:03.456789'>"
+    spark.range(1).select(literal).collect()
+    # ... Project.expressions=3; Expression.literal=1, Literal.time=26 
(nano=1, precision=2).
+    plan = 
parse_message(parse_message(service.execute_requests[-1])[3][0])[1][0]
+    expression = parse_message(parse_message(parse_message(plan)[3][0])[3][0])
+    assert parse_message(parse_message(expression[1][0])[26][0]) == {1: 
[nanos], 2: [6]}
+
+
+def test_time_types_parse_as_pyspark():
+    # As PySpark's: a bare time is TimeType(), of the default precision 6, in 
a DDL string and
+    # in the JSON of a type.
+    assert T.DataType.fromDDL("t time") == T.StructType([T.StructField("t", 
T.TimeType(6))])
+    assert T.DataType.fromDDL("t time(3)") == T.StructType([T.StructField("t", 
T.TimeType(3))])
+    assert T._parse_datatype_json_string('"time"') == T.TimeType(6)
+    assert T._parse_datatype_json_string('"time(0)"') == T.TimeType(0)
+
+
 def 
test_decimal_literal_is_of_the_default_decimal_type_as_pyspark(mock_connect):
     # As PySpark's LiteralExpression of a Decimal, of DecimalType(): precision 
10 and scale
     # 0, which the server widens to those of the value, wherever the literal 
is.


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to