This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 774f939 [SPARK-60947][CONNECT][PYTHON] Send and collect TIME values,
and parse the TIME type of DDL strings, as PySpark does
774f939 is described below
commit 774f939fce453df1d0fbe1872bdb191edff24f96
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Fri Nov 6 23:24:14 2026 +0900
[SPARK-60947][CONNECT][PYTHON] Send and collect TIME values, and parse the
TIME type of DDL strings, as PySpark does
### What changes were proposed in this pull request?
[SPARK-60947](https://issues.apache.org/jira/browse/SPARK-60947): the
Python client now sends and collects TIME values, and parses the TIME type of
DDL strings, as PySpark Connect does.
- `createDataFrame` infers `TimeType()` of a `datetime.time`, and sends a
TIME value as PyArrow's `time64[ns]` of it, as `to_arrow_type` declares. That
type also takes the nanoseconds of an int.
- `lit()` of a `datetime.time` is a time literal of its nanoseconds since
midnight.
- Collected TIME values are `datetime.time`, as `TimeType.fromInternal`
reads them.
- A bare `time`, in a DDL string or in the JSON of a type, is `TimeType(6)`.
So, as for PySpark, the server raises `UNSUPPORTED_TIME_TYPE` unless the
session enables `spark.sql.timeType.enabled`, and with it TIME values
round-trip. The core values have `Time`, in nanoseconds since midnight, for
these.
### Why are the changes needed?
TIME values were unsupported, and differed from PySpark 4.3.0:
- `createDataFrame([(datetime.time(1, 2, 3),)])` raised `TypeError:
unsupported value type for createDataFrame: time`, rather than the server's
`UNSUPPORTED_TIME_TYPE`.
- `createDataFrame(..., "t time")` raised `ValueError: invalid type string
't time'`.
- `lit(datetime.time(...))` raised `UNSUPPORTED_DATA_TYPE`.
- Collected TIME values, such as of `SELECT TIME'12:34:56.789'`, were
strings.
### Does this PR introduce _any_ user-facing change?
Yes. TIME values work as in PySpark 4.3.0.
### How was this patch tested?
- A new live test round-trips TIME values of `createDataFrame`, inferred
and of a DDL schema, of `lit()` and of SQL, with the TIME type enabled. It also
checks `UNSUPPORTED_TIME_TYPE` without the TIME type. It passes with official
PySpark 4.3.0, and fails without this change.
- A new offline test checks, against a mock Connect server, the
`time64[ns]` values and the `time(6)` schema that `createDataFrame` sends, and
the time literal of `lit()`. Another checks the parsing of `time` and
`time(p)`. They fail without this change.
- New lib tests check the decoding of Arrow times, the encoding of TIME
values, and the parsing of `time`.
- The lib tests and every integration test, e2e included, against a local
Spark 4.3.0 server (664 passed, and 125 of the transport crate), the build of
all tests with `wasm-udf`, the offline Python suites (1330 passed, 8 skipped),
and the live drop-in and Arrow suites (88 passed) and live cases (10 passed).
Clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #214 from HyukjinKwon/SPARK-60947-time.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/functions.rs | 8 ++++
crates/pyspark-rs/src/infer.rs | 26 +++++++++++++
crates/pyspark-rs/src/row.rs | 15 ++++++++
crates/pyspark-rs/src/session.rs | 11 ++++++
crates/spark-connect/src/dataframe.rs | 71 ++++++++++++-----------------------
crates/spark-connect/src/row.rs | 4 ++
crates/spark-connect/src/session.rs | 31 ++++++++++++++-
crates/spark-connect/src/types.rs | 14 ++++++-
python/tests/test_dropin_e2e.py | 24 ++++++++++++
python/tests/test_dropin_offline.py | 52 +++++++++++++++++++++++++
10 files changed, 208 insertions(+), 48 deletions(-)
diff --git a/crates/pyspark-rs/src/functions.rs
b/crates/pyspark-rs/src/functions.rs
index 32f2bc8..2952a07 100644
--- a/crates/pyspark-rs/src/functions.rs
+++ b/crates/pyspark-rs/src/functions.rs
@@ -305,6 +305,14 @@ fn scalar_literal(obj: &Bound<'_, PyAny>) ->
PyResult<Option<LiteralExpression>>
let micros = crate::session::datetime_to_micros(obj)?;
return Ok(Some(LiteralExpression::Timestamp(micros)));
}
+ // datetime.time -> TimeType(), as PySpark's literal of one: its
nanoseconds since
+ // midnight, of the default precision.
+ if obj.is_instance(&dt_mod.getattr("time")?)? {
+ return Ok(Some(LiteralExpression::Time {
+ nano: crate::infer::time_nanos(obj)?,
+ precision: 6,
+ }));
+ }
// datetime.timedelta -> DayTimeIntervalType, as PySpark's literal of
one.
if obj.is_instance(&dt_mod.getattr("timedelta")?)? {
let micros = crate::infer::timedelta_micros(obj)?;
diff --git a/crates/pyspark-rs/src/infer.rs b/crates/pyspark-rs/src/infer.rs
index 0119d54..7b3d0fe 100644
--- a/crates/pyspark-rs/src/infer.rs
+++ b/crates/pyspark-rs/src/infer.rs
@@ -1051,10 +1051,36 @@ pub(crate) fn convert(obj: &Bound<'_, PyAny>, dt:
&DataType, nullable: bool) ->
let ordinal: i64 = obj.call_method0(intern!(py,
"toordinal"))?.extract()?;
Ok(Value::Date((ordinal - 719_163) as i32))
}
+ // As PyArrow's `time64[ns]`, to which PySpark passes the value as it
is: a time, and
+ // the nanoseconds of an int, or of a float, truncated.
+ DataType::Time { .. } => {
+ if obj.is_instance(TIME.import(py, "datetime", "time")?.as_any())?
{
+ Ok(Value::Time(time_nanos(obj)?))
+ } else if obj.is_instance_of::<PyInt>() {
+ Ok(Value::Time(obj.extract()?))
+ } else if obj.is_instance_of::<PyFloat>() {
+ Ok(Value::Time(obj.extract::<f64>()? as i64))
+ } else {
+ Err(PyTypeError::new_err(format!(
+ "object of type {} cannot be converted to int",
+ obj.get_type().repr()?
+ )))
+ }
+ }
_ => Err(unsupported(obj)?),
}
}
+/// The nanoseconds since midnight of a `datetime.time`, as PySpark's
`TimeType.toInternal`.
+pub(crate) fn time_nanos(obj: &Bound<'_, PyAny>) -> PyResult<i64> {
+ let py = obj.py();
+ let field = |name: &Bound<'_, PyString>| -> PyResult<i64> {
obj.getattr(name)?.extract() };
+ let seconds = field(intern!(py, "hour"))? * 3600
+ + field(intern!(py, "minute"))? * 60
+ + field(intern!(py, "second"))?;
+ Ok(seconds * 1_000_000_000 + field(intern!(py, "microsecond"))? * 1_000)
+}
+
/// A value of a day-time interval column, as PyArrow converts one to
`duration[us]`: a
/// timedelta to its microseconds, as `DayTimeIntervalType.toInternal`, and an
int, or a
/// float truncated, as microseconds. Another value raises PyArrow's error.
diff --git a/crates/pyspark-rs/src/row.rs b/crates/pyspark-rs/src/row.rs
index 66b2956..669a073 100644
--- a/crates/pyspark-rs/src/row.rs
+++ b/crates/pyspark-rs/src/row.rs
@@ -211,6 +211,21 @@ pub fn value_to_py<'py>(py: Python<'py>, v: &Value) ->
PyResult<Bound<'py, PyAny
.call_method1("fromtimestamp", (seconds,))?
.into_any()
}
+ Value::Time(nanos) => {
+ // As PySpark's `TimeType.fromInternal`: a datetime.time of the
microseconds.
+ let (hours, remainder) = (
+ nanos.div_euclid(3_600_000_000_000),
+ nanos.rem_euclid(3_600_000_000_000),
+ );
+ let (minutes, remainder) = (remainder / 60_000_000_000, remainder
% 60_000_000_000);
+ let (seconds, remainder) = (remainder / 1_000_000_000, remainder %
1_000_000_000);
+ py.import("datetime")?.getattr("time")?.call1((
+ hours,
+ minutes,
+ seconds,
+ remainder / 1_000,
+ ))?
+ }
Value::DayTimeInterval(micros) => {
// Normalize using integer arithmetic: neither floats nor an
intermediate
// nanosecond duration can preserve the full signed i64
microsecond range.
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index 96a092a..baf3515 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -1668,6 +1668,8 @@ fn value_to_datatype(v: &Value) -> DataType {
Value::Binary(_) => DataType::Binary,
Value::Date(_) => DataType::Date,
Value::Timestamp(_) => DataType::Timestamp,
+ // As PySpark infers TimeType() of a datetime.time.
+ Value::Time(_) => DataType::Time { precision: 6 },
Value::DayTimeInterval(_) => DataType::DayTimeInterval {
start_field: 0,
end_field: 3,
@@ -1912,6 +1914,15 @@ pub(crate) fn py_to_value(obj: &Bound<'_, PyAny>) ->
PyResult<Value> {
}
}
+ // datetime.time -> nanoseconds since midnight, as TimeType.toInternal.
+ if let Some(dt_mod) = &datetime_mod {
+ if let Ok(time_cls) = dt_mod.getattr("time") {
+ if obj.is_instance(&time_cls)? {
+ return Ok(Value::Time(crate::infer::time_nanos(obj)?));
+ }
+ }
+ }
+
// datetime.timedelta -> a day-time interval, as
DayTimeIntervalType.toInternal.
if let Some(dt_mod) = &datetime_mod {
if let Ok(timedelta_cls) = dt_mod.getattr("timedelta") {
diff --git a/crates/spark-connect/src/dataframe.rs
b/crates/spark-connect/src/dataframe.rs
index 6eb2b14..1fd7d8e 100644
--- a/crates/spark-connect/src/dataframe.rs
+++ b/crates/spark-connect/src/dataframe.rs
@@ -2478,19 +2478,15 @@ impl<'a> ArrowValueLayout<'a> {
scale: Some(arr.scale() as i32),
}),
ArrowValueDecoder::FixedSizeBinary(arr) =>
Ok(Value::Binary(arr.value(index).to_vec())),
- // TimeType has no dedicated Value variant, so render as an ISO
time string.
- ArrowValueDecoder::Time64Microsecond(arr) => {
- Ok(Value::String(micros_to_time_string(arr.value(index))))
+ // A TIME, in nanoseconds since midnight, as PySpark's
`TimeType.fromInternal` reads.
+ ArrowValueDecoder::Time64Microsecond(arr) =>
Ok(Value::Time(arr.value(index) * 1_000)),
+ ArrowValueDecoder::Time64Nanosecond(arr) =>
Ok(Value::Time(arr.value(index))),
+ ArrowValueDecoder::Time32Millisecond(arr) => {
+ Ok(Value::Time(arr.value(index) as i64 * 1_000_000))
+ }
+ ArrowValueDecoder::Time32Second(arr) => {
+ Ok(Value::Time(arr.value(index) as i64 * 1_000_000_000))
}
- ArrowValueDecoder::Time64Nanosecond(arr) =>
Ok(Value::String(micros_to_time_string(
- arr.value(index) / 1_000,
- ))),
- ArrowValueDecoder::Time32Millisecond(arr) =>
Ok(Value::String(micros_to_time_string(
- arr.value(index) as i64 * 1_000,
- ))),
- ArrowValueDecoder::Time32Second(arr) =>
Ok(Value::String(micros_to_time_string(
- arr.value(index) as i64 * 1_000_000,
- ))),
// Intervals likewise have no dedicated Value variant, so render
as strings.
ArrowValueDecoder::IntervalYearMonth(arr) => {
let months = arr.value(index);
@@ -3128,18 +3124,6 @@ mod nested_decode_tests {
}
}
-/// Render microseconds-since-midnight as an ISO time string
`HH:MM:SS[.ffffff]`.
-fn micros_to_time_string(micros: i64) -> String {
- let total_secs = micros.div_euclid(1_000_000);
- let us = micros.rem_euclid(1_000_000);
- let (h, m, s) = (total_secs / 3600, (total_secs % 3600) / 60, total_secs %
60);
- if us == 0 {
- format!("{h:02}:{m:02}:{s:02}")
- } else {
- format!("{h:02}:{m:02}:{s:02}.{us:06}")
- }
-}
-
#[cfg(test)]
mod cache_tests {
use super::*;
@@ -4187,11 +4171,11 @@ mod arrow_value_tests {
),
(
Arc::new(Time64NanosecondArray::from(vec![1_234_567])),
- Value::String("00:00:00.001234".into()),
+ Value::Time(1_234_567),
),
(
Arc::new(Time32MillisecondArray::from(vec![1])),
- Value::String("00:00:00.001000".into()),
+ Value::Time(1_000_000),
),
];
for (array, expected) in cases {
@@ -4341,23 +4325,24 @@ mod arrow_value_tests {
}
#[test]
- fn time_types_render_as_string() {
- assert!(matches!(
+ fn time_types_decode_as_nanoseconds_since_midnight() {
+ // As PySpark's internal value of TimeType, whatever the unit of the
Arrow time.
+ assert_eq!(
arrow_value_at(&Time64MicrosecondArray::from(vec![1i64]),
0).unwrap(),
- Value::String(_)
- ));
- assert!(matches!(
- arrow_value_at(&Time64NanosecondArray::from(vec![1000i64]),
0).unwrap(),
- Value::String(_)
- ));
- assert!(matches!(
+ Value::Time(1_000)
+ );
+ assert_eq!(
+ arrow_value_at(&Time64NanosecondArray::from(vec![1001i64]),
0).unwrap(),
+ Value::Time(1_001)
+ );
+ assert_eq!(
arrow_value_at(&Time32MillisecondArray::from(vec![1i32]),
0).unwrap(),
- Value::String(_)
- ));
- assert!(matches!(
+ Value::Time(1_000_000)
+ );
+ assert_eq!(
arrow_value_at(&Time32SecondArray::from(vec![1i32]), 0).unwrap(),
- Value::String(_)
- ));
+ Value::Time(1_000_000_000)
+ );
}
#[test]
@@ -4444,12 +4429,6 @@ mod arrow_value_tests {
assert_eq!(i128_to_decimal_string(5, 4), "0.0005");
assert_eq!(i128_to_decimal_string(-5, 4), "-0.0005");
}
-
- #[test]
- fn micros_to_time_string_branches() {
- assert_eq!(micros_to_time_string(0), "00:00:00");
- assert!(micros_to_time_string(1).contains('.'));
- }
}
#[cfg(test)]
diff --git a/crates/spark-connect/src/row.rs b/crates/spark-connect/src/row.rs
index 38dab85..219f916 100644
--- a/crates/spark-connect/src/row.rs
+++ b/crates/spark-connect/src/row.rs
@@ -33,6 +33,8 @@ pub enum Value {
Date(i32),
/// Timestamp (microseconds since epoch)
Timestamp(i64),
+ /// TIME, in nanoseconds since midnight, as PySpark's internal value of
`TimeType`.
+ Time(i64),
/// Spark day-time interval, in signed microseconds. The full i64 range is
/// representable by Python's datetime.timedelta without nanosecond
conversion.
/// Display uses raw microseconds, like Date and Timestamp use their raw
units.
@@ -72,6 +74,7 @@ impl fmt::Debug for Value {
Self::Binary(v) => f.debug_tuple("Binary").field(v).finish(),
Self::Date(v) => f.debug_tuple("Date").field(v).finish(),
Self::Timestamp(v) => f.debug_tuple("Timestamp").field(v).finish(),
+ Self::Time(v) => f.debug_tuple("Time").field(v).finish(),
Self::DayTimeInterval(v) =>
f.debug_tuple("DayTimeInterval").field(v).finish(),
Self::Decimal {
value,
@@ -174,6 +177,7 @@ impl fmt::Display for Value {
Value::Binary(b) => write!(f, "{:?}", b),
Value::Date(d) => write!(f, "{}", d),
Value::Timestamp(t) => write!(f, "{}", t),
+ Value::Time(t) => write!(f, "{}", t),
Value::DayTimeInterval(t) => write!(f, "{}", t),
Value::Decimal {
value,
diff --git a/crates/spark-connect/src/session.rs
b/crates/spark-connect/src/session.rs
index 5727e19..3bee47d 100644
--- a/crates/spark-connect/src/session.rs
+++ b/crates/spark-connect/src/session.rs
@@ -964,7 +964,8 @@ fn datatype_to_arrow(dt: &DataType) ->
Result<arrow::datatypes::DataType> {
A::Timestamp(TimeUnit::Nanosecond, Some("UTC".into()))
}
DataType::TimestampNtzNanos { .. } =>
A::Timestamp(TimeUnit::Nanosecond, None),
- DataType::Time { .. } => A::Time64(TimeUnit::Microsecond),
+ // As PySpark's `to_arrow_type` of TimeType.
+ DataType::Time { .. } => A::Time64(TimeUnit::Nanosecond),
// As Spark's Arrow conversion of DayTimeIntervalType.
DataType::DayTimeInterval { .. } => A::Duration(TimeUnit::Microsecond),
DataType::Decimal { precision, scale } => A::Decimal128(*precision as
u8, *scale as i8),
@@ -1109,6 +1110,8 @@ fn values_to_arrow(values: &[&Value], dt: &DataType) ->
Result<Arc<dyn arrow::ar
Ok(Arc::new(BinaryArray::from(vals?)))
}
DataType::Date => prim!(Date32Array, Value::Date),
+ // As PySpark's `to_arrow_type`: nanoseconds since midnight, as
`TimeType.toInternal`.
+ DataType::Time { .. } => prim!(Time64NanosecondArray, Value::Time),
DataType::DayTimeInterval { .. } => prim!(DurationMicrosecondArray,
Value::DayTimeInterval),
DataType::Timestamp | DataType::TimestampNtz => {
let vals: Result<Vec<Option<i64>>> = values
@@ -2202,6 +2205,32 @@ mod tests {
);
}
+ #[test]
+ fn input_ipc_sends_times_as_nanoseconds_as_pyspark() {
+ // As PySpark's `to_arrow_type` of TimeType: time64[ns] of the
nanoseconds since
+ // midnight, whatever the precision.
+ use arrow::array::Time64NanosecondArray;
+ use arrow::datatypes::{DataType as A, TimeUnit};
+ let schema = schema(vec![DataType::Time { precision: 3 }]);
+ let rows = vec![
+ row(vec![Value::Time(3_723_456_789_000)]),
+ row(vec![Value::Null]),
+ ];
+ let batch = read_batch(&rows, &schema);
+ assert_eq!(
+ batch.schema().field(0).data_type(),
+ &A::Time64(TimeUnit::Nanosecond)
+ );
+ assert_eq!(
+ batch
+ .column(0)
+ .as_any()
+ .downcast_ref::<Time64NanosecondArray>()
+ .unwrap(),
+ &Time64NanosecondArray::from(vec![Some(3_723_456_789_000), None])
+ );
+ }
+
#[test]
fn input_ipc_scales_nanos_timestamps_and_keeps_zone() {
use arrow::array::{ListArray, TimestampNanosecondArray};
diff --git a/crates/spark-connect/src/types.rs
b/crates/spark-connect/src/types.rs
index c357273..043b951 100644
--- a/crates/spark-connect/src/types.rs
+++ b/crates/spark-connect/src/types.rs
@@ -1198,6 +1198,8 @@ fn parse_json_type_string(s: &str) -> Result<DataType> {
"date" => Ok(DataType::Date),
"timestamp" => Ok(DataType::Timestamp),
"timestamp_ntz" => Ok(DataType::TimestampNtz),
+ // A bare `time`, as `TimeType().typeName()` names it, is the default
time(6).
+ "time" => Ok(DataType::Time { precision: 6 }),
"interval" | "calendarinterval" => Ok(DataType::CalendarInterval),
"variant" => Ok(DataType::Variant),
// A bare `decimal`, as `DecimalType().typeName()` names it, is the
default decimal(10,0).
@@ -1741,6 +1743,8 @@ fn parse_single_type(input: &str) -> Result<DataType> {
"date" => Ok(DataType::Date),
"timestamp" => Ok(DataType::Timestamp),
"timestamp_ntz" => Ok(DataType::TimestampNtz),
+ // As Spark's `TIME`, of the default precision.
+ "time" => Ok(DataType::Time { precision: 6 }),
"interval" => Ok(DataType::CalendarInterval),
"variant" => Ok(DataType::Variant),
_ => {
@@ -2574,11 +2578,19 @@ mod tests {
DataType::Varchar { length: 100 }
);
- // Test time
+ // Test time; a bare time is of the default precision, as Spark's TIME.
assert_eq!(
DataType::from_ddl("time(6)").unwrap(),
DataType::Time { precision: 6 }
);
+ assert_eq!(
+ DataType::from_ddl("time").unwrap(),
+ DataType::Time { precision: 6 }
+ );
+ assert_eq!(
+ DataType::from_json_str("\"time\"").unwrap(),
+ DataType::Time { precision: 6 }
+ );
}
#[test]
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 64217f5..bdca686 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -634,6 +634,30 @@ def
test_collect_reads_a_result_across_stream_ends(session):
assert session.range(n).toArrow().num_rows == n
+def test_time_values_round_trip_as_pyspark(session):
+ # As PySpark's: TIME values are sent, and collected, as datetime.time,
when the session
+ # enables the TIME type.
+ import datetime
+
+ t = datetime.time(1, 2, 3, 456789)
+ session.conf.set("spark.sql.timeType.enabled", "true")
+ try:
+ assert session.createDataFrame([(t,)]).collect() == [Row(_1=t)]
+ assert session.createDataFrame([(t,)], "t time").collect() ==
[Row(t=t)]
+ assert session.range(1).select(F.lit(t).alias("t")).collect() ==
[Row(t=t)]
+ assert session.sql("SELECT TIME'12:34:56.789' AS t").collect() == [
+ Row(t=datetime.time(12, 34, 56, 789000))
+ ]
+ # Without it, the server does not support the type, as for PySpark. It
is set to
+ # false, as the server enables it by default when SPARK_TESTING is set.
+ session.conf.set("spark.sql.timeType.enabled", "false")
+ with pytest.raises(AnalysisException) as error:
+ session.createDataFrame([(t,)], "t time").collect()
+ assert error.value.getCondition() == "UNSUPPORTED_TIME_TYPE"
+ finally:
+ session.conf.unset("spark.sql.timeType.enabled")
+
+
def test_timedelta_is_a_day_time_interval_as_pyspark(session):
import datetime
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 4e12e66..16798f0 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -930,6 +930,58 @@ def test_udtfs_are_created_as_pyspark(mock_connect):
assert service.read_confs == [arrow_conf, arrow_conf]
+def test_time_values_are_sent_as_pyspark(mock_connect):
+ # As PySpark: a datetime.time is of TimeType(), and createDataFrame sends
it as PyArrow's
+ # time64[ns] of the value, which takes the nanoseconds of an int too, and
lit() of it is
+ # a time literal of its nanoseconds since midnight.
+ import datetime
+
+ import pyarrow as pa
+ from pyspark.sql import functions as F
+
+ spark, service = mock_connect
+ t = datetime.time(1, 2, 3, 456789)
+ nanos = 3_723_456_789_000
+
+ def local_relation(df):
+ df.toArrow()
+ request = parse_message(service.execute_requests[-1])
+ relation = parse_message(parse_message(request[3][0])[1][0])
+ while 18 in relation: # Relation.to_df -> ToDF.input.
+ relation = parse_message(parse_message(relation[18][0])[1][0])
+ # Relation.local_relation=11 -> LocalRelation.data=1, schema=2.
+ local = parse_message(relation[11][0])
+ table = pa.ipc.open_stream(local[1][0]).read_all()
+ return table, [field["type"] for field in
json.loads(local[2][0])["fields"]]
+
+ for schema in (None, "t time"):
+ table, types = local_relation(spark.createDataFrame([(t,), (None,)],
schema))
+ assert types == ["time(6)"]
+ assert table.schema.field(0).type == pa.time64("ns")
+ assert table.column(0).cast(pa.int64()).to_pylist() == [nanos, None]
+ table, _ = local_relation(spark.createDataFrame([(5,)], "t time"))
+ assert table.column(0).cast(pa.int64()).to_pylist() == [5]
+ with pytest.raises(TypeError, match="cannot be converted to int"):
+ spark.createDataFrame([("01:02:03",)], "t time")
+
+ literal = F.lit(t)
+ assert repr(literal) == "Column<'01:02:03.456789'>"
+ spark.range(1).select(literal).collect()
+ # ... Project.expressions=3; Expression.literal=1, Literal.time=26
(nano=1, precision=2).
+ plan =
parse_message(parse_message(service.execute_requests[-1])[3][0])[1][0]
+ expression = parse_message(parse_message(parse_message(plan)[3][0])[3][0])
+ assert parse_message(parse_message(expression[1][0])[26][0]) == {1:
[nanos], 2: [6]}
+
+
+def test_time_types_parse_as_pyspark():
+ # As PySpark's: a bare time is TimeType(), of the default precision 6, in
a DDL string and
+ # in the JSON of a type.
+ assert T.DataType.fromDDL("t time") == T.StructType([T.StructField("t",
T.TimeType(6))])
+ assert T.DataType.fromDDL("t time(3)") == T.StructType([T.StructField("t",
T.TimeType(3))])
+ assert T._parse_datatype_json_string('"time"') == T.TimeType(6)
+ assert T._parse_datatype_json_string('"time(0)"') == T.TimeType(0)
+
+
def
test_decimal_literal_is_of_the_default_decimal_type_as_pyspark(mock_connect):
# As PySpark's LiteralExpression of a Decimal, of DecimalType(): precision
10 and scale
# 0, which the server widens to those of the value, wherever the literal
is.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]