This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new f70b87b9 [SPARK-61134][CONNECT][PYTHON] Make DayTimeIntervalType and 
YearMonthIntervalType check their fields, and the types without parameters 
their arguments, in the Python client as PySpark does
f70b87b9 is described below

commit f70b87b9ea8e9ef04ae6f6f036ca129ac357a110
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 12:42:48 2026 +0900

    [SPARK-61134][CONNECT][PYTHON] Make DayTimeIntervalType and 
YearMonthIntervalType check their fields, and the types without parameters 
their arguments, in the Python client as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61134](https://issues.apache.org/jira/browse/SPARK-61134): The 
constructors of the client's types now check their arguments as PySpark's do.
    
    - **`DayTimeIntervalType(startField=None, endField=None)` and 
`YearMonthIntervalType(...)`** take the fields of their types, as PySpark's do:
      - without any field, those of the whole type (`DAY` to `SECOND`, `YEAR` 
to `MONTH`)
      - without an end field, the start field
    
      For a field that is not one of the type, they raise `PySparkRuntimeError` 
`INVALID_INTERVAL_CASTING`, of the `str` of the start and end fields: 
`DayTimeIntervalType(endField=SECOND)` raises with `start_field` "None", and 
`DayTimeIntervalType(123)` with "123". As with PySpark's membership check, a 
float equal to a field, such as `1.0`, is that field.
    - **The types without parameters**, `NullType` through 
`CalendarIntervalType`, are PySpark's singletons of `DataTypeSingleton`. For an 
argument they now raise its `TypeError`, as Python words it: 
`DataTypeSingleton.__call__() got an unexpected keyword argument 'x'` for the 
first keyword, or else `DataTypeSingleton.__call__() takes 1 positional 
argument but 2 were given`.
    
    ### Why are the changes needed?
    
    - The client took any integer as a field, and filled a missing start field: 
`DayTimeIntervalType(endField=3)` was `DayTimeIntervalType(0, 3)`, and 
`DayTimeIntervalType(123)` an interval of no such field.
    - The types without parameters raised the `TypeError` of their own 
`__new__`.
    
    Apache Spark's `test_daytime_interval_type_constructor`, 
`test_yearmonth_interval_type_constructor` and 
`test_calendar_interval_type_constructor` failed on this client.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. Interval types of invalid fields raise `INVALID_INTERVAL_CASTING`, and 
calls with arguments of the types without parameters raise PySpark's 
`TypeError`, as with PySpark 4.3.0.
    
    ### How was this patch tested?
    
    - A new offline test checks:
      - the fields of the interval types and their `INVALID_INTERVAL_CASTING` 
errors, with their parameters
      - the `TypeError`s of the types without parameters, which stay singletons
    
      It passes with official PySpark 4.3.0, and fails without this change.
    - Apache Spark's interval tests of `test_parity_types.py` run on this 
client against a local server: 7 pass and 2 are skipped, including the three 
tests above.
    - The lib tests of both crates (590 passed), the build of all tests, also 
with `wasm-udf`, the offline Python suites (2202 passed, 8 skipped), and the 
live drop-in and Arrow suites (184 passed) and live cases (10 passed) against a 
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #329 from HyukjinKwon/SPARK-61134-type-constructors.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/types.rs      | 274 +++++++++++++++++++++++++++++-------
 python/tests/test_dropin_offline.py |  39 +++++
 2 files changed, 264 insertions(+), 49 deletions(-)

diff --git a/crates/pyspark-rs/src/types.rs b/crates/pyspark-rs/src/types.rs
index 6ed29d0f..6c58da2e 100644
--- a/crates/pyspark-rs/src/types.rs
+++ b/crates/pyspark-rs/src/types.rs
@@ -67,6 +67,80 @@ pub struct PyDataType {
     pub(crate) inner: DataType,
 }
 
+/// What PySpark's `DataTypeSingleton.__call__(cls)` raises, as Python does, 
for the arguments
+/// of a call of a type without parameters: a `TypeError` of the first keyword 
argument, or else
+/// of the positional ones.
+fn singleton_arguments(
+    args: &Bound<'_, pyo3::types::PyTuple>,
+    kwargs: Option<&Bound<'_, pyo3::types::PyDict>>,
+) -> PyResult<()> {
+    if let Some((key, _)) = kwargs.and_then(|kwargs| kwargs.iter().next()) {
+        return Err(pyo3::exceptions::PyTypeError::new_err(format!(
+            "DataTypeSingleton.__call__() got an unexpected keyword argument 
'{}'",
+            key.str()?
+        )));
+    }
+    if !args.is_empty() {
+        return Err(pyo3::exceptions::PyTypeError::new_err(format!(
+            "DataTypeSingleton.__call__() takes 1 positional argument but {} 
were given",
+            args.len() + 1
+        )));
+    }
+    Ok(())
+}
+
+/// The fields of an interval type, as PySpark's `DayTimeIntervalType` and
+/// `YearMonthIntervalType` take them: both of `default` without any, the 
start field for the
+/// end without one, and a `PySparkRuntimeError` of `INVALID_INTERVAL_CASTING` 
for a field that
+/// is not one of the type, of the `str` of each.
+fn interval_fields(
+    py: Python<'_>,
+    start: Option<&Bound<'_, PyAny>>,
+    end: Option<&Bound<'_, PyAny>>,
+    default: (i32, i32),
+    last: i32,
+) -> PyResult<(i32, i32)> {
+    let field = |value: Option<&Bound<'_, PyAny>>| -> Option<i32> {
+        let value = value?;
+        if value.is_instance_of::<pyo3::types::PyFloat>() {
+            // `1.0 in {0, 1}` of PySpark, of a float that equals a field.
+            let float: f64 = value.extract().ok()?;
+            return (float.fract() == 0.0 && 
(0.0..=f64::from(last)).contains(&float))
+                .then_some(float as i32);
+        }
+        value
+            .extract::<i64>()
+            .ok()
+            .filter(|field| (0..=i64::from(last)).contains(field))
+            .map(|field| field as i32)
+    };
+    let (start, end) = match (start, end) {
+        (None, None) => return Ok(default),
+        (Some(start), None) => (Some(start), Some(start)),
+        (start, end) => (start, end),
+    };
+    match (field(start), field(end)) {
+        (Some(start), Some(end)) => Ok((start, end)),
+        _ => {
+            let text = |value: Option<&Bound<'_, PyAny>>| -> PyResult<String> {
+                Ok(match value {
+                    Some(value) => value.str()?.to_string(),
+                    None => "None".to_string(),
+                })
+            };
+            Err(crate::column::raise_pyspark(
+                py,
+                "PySparkRuntimeError",
+                "INVALID_INTERVAL_CASTING",
+                &[
+                    ("start_field", text(start)?.as_str()),
+                    ("end_field", text(end)?.as_str()),
+                ],
+            ))
+        }
+    }
+}
+
 /// The abstract intermediate base classes of the type hierarchy (mirror 
pyspark's
 /// AtomicType/NumericType/IntegralType/... ). They carry no data — a concrete 
type's
 /// `__new__` builds the initializer chain `PyDataType{inner} -> ... -> 
Concrete` — but
@@ -479,9 +553,15 @@ impl PyNullType {
         type_reduce(py, "\"void\"")
     }
     /// The one instance of the type, as PySpark's `DataTypeSingleton`: every 
call of the class
-    /// returns it.
+    /// returns it, and raises its `TypeError` of an argument.
     #[new]
-    fn new(py: Python<'_>) -> PyResult<Py<Self>> {
+    #[pyo3(signature = (*args, **kwargs))]
+    fn new(
+        py: Python<'_>,
+        args: &Bound<'_, pyo3::types::PyTuple>,
+        kwargs: Option<&Bound<'_, pyo3::types::PyDict>>,
+    ) -> PyResult<Py<Self>> {
+        singleton_arguments(args, kwargs)?;
         static INSTANCE: PyOnceLock<Py<PyNullType>> = PyOnceLock::new();
         INSTANCE
             .get_or_try_init(py, || {
@@ -544,9 +624,15 @@ impl PyBooleanType {
         type_reduce(py, "\"boolean\"")
     }
     /// The one instance of the type, as PySpark's `DataTypeSingleton`: every 
call of the class
-    /// returns it.
+    /// returns it, and raises its `TypeError` of an argument.
     #[new]
-    fn new(py: Python<'_>) -> PyResult<Py<Self>> {
+    #[pyo3(signature = (*args, **kwargs))]
+    fn new(
+        py: Python<'_>,
+        args: &Bound<'_, pyo3::types::PyTuple>,
+        kwargs: Option<&Bound<'_, pyo3::types::PyDict>>,
+    ) -> PyResult<Py<Self>> {
+        singleton_arguments(args, kwargs)?;
         static INSTANCE: PyOnceLock<Py<PyBooleanType>> = PyOnceLock::new();
         INSTANCE
             .get_or_try_init(py, || {
@@ -606,9 +692,15 @@ impl PyByteType {
         type_reduce(py, "\"byte\"")
     }
     /// The one instance of the type, as PySpark's `DataTypeSingleton`: every 
call of the class
-    /// returns it.
+    /// returns it, and raises its `TypeError` of an argument.
     #[new]
-    fn new(py: Python<'_>) -> PyResult<Py<Self>> {
+    #[pyo3(signature = (*args, **kwargs))]
+    fn new(
+        py: Python<'_>,
+        args: &Bound<'_, pyo3::types::PyTuple>,
+        kwargs: Option<&Bound<'_, pyo3::types::PyDict>>,
+    ) -> PyResult<Py<Self>> {
+        singleton_arguments(args, kwargs)?;
         static INSTANCE: PyOnceLock<Py<PyByteType>> = PyOnceLock::new();
         INSTANCE
             .get_or_try_init(py, || {
@@ -672,9 +764,15 @@ impl PyShortType {
         type_reduce(py, "\"short\"")
     }
     /// The one instance of the type, as PySpark's `DataTypeSingleton`: every 
call of the class
-    /// returns it.
+    /// returns it, and raises its `TypeError` of an argument.
     #[new]
-    fn new(py: Python<'_>) -> PyResult<Py<Self>> {
+    #[pyo3(signature = (*args, **kwargs))]
+    fn new(
+        py: Python<'_>,
+        args: &Bound<'_, pyo3::types::PyTuple>,
+        kwargs: Option<&Bound<'_, pyo3::types::PyDict>>,
+    ) -> PyResult<Py<Self>> {
+        singleton_arguments(args, kwargs)?;
         static INSTANCE: PyOnceLock<Py<PyShortType>> = PyOnceLock::new();
         INSTANCE
             .get_or_try_init(py, || {
@@ -738,9 +836,15 @@ impl PyIntegerType {
         type_reduce(py, "\"integer\"")
     }
     /// The one instance of the type, as PySpark's `DataTypeSingleton`: every 
call of the class
-    /// returns it.
+    /// returns it, and raises its `TypeError` of an argument.
     #[new]
-    fn new(py: Python<'_>) -> PyResult<Py<Self>> {
+    #[pyo3(signature = (*args, **kwargs))]
+    fn new(
+        py: Python<'_>,
+        args: &Bound<'_, pyo3::types::PyTuple>,
+        kwargs: Option<&Bound<'_, pyo3::types::PyDict>>,
+    ) -> PyResult<Py<Self>> {
+        singleton_arguments(args, kwargs)?;
         static INSTANCE: PyOnceLock<Py<PyIntegerType>> = PyOnceLock::new();
         INSTANCE
             .get_or_try_init(py, || {
@@ -804,9 +908,15 @@ impl PyLongType {
         type_reduce(py, "\"long\"")
     }
     /// The one instance of the type, as PySpark's `DataTypeSingleton`: every 
call of the class
-    /// returns it.
+    /// returns it, and raises its `TypeError` of an argument.
     #[new]
-    fn new(py: Python<'_>) -> PyResult<Py<Self>> {
+    #[pyo3(signature = (*args, **kwargs))]
+    fn new(
+        py: Python<'_>,
+        args: &Bound<'_, pyo3::types::PyTuple>,
+        kwargs: Option<&Bound<'_, pyo3::types::PyDict>>,
+    ) -> PyResult<Py<Self>> {
+        singleton_arguments(args, kwargs)?;
         static INSTANCE: PyOnceLock<Py<PyLongType>> = PyOnceLock::new();
         INSTANCE
             .get_or_try_init(py, || {
@@ -870,9 +980,15 @@ impl PyFloatType {
         type_reduce(py, "\"float\"")
     }
     /// The one instance of the type, as PySpark's `DataTypeSingleton`: every 
call of the class
-    /// returns it.
+    /// returns it, and raises its `TypeError` of an argument.
     #[new]
-    fn new(py: Python<'_>) -> PyResult<Py<Self>> {
+    #[pyo3(signature = (*args, **kwargs))]
+    fn new(
+        py: Python<'_>,
+        args: &Bound<'_, pyo3::types::PyTuple>,
+        kwargs: Option<&Bound<'_, pyo3::types::PyDict>>,
+    ) -> PyResult<Py<Self>> {
+        singleton_arguments(args, kwargs)?;
         static INSTANCE: PyOnceLock<Py<PyFloatType>> = PyOnceLock::new();
         INSTANCE
             .get_or_try_init(py, || {
@@ -936,9 +1052,15 @@ impl PyDoubleType {
         type_reduce(py, "\"double\"")
     }
     /// The one instance of the type, as PySpark's `DataTypeSingleton`: every 
call of the class
-    /// returns it.
+    /// returns it, and raises its `TypeError` of an argument.
     #[new]
-    fn new(py: Python<'_>) -> PyResult<Py<Self>> {
+    #[pyo3(signature = (*args, **kwargs))]
+    fn new(
+        py: Python<'_>,
+        args: &Bound<'_, pyo3::types::PyTuple>,
+        kwargs: Option<&Bound<'_, pyo3::types::PyDict>>,
+    ) -> PyResult<Py<Self>> {
+        singleton_arguments(args, kwargs)?;
         static INSTANCE: PyOnceLock<Py<PyDoubleType>> = PyOnceLock::new();
         INSTANCE
             .get_or_try_init(py, || {
@@ -1189,9 +1311,15 @@ impl PyBinaryType {
         type_reduce(py, "\"binary\"")
     }
     /// The one instance of the type, as PySpark's `DataTypeSingleton`: every 
call of the class
-    /// returns it.
+    /// returns it, and raises its `TypeError` of an argument.
     #[new]
-    fn new(py: Python<'_>) -> PyResult<Py<Self>> {
+    #[pyo3(signature = (*args, **kwargs))]
+    fn new(
+        py: Python<'_>,
+        args: &Bound<'_, pyo3::types::PyTuple>,
+        kwargs: Option<&Bound<'_, pyo3::types::PyDict>>,
+    ) -> PyResult<Py<Self>> {
+        singleton_arguments(args, kwargs)?;
         static INSTANCE: PyOnceLock<Py<PyBinaryType>> = PyOnceLock::new();
         INSTANCE
             .get_or_try_init(py, || {
@@ -1254,9 +1382,15 @@ impl PyDateType {
         type_reduce(py, "\"date\"")
     }
     /// The one instance of the type, as PySpark's `DataTypeSingleton`: every 
call of the class
-    /// returns it.
+    /// returns it, and raises its `TypeError` of an argument.
     #[new]
-    fn new(py: Python<'_>) -> PyResult<Py<Self>> {
+    #[pyo3(signature = (*args, **kwargs))]
+    fn new(
+        py: Python<'_>,
+        args: &Bound<'_, pyo3::types::PyTuple>,
+        kwargs: Option<&Bound<'_, pyo3::types::PyDict>>,
+    ) -> PyResult<Py<Self>> {
+        singleton_arguments(args, kwargs)?;
         static INSTANCE: PyOnceLock<Py<PyDateType>> = PyOnceLock::new();
         INSTANCE
             .get_or_try_init(py, || {
@@ -1316,9 +1450,15 @@ impl PyTimestampType {
         type_reduce(py, "\"timestamp\"")
     }
     /// The one instance of the type, as PySpark's `DataTypeSingleton`: every 
call of the class
-    /// returns it.
+    /// returns it, and raises its `TypeError` of an argument.
     #[new]
-    fn new(py: Python<'_>) -> PyResult<Py<Self>> {
+    #[pyo3(signature = (*args, **kwargs))]
+    fn new(
+        py: Python<'_>,
+        args: &Bound<'_, pyo3::types::PyTuple>,
+        kwargs: Option<&Bound<'_, pyo3::types::PyDict>>,
+    ) -> PyResult<Py<Self>> {
+        singleton_arguments(args, kwargs)?;
         static INSTANCE: PyOnceLock<Py<PyTimestampType>> = PyOnceLock::new();
         INSTANCE
             .get_or_try_init(py, || {
@@ -1382,9 +1522,15 @@ impl PyTimestampNTZType {
         type_reduce(py, "\"timestamp_ntz\"")
     }
     /// The one instance of the type, as PySpark's `DataTypeSingleton`: every 
call of the class
-    /// returns it.
+    /// returns it, and raises its `TypeError` of an argument.
     #[new]
-    fn new(py: Python<'_>) -> PyResult<Py<Self>> {
+    #[pyo3(signature = (*args, **kwargs))]
+    fn new(
+        py: Python<'_>,
+        args: &Bound<'_, pyo3::types::PyTuple>,
+        kwargs: Option<&Bound<'_, pyo3::types::PyDict>>,
+    ) -> PyResult<Py<Self>> {
+        singleton_arguments(args, kwargs)?;
         static INSTANCE: PyOnceLock<Py<PyTimestampNTZType>> = 
PyOnceLock::new();
         INSTANCE
             .get_or_try_init(py, || {
@@ -2680,9 +2826,15 @@ impl PyCalendarIntervalType {
         type_reduce(py, "\"calendarinterval\"")
     }
     /// The one instance of the type, as PySpark's `DataTypeSingleton`: every 
call of the class
-    /// returns it.
+    /// returns it, and raises its `TypeError` of an argument.
     #[new]
-    fn new(py: Python<'_>) -> PyResult<Py<Self>> {
+    #[pyo3(signature = (*args, **kwargs))]
+    fn new(
+        py: Python<'_>,
+        args: &Bound<'_, pyo3::types::PyTuple>,
+        kwargs: Option<&Bound<'_, pyo3::types::PyDict>>,
+    ) -> PyResult<Py<Self>> {
+        singleton_arguments(args, kwargs)?;
         static INSTANCE: PyOnceLock<Py<PyCalendarIntervalType>> = 
PyOnceLock::new();
         INSTANCE
             .get_or_try_init(py, || {
@@ -2769,13 +2921,18 @@ impl PyYearMonthIntervalType {
         type_reduce(py, &dt.json())
     }
     /// Fields: YEAR=0, MONTH=1. Defaults to the full YEAR..MONTH range.
+    /// As PySpark's `YearMonthIntervalType(startField=None, endField=None)`, 
of the fields YEAR
+    /// and MONTH: `INVALID_INTERVAL_CASTING` for another.
     #[new]
     #[pyo3(signature = (startField=None, endField=None))]
     #[allow(non_snake_case)]
-    fn new(startField: Option<i32>, endField: Option<i32>) -> 
pyo3::PyClassInitializer<Self> {
-        let start = startField.unwrap_or(0);
-        let end = endField.unwrap_or(if startField.is_some() { start } else { 
1 });
-        init_chain!(
+    fn new(
+        py: Python<'_>,
+        startField: Option<&Bound<'_, PyAny>>,
+        endField: Option<&Bound<'_, PyAny>>,
+    ) -> PyResult<pyo3::PyClassInitializer<Self>> {
+        let (start, end) = interval_fields(py, startField, endField, (0, 1), 
1)?;
+        Ok(init_chain!(
             DataType::YearMonthInterval {
                 start_field: start,
                 end_field: end,
@@ -2786,7 +2943,7 @@ impl PyYearMonthIntervalType {
             },
             [PyAtomicType, PyAnsiIntervalType],
             value
-        )
+        ))
     }
     fn __repr__(&self) -> String {
         format!(
@@ -2861,13 +3018,18 @@ impl PyDayTimeIntervalType {
         type_reduce(py, &dt.json())
     }
     /// Fields: DAY=0, HOUR=1, MINUTE=2, SECOND=3. Defaults to the full 
DAY..SECOND range.
+    /// As PySpark's `DayTimeIntervalType(startField=None, endField=None)`, of 
the fields DAY,
+    /// HOUR, MINUTE and SECOND: `INVALID_INTERVAL_CASTING` for another.
     #[new]
     #[pyo3(signature = (startField=None, endField=None))]
     #[allow(non_snake_case)]
-    fn new(startField: Option<i32>, endField: Option<i32>) -> 
pyo3::PyClassInitializer<Self> {
-        let start = startField.unwrap_or(0);
-        let end = endField.unwrap_or(if startField.is_some() { start } else { 
3 });
-        init_chain!(
+    fn new(
+        py: Python<'_>,
+        startField: Option<&Bound<'_, PyAny>>,
+        endField: Option<&Bound<'_, PyAny>>,
+    ) -> PyResult<pyo3::PyClassInitializer<Self>> {
+        let (start, end) = interval_fields(py, startField, endField, (0, 3), 
3)?;
+        Ok(init_chain!(
             DataType::DayTimeInterval {
                 start_field: start,
                 end_field: end,
@@ -2878,7 +3040,7 @@ impl PyDayTimeIntervalType {
             },
             [PyAtomicType, PyAnsiIntervalType],
             value
-        )
+        ))
     }
     fn __repr__(&self) -> String {
         format!(
@@ -3172,14 +3334,20 @@ pub(crate) fn data_type_to_py(py: Python<'_>, dt: 
&DataType) -> PyResult<Py<PyAn
         };
     }
     let o = match dt {
-        DataType::Null => PyNullType::new(py)?.into_any(),
-        DataType::Boolean => PyBooleanType::new(py)?.into_any(),
-        DataType::Byte => PyByteType::new(py)?.into_any(),
-        DataType::Short => PyShortType::new(py)?.into_any(),
-        DataType::Integer => PyIntegerType::new(py)?.into_any(),
-        DataType::Long => PyLongType::new(py)?.into_any(),
-        DataType::Float => PyFloatType::new(py)?.into_any(),
-        DataType::Double => PyDoubleType::new(py)?.into_any(),
+        DataType::Null => PyNullType::new(py, 
&pyo3::types::PyTuple::empty(py), None)?.into_any(),
+        DataType::Boolean => {
+            PyBooleanType::new(py, &pyo3::types::PyTuple::empty(py), 
None)?.into_any()
+        }
+        DataType::Byte => PyByteType::new(py, 
&pyo3::types::PyTuple::empty(py), None)?.into_any(),
+        DataType::Short => PyShortType::new(py, 
&pyo3::types::PyTuple::empty(py), None)?.into_any(),
+        DataType::Integer => {
+            PyIntegerType::new(py, &pyo3::types::PyTuple::empty(py), 
None)?.into_any()
+        }
+        DataType::Long => PyLongType::new(py, 
&pyo3::types::PyTuple::empty(py), None)?.into_any(),
+        DataType::Float => PyFloatType::new(py, 
&pyo3::types::PyTuple::empty(py), None)?.into_any(),
+        DataType::Double => {
+            PyDoubleType::new(py, &pyo3::types::PyTuple::empty(py), 
None)?.into_any()
+        }
         DataType::Decimal { precision, scale } => {
             let (precision, scale) = (*precision, *scale);
             obj!(init_chain!(
@@ -3199,10 +3367,16 @@ pub(crate) fn data_type_to_py(py: Python<'_>, dt: 
&DataType) -> PyResult<Py<PyAn
             [PyAtomicType],
             value
         )),
-        DataType::Binary => PyBinaryType::new(py)?.into_any(),
-        DataType::Date => PyDateType::new(py)?.into_any(),
-        DataType::Timestamp => PyTimestampType::new(py)?.into_any(),
-        DataType::TimestampNtz => PyTimestampNTZType::new(py)?.into_any(),
+        DataType::Binary => {
+            PyBinaryType::new(py, &pyo3::types::PyTuple::empty(py), 
None)?.into_any()
+        }
+        DataType::Date => PyDateType::new(py, 
&pyo3::types::PyTuple::empty(py), None)?.into_any(),
+        DataType::Timestamp => {
+            PyTimestampType::new(py, &pyo3::types::PyTuple::empty(py), 
None)?.into_any()
+        }
+        DataType::TimestampNtz => {
+            PyTimestampNTZType::new(py, &pyo3::types::PyTuple::empty(py), 
None)?.into_any()
+        }
         DataType::Array {
             element_type,
             contains_null,
@@ -3292,7 +3466,9 @@ pub(crate) fn data_type_to_py(py: Python<'_>, dt: 
&DataType) -> PyResult<Py<PyAn
                 value
             ))
         }
-        DataType::CalendarInterval => 
PyCalendarIntervalType::new(py)?.into_any(),
+        DataType::CalendarInterval => {
+            PyCalendarIntervalType::new(py, &pyo3::types::PyTuple::empty(py), 
None)?.into_any()
+        }
         DataType::YearMonthInterval {
             start_field,
             end_field,
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 08f61f51..2612868b 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -104,6 +104,45 @@ def test_decimal_precision_scale():
     assert d.precision == 10 and d.scale == 2
 
 
+def test_type_constructors_check_their_arguments_as_pyspark():
+    # As PySpark's: DayTimeIntervalType and YearMonthIntervalType take the 
fields of their
+    # types, both of the defaults without any and the start field for the end 
without one, and
+    # raise INVALID_INTERVAL_CASTING of the str of each for another; the types 
without
+    # parameters, singletons of DataTypeSingleton, raise its TypeError of an 
argument.
+    from pyspark.errors import PySparkRuntimeError
+
+    D, Y = T.DayTimeIntervalType, T.YearMonthIntervalType
+    assert D().simpleString() == "interval day to second"
+    assert D(D.HOUR).simpleString() == "interval hour"
+    assert D(D.MINUTE, D.HOUR).simpleString() == "interval minute to hour"
+    assert D(1.0).simpleString() == "interval hour"
+    assert Y().simpleString() == "interval year to month"
+    assert Y(Y.MONTH).simpleString() == "interval month"
+    for make, params in [
+        (lambda: D(endField=D.SECOND), ("None", "3")),
+        (lambda: D(123), ("123", "123")),
+        (lambda: D("a"), ("a", "a")),
+        (lambda: Y(endField=3), ("None", "3")),
+        (lambda: Y(Y.YEAR, 5), ("0", "5")),
+    ]:
+        with pytest.raises(PySparkRuntimeError) as e:
+            make()
+        assert e.value.getCondition() == "INVALID_INTERVAL_CASTING"
+        assert e.value.getMessageParameters() == {"start_field": params[0], 
"end_field": params[1]}
+
+    assert T.CalendarIntervalType() is T.CalendarIntervalType()
+    for make, message in [
+        (lambda: T.CalendarIntervalType(3), "takes 1 positional argument but 2 
were given"),
+        (lambda: T.IntegerType(1, 2), "takes 1 positional argument but 3 were 
given"),
+        (lambda: T.NullType(x=1), "got an unexpected keyword argument 'x'"),
+        (lambda: T.LongType(1, y=2), "got an unexpected keyword argument 'y'"),
+    ]:
+        with pytest.raises(TypeError, match=r"^DataTypeSingleton\.__call__\(\) 
" + message):
+            make()
+
+
+
+
 def test_nested_types_pickle_roundtrip():
     nested = [
         T.ArrayType(T.IntegerType()),


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to