This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new f70b87b9 [SPARK-61134][CONNECT][PYTHON] Make DayTimeIntervalType and
YearMonthIntervalType check their fields, and the types without parameters
their arguments, in the Python client as PySpark does
f70b87b9 is described below
commit f70b87b9ea8e9ef04ae6f6f036ca129ac357a110
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 12:42:48 2026 +0900
[SPARK-61134][CONNECT][PYTHON] Make DayTimeIntervalType and
YearMonthIntervalType check their fields, and the types without parameters
their arguments, in the Python client as PySpark does
### What changes were proposed in this pull request?
[SPARK-61134](https://issues.apache.org/jira/browse/SPARK-61134): The
constructors of the client's types now check their arguments as PySpark's do.
- **`DayTimeIntervalType(startField=None, endField=None)` and
`YearMonthIntervalType(...)`** take the fields of their types, as PySpark's do:
- without any field, those of the whole type (`DAY` to `SECOND`, `YEAR`
to `MONTH`)
- without an end field, the start field
For a field that is not one of the type, they raise `PySparkRuntimeError`
`INVALID_INTERVAL_CASTING`, of the `str` of the start and end fields:
`DayTimeIntervalType(endField=SECOND)` raises with `start_field` "None", and
`DayTimeIntervalType(123)` with "123". As with PySpark's membership check, a
float equal to a field, such as `1.0`, is that field.
- **The types without parameters**, `NullType` through
`CalendarIntervalType`, are PySpark's singletons of `DataTypeSingleton`. For an
argument they now raise its `TypeError`, as Python words it:
`DataTypeSingleton.__call__() got an unexpected keyword argument 'x'` for the
first keyword, or else `DataTypeSingleton.__call__() takes 1 positional
argument but 2 were given`.
### Why are the changes needed?
- The client took any integer as a field, and filled a missing start field:
`DayTimeIntervalType(endField=3)` was `DayTimeIntervalType(0, 3)`, and
`DayTimeIntervalType(123)` an interval of no such field.
- The types without parameters raised the `TypeError` of their own
`__new__`.
Apache Spark's `test_daytime_interval_type_constructor`,
`test_yearmonth_interval_type_constructor` and
`test_calendar_interval_type_constructor` failed on this client.
### Does this PR introduce _any_ user-facing change?
Yes. Interval types of invalid fields raise `INVALID_INTERVAL_CASTING`, and
calls with arguments of the types without parameters raise PySpark's
`TypeError`, as with PySpark 4.3.0.
### How was this patch tested?
- A new offline test checks:
- the fields of the interval types and their `INVALID_INTERVAL_CASTING`
errors, with their parameters
- the `TypeError`s of the types without parameters, which stay singletons
It passes with official PySpark 4.3.0, and fails without this change.
- Apache Spark's interval tests of `test_parity_types.py` run on this
client against a local server: 7 pass and 2 are skipped, including the three
tests above.
- The lib tests of both crates (590 passed), the build of all tests, also
with `wasm-udf`, the offline Python suites (2202 passed, 8 skipped), and the
live drop-in and Arrow suites (184 passed) and live cases (10 passed) against a
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #329 from HyukjinKwon/SPARK-61134-type-constructors.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/types.rs | 274 +++++++++++++++++++++++++++++-------
python/tests/test_dropin_offline.py | 39 +++++
2 files changed, 264 insertions(+), 49 deletions(-)
diff --git a/crates/pyspark-rs/src/types.rs b/crates/pyspark-rs/src/types.rs
index 6ed29d0f..6c58da2e 100644
--- a/crates/pyspark-rs/src/types.rs
+++ b/crates/pyspark-rs/src/types.rs
@@ -67,6 +67,80 @@ pub struct PyDataType {
pub(crate) inner: DataType,
}
+/// What PySpark's `DataTypeSingleton.__call__(cls)` raises, as Python does,
for the arguments
+/// of a call of a type without parameters: a `TypeError` of the first keyword
argument, or else
+/// of the positional ones.
+fn singleton_arguments(
+ args: &Bound<'_, pyo3::types::PyTuple>,
+ kwargs: Option<&Bound<'_, pyo3::types::PyDict>>,
+) -> PyResult<()> {
+ if let Some((key, _)) = kwargs.and_then(|kwargs| kwargs.iter().next()) {
+ return Err(pyo3::exceptions::PyTypeError::new_err(format!(
+ "DataTypeSingleton.__call__() got an unexpected keyword argument
'{}'",
+ key.str()?
+ )));
+ }
+ if !args.is_empty() {
+ return Err(pyo3::exceptions::PyTypeError::new_err(format!(
+ "DataTypeSingleton.__call__() takes 1 positional argument but {}
were given",
+ args.len() + 1
+ )));
+ }
+ Ok(())
+}
+
+/// The fields of an interval type, as PySpark's `DayTimeIntervalType` and
+/// `YearMonthIntervalType` take them: both of `default` without any, the
start field for the
+/// end without one, and a `PySparkRuntimeError` of `INVALID_INTERVAL_CASTING`
for a field that
+/// is not one of the type, of the `str` of each.
+fn interval_fields(
+ py: Python<'_>,
+ start: Option<&Bound<'_, PyAny>>,
+ end: Option<&Bound<'_, PyAny>>,
+ default: (i32, i32),
+ last: i32,
+) -> PyResult<(i32, i32)> {
+ let field = |value: Option<&Bound<'_, PyAny>>| -> Option<i32> {
+ let value = value?;
+ if value.is_instance_of::<pyo3::types::PyFloat>() {
+ // `1.0 in {0, 1}` of PySpark, of a float that equals a field.
+ let float: f64 = value.extract().ok()?;
+ return (float.fract() == 0.0 &&
(0.0..=f64::from(last)).contains(&float))
+ .then_some(float as i32);
+ }
+ value
+ .extract::<i64>()
+ .ok()
+ .filter(|field| (0..=i64::from(last)).contains(field))
+ .map(|field| field as i32)
+ };
+ let (start, end) = match (start, end) {
+ (None, None) => return Ok(default),
+ (Some(start), None) => (Some(start), Some(start)),
+ (start, end) => (start, end),
+ };
+ match (field(start), field(end)) {
+ (Some(start), Some(end)) => Ok((start, end)),
+ _ => {
+ let text = |value: Option<&Bound<'_, PyAny>>| -> PyResult<String> {
+ Ok(match value {
+ Some(value) => value.str()?.to_string(),
+ None => "None".to_string(),
+ })
+ };
+ Err(crate::column::raise_pyspark(
+ py,
+ "PySparkRuntimeError",
+ "INVALID_INTERVAL_CASTING",
+ &[
+ ("start_field", text(start)?.as_str()),
+ ("end_field", text(end)?.as_str()),
+ ],
+ ))
+ }
+ }
+}
+
/// The abstract intermediate base classes of the type hierarchy (mirror
pyspark's
/// AtomicType/NumericType/IntegralType/... ). They carry no data — a concrete
type's
/// `__new__` builds the initializer chain `PyDataType{inner} -> ... ->
Concrete` — but
@@ -479,9 +553,15 @@ impl PyNullType {
type_reduce(py, "\"void\"")
}
/// The one instance of the type, as PySpark's `DataTypeSingleton`: every
call of the class
- /// returns it.
+ /// returns it, and raises its `TypeError` of an argument.
#[new]
- fn new(py: Python<'_>) -> PyResult<Py<Self>> {
+ #[pyo3(signature = (*args, **kwargs))]
+ fn new(
+ py: Python<'_>,
+ args: &Bound<'_, pyo3::types::PyTuple>,
+ kwargs: Option<&Bound<'_, pyo3::types::PyDict>>,
+ ) -> PyResult<Py<Self>> {
+ singleton_arguments(args, kwargs)?;
static INSTANCE: PyOnceLock<Py<PyNullType>> = PyOnceLock::new();
INSTANCE
.get_or_try_init(py, || {
@@ -544,9 +624,15 @@ impl PyBooleanType {
type_reduce(py, "\"boolean\"")
}
/// The one instance of the type, as PySpark's `DataTypeSingleton`: every
call of the class
- /// returns it.
+ /// returns it, and raises its `TypeError` of an argument.
#[new]
- fn new(py: Python<'_>) -> PyResult<Py<Self>> {
+ #[pyo3(signature = (*args, **kwargs))]
+ fn new(
+ py: Python<'_>,
+ args: &Bound<'_, pyo3::types::PyTuple>,
+ kwargs: Option<&Bound<'_, pyo3::types::PyDict>>,
+ ) -> PyResult<Py<Self>> {
+ singleton_arguments(args, kwargs)?;
static INSTANCE: PyOnceLock<Py<PyBooleanType>> = PyOnceLock::new();
INSTANCE
.get_or_try_init(py, || {
@@ -606,9 +692,15 @@ impl PyByteType {
type_reduce(py, "\"byte\"")
}
/// The one instance of the type, as PySpark's `DataTypeSingleton`: every
call of the class
- /// returns it.
+ /// returns it, and raises its `TypeError` of an argument.
#[new]
- fn new(py: Python<'_>) -> PyResult<Py<Self>> {
+ #[pyo3(signature = (*args, **kwargs))]
+ fn new(
+ py: Python<'_>,
+ args: &Bound<'_, pyo3::types::PyTuple>,
+ kwargs: Option<&Bound<'_, pyo3::types::PyDict>>,
+ ) -> PyResult<Py<Self>> {
+ singleton_arguments(args, kwargs)?;
static INSTANCE: PyOnceLock<Py<PyByteType>> = PyOnceLock::new();
INSTANCE
.get_or_try_init(py, || {
@@ -672,9 +764,15 @@ impl PyShortType {
type_reduce(py, "\"short\"")
}
/// The one instance of the type, as PySpark's `DataTypeSingleton`: every
call of the class
- /// returns it.
+ /// returns it, and raises its `TypeError` of an argument.
#[new]
- fn new(py: Python<'_>) -> PyResult<Py<Self>> {
+ #[pyo3(signature = (*args, **kwargs))]
+ fn new(
+ py: Python<'_>,
+ args: &Bound<'_, pyo3::types::PyTuple>,
+ kwargs: Option<&Bound<'_, pyo3::types::PyDict>>,
+ ) -> PyResult<Py<Self>> {
+ singleton_arguments(args, kwargs)?;
static INSTANCE: PyOnceLock<Py<PyShortType>> = PyOnceLock::new();
INSTANCE
.get_or_try_init(py, || {
@@ -738,9 +836,15 @@ impl PyIntegerType {
type_reduce(py, "\"integer\"")
}
/// The one instance of the type, as PySpark's `DataTypeSingleton`: every
call of the class
- /// returns it.
+ /// returns it, and raises its `TypeError` of an argument.
#[new]
- fn new(py: Python<'_>) -> PyResult<Py<Self>> {
+ #[pyo3(signature = (*args, **kwargs))]
+ fn new(
+ py: Python<'_>,
+ args: &Bound<'_, pyo3::types::PyTuple>,
+ kwargs: Option<&Bound<'_, pyo3::types::PyDict>>,
+ ) -> PyResult<Py<Self>> {
+ singleton_arguments(args, kwargs)?;
static INSTANCE: PyOnceLock<Py<PyIntegerType>> = PyOnceLock::new();
INSTANCE
.get_or_try_init(py, || {
@@ -804,9 +908,15 @@ impl PyLongType {
type_reduce(py, "\"long\"")
}
/// The one instance of the type, as PySpark's `DataTypeSingleton`: every
call of the class
- /// returns it.
+ /// returns it, and raises its `TypeError` of an argument.
#[new]
- fn new(py: Python<'_>) -> PyResult<Py<Self>> {
+ #[pyo3(signature = (*args, **kwargs))]
+ fn new(
+ py: Python<'_>,
+ args: &Bound<'_, pyo3::types::PyTuple>,
+ kwargs: Option<&Bound<'_, pyo3::types::PyDict>>,
+ ) -> PyResult<Py<Self>> {
+ singleton_arguments(args, kwargs)?;
static INSTANCE: PyOnceLock<Py<PyLongType>> = PyOnceLock::new();
INSTANCE
.get_or_try_init(py, || {
@@ -870,9 +980,15 @@ impl PyFloatType {
type_reduce(py, "\"float\"")
}
/// The one instance of the type, as PySpark's `DataTypeSingleton`: every
call of the class
- /// returns it.
+ /// returns it, and raises its `TypeError` of an argument.
#[new]
- fn new(py: Python<'_>) -> PyResult<Py<Self>> {
+ #[pyo3(signature = (*args, **kwargs))]
+ fn new(
+ py: Python<'_>,
+ args: &Bound<'_, pyo3::types::PyTuple>,
+ kwargs: Option<&Bound<'_, pyo3::types::PyDict>>,
+ ) -> PyResult<Py<Self>> {
+ singleton_arguments(args, kwargs)?;
static INSTANCE: PyOnceLock<Py<PyFloatType>> = PyOnceLock::new();
INSTANCE
.get_or_try_init(py, || {
@@ -936,9 +1052,15 @@ impl PyDoubleType {
type_reduce(py, "\"double\"")
}
/// The one instance of the type, as PySpark's `DataTypeSingleton`: every
call of the class
- /// returns it.
+ /// returns it, and raises its `TypeError` of an argument.
#[new]
- fn new(py: Python<'_>) -> PyResult<Py<Self>> {
+ #[pyo3(signature = (*args, **kwargs))]
+ fn new(
+ py: Python<'_>,
+ args: &Bound<'_, pyo3::types::PyTuple>,
+ kwargs: Option<&Bound<'_, pyo3::types::PyDict>>,
+ ) -> PyResult<Py<Self>> {
+ singleton_arguments(args, kwargs)?;
static INSTANCE: PyOnceLock<Py<PyDoubleType>> = PyOnceLock::new();
INSTANCE
.get_or_try_init(py, || {
@@ -1189,9 +1311,15 @@ impl PyBinaryType {
type_reduce(py, "\"binary\"")
}
/// The one instance of the type, as PySpark's `DataTypeSingleton`: every
call of the class
- /// returns it.
+ /// returns it, and raises its `TypeError` of an argument.
#[new]
- fn new(py: Python<'_>) -> PyResult<Py<Self>> {
+ #[pyo3(signature = (*args, **kwargs))]
+ fn new(
+ py: Python<'_>,
+ args: &Bound<'_, pyo3::types::PyTuple>,
+ kwargs: Option<&Bound<'_, pyo3::types::PyDict>>,
+ ) -> PyResult<Py<Self>> {
+ singleton_arguments(args, kwargs)?;
static INSTANCE: PyOnceLock<Py<PyBinaryType>> = PyOnceLock::new();
INSTANCE
.get_or_try_init(py, || {
@@ -1254,9 +1382,15 @@ impl PyDateType {
type_reduce(py, "\"date\"")
}
/// The one instance of the type, as PySpark's `DataTypeSingleton`: every
call of the class
- /// returns it.
+ /// returns it, and raises its `TypeError` of an argument.
#[new]
- fn new(py: Python<'_>) -> PyResult<Py<Self>> {
+ #[pyo3(signature = (*args, **kwargs))]
+ fn new(
+ py: Python<'_>,
+ args: &Bound<'_, pyo3::types::PyTuple>,
+ kwargs: Option<&Bound<'_, pyo3::types::PyDict>>,
+ ) -> PyResult<Py<Self>> {
+ singleton_arguments(args, kwargs)?;
static INSTANCE: PyOnceLock<Py<PyDateType>> = PyOnceLock::new();
INSTANCE
.get_or_try_init(py, || {
@@ -1316,9 +1450,15 @@ impl PyTimestampType {
type_reduce(py, "\"timestamp\"")
}
/// The one instance of the type, as PySpark's `DataTypeSingleton`: every
call of the class
- /// returns it.
+ /// returns it, and raises its `TypeError` of an argument.
#[new]
- fn new(py: Python<'_>) -> PyResult<Py<Self>> {
+ #[pyo3(signature = (*args, **kwargs))]
+ fn new(
+ py: Python<'_>,
+ args: &Bound<'_, pyo3::types::PyTuple>,
+ kwargs: Option<&Bound<'_, pyo3::types::PyDict>>,
+ ) -> PyResult<Py<Self>> {
+ singleton_arguments(args, kwargs)?;
static INSTANCE: PyOnceLock<Py<PyTimestampType>> = PyOnceLock::new();
INSTANCE
.get_or_try_init(py, || {
@@ -1382,9 +1522,15 @@ impl PyTimestampNTZType {
type_reduce(py, "\"timestamp_ntz\"")
}
/// The one instance of the type, as PySpark's `DataTypeSingleton`: every
call of the class
- /// returns it.
+ /// returns it, and raises its `TypeError` of an argument.
#[new]
- fn new(py: Python<'_>) -> PyResult<Py<Self>> {
+ #[pyo3(signature = (*args, **kwargs))]
+ fn new(
+ py: Python<'_>,
+ args: &Bound<'_, pyo3::types::PyTuple>,
+ kwargs: Option<&Bound<'_, pyo3::types::PyDict>>,
+ ) -> PyResult<Py<Self>> {
+ singleton_arguments(args, kwargs)?;
static INSTANCE: PyOnceLock<Py<PyTimestampNTZType>> =
PyOnceLock::new();
INSTANCE
.get_or_try_init(py, || {
@@ -2680,9 +2826,15 @@ impl PyCalendarIntervalType {
type_reduce(py, "\"calendarinterval\"")
}
/// The one instance of the type, as PySpark's `DataTypeSingleton`: every
call of the class
- /// returns it.
+ /// returns it, and raises its `TypeError` of an argument.
#[new]
- fn new(py: Python<'_>) -> PyResult<Py<Self>> {
+ #[pyo3(signature = (*args, **kwargs))]
+ fn new(
+ py: Python<'_>,
+ args: &Bound<'_, pyo3::types::PyTuple>,
+ kwargs: Option<&Bound<'_, pyo3::types::PyDict>>,
+ ) -> PyResult<Py<Self>> {
+ singleton_arguments(args, kwargs)?;
static INSTANCE: PyOnceLock<Py<PyCalendarIntervalType>> =
PyOnceLock::new();
INSTANCE
.get_or_try_init(py, || {
@@ -2769,13 +2921,18 @@ impl PyYearMonthIntervalType {
type_reduce(py, &dt.json())
}
/// Fields: YEAR=0, MONTH=1. Defaults to the full YEAR..MONTH range.
+ /// As PySpark's `YearMonthIntervalType(startField=None, endField=None)`,
of the fields YEAR
+ /// and MONTH: `INVALID_INTERVAL_CASTING` for another.
#[new]
#[pyo3(signature = (startField=None, endField=None))]
#[allow(non_snake_case)]
- fn new(startField: Option<i32>, endField: Option<i32>) ->
pyo3::PyClassInitializer<Self> {
- let start = startField.unwrap_or(0);
- let end = endField.unwrap_or(if startField.is_some() { start } else {
1 });
- init_chain!(
+ fn new(
+ py: Python<'_>,
+ startField: Option<&Bound<'_, PyAny>>,
+ endField: Option<&Bound<'_, PyAny>>,
+ ) -> PyResult<pyo3::PyClassInitializer<Self>> {
+ let (start, end) = interval_fields(py, startField, endField, (0, 1),
1)?;
+ Ok(init_chain!(
DataType::YearMonthInterval {
start_field: start,
end_field: end,
@@ -2786,7 +2943,7 @@ impl PyYearMonthIntervalType {
},
[PyAtomicType, PyAnsiIntervalType],
value
- )
+ ))
}
fn __repr__(&self) -> String {
format!(
@@ -2861,13 +3018,18 @@ impl PyDayTimeIntervalType {
type_reduce(py, &dt.json())
}
/// Fields: DAY=0, HOUR=1, MINUTE=2, SECOND=3. Defaults to the full
DAY..SECOND range.
+ /// As PySpark's `DayTimeIntervalType(startField=None, endField=None)`, of
the fields DAY,
+ /// HOUR, MINUTE and SECOND: `INVALID_INTERVAL_CASTING` for another.
#[new]
#[pyo3(signature = (startField=None, endField=None))]
#[allow(non_snake_case)]
- fn new(startField: Option<i32>, endField: Option<i32>) ->
pyo3::PyClassInitializer<Self> {
- let start = startField.unwrap_or(0);
- let end = endField.unwrap_or(if startField.is_some() { start } else {
3 });
- init_chain!(
+ fn new(
+ py: Python<'_>,
+ startField: Option<&Bound<'_, PyAny>>,
+ endField: Option<&Bound<'_, PyAny>>,
+ ) -> PyResult<pyo3::PyClassInitializer<Self>> {
+ let (start, end) = interval_fields(py, startField, endField, (0, 3),
3)?;
+ Ok(init_chain!(
DataType::DayTimeInterval {
start_field: start,
end_field: end,
@@ -2878,7 +3040,7 @@ impl PyDayTimeIntervalType {
},
[PyAtomicType, PyAnsiIntervalType],
value
- )
+ ))
}
fn __repr__(&self) -> String {
format!(
@@ -3172,14 +3334,20 @@ pub(crate) fn data_type_to_py(py: Python<'_>, dt:
&DataType) -> PyResult<Py<PyAn
};
}
let o = match dt {
- DataType::Null => PyNullType::new(py)?.into_any(),
- DataType::Boolean => PyBooleanType::new(py)?.into_any(),
- DataType::Byte => PyByteType::new(py)?.into_any(),
- DataType::Short => PyShortType::new(py)?.into_any(),
- DataType::Integer => PyIntegerType::new(py)?.into_any(),
- DataType::Long => PyLongType::new(py)?.into_any(),
- DataType::Float => PyFloatType::new(py)?.into_any(),
- DataType::Double => PyDoubleType::new(py)?.into_any(),
+ DataType::Null => PyNullType::new(py,
&pyo3::types::PyTuple::empty(py), None)?.into_any(),
+ DataType::Boolean => {
+ PyBooleanType::new(py, &pyo3::types::PyTuple::empty(py),
None)?.into_any()
+ }
+ DataType::Byte => PyByteType::new(py,
&pyo3::types::PyTuple::empty(py), None)?.into_any(),
+ DataType::Short => PyShortType::new(py,
&pyo3::types::PyTuple::empty(py), None)?.into_any(),
+ DataType::Integer => {
+ PyIntegerType::new(py, &pyo3::types::PyTuple::empty(py),
None)?.into_any()
+ }
+ DataType::Long => PyLongType::new(py,
&pyo3::types::PyTuple::empty(py), None)?.into_any(),
+ DataType::Float => PyFloatType::new(py,
&pyo3::types::PyTuple::empty(py), None)?.into_any(),
+ DataType::Double => {
+ PyDoubleType::new(py, &pyo3::types::PyTuple::empty(py),
None)?.into_any()
+ }
DataType::Decimal { precision, scale } => {
let (precision, scale) = (*precision, *scale);
obj!(init_chain!(
@@ -3199,10 +3367,16 @@ pub(crate) fn data_type_to_py(py: Python<'_>, dt:
&DataType) -> PyResult<Py<PyAn
[PyAtomicType],
value
)),
- DataType::Binary => PyBinaryType::new(py)?.into_any(),
- DataType::Date => PyDateType::new(py)?.into_any(),
- DataType::Timestamp => PyTimestampType::new(py)?.into_any(),
- DataType::TimestampNtz => PyTimestampNTZType::new(py)?.into_any(),
+ DataType::Binary => {
+ PyBinaryType::new(py, &pyo3::types::PyTuple::empty(py),
None)?.into_any()
+ }
+ DataType::Date => PyDateType::new(py,
&pyo3::types::PyTuple::empty(py), None)?.into_any(),
+ DataType::Timestamp => {
+ PyTimestampType::new(py, &pyo3::types::PyTuple::empty(py),
None)?.into_any()
+ }
+ DataType::TimestampNtz => {
+ PyTimestampNTZType::new(py, &pyo3::types::PyTuple::empty(py),
None)?.into_any()
+ }
DataType::Array {
element_type,
contains_null,
@@ -3292,7 +3466,9 @@ pub(crate) fn data_type_to_py(py: Python<'_>, dt:
&DataType) -> PyResult<Py<PyAn
value
))
}
- DataType::CalendarInterval =>
PyCalendarIntervalType::new(py)?.into_any(),
+ DataType::CalendarInterval => {
+ PyCalendarIntervalType::new(py, &pyo3::types::PyTuple::empty(py),
None)?.into_any()
+ }
DataType::YearMonthInterval {
start_field,
end_field,
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 08f61f51..2612868b 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -104,6 +104,45 @@ def test_decimal_precision_scale():
assert d.precision == 10 and d.scale == 2
+def test_type_constructors_check_their_arguments_as_pyspark():
+ # As PySpark's: DayTimeIntervalType and YearMonthIntervalType take the
fields of their
+ # types, both of the defaults without any and the start field for the end
without one, and
+ # raise INVALID_INTERVAL_CASTING of the str of each for another; the types
without
+ # parameters, singletons of DataTypeSingleton, raise its TypeError of an
argument.
+ from pyspark.errors import PySparkRuntimeError
+
+ D, Y = T.DayTimeIntervalType, T.YearMonthIntervalType
+ assert D().simpleString() == "interval day to second"
+ assert D(D.HOUR).simpleString() == "interval hour"
+ assert D(D.MINUTE, D.HOUR).simpleString() == "interval minute to hour"
+ assert D(1.0).simpleString() == "interval hour"
+ assert Y().simpleString() == "interval year to month"
+ assert Y(Y.MONTH).simpleString() == "interval month"
+ for make, params in [
+ (lambda: D(endField=D.SECOND), ("None", "3")),
+ (lambda: D(123), ("123", "123")),
+ (lambda: D("a"), ("a", "a")),
+ (lambda: Y(endField=3), ("None", "3")),
+ (lambda: Y(Y.YEAR, 5), ("0", "5")),
+ ]:
+ with pytest.raises(PySparkRuntimeError) as e:
+ make()
+ assert e.value.getCondition() == "INVALID_INTERVAL_CASTING"
+ assert e.value.getMessageParameters() == {"start_field": params[0],
"end_field": params[1]}
+
+ assert T.CalendarIntervalType() is T.CalendarIntervalType()
+ for make, message in [
+ (lambda: T.CalendarIntervalType(3), "takes 1 positional argument but 2
were given"),
+ (lambda: T.IntegerType(1, 2), "takes 1 positional argument but 3 were
given"),
+ (lambda: T.NullType(x=1), "got an unexpected keyword argument 'x'"),
+ (lambda: T.LongType(1, y=2), "got an unexpected keyword argument 'y'"),
+ ]:
+ with pytest.raises(TypeError, match=r"^DataTypeSingleton\.__call__\(\)
" + message):
+ make()
+
+
+
+
def test_nested_types_pickle_roundtrip():
nested = [
T.ArrayType(T.IntegerType()),
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]