Hyukjin Kwon created SPARK-61134:
------------------------------------
Summary: Make DayTimeIntervalType and YearMonthIntervalType check
their fields, and the types without parameters their arguments, in the Python
client as PySpark does
Key: SPARK-61134
URL: https://issues.apache.org/jira/browse/SPARK-61134
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
PySpark's DayTimeIntervalType(startField, endField) and
YearMonthIntervalType(startField, endField) raise PySparkRuntimeError
INVALID_INTERVAL_CASTING, with start_field and end_field, for a field that is
not one of theirs. For example,
DayTimeIntervalType(endField=DayTimeIntervalType.SECOND) raises with
start_field "None", and DayTimeIntervalType(123) with "123". The client accepts
them: DayTimeIntervalType(endField=3) is DayTimeIntervalType(0, 3).
PySpark's types without parameters, NullType through CalendarIntervalType, are
the singletons of the DataTypeSingleton metaclass. Its __call__ raises
TypeError "DataTypeSingleton.__call__() takes 1 positional argument but 2 were
given" for an argument, and "got an unexpected keyword argument" for a keyword.
The client's raise "CalendarIntervalType.__new__() takes 0 positional arguments
but 1 was given".
Apache Spark's test_daytime_interval_type_constructor,
test_yearmonth_interval_type_constructor and
test_calendar_interval_type_constructor fail on this client. The client should
check the fields and arguments of these types as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
