Hyukjin Kwon created SPARK-61156:
------------------------------------
Summary: createDataFrame of local rows with a year-month or
calendar interval raises the wrong error
Key: SPARK-61156
URL: https://issues.apache.org/jira/browse/SPARK-61156
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
createDataFrame of local rows whose schema has a YEAR-MONTH or CALENDAR
interval type raises a plain TypeError ("unsupported value type for
createDataFrame: int") for a value, and for an all-None column a
PySparkRuntimeError that carries Rust Debug text ("Unsupported type for Arrow
createDataFrame conversion: YearMonthInterval { start_field: 0, end_field: 1
}"). Repro: spark.createDataFrame([(1,)], "a interval year to month"). PySpark
raises PySparkTypeError UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION
("YearMonthIntervalType(0, 1) is not supported in conversion to Arrow.", and
"CalendarIntervalType() ...") for values and None alike, top-level, nested and
atomic.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
