Max Gekk created SPARK-57783:
--------------------------------
Summary: Document the TIME data type as unsupported in SparkR
Key: SPARK-57783
URL: https://issues.apache.org/jira/browse/SPARK-57783
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 4.3.0
Environment: h2. Background
Umbrella: SPARK-57550 (Extend support for the TIME data type), SPIP SPARK-51162.
The TIME data type ({{TimeType}}) is being readied to enable by default
({{spark.sql.timeType.enabled}}). PySpark (SPARK-57565 / SPARK-57567 /
SPARK-57574 /
SPARK-57696) and Spark Connect (SPARK-57566) coverage are tracked, but there is
no
equivalent work for SparkR.
h2. Problem
SparkR's R <-> JVM serializer handles only the legacy {{java.sql}} datetime
types:
{code:scala}
// core/src/main/scala/org/apache/spark/api/r/SerDe.scala
import java.sql.{Date, Time, Timestamp}
{code}
{{writeObject}} matches {{java.sql.Date}} / {{java.sql.Time}} /
{{java.sql.Timestamp}}
but not {{java.time.LocalTime}}, which is the external type of {{TimeType}}. As
a result,
collecting or creating TIME columns through SparkR (non-Arrow path) falls
through to the
{{case _ => "Invalid type"}} branch, and there is no SparkR TIME test coverage.
h2. Scope
Rather than add full SparkR TIME support, document TIME as unsupported in
SparkR, matching
how Hive interop is already marked. Specifically:
* In {{docs/sql-ref-datatypes.md}}, mark {{TimeType}} as "Not supported" in the
R <-> Spark
type mapping table (the same way {{TimeType}} is shown as "Not supported" for
Hive).
* Optionally add a short note in the SparkR docs / migration guide.
h2. Acceptance
* The data types reference clearly states TIME is not supported in SparkR.
* No behavioral code change in SparkR is required by this ticket.
Reporter: Max Gekk
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
