[
https://issues.apache.org/jira/browse/SPARK-57783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-57783:
-----------------------------------
Labels: pull-request-available (was: )
> Document the TIME data type as unsupported in SparkR
> ----------------------------------------------------
>
> Key: SPARK-57783
> URL: https://issues.apache.org/jira/browse/SPARK-57783
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 4.3.0
> Environment: h2. Background
> Umbrella: SPARK-57550 (Extend support for the TIME data type), SPIP
> SPARK-51162.
> The TIME data type ({{TimeType}}) is being readied to enable by default
> ({{spark.sql.timeType.enabled}}). PySpark (SPARK-57565 / SPARK-57567 /
> SPARK-57574 /
> SPARK-57696) and Spark Connect (SPARK-57566) coverage are tracked, but there
> is no
> equivalent work for SparkR.
> h2. Problem
> SparkR's R <-> JVM serializer handles only the legacy {{java.sql}} datetime
> types:
> {code:scala}
> // core/src/main/scala/org/apache/spark/api/r/SerDe.scala
> import java.sql.{Date, Time, Timestamp}
> {code}
> {{writeObject}} matches {{java.sql.Date}} / {{java.sql.Time}} /
> {{java.sql.Timestamp}}
> but not {{java.time.LocalTime}}, which is the external type of {{TimeType}}.
> As a result,
> collecting or creating TIME columns through SparkR (non-Arrow path) falls
> through to the
> {{case _ => "Invalid type"}} branch, and there is no SparkR TIME test
> coverage.
> h2. Scope
> Rather than add full SparkR TIME support, document TIME as unsupported in
> SparkR, matching
> how Hive interop is already marked. Specifically:
> * In {{docs/sql-ref-datatypes.md}}, mark {{TimeType}} as "Not supported" in
> the R <-> Spark
> type mapping table (the same way {{TimeType}} is shown as "Not supported"
> for Hive).
> * Optionally add a short note in the SparkR docs / migration guide.
> h2. Acceptance
> * The data types reference clearly states TIME is not supported in SparkR.
> * No behavioral code change in SparkR is required by this ticket.
> Reporter: Max Gekk
> Priority: Major
> Labels: pull-request-available
>
> h2. Background
> Umbrella: SPARK-57550 (Extend support for the TIME data type), SPIP
> SPARK-51162.
> The TIME data type ({{TimeType}}) is being readied to enable by default
> ({{spark.sql.timeType.enabled}}). PySpark (SPARK-57565 / SPARK-57567 /
> SPARK-57574 /
> SPARK-57696) and Spark Connect (SPARK-57566) coverage are tracked, but there
> is no
> equivalent work for SparkR.
> h2. Problem
> SparkR's R <-> JVM serializer handles only the legacy {{java.sql}} datetime
> types:
> {code:scala}
> // core/src/main/scala/org/apache/spark/api/r/SerDe.scala
> import java.sql.{Date, Time, Timestamp}
> {code}
> {{writeObject}} matches {{java.sql.Date}} / {{java.sql.Time}} /
> {{java.sql.Timestamp}}
> but not {{java.time.LocalTime}}, which is the external type of {{TimeType}}.
> As a result,
> collecting or creating TIME columns through SparkR (non-Arrow path) falls
> through to the
> {{case _ => "Invalid type"}} branch, and there is no SparkR TIME test
> coverage.
> h2. Scope
> Rather than add full SparkR TIME support, document TIME as unsupported in
> SparkR, matching
> how Hive interop is already marked. Specifically:
> * In {{docs/sql-ref-datatypes.md}}, mark {{TimeType}} as "Not supported" in
> the R <-> Spark
> type mapping table (the same way {{TimeType}} is shown as "Not supported"
> for Hive).
> * Optionally add a short note in the SparkR docs / migration guide.
> h2. Acceptance
> * The data types reference clearly states TIME is not supported in SparkR.
> * No behavioral code change in SparkR is required by this ticket.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
