Max Gekk created SPARK-57555:
--------------------------------
Summary: Support the TIME data type in the built-in JDBC data
source
Key: SPARK-57555
URL: https://issues.apache.org/jira/browse/SPARK-57555
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 4.3.0
Reporter: Max Gekk
h2. What
Add native {{TIME}} support to the built-in JDBC data source so that SQL
{{TIME}} columns
in external databases map to Spark's {{TimeType}} on read, and {{TimeType}}
columns map to
SQL {{TIME}} on write.
h2. Why
This is the core motivation of the SPIP (SPARK-51162, Q5): simplify migration
from
PostgreSQL, Snowflake, Redshift, Teradata, DB2, etc. Today JDBC cannot
represent TIME:
* On read, {{JdbcUtils.getCatalystType}} maps {{java.sql.Types.TIME}} to a
*timestamp*
type, not {{TimeType}}.
* On write, {{JdbcUtils.getJdbcType}} has no {{case TimeType}}.
* No dialect maps TIME.
h2. Scope
* {{JdbcUtils.getCatalystType}}: map {{java.sql.Types.TIME}} to {{TimeType}}
(gated by the
TIME feature flag and a JDBC option for back-compat).
* {{JdbcUtils.getJdbcType}}: add {{case TimeType => JdbcType("TIME",
java.sql.Types.TIME)}}.
* Value conversion in {{JdbcUtils}} (read {{ResultSet.getTime}}/getObject ->
nanos-of-day;
write nanos-of-day -> {{java.sql.Time}}/{{LocalTime}}), preserving sub-second
precision
where the driver allows.
* Per-dialect overrides where the SQL type name/precision differs
(PostgreSQL, MySQL, MS SQL Server, Oracle, DB2, H2) in {{JdbcDialects}} and
dialect
classes. Consider splitting per-dialect work into sub-tasks.
h2. Compatibility
* A back-compat switch so existing pipelines that read TIME-as-timestamp do not
break;
document the new behavior.
h2. Acceptance criteria
* Read/write round-trip of TIME columns against the JDBC integration-test
databases
(Postgres/MySQL/MS SQL/Oracle/DB2) preserves values.
* Schema inference reports {{TimeType}} for TIME columns.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
