[
https://issues.apache.org/jira/browse/SPARK-58825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uroš Bojanić resolved SPARK-58825.
----------------------------------
Fix Version/s: 4.3.0
Resolution: Fixed
Issue resolved by pull request 58082
[https://github.com/apache/spark/pull/58082]
> Support TimestampNTZType as a JDBC partition column
> ---------------------------------------------------
>
> Key: SPARK-58825
> URL: https://issues.apache.org/jira/browse/SPARK-58825
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: Aleksandar Trajković
> Priority: Minor
> Labels: pull-request-available
> Fix For: 4.3.0
>
>
> h3. What
> {{JDBCRelation}} restricts the {{partitionColumn}} option to numeric,
> {{DateType}}, and
> {{TimestampType}} columns (SPARK-22814). This extends that set to include
> {{TimestampNTZType}}:
> * {{verifyAndGetNormalizedPartitionColumn}} accepts {{TimestampNTZType}}.
> * Bound parsing uses {{stringToTimestampWithoutTimeZone(_, allowTimeZone =
> false)}}, so
> {{lowerBound}}/{{upperBound}} are parsed as zoneless wall-clock values.
> * Bound-to-string conversion formats the NTZ micros in UTC, so no
> session-time-zone shift is
> applied to the generated partition predicates.
> Behavior for existing numeric/{{DateType}}/{{TimestampType}} partition
> columns is unchanged.
> h3. Why
> More JDBC sources map zoneless database types (timestamp-without-time-zone) to
> {{TimestampNTZType}}. Such a column is a valid, orderable partition column,
> but today it is
> rejected with {{invalidPartitionColumnTypeError}}.
> h3. Example
> {code:scala}
> spark.read.format("jdbc")
> .option("url", url)
> .option("dbtable", "events")
> .option("partitionColumn", "ts_ntz") // TimestampNTZType column
> .option("lowerBound", "2018-07-06 00:00:00")
> .option("upperBound", "2018-07-07 00:00:00")
> .option("numPartitions", "4")
> .load()
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
