[
https://issues.apache.org/jira/browse/SPARK-46769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenchen Fan resolved SPARK-46769.
---------------------------------
Fix Version/s: 3.5.1
4.0.0
Resolution: Fixed
Issue resolved by pull request 44800
[https://github.com/apache/spark/pull/44800]
> Fix inferring of TIMESTAMP_NTZ in CSV/JSON
> ------------------------------------------
>
> Key: SPARK-46769
> URL: https://issues.apache.org/jira/browse/SPARK-46769
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.0.0
> Reporter: Max Gekk
> Assignee: Max Gekk
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.5.1, 4.0.0
>
>
> After the PR https://github.com/apache/spark/pull/43243, the TIMESTAMP_NTZ
> type inference in CSV/JSON datasource got 2 new guards which means
> TIMESTAMP_NTZ should be inferred either if:
> 1. the SQL config `spark.sql.legacy.timeParserPolicy` is set to `LEGACY` or
> 2. `spark.sql.timestampType` is set to `TIMESTAMP_NTZ`.
> otherwise CSV/JSON should try to infer `TIMESTAMP_LTZ`.
> Both guards are unnecessary because:
> 1. when `spark.sql.legacy.timeParserPolicy` is `LEGACY` that only means Spark
> should use a legacy Java 7- parser: `FastDateFormat` or `SimpleDateFormat`.
> Both parser are applicable for parsing `TIMESTAMP_NTZ`.
> 2. when `spark.sql.timestampType` is set to `TIMESTAMP_LTZ`, it doesn't mean
> that we should skip inferring of `TIMESTAMP_NTZ` types in CSV/JSON, and try
> to parse the timestamp string value w/o time zone like
> `2024-01-19T09:10:11.123` using a LTZ format **with timezone** like
> `yyyy-MM-dd'T'HH:mm:ss.SSSXXX`. _The last one cannot match any NTZ values for
> sure._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
