[
https://issues.apache.org/jira/browse/SPARK-46769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenchen Fan updated SPARK-46769:
--------------------------------
Description: (was: After the PR
https://github.com/apache/spark/pull/43243, the TIMESTAMP_NTZ type inference in
CSV/JSON datasource got 2 new guards which means TIMESTAMP_NTZ should be
inferred either if:
1. the SQL config `spark.sql.legacy.timeParserPolicy` is set to `LEGACY` or
2. `spark.sql.timestampType` is set to `TIMESTAMP_NTZ`.
otherwise CSV/JSON should try to infer `TIMESTAMP_LTZ`.
Both guards are unnecessary because:
1. when `spark.sql.legacy.timeParserPolicy` is `LEGACY` that only means Spark
should use a legacy Java 7- parser: `FastDateFormat` or `SimpleDateFormat`.
Both parser are applicable for parsing `TIMESTAMP_NTZ`.
2. when `spark.sql.timestampType` is set to `TIMESTAMP_LTZ`, it doesn't mean
that we should skip inferring of `TIMESTAMP_NTZ` types in CSV/JSON, and try to
parse the timestamp string value w/o time zone like `2024-01-19T09:10:11.123`
using a LTZ format **with timezone** like `yyyy-MM-dd'T'HH:mm:ss.SSSXXX`. _The
last one cannot match any NTZ values for sure._)
> Refine timestamp related schema inference
> -----------------------------------------
>
> Key: SPARK-46769
> URL: https://issues.apache.org/jira/browse/SPARK-46769
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.0.0
> Reporter: Max Gekk
> Assignee: Wenchen Fan
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0, 3.5.1
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
