[
https://issues.apache.org/jira/browse/SPARK-45433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun closed SPARK-45433.
---------------------------------
> CSV/JSON schema inference when timestamps do not match specified
> timestampFormat with only one row on each partition report error
> ---------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-45433
> URL: https://issues.apache.org/jira/browse/SPARK-45433
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.3.0, 3.4.0, 3.5.0
> Reporter: Jia Fan
> Assignee: Jia Fan
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.2, 3.5.1, 4.0.0
>
>
> CSV/JSON schema inference when timestamps do not match specified
> timestampFormat with `only one row on each partition` report error.
> {code:java}
> //eg
> val csv = spark.read.option("timestampFormat", "yyyy-MM-dd'T'HH:mm:ss")
> .option("inferSchema", true).csv(Seq("2884-06-24T02:45:51.138").toDS())
> csv.show() {code}
> {code:java}
> //error
> Caused by: java.time.format.DateTimeParseException: Text
> '2884-06-24T02:45:51.138' could not be parsed, unparsed text found at index
> 19 {code}
> This bug affect 3.3/3.4/3.5. Unlike
> https://issues.apache.org/jira/browse/SPARK-45424 , this is a different bug
> but has the same error message
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
