sadikovi commented on PR #36871: URL: https://github.com/apache/spark/pull/36871#issuecomment-1192185550
@cloud-fan @Jonathancui123 Wouldn't this patch cause correctness issues? This is what I found when working on https://github.com/apache/spark/pull/37147: The "SPARK-39469: Infer schema for date type" test in CSVSuite highlights the issue when run together with my patch which attempts to forbid users to fall back to the default parser when the timestamp format is provided as it could lead to correctness issues. Because "1765-03-28" does not match timestamp pattern and the column is inferred as TimestampType, it should be returned as `null`. However, in the test it is returned as `1765-03-28 00:00:00.0`. This is at very least is confusing. Can someone take a look and clarify this one? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
