matthewbayer opened a new pull request, #59406: URL: https://github.com/apache/spark/pull/59406
### What changes were proposed in this pull request? For XML date reads with an explicit `dateFormat` and `spark.sql.legacy.timeParserPolicy=LEGACY`, try ISO date parsing before the existing configured-format parser. Use an XML-specific helper shared by reading and inference, preserving spark-xml's `java.sql.Date` conversion for historical dates. No shared date formatter or global configuration is changed. ### Why are the changes needed? [SPARK-60201](https://issues.apache.org/jira/browse/SPARK-60201): [spark-xml 0.17.0](https://github.com/databricks/spark-xml/blob/v0.17.0/src/main/scala/com/databricks/spark/xml/util/TypeCast.scala) tried ISO dates before custom formats. The built-in XML reader instead interprets `2024-03-04` with `dateFormat=yyyy-dd-MM` as April 3 rather than March 4. Both interpretations parse successfully, so `FAILFAST` does not detect the changed value. This provides a compatibility path under `LEGACY` without changing the default parser policy. ### Does this PR introduce _any_ user-facing change? Yes. With `LEGACY`, the example above returns `2024-03-04` instead of `2024-04-03`, matching spark-xml's ISO-first precedence. `CORRECTED` and `EXCEPTION` still return `2024-04-03`. Non-ISO custom-format fallback and locale handling, reads without an explicit `dateFormat`, inference's existing `preferDate` rules, XML writes, timestamps, and other formats remain unchanged. This is a precedence fix, not a broader rewrite of legacy date parsing. ### How was this patch tested? - The new `from_xml` regression fails before the fix and passes afterward. - All 300 tests passed across `XmlExpressionsSuite`, `XmlInferSchemaTypeCastingSuite`, `XmlSuite`, `XmlInferSchemaSuite`, and `XmlFunctionsSuite` (one existing ignored test). New coverage includes file reads, all three parser policies, `PERMISSIVE`/`FAILFAST`, custom-format fallback, historical dates, offsets, and locale handling. - Catalyst main/test and SQL-test Scalastyle passed. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
