Matthew Bayer created SPARK-60201:
-------------------------------------
Summary: Spark 4 XML silently changes ISO date interpretation when
a custom dateFormat is configured
Key: SPARK-60201
URL: https://issues.apache.org/jira/browse/SPARK-60201
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.2.0
Reporter: Matthew Bayer
h3. Description
Migrating from \{{spark-xml}} to Spark 4’s built-in XML reader can silently
change date values when an input matches both ISO format and a configured
\{{dateFormat}}.
The previous connector tries ISO parsing first, falling back to the custom
format. Spark 4 uses the configured format directly.
For example, \{{2024-03-04}} with \{{dateFormat=yyyy-dd-MM}} changes from
*March 4* to *April 3*, without an error.
April 3 is a valid interpretation of the configured format. This report
concerns the backward-incompatible change in parsing precedence, rather than an
incorrectly implemented format.
h3. Reproduction
Compare \{{spark-xml}} 0.17.0 with Spark 4’s built-in XML reader, using the
same input and options.
{\{input.xml}}:
{code:xml}
<ROW><event_date>2024-03-04</event_date></ROW>
{code}
{code:python}
for policy in ["LEGACY", "CORRECTED", "EXCEPTION"]:
spark.conf.set("spark.sql.legacy.timeParserPolicy", policy)
result = (
spark.read.format("xml")
.option("rowTag", "ROW")
.option("dateFormat", "yyyy-dd-MM")
.option("mode", "FAILFAST")
.schema("event_date DATE")
.load("/path/to/input.xml")
.first()
)
print(policy, result.event_date)
{code}
h3. Actual behavior
|| Reader || LEGACY || CORRECTED || EXCEPTION ||
| \{{spark-xml}} 0.17.0 | \{{2024-03-04}} | \{{2024-03-04}} | \{{2024-03-04}} |
| Spark 4 built-in XML | \{{2024-04-03}} | \{{2024-04-03}} | \{{2024-04-03}} |
{\{FAILFAST}} does not protect against this change because parsing succeeds.
Retaining \{{spark.sql.legacy.timeParserPolicy=LEGACY}} also does not preserve
the previous result.
h3. Expected behavior
Provide a compatibility path that preserves the previous interpretation, or
fails explicitly instead of silently returning a different date.
A narrowly scoped solution would restore the old XML date-parsing precedence
under \{{LEGACY}}: try ISO first, then the explicitly configured custom format.
Reading and schema inference should use the same parsing rules, without
changing shared date formatters or other data sources.
h3. Impact
Existing pipelines containing ISO dates alongside a custom date-format option
can produce different dates after upgrading. Ambiguous day/month values can
affect date filters, joins, partitioning, and aggregations without failing the
build.
For ISO-only inputs, removing \{{dateFormat}} or setting it to \{{yyyy-MM-dd}}
is a workaround. That does not preserve the old behavior for inputs mixing ISO
and custom formats.
h3. Reference
The previous ISO-first behavior is implemented in [spark-xml 0.17.0’s
parseXmlDate|https://github.com/databricks/spark-xml/blob/v0.17.0/src/main/scala/com/databricks/spark/xml/util/TypeCast.scala].
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
