HyukjinKwon commented on code in PR #59404:
URL: https://github.com/apache/spark/pull/59404#discussion_r4277892710
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/StaxXmlParser.scala:
##########
@@ -1172,6 +1189,73 @@ class XmlTokenizer(
}
object StaxXmlParser {
+ private val legacyZonedTimestampFormatters = Seq(
+ DateTimeFormatter.ISO_OFFSET_DATE_TIME,
+ DateTimeFormatter.ISO_INSTANT)
+
+ private val legacyLocalTimestampFormatters = Seq(
+ new DateTimeFormatterBuilder()
+ .parseCaseInsensitive()
+ .append(DateTimeFormatter.ISO_LOCAL_DATE)
+ .appendLiteral(' ')
+ .append(DateTimeFormatter.ISO_LOCAL_TIME)
+ .toFormatter()
Review Comment:
**Nit (P3):** `DateTimeFormatterBuilder.toFormatter()` uses
`ResolverStyle.SMART`, and appending `ISO_LOCAL_DATE` does not carry over its
STRICT resolver. This space-separated formatter therefore clamps impossible
dates instead of rejecting them, while its `T`-separated sibling
`ISO_LOCAL_DATE_TIME` stays STRICT. Checked with jshell:
- `2024-02-30 12:34:56` resolves to `2024-02-29T12:34:56Z`.
- `2023-04-31 00:00:00` resolves to `2023-04-30T00:00:00Z`.
- `2024-02-29 24:00:00` resolves to `2024-03-01T00:00:00Z`.
The `T` forms of all three are rejected. Under LEGACY, such values arrive
here when there is no `timestampFormat`, or when the value does not match it.
They now read as a different valid instant in both PERMISSIVE and FAILFAST, and
inference reports TIMESTAMP. Before this PR they were rejected, and the PR
description says malformed ISO values are rejected. The rejection test only
uses `T` values with an offset, so it does not catch this.
Adding `.withResolverStyle(ResolverStyle.STRICT)` here matches the `T` path.
Valid values such as `2024-02-29 12:34:56.123456` still parse.
**Verification:**
- **Regression:** LEGACY XML reads and inference reject space-separated
impossible dates without a matching timestampFormat, in PERMISSIVE and FAILFAST.
- **Compatibility:** Existing LEGACY expectations for valid space-separated
and 'T'-separated zone-less ISO values, including fraction preservation, still
hold.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
