MaxGekk commented on code in PR #56854:
URL: https://github.com/apache/spark/pull/56854#discussion_r3489805481
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/XmlInferSchema.scala:
##########
@@ -641,8 +650,44 @@ object XmlInferSchema {
(t1: DataType, t2: DataType): DataType = {
// TODO: Optimise this logic.
+ // AnyTimestampNanoType extends DatetimeType but is not covered by
findWiderDateTimeType;
+ // handle it first to avoid a MatchError inside
TypeCoercion.findTightestCommonType.
+ // StructType and ArrayType are also handled here so that compatibleType
is used recursively
+ // for nested field types, preserving the nano-timestamp downgrade logic
at all nesting levels.
+ // (TypeCoercion.findTightestCommonType handles same-structure
StructType/ArrayType via
+ // findTypeForComplex, which calls findWiderDateTimeType and would bypass
the custom logic.)
+ (t1, t2) match {
+ case (n1: TimestampNTZNanosType, n2: TimestampNTZNanosType) =>
+ return TimestampNTZNanosType(math.max(n1.precision, n2.precision))
+ case (n1: TimestampLTZNanosType, n2: TimestampLTZNanosType) =>
+ return TimestampLTZNanosType(math.max(n1.precision, n2.precision))
+ case (_: TimestampNTZNanosType, TimestampNTZType) |
+ (TimestampNTZType, _: TimestampNTZNanosType) =>
+ return TimestampNTZType
+ case (_: AnyTimestampNanoType, _) | (_, _: AnyTimestampNanoType) =>
Review Comment:
This catch-all matches a nano type paired with *any* type, and it precedes
the `StructType`/`ArrayType` cases below. So a field inferred as a nano
timestamp in one record and a non-datetime (`Long`, `Decimal`, `String`,
`Struct`, `Array`) in another widens to `TimestampType`, whereas the
micro-precision path and every other incompatible combination fall through to
`case (_, _) => StringType` (line 747). The non-timestamp rows would then fail
/ become `null` at read.
The `MatchError`-avoidance motive only needs nano-vs-*datetime* to be
intercepted before `findTightestCommonType`, so narrowing the other side to
`DatetimeType` keeps that protection while letting nano-vs-non-datetime fall
through to the `StringType` default, matching the micro path:
```suggestion
case (_: AnyTimestampNanoType, _: DatetimeType) | (_: DatetimeType, _:
AnyTimestampNanoType) =>
```
(`DatetimeType` is already in scope via the `org.apache.spark.sql.types._`
import.)
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/StaxXmlParser.scala:
##########
@@ -601,6 +601,9 @@ class StaxXmlParser(
Decimal(decimalParser(datum), dt.precision, dt.scale)
case _: TimestampType => parseXmlTimestamp(datum, options)
case _: TimestampNTZType =>
timestampNTZFormatter.parseWithoutTimeZone(datum, false)
+ case t: TimestampLTZNanosType => timestampFormatter.parseNanos(datum,
t.precision)
+ case t: TimestampNTZNanosType =>
+ timestampNTZFormatter.parseWithoutTimeZoneNanos(datum, t.precision)
Review Comment:
This calls the 2-arg `parseWithoutTimeZoneNanos(s, precision)` overload,
which fixes `allowTimeZone = true` (`TimestampFormatter.scala:240`). So a
zone-bearing value (e.g. `2025-06-15T12:30:45.123456789+05:00`) read into a
`TIMESTAMP_NTZ(p)` column is silently accepted with the zone dropped —
`extractNanosNTZ` only throws when `!allowTimeZone`.
The adjacent micro `TimestampNTZType` path (line 603,
`parseWithoutTimeZone(datum, false)`), the inference path
(`parseWithoutTimeZoneNanosOptional(field, 9, false)`), and the CSV analogue
(`UnivocityParser.scala`, `parseWithoutTimeZoneNanos(datum, t.precision,
false)`) all pass `false` and reject zoned input as a malformed record. Suggest
matching that contract here:
```suggestion
timestampNTZFormatter.parseWithoutTimeZoneNanos(datum,
t.precision, false)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
