cloud-fan commented on code in PR #56966:
URL: https://github.com/apache/spark/pull/56966#discussion_r3538270890
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/XmlInferSchema.scala:
##########
@@ -682,16 +682,20 @@ class XmlInferSchema(private val options: XmlOptions,
private val caseSensitive:
} else if (isTimeTypeEnabled && isTime(field)) {
// TIME is tried ahead of date/timestamp, matching the ordering of the
previous cascade.
TimeType(TimeType.DEFAULT_PRECISION)
- } else {
+ } else if (options.preferDate) {
tryParseDate(field)
+ } else {
+ tryParseTimestampNTZ(field)
}
}
private def tryParseTime(field: String): DataType = {
if (isTimeTypeEnabled && isTime(field)) {
TimeType(TimeType.DEFAULT_PRECISION)
- } else {
+ } else if (options.preferDate) {
tryParseDate(field)
+ } else {
+ tryParseTimestampNTZ(field)
}
Review Comment:
Good catch, both gaps are now covered in `XmlInferSchemaTypeCastingSuite`:
- `TIME inference precedes the preferDate gate` — asserts a TIME-shaped
value infers as `TimeType` for both `preferDate` values, via both the fresh
path (`tryParseDouble`) and the incremental temporal re-entry (`tryParseTime`).
This guards against moving the TIME guard inside the `preferDate` branch.
- `preferDate gates the incremental temporal re-entry (tryParseTime)` —
asserts a `Date`-so-far field seeing a date-only value yields `DateType` under
`preferDate=true` and `TimestampType` under `preferDate=false`, exercising the
`tryParseTime` branch that the incremental-vs-legacy parity test (default
`preferDate=true`) did not reach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
