alamb commented on issue #27023:
URL: https://github.com/apache/datafusion/issues/27023#issuecomment-6616516002
I reproduced this and confirmed the timestamp case is a regression, so I
have marked this issue as such.
I ran the program from the description (plus a `Utf8` / `Utf8View` case)
against datafusion 53.1.0, 54.1.0, 55.2.0 and current `main` (181a87af61):
```
== datafusion 53.1.0 ==
OK declared=Timestamp(Microsecond, Some("UTC"))
executed=Timestamp(Microsecond, Some("UTC"))
MISMATCH declared=Utf8 executed=LargeUtf8
MISMATCH declared=Utf8 executed=Utf8View
== datafusion 54.1.0 ==
OK declared=Timestamp(Microsecond, Some("UTC"))
executed=Timestamp(Microsecond, Some("UTC"))
MISMATCH declared=Utf8 executed=LargeUtf8
MISMATCH declared=Utf8 executed=Utf8View
== datafusion 55.2.0 ==
MISMATCH declared=Timestamp(Microsecond, Some("UTC"))
executed=Timestamp(Nanosecond, Some("UTC"))
MISMATCH declared=Utf8 executed=LargeUtf8
MISMATCH declared=Utf8 executed=Utf8View
== datafusion main ==
MISMATCH declared=Timestamp(Microsecond, Some("UTC"))
executed=Timestamp(Nanosecond, Some("UTC"))
MISMATCH declared=Utf8 executed=LargeUtf8
MISMATCH declared=Utf8 executed=Utf8View
```
Summary:
- **Timestamp case: regression in 55.0.0** (not 54.0 as the description
guesses). It was introduced by #22759 (51673d07f8, "fix: preserve timestamp
precision when coercing mixed time units"), which changed `timeunit_coercion`
to pick the finer unit. Before that change `Timestamp(us, UTC)` UNION ALL
`Timestamp(ns, None)` coerced to `Timestamp(us, UTC)`, which happened to match
the first input and so the declared schema. Now it coerces to `Timestamp(ns,
UTC)` and the declared schema no longer matches.
- **`Utf8` / `LargeUtf8` and `Utf8` / `Utf8View` cases: not a regression.**
They mismatch on every version tested, back to 53.1.0. That is the
long-standing root cause tracked in #14380: the `Union` node is built with the
first input's schema and only coerced later by the `TypeCoercion` analyzer.
So the underlying bug is old, but #22759 widened the set of queries where it
is visible, and in particular broke the timestamp query above which passed on
53.1 and 54.1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
