mikewolfd opened a new issue, #27023:
URL: https://github.com/apache/datafusion/issues/27023
### Describe the bug
For a `UNION ALL` whose inputs have different column types,
`DataFrame::schema()` (the plan as built by the SQL planner) reports the
**first input's** type, but the batches the query produces have the type chosen
later by the `TypeCoercion` analyzer. A caller that checks results against the
declared schema sees a type change between planning and execution.
#14380 describes the root cause (the `Union` node takes the left input's
schema and is only coerced later). This issue is about a user-visible effect,
made more common by a 54.0 change: union coercion now uses
`type_union_coercion`, which picks the finer timestamp unit. 53.1 kept the left
unit in the case below, so declared and executed types used to agree.
### To Reproduce
```rust
use datafusion::prelude::*;
#[tokio::main]
async fn main() -> datafusion::error::Result<()> {
let ctx = SessionContext::new();
for sql in [
// Timestamp(us, UTC) UNION ALL CAST(NULL AS TIMESTAMP), i.e.
Timestamp(ns, None)
"SELECT t FROM (SELECT arrow_cast(arrow_cast(1,
'Timestamp(Microsecond, None)'), \
'Timestamp(Microsecond, Some(\"UTC\"))') AS t UNION ALL SELECT
CAST(NULL AS TIMESTAMP) AS t) u",
// Utf8 UNION ALL LargeUtf8
"SELECT s FROM (SELECT arrow_cast('a', 'Utf8') AS s \
UNION ALL SELECT arrow_cast('b', 'LargeUtf8') AS s) u",
] {
let df = ctx.sql(sql).await?;
let declared = df.schema().field(0).data_type().clone();
let executed =
df.collect().await?[0].schema().field(0).data_type().clone();
println!("declared={declared:?} executed={executed:?}");
}
Ok(())
}
```
Output on 55.2.0:
```
declared=Timestamp(Microsecond, Some("UTC")) executed=Timestamp(Nanosecond,
Some("UTC"))
declared=Utf8 executed=LargeUtf8
```
Output on 53.1.0:
```
declared=Timestamp(Microsecond, Some("UTC")) executed=Timestamp(Microsecond,
Some("UTC"))
declared=Utf8 executed=LargeUtf8
```
(`Utf8` UNION ALL `Utf8View` behaves the same way: declared `Utf8`, executed
`Utf8View`, on both versions.)
### Expected behavior
`DataFrame::schema()` should describe the batches the query returns: either
the `Union` is built with the coerced schema (#14380), or the declared schema
reflects the analyzer's coercion.
### Additional context
Found while upgrading an embedded engine from 53.1 to 55.2. Its export path
checks every executed batch against the declared schema, and this mismatch
refused queries that had passed on 53.1. The workaround is to cast every union
branch to one explicit type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
