zhuqi-lucas opened a new issue, #26358: URL: https://github.com/apache/datafusion/issues/26358
### Is your feature request related to a problem or challenge? `UnionExec` aligns its inputs to the merged union schema. When an input differs only in nullability, it wraps that input in a `ProjectionExec` of `CastExpr::new_with_target_field`, which keeps the data type and changes only the field's nullability. `CastExpr`'s `Display` prints only the data type, so in EXPLAIN this cast looks like a no-op: ``` ProjectionExec: expr=[..., CAST(trf_id@11 AS Int32) as trf_id, ...] ``` `trf_id` is already `Int32`. The only thing this cast does is make the column nullable, because another union input is nullable. Nothing in the plan says that, so a reader has to go back to the source code to see why the cast exists. We hit this when we changed one table's column from nullable to `NOT NULL`. A new same-type cast showed up in every plan that unions that table with a nullable input. The same thing happens in DataFusion's own plans, for example in `cte.slt`: ``` ProjectionExec: expr=[CAST(1 AS Int64) as id] ``` ### Describe the solution you'd like When the cast has an explicit target field that is nullable, print the nullability the way schemas already do, with `;N`: ``` ProjectionExec: expr=[CAST(1 AS Int64;N) as id] ``` Casts built from a bare type (`CastExpr::new`) are unchanged. Their nullability follows the input, so there is nothing extra to show. `fmt_sql` is also unchanged. I have a small patch ready. It updates goldens in 6 slt files, 1 insta snapshot and the pruning predicate unit tests. In every one of them the only change is the added `;N`. ### Describe alternatives you've considered - Rely on `datafusion.explain.show_schema`. That shows the projection's output schema, but it is verbose and off by default. It also doesn't tie the nullability to the cast that changed it. - Print `;N` only when the nullability actually changes. `Display` has no access to the input schema, so it can't tell. - Use a different notation, for example `CAST(a AS Int32 NULL)`. `;N` matches the existing schema display, but I'm happy to change it. ### Additional context The patch also shows a case that was hard to see before. `rewrite_expr_to_prunable` rebuilds a plain `CAST(col AS T)` as `CastExpr::new_with_target_field(.., cast.target_field(), None)`. For a plain cast, `target_field()` is a placeholder nullable field. So the rewrite turns a cast whose nullability follows its input into one that is always nullable, and it drops the original cast options (`None`). This is harmless for pruning, since statistics columns are nullable anyway. It is visible now as `CAST(c1_min@0 AS Int64;N)` in the pruning predicate display. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
