paleolimbot commented on issue #22079:
URL: https://github.com/apache/datafusion/issues/22079#issuecomment-4454223068
I'm partway through working on this...the "issue" is that there is a
different answer for metadata and nullability depending on what you ask:
- `Expr::Cast(Cast { field, ...}) => field.metadata()` is the target cast
field. When I added this I intended for its nullability to be ignored and its
metadata to be authoritative, but this was modified shortly afterward (I think
before DataFusion 53 was released).
- `logical_expr.to_field()` currently returns the source field's nullability
and metadata (i.e., everything about the logical target field is ignored except
its data type). This is the part causing me problems because enclosing
expressions can't detect that something is the right (extension) type (and this
happens before I can run an optimizer rule to fix it).
- `physical_expr.return_field()` currently returns `Expr::Cast(Cast { field,
...}) => field` (i.e., ignores any computation done in `to_field()`). The
answer is different if `is_default_target_field()` (which most casts are,
because they only modify data type and are created via `Cast::new()` which
defines the field in this form)...this is the version that propagates metadata
and nullability from the source. This is the version that agrees with
`logical_expr.to_field()`.
- `physical_expr.nullable()` currently returns the wider of the source +
destination nullability. In practice this frequently disagrees with
`return_field().is_nullable()` but probably few enough places call it that it
doesn't matter.
Touching return field casting causes a lot of downstream failures but I will
try to work through what should happen here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
