joseph-isaacs commented on issue #25450:
URL: https://github.com/apache/datafusion/issues/25450#issuecomment-6450195754
Sounds good. I think we can make this change with a migration guide.
Two open questions about extensions pushdown safety:
1. I would think we want to disallow the optimiser from moving these
expression past other nodes, not leave it up to the `TableProvider/FileSource`.
a. `1 / x` this does feel like something that other parts of the
optimiser will care about, so I am not sure if we want a `is_safe_to_move` (on
method or a `is_fallible` method (and make `is_safe_to_move == !is_volatile &&
!is_fallible`). These are both on Expr (and ScalarUDF, etc).
2. We need a way to have external crates extend the `TableProvider` and/or
the `FileSource `. I think there are two possible options:
a. Add a `{TableProvider/FileSource}Factory` adaptor this will override
the methods on these traits allowing third party crates to add more
functionality. This could be a session registered adaptor.
b. Add a extra registry on the Datafusion session that is consulted by
the file format (or by the optimisation rule) to force placement style movement
of expressions (foreign to the crate implementing the FileFormat).
If we have 2. then functions-json can still use same `PhysicalExprAdapter`
trick and we could in a later design to add a way to accept unknown custom
PhysicalExpr and interp them for each specific TP/FS?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
