joseph-isaacs commented on issue #25450:
URL: https://github.com/apache/datafusion/issues/25450#issuecomment-5778363964
This functionally will be great for vortex! We want to able to receive most
scalar/linear functions and (partial)-aggregations [coming soon].
## Aside
Regarding
```sql
EXPLAIN SELECT "CounterID", AVG(octet_length("URL")) AS l, COUNT(*) AS c
FROM hits WHERE "URL" <> '' GROUP BY "CounterID" HAVING COUNT(*) > 100000 ORDER
BY l DESC LIMIT 25;
```
if aggregations could push down part of expr in `agg` (here scalar fn
`octet_length("URL")`) using `try_swapping_with_projection`, then vortex could
consume the filter and also octet_len. This might not need an API change,
depending on `RepartitionExec` handling.
## Proposed solution
Regarding `supports_leaf_expressions` I have a few thoughts:
- There is likely more info needed other than ("absorb it" and "leave it
above me") since some expr are sub-linear cost (`col + 1`[1]) while others are
negative cost (`octet_len`[2]) if pushed down and other are neutral. sub-linear
costs might mean that its faster to return `col` apply row reducing operator (a
filter that cannot be pushed down) and then apply `x + 1` in datafusion..
- this is likely expr slice sensitive if there are two expensive
expressions over a column then we might accept them if they have similar
compute patterns or not.
[1] We can at lower cost than two passes over that data materialise this.
[2] We only load in the row lengths
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
