Braedon-Wooding-Displayr commented on PR #24441:
URL: https://github.com/apache/datafusion/pull/24441#issuecomment-5342821699
Note: my optimisations to `SELECT *` could apply to this codebase as well
(if interested).
The idea is that in lots of cases we don't need to expand the projection of
* until physical planning, this is because it doesn't effect anything.
This means that `*` doesn't become a performance cost. In my case I have a
special command `EXPORT *` just because it restricts the edgecases to consider
(no nesting).
But conceptually an expression like;
```sql
SELECT x from (
SELECT * from foo
)
```
Still doesn't need to expand `*` it just needs to ensure that `*` holds `x`
that is effectively it just needs in a "logical" plan to be something like;
```sql
SELECT x from (
SELECT x, * from foo
)
```
(I'm saying "something like" because obviously this would duplicate the `x`
column but hopefully you understand what I mean).
This results in massive performance improvements for wide queries because a
lot of the cost is in planning and quadratic costs (well that and parquet
metadata is terrible for wide formats, but that's something ultimately
unfixable without caching / having a custom footer through something like
delta/iceberg/... though they also don't handle it great).
In my case I ended up with sublinear scaling which is nice, and physical
plan is very cheap. Now the risk here is that there might be some
optimisations that rely on `*` expanding but honestly I can't really imagine
that to be true.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
