ole-baranov opened a new pull request, #26190:
URL: https://github.com/apache/datafusion/pull/26190
`ensure_schema_compatibility` and `apply_projection` looked up every field
of the Substrait base schema with a linear scan of the table schema, making
consumption of a read over a wide table O(m*n). Build a name->index map of the
table schema once instead. Missing or duplicated names fall back to the
existing DFSchema lookups, so errors and disambiguation are unchanged.
Add a `consume_wide_read` benchmark. Consume time for `SELECT c0` over a
table with N columns:
columns before after
100 58 us 25.5 us
1000 2.33 ms 276 us
10000 211 ms 3.16 ms
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
