andygrove opened a new issue, #5708: URL: https://github.com/apache/datafusion-comet/issues/5708
### Describe the problem Reported during review of #5262: reading two visible values out of a sliced array allocated approximately **8 MiB**, because DataFusion normalizes the entire backing buffer before slicing. Slicing first reduced the largest allocation to 416 bytes. This is a component allocation measurement, not an end-to-end query timing. ### Describe the potential solution Slice before normalizing so the work is proportional to the visible range rather than the backing buffer. A benchmark demonstrating the allocation should land with the fix so the improvement is measurable and protected. ### Additional context Reported by @sunchao on #5262. The other allocation observation from the same review, the per-file Iceberg schema helper that built a field-id map for every task even with no required fields, was fixed in #5262 itself: presence is now tested per required id, so an empty required set does no work. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
