alamb opened a new issue, #10774: URL: https://github.com/apache/arrow-rs/issues/10774
### Is your feature request related to a problem or challenge? While working on https://github.com/apache/datafusion/pull/24426#issuecomment-5342167688 I am profiling ClickBench Q25 (the one query that is going slower for this setup) ```sql SET datafusion.execution.parquet.binary_as_string = true; CREATE EXTERNAL TABLE hits_raw STORED AS PARQUET LOCATION '/Users/andrewlamb/Software/datafusion/benchmarks/data/hits_partitioned/'; CREATE VIEW hits AS SELECT * EXCEPT ("EventDate"), CAST(CAST("EventDate" AS INTEGER) AS DATE) AS "EventDate" FROM hits_raw; SET datafusion.execution.parquet.pushdown_filters = true; SELECT "SearchPhrase" FROM hits WHERE "SearchPhrase" <> '' ORDER BY "SearchPhrase" LIMIT 10; ``` About 5% of the overall time is going to reassembling the correct rows in the cached array reader <img width="1920" height="957" alt="Image" src="https://github.com/user-attachments/assets/528a7da5-7d7e-4952-a515-385e32db9cbb"; /> ### Describe the solution you'd like Make it faster ### Describe alternatives you've considered It looks like the CachedArrayreader is currently using the classic "filter + concat" operation https://github.com/apache/arrow-rs/blob/03b940e1c7e3e7af9b636940fb9f1ee195891a1f/parquet/src/arrow/array_reader/cached_array_reader.rs#L315-L316 https://github.com/apache/arrow-rs/blob/03b940e1c7e3e7af9b636940fb9f1ee195891a1f/parquet/src/arrow/array_reader/cached_array_reader.rs#L335-L336 Which is what the `coalesce` kernel is designed to improve (avoids a copy)(: https://docs.rs/arrow/latest/arrow/compute/kernels/coalesce/index.html So I think we could switch the CachedArrayReader to use coalesce to improve the performance ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
