nuno-faria opened a new issue, #19875: URL: https://github.com/apache/datafusion/issues/19875
### Is your feature request related to a problem or challenge? When scanning a Parquet file, we can see how many rows have been matched by the page index. However, if we are not familiar with its structure, we don't know how many pages have been retrieved. Additionally, even in the same file, some pages have different sizes, for example: ```sql -- sample parquet file created with copy (select i from generate_series(1, 10000000) t(i)) to 't.parquet'; ``` With `i=1`: ```sql predicate=i@0 = 1 page_index_rows_pruned=1.05 M total → 20.48 K matched ``` With `i=123456`: ```sql predicate=i@0 = 123456 page_index_rows_pruned=1.05 M total → 8.19 K matched ``` In this case, both queries retrieved just one page, even though one of them has 2.5x more rows (dictionary size was exceeded). On one hand I would like to have access to this information in order to be able to see if some file/query needs to be optimized. On the other the `DataSourceExec` operator already contains a lot of metrics, so I would like to put this up for discussion before opening up a PR. ### Describe the solution you'd like A metric similar to `page_index_rows_pruned`. For example: ```sql page_index_pages_pruned=497 total → 1 matched ``` ### Describe alternatives you've considered Manually inspecting the file metadata. ### Additional context The number of matched pages is fairly easy to retrieve, so it wouldn't add a lot of code. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
