alamb commented on issue #8078: URL: https://github.com/apache/datafusion/issues/8078#issuecomment-3392221997
> TLDR is it's expensive to randomly sample compressed columnar storage like Parquet, but if you store a pre-sampled portion of the file e.g. as the last row group you can get very good estimates for all kinds of things (filter selectivity, cardinality of any column, etc.) Sounds like a good usecase for https://datafusion.apache.org/blog/2025/07/14/user-defined-parquet-indexes/ 🤔 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
