Dandandan commented on issue #20325: URL: https://github.com/apache/datafusion/issues/20325#issuecomment-3912873585
Just thinking out loud: I think what also would help most ClickBench if instead of materializing the Parquet data before evaluating the filter, the filter could be evaluated _during decoding_. We could implement Parquet decoders that push down e.g. scalar (in)equality that produces only the arrow array for the rows matching this predicate (e.g. `<> ''`) and a boolean mask (when required) for the matching rows. This would * save overhead of decoding / copy / converting rows to e.g. string views * save time post-filtering * save memory overhead to cache the (unfiltered) column The downside mainly being is that it only works for the predicates implemented (e.g. equality / inequality), but it seems it would be general enough to be worth it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
