wgtmac commented on issue #37559: URL: https://github.com/apache/arrow/issues/37559#issuecomment-1709352541
+1 for @emkornfield 's suggestion. It would be good to have a draft design in google doc and share to the public. I have just read the paper and it simply focuses on optimization row-level filtering. However, without predicate pushdown (which is a prerequisite in this case), it still involves a lot of unnecessary I/O and filter evaluation (even using BMI on encoded values) on pages that can be filtered by page index. But these features (i.e. predicate pushdown and selection pushdown) are orthogonal, therefore I am not objecting to implement selection pushdown using BMI. We need to choose expression and selection vector which can be used in the low-level parquet-cpp library and then integrate into the arrow layer. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
