yordan-pavlov commented on a change in pull request #9704:
URL: https://github.com/apache/arrow/pull/9704#discussion_r593965090
##########
File path: rust/datafusion/src/physical_plan/parquet.rs
##########
@@ -645,8 +654,29 @@ fn build_predicate_expression(
use crate::logical_plan;
// predicate expression can only be a binary expression
let (left, op, right) = match expr {
- Expr::BinaryExpr { left, op, right } => (left, *op, right),
+ Expr::BinaryExpr { left, op, right } => (left.clone(), *op,
right.clone()),
+ Expr::Between {
+ expr,
+ negated,
+ low,
+ high,
+ } if !negated // NOT BETWEEN not supported
+ => {
+ let left = Box::new(Expr::BinaryExpr {
+ left: expr.clone(),
+ op: Operator::GtEq,
+ right: low.clone(),
+ });
+ let right = Box::new(Expr::BinaryExpr {
+ left: expr.clone(),
+ op: Operator::LtEq,
+ right: high.clone(),
+ });
+
+ (left, Operator::And, right)
+ }
_ => {
+ debug!("Filter expression not supported in ParquetExec {:?}",
expr);
Review comment:
yes, filter push-down into parquet works best when the filter is over
sorted data, if the data is randomly distributed across files it can't do much
for performance as you have found
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
