Dandandan commented on a change in pull request #9043:
URL: https://github.com/apache/arrow/pull/9043#discussion_r549804455
##########
File path: rust/datafusion/src/physical_plan/planner.rs
##########
@@ -110,6 +111,16 @@ impl DefaultPhysicalPlanner {
// leaf node, children cannot be replaced
Ok(plan.clone())
} else {
+ // wrap filter in coalesce batches
+ let plan = if plan.as_any().downcast_ref::<FilterExec>().is_some()
{
+ let target_batch_size = ctx_state.config.batch_size;
+ Arc::new(CoalesceBatchesExec::new(plan.clone(),
target_batch_size))
Review comment:
I guess this could at some point be part of a cost based on optimization
based on the nr. of rows and selectivity of the filters?
Would it also make sense to wrap joins in the coalescebatchexec, as it can
also reduce/increase the size of the batches? E.g. what is/would be the effect
on tcph query 5?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
