andygrove commented on code in PR #1647:
URL:
https://github.com/apache/datafusion-ballista/pull/1647#discussion_r3213579614
##########
ballista/scheduler/src/planner.rs:
##########
@@ -125,17 +140,76 @@ impl DefaultDistributedPlanner {
job_id: &'a str,
execution_plan: Arc<dyn ExecutionPlan>,
config: &ConfigOptions,
+ broadcast_threshold_bytes: usize,
Review Comment:
Done in 0cbe593d. Dropped the parameter and let maybe_promote_to_broadcast
read the threshold from the &ConfigOptions it already gets.
##########
ballista/scheduler/src/state/aqe/mod.rs:
##########
@@ -42,6 +42,13 @@ use std::sync::Arc;
use std::time::{SystemTime, UNIX_EPOCH};
use std::vec;
+// TODO: the AQE planner does not yet apply the broadcast-join lowering
Review Comment:
Yeah still relevant. AdaptivePlanner runs DefaultPhysicalPlanner with a list
of PhysicalOptimizerRules and never goes through
DefaultDistributedPlanner::plan_query_stages_internal, so neither
maybe_promote_to_broadcast nor the HashJoinExec(CollectLeft) shuffle lowering
fire under AQE. Tightened the comment to spell that out in 0cbe593d. Moving the
lowering into an AQE optimizer rule is the follow-up you flagged separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
