andygrove commented on code in PR #2852:
URL: 
https://github.com/apache/datafusion-ballista/pull/2852#discussion_r4643077828


##########
ballista/scheduler/src/cluster/mod.rs:
##########
@@ -621,11 +663,13 @@ fn bind_one(
     // one vcore and leave the rest for other stages' tasks on this executor.
     let vcores_consumed = if is_collapse {
         1
+    } else if is_unsplittable {
+        (input_partition_ids.len() as u32).min(budget.vcores)

Review Comment:
   Not for this PR, but a packed union stage takes the first executor with any 
free vcore, the same as a collapse stage. A wide stage could end up on a 
1-vcore executor while another has plenty free. Maybe worth noting next to 
#2854 that packed stages should prefer the executor with the most free vcores?



##########
ballista/scheduler/src/state/task_manager.rs:
##########
@@ -171,6 +171,9 @@ pub struct JobInfoCache {
     pub execution_graph: Arc<RwLock<ExecutionGraphBox>>,
     /// Cached job status for quick access.
     pub status: Option<job_status::Status>,
+    /// The resolver the job's tasks are restricted with, so binding can tell
+    /// which stages a task may run only part of.
+    pub(crate) partition_sliceable_resolver: Option<Arc<dyn 
PartitionSliceableResolver>>,

Review Comment:
   Small API note. `JobInfoCache` and `RunningStage` are both reachable through 
`ballista_scheduler::state` and had only `pub` fields until now. So the new 
`pub(crate)` fields stop downstream code from building them with a struct 
literal. I doubt anyone does, and `new` covers it. Could we either add the `api 
change` label with a one-line upgrade note, or mark both `#[non_exhaustive]` so 
later fields don't break anything?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to