andygrove commented on code in PR #2904:
URL: 
https://github.com/apache/datafusion-ballista/pull/2904#discussion_r4646848431


##########
ballista/scheduler/src/scheduler_server/mod.rs:
##########
@@ -1014,6 +1016,51 @@ mod test {
         assert_event(case, job_id, metrics, MetricEventKind::Failed);
     }
 
+    /// A task that reports released vcores while it runs still completes on
+    /// its terminal report, and gives back every vcore exactly once.
+    #[tokio::test]
+    async fn running_reports_release_vcores_and_leave_the_task_running() -> 
Result<()> {

Review Comment:
   I think this test would still pass if the running refunds were ignored, 
because the terminal status returns everything and the test only checks `free 
== 4` at the end.
   
   Could it block a task after its running report and assert that free vcores 
rise, or that a second task gets bound, before the terminal status arrives?



##########
ballista/core/src/config.rs:
##########
@@ -642,6 +647,19 @@ static CONFIG_ENTRIES: LazyLock<HashMap<String, 
ConfigEntry>> = LazyLock::new(||
             DataType::UInt64,
             Some(0.to_string()),
         ),
+        ConfigEntry::new(
+            BALLISTA_SCHEDULER_RELEASE_PARTITION_VCORES.to_string(),
+            "Hands back the vcores of a task's finished partitions while its \
+             other partitions are still running, instead of holding every \
+             vcore until the whole task ends, so that other tasks can start on 
\
+             them. The task's memory pool keeps the size it was given for all \
+             of its partitions, so while a task with released vcores runs, the 
\
+             executor's task pools can add up to more than its memory. Does \
+             not apply to pipelined jobs, whose vcores are reserved per job."
+                .to_string(),
+            DataType::Boolean,
+            Some(true.to_string()),

Review Comment:
   My main concern is memory. The task's pool keeps the size it was given at 
launch after vcores are released. With `FairSpillPool` the straggler's fair 
share actually grows as the finished partitions unregister their consumers.
   
   For example, take an 8-vcore executor with 32 GB, so 4 GB per vcore. An 
8-partition task gets a 32 GB pool. Seven partitions finish, and the scheduler 
binds a 7-vcore task with a 28 GB pool on the released vcores. The straggler 
can now use the whole 32 GB, so the executor can be asked for about 60 GB. The 
executor sizes the pool from host memory by default, so this applies out of the 
box. Skew is also exactly when the straggler tends to be the memory-heavy 
partition, like a hot key in an aggregate or join.
   
   Could the release also shrink the task's pool? One option is a pool wrapper 
with a lowerable limit that the releaser decrements by `per_vcore * released`. 
Another is to only release a vcore once its share of the pool is actually 
unreserved.
   
   If that's a bigger change, would you be open to defaulting 
`ballista.scheduler.release_partition_vcores` to `false` until the pool follows 
the vcores?



##########
ballista/executor/src/executor_server.rs:
##########
@@ -450,6 +454,46 @@ impl<T: 'static + AsLogicalPlan, U: 'static + 
AsExecutionPlan> ExecutorServer<T,
             )
         };
 
+        // A pipelined task holds no vcores of its own to release.

Review Comment:
   Would it be worth a small test for the push side, like the pull-mode one in 
`execution_loop.rs`? It could check that a `RunningTask` status with 
`vcores_released` is sent, and that pipelined tasks and tasks with 
`vcores_consumed <= 1` don't get a releaser.



##########
ballista/core/src/execution_plans/shuffle_writer.rs:
##########
@@ -834,6 +836,7 @@ impl ShuffleWriterExec {
                     let rows = stats.num_rows.unwrap_or(0) as usize;
                     write_metrics.input_rows.add(rows);
                     write_metrics.output_rows.add(rows);
+                    vcore_release::partition_completed(&context, 
num_partitions);

Review Comment:
   Question here. `num_partitions` is the child's output partition count. The 
comment above says coordinating operators like `DynamicRangeRepartitionExec` 
feed all K outputs from shared scatter tasks. In that case, does one output 
reaching EOF actually free a thread? Or can the scatter side still be busy on 
all inputs?
   
   K also may not match the vcores the task was charged, so I wanted to check 
the assumption holds for that shape.



##########
ballista/core/src/serde/generated/ballista.rs:
##########
@@ -1428,6 +1428,10 @@ pub struct ExecutorResourcePair {
 pub struct RunningTask {
     #[prost(string, tag = "1")]
     pub executor_id: ::prost::alloc::string::String,
+    /// Vcores the task's finished partitions no longer need, released since 
its
+    /// previous status. The task keeps the rest until its terminal status.
+    #[prost(uint32, tag = "2")]
+    pub vcores_released: u32,

Review Comment:
   Small one. The wire change is additive, but the new field breaks downstream 
struct literals and exhaustive patterns like `RunningTask { executor_id }`, the 
same way the in-tree call sites needed updating. Maybe worth a one-line note in 
`docs/source/upgrading/55.0.0.md` for library users?



##########
ballista/scheduler/src/state/task_manager.rs:
##########
@@ -636,36 +636,44 @@ impl<T: 'static + AsLogicalPlan, U: 'static + 
AsExecutionPlan> TaskManager<T, U>
         }
     }
 
-    /// Consume reservations for terminal task reports exactly once, including
-    /// reports received after the entire job graph has been evicted.
+    /// Consume reservations for task reports, including reports received
+    /// after the entire job graph has been evicted. A terminal report takes
+    /// what is left of the task's reservation, exactly once. A running report
+    /// takes the vcores its finished partitions released, and leaves the task
+    /// at least one until its terminal report.
     pub(crate) fn take_vcores_for_statuses(
         &self,
         executor_id: &str,
         statuses: &[TaskStatus],
     ) -> u32 {
         statuses
             .iter()
-            .filter(|status| {
-                matches!(
-                    status.status,
-                    Some(
-                        task_status::Status::Successful(_)
-                            | task_status::Status::Failed(_)
-                    )
-                )
-            })
             .filter_map(|status| {
-                self.outstanding_task_reservations.remove(&(
+                let reservation = (

Review Comment:
   Just to confirm my reading. Since the key includes the attempt number, a 
late running report from an old attempt can't refund vcores from a new one. 
That relies on task ids never being reused within a stage attempt. Is that 
guaranteed?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to