Dandandan commented on code in PR #378:
URL: https://github.com/apache/arrow-ballista/pull/378#discussion_r997405760
##########
ballista/executor/src/execution_loop.rs:
##########
@@ -63,20 +63,28 @@ pub async fn poll_loop<T: 'static + AsLogicalPlan, U:
'static + AsExecutionPlan>
info!("Starting poll work loop with scheduler");
loop {
- let task_status: Vec<TaskStatus> =
- sample_tasks_status(&mut task_status_receiver).await;
-
// Keeps track of whether we received task in last iteration
// to avoid going in sleep mode between polling
let mut active_job = false;
+ let can_accept_task = available_tasks_slots.load(Ordering::SeqCst) > 0;
+
+ // Don't poll for work if we can not accept any tasks
+ if !can_accept_task {
+ tokio::time::sleep(Duration::from_millis(1)).await;
Review Comment:
I think checking the value is really cheap (e.g. compared to grpc requests),
so I did use the minimum to reduce the latency as much as possible when a job
is running.
I wonder if we can use some lock or channel instead of this loop in the
future to rewrite this without `sleep` 🤔 and use resources more efficiently...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
