zhzy0077 commented on issue #714:
URL: https://github.com/apache/arrow-ballista/issues/714#issuecomment-1486604900
Interestingly, there are quite a few cases where Ballista pools results from
executor and the pool timeout is fixed 100ms.
To compare, I ran the same code and see 250ms on out of box code, and see
160ms per loop after applying this patch:
```diff
--- a/ballista/core/src/execution_plans/distributed_query.rs
+++ b/ballista/core/src/execution_plans/distributed_query.rs
@@ -259,7 +259,7 @@ async fn execute_query(
.map_err(|e| DataFusionError::Execution(format!("{e:?}")))?
.into_inner();
let status = status.and_then(|s| s.status);
- let wait_future = tokio::time::sleep(Duration::from_millis(100));
+ let wait_future = tokio::time::sleep(Duration::from_millis(1));
let has_status_change = prev_status != status;
match status {
None => {
```
And further to around 70ms per loop with this patch:
```diff
--- a/ballista/executor/src/execution_loop.rs
+++ b/ballista/executor/src/execution_loop.rs
@@ -127,7 +127,7 @@ pub async fn poll_loop<T: 'static + AsLogicalPlan, U:
'static + AsExecutionPlan>
}
if !active_job {
- tokio::time::sleep(Duration::from_millis(100)).await;
+ tokio::time::sleep(Duration::from_millis(1)).await;
}
}
}
```
Maybe Ballista should support configuring this wait period?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
