thinkharderdev commented on code in PR #728:
URL: https://github.com/apache/arrow-ballista/pull/728#discussion_r1152981165
##########
ballista/scheduler/src/cluster/kv.rs:
##########
@@ -57,6 +56,8 @@ pub struct KeyValueState<
> {
/// Underlying `KeyValueStore`
store: S,
+ /// ExecutorHeartbeat cache, executor_id -> ExecutorHeartbeat
+ executor_heartbeats: Arc<DashMap<String, ExecutorHeartbeat>>,
Review Comment:
Ah, yeah I don't think `ExecutorData` is used at all anymore actually. We
use `ExecutorTaskSlots` to store the task slots for storing available slots for
reservation purposes. So I think we should be fine just caching
`ExecutorMetadata`
> Actually, I don't think it's necessary to introduce multiple active
schedulers in the Ballista cluster.
We run multiple active schedulers. The scheduler is doing a non-trivial
amount of work in scheduling and it's important to us to be able to scale that
layer horizontally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
