yahoNanJing commented on code in PR #728:
URL: https://github.com/apache/arrow-ballista/pull/728#discussion_r1152647112
##########
ballista/scheduler/src/cluster/kv.rs:
##########
@@ -57,6 +56,8 @@ pub struct KeyValueState<
> {
/// Underlying `KeyValueStore`
store: S,
+ /// ExecutorHeartbeat cache, executor_id -> ExecutorHeartbeat
+ executor_heartbeats: Arc<DashMap<String, ExecutorHeartbeat>>,
Review Comment:
For `ExecutorMetadata`, it can be cached. However, for `ExecutorData`, I
don't think it should be cached, as it's frequently changed and it's hard to
make it consistent with the one stored in backend storage when there are
multiple active schedulers.
Actually, I don't think it's necessary to introduce multiple active
schedulers in the Ballista cluster. Maybe only HA is needed. If so, we can
cache the `ExecutorData` with executor available slots to avoid frequent visit
to the backend storage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
