[
https://issues.apache.org/jira/browse/HDDS-15058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Andika updated HDDS-15058:
-------------------------------
Description:
Currently, we use normal gRPC blocking service as a heartbeat mechanism. The
SCM runs a separate thread to mark a DN as stale or dead based on DN last
heartbeat time. This is simple to implement, but seems to have drawbacks
* There is no way to differentiate between crashed datanodes and restarted
datanodes
* Detection of crashed datanodes might be too slow
* The stale and dead time might be too short or too long for a given cluster
(e.g. a cluster with dense datanodes with a lot of containers per DN might
require DN to have longer grace period).
An idea is to treat these datanodes similar to ZK Ephemeral node where a node
exists as long as the session is still active and ZK is able to detect and
generate an event if the ephemeral node down (session is terminated). A DN will
create a persistent gRPC stream (or "session" in ZK term). If the DN is
restarted normally, we can call onCompleted() and SCM can mark datanodes to
status other than DEAD. If the DN crashed, it will trigger onError(). Therefore
we can handle DN lifecycle in higher granularity.
was:
Currently, we use normal gRPC blocking service as a heartbeat mechanism. The
SCM runs a separate thread to mark a DN as stale or dead based on DN last
heartbeat time. This is simple to implement, but seems to have drawbacks
* There is no way to differentiate between crashed datanodes and restarted
datanodes
* Detection of crashed datanodes is pretty slow
* The stale and dead time might be too short or too long for a given cluster
(e.g. a cluster with dense datanodes with a lot of containers per DN might
require DN to have longer grace period).
An idea is to treat these datanodes similar to ZK Ephemeral node where a node
exists as long as the session is still active and ZK is able to detect and
generate an event if the ephemeral node down (session is terminated). A DN will
create a persistent gRPC stream (or "session" in ZK term). If the DN is
restarted normally, we can call onCompleted() and SCM can mark datanodes to
status other than DEAD. If the DN crashed, it will trigger onError(). Therefore
we can handle DN lifecycle in higher granularity.
> Use gRPC stream for DN heartbeat mechanism
> ------------------------------------------
>
> Key: HDDS-15058
> URL: https://issues.apache.org/jira/browse/HDDS-15058
> Project: Apache Ozone
> Issue Type: New Feature
> Reporter: Ivan Andika
> Assignee: Ivan Andika
> Priority: Major
>
> Currently, we use normal gRPC blocking service as a heartbeat mechanism. The
> SCM runs a separate thread to mark a DN as stale or dead based on DN last
> heartbeat time. This is simple to implement, but seems to have drawbacks
> * There is no way to differentiate between crashed datanodes and restarted
> datanodes
> * Detection of crashed datanodes might be too slow
> * The stale and dead time might be too short or too long for a given cluster
> (e.g. a cluster with dense datanodes with a lot of containers per DN might
> require DN to have longer grace period).
> An idea is to treat these datanodes similar to ZK Ephemeral node where a node
> exists as long as the session is still active and ZK is able to detect and
> generate an event if the ephemeral node down (session is terminated). A DN
> will create a persistent gRPC stream (or "session" in ZK term). If the DN is
> restarted normally, we can call onCompleted() and SCM can mark datanodes to
> status other than DEAD. If the DN crashed, it will trigger onError().
> Therefore we can handle DN lifecycle in higher granularity.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]