smengcl commented on PR #11695: URL: https://github.com/apache/ozone/pull/11695#issuecomment-6421066985
Thanks @ivandika3 for the review. > The current solution seems to only work if recon heartbeat interval is higher than the datanode heartbeat interval. Yes, that is intended. By default, `hdds.heartbeat.recon.interval` (60s) is longer than `hdds.heartbeat.interval` (30s), so the default case works. A shorter Recon interval was never possible. Before this patch, every cycle heartbeated Recon at the SCM interval, whatever the key said. The patch keeps that limit explicit. Is there a use case for heartbeating Recon more often than SCM? Achieving that would need a heartbeat thread per endpoint, as you suggested. Recon only needs heartbeats often enough not to mark the datanode stale. > This also creates another concern where a single stuck SCM RPC can cause heartbeats to the other unstuck SCMs to be delayed. The regular heartbeats to the other SCMs and Recon are still sent every `hdds.heartbeat.interval`. Each endpoint runs its tasks on its own single thread executor, `RunningDatanodeState` waits for each endpoint at most one interval, and `DatanodeStateMachine` sets the next cycle time before it runs the cycle (existing TestRunningDatanodeState#testAwait` covers a slow endpoint). The exception is the shared `DatanodeStateMachine` thread pool (one thread per endpoint). Each wait on the stuck SCM holds a thread for up to one interval, and each `triggerHeartbeat()` cycle adds another. Once they hold all threads, tasks for the other SCMs are queued and cycles are skipped, so their heartbeats can be up to about one interval late. Let's fix this in another Jira HDDS-17488. The stuck SCM's own endpoint also builds up a queue of heartbeat tasks. With dropped packets as in HDDS-16376, each new connection blocks for about 15 minutes on connect timeout retries. That is HDDS-17322, and HDDS-17269 covers the queued tasks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
