Siyao Meng created HDDS-17488:
---------------------------------
Summary: Waits on a stuck SCM or Recon endpoint can take all
datanode state machine threads and delay heartbeats to the other SCMs
Key: HDDS-17488
URL: https://issues.apache.org/jira/browse/HDDS-17488
Project: Apache Ozone
Issue Type: Bug
Components: Ozone Datanode
Reporter: Siyao Meng
{{RunningDatanodeState.execute}} submits one wrapper per endpoint to the
{{DatanodeStateMachine}} thread pool. The wrapper submits the endpoint task to
the endpoint's own executor and blocks on it for up to one heartbeat interval.
The pool has one thread per endpoint (SCMs + 1 Recon,
{{getEndPointTaskThreadPoolSize}}), and {{StateContext.execute}} skips a cycle
when a task is waiting in the pool queue ({{isThreadPoolAvailable}}).
HDDS-4386 moved the endpoint tasks to their own executors so that a slow Recon
or SCM does not take the threads used for the others. The wait for the task
still holds a pool thread, though, and so a stuck endpoint can still delay the
others:
* Every cycle adds one wait on the stuck endpoint, held for up to one interval.
* With regular cycles only, that wait ends about when the next cycle starts, so
the other SCMs and Recon are still heartbeated every interval.
* {{DatanodeStateMachine.triggerHeartbeat()}} (close container action, pipeline
action, ICR) interrupts the cycle and starts the next one at once. Each such
extra cycle adds another wait.
* With as many extra cycles within one interval as there are other endpoints (3
with 3 SCMs and Recon), all threads are held by waits on the stuck endpoint.
The tasks for the other SCMs wait in the queue until the oldest wait times out,
and every cycle started meanwhile is skipped.
So heartbeats to the healthy SCMs, including the immediate ones that carry
close container and pipeline actions, can be up to about one heartbeat interval
late. On its own that is not enough to mark the datanode stale. The stuck
endpoint can be Recon as well, which is the case HDDS-4386 fixed. This is from
code reading; not reproduced.
Proposed fix: do not block a pool thread for the endpoint task. For example,
keep the futures from the endpoint executors and wait for them in
{{RunningDatanodeState.await}} with the cycle deadline, or skip an endpoint
whose previous task is still running.
Related: HDDS-4386, HDDS-17269 (queued tasks on the stuck endpoint), HDDS-17322
(why an endpoint can be stuck for about 15 minutes)
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
