runzhiwang opened a new pull request #1185: URL: https://github.com/apache/hadoop-ozone/pull/1185
## What changes were proposed in this pull request? **What's problem ?** Datanode creates more than 20K Datanode State Machine Thread, then OOM happened.  **What's the reason ?** 20K Datanode State Machine Thread were created by newCachedThreadPool  Almost all of them were wait lock.  Only one Datanode State Machine Thread got the lock, and block when submitRequest. Because this thread was blocked and can not free the lock, newCachedThreadPool will create new thread infinitely.  **How to fix ?** 1. Avoid use newCachedThreadPool, because it will create new thread infinitely, if no thread available in pool. 2. Cancel future when task time out. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-3933 ## How was this patch tested? Existed UT. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
