keith-turner opened a new issue #2423:
URL: https://github.com/apache/accumulo/issues/2423
**Describe the bug**
When running Accumulo 2.1.0-SNAPSHOT for extended periods (like 24+hours) I
would see the manager process die with an out of memory error.
**Versions (OS, Maven, Java, and others, as appropriate):**
- Affected version(s) of this project: 2.1.0-SNAPSHOT
**To Reproduce**
Run the Accumulo 2.1.0 Manager for an extended period of time with 512M of
RAM. 512M is the default config set by Muchos for the manager. It has been
this size for a very long time and I don't recall seeing this error before.
**Expected behavior**
No OOME
**Screenshots**
After the first time this happened, I modified the manager process to heap
dump on OOME. Below is a screenshot of analyzing the heap dump. I saw a lot
of micrometer objects in the heap dump. Randomly poking around, a lot of these
seemed to have `exectuor` in the name. Based on this I looked in the manager
code and found it was [periodically creating thread pools to get tserver
info](https://github.com/apache/accumulo/blob/1f80b7445d1320efaa8db1fb58c7b67d73c9fdbc/server/manager/src/main/java/org/apache/accumulo/manager/Manager.java#L920)
and this [may register the thread pool statically for
monitoring](https://github.com/apache/accumulo/blob/1f80b7445d1320efaa8db1fb58c7b67d73c9fdbc/core/src/main/java/org/apache/accumulo/core/util/threads/ThreadPools.java#L287).
I have yet to connect the objects in the heap dump the code linked Manager
code. The objects seem to be statically referenced so I am guessing the refs
from Accumulo code are gone and only the static metric registry re
fs are left. I am not sure yet if the Manager code I linked is the culprit
though.
**Additional context**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
