[
https://issues.apache.org/jira/browse/BROOKLYN-375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15644020#comment-15644020
]
ASF GitHub Bot commented on BROOKLYN-375:
-----------------------------------------
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/brooklyn-docs/pull/122#discussion_r86764032
--- Diff: guide/ops/troubleshooting/memory-usage.md ---
@@ -0,0 +1,94 @@
+---
+layout: website-normal
+title: "Troubleshooting: Monitoring Memory Usage"
+toc: /guide/toc.json
+---
+
+## Memory Usage
+
+Brooklyn tries to keep in memory as much history of its activity as
possible,
+for displaying through the UI, so it is normal for it to consume as much
memory
+as it can. It uses "soft references" so these objects will be cleared if
needed,
+but **it is not a sign of anything unusual if Brooklyn is using all its
available memory**.
+
+The number of active tasks, CPU usage, thread counts, and
+retention of soft reference objects are a much better indication of load.
+This information can be found by looking in the log for lines containing
+`brooklyn gc`, such as:
+
+ 2016-09-16 16:19:43,337 DEBUG o.a.b.c.m.i.BrooklynGarbageCollector
[brooklyn-gc]: brooklyn gc (before) - using 910 MB / 3.76 GB memory; 98%
soft-reference maybe retention (of 362); 35 threads; tasks: 0 active, 2
unfinished; 31 remembered, 1013 total submitted)
+
+The soft-reference figure is indicative, but the lower this is, the more
+the JVM has decided to get rid of items that were desired to be kept but
optional.
+It only tracks some soft-references (those wrapped in `Maybe`),
+and of course if there are many many such items the JVM will have to get
rid
+of some, so a lower figure does not necessarily mean a problem.
+Typically however if there's no `OutOfMemoryError` (OOME) reported,
+there's no problem.
+
+If you are concerned about memory usage, or doing evaluation on test
environments,
+the following method (in the Groovy console) can be invoked to force the
system to
+reclaim as much memory as possible, including *all* soft references:
+
+
org.apache.brooklyn.util.javalang.MemoryUsageTracker.forceClearSoftReferences()
+
+If things are happy usage should return to a small level. This is quite
disruptive
+to the system however so use with care.
+
+The above method can also be configured to run automatically when memory
usage
--- End diff --
I'd go further and say that we discourage this for any production use-case
(because it has not had expensive testing).
A better solution for production use-cases is to
use`-XX:SoftRefLRUPolicyMSPerMB=1`. See the discussion in
https://issues.apache.org/jira/browse/BROOKLYN-375.
We should also describe the symptoms of BROOKLYN-375 - that the CPU usage
goes very high, with almost all of that time being taken up by garbage
collection.
We need to do more on BROOKLYN-375 - e.g. implement the cache, to stop
using soft references!
> Brooklyn intermittently uses high CPU levels and becomes unresponsive
> ---------------------------------------------------------------------
>
> Key: BROOKLYN-375
> URL: https://issues.apache.org/jira/browse/BROOKLYN-375
> Project: Brooklyn
> Issue Type: Bug
> Environment: OSX Sierra, Java 1.7
> Reporter: Duncan Godwin
>
> Intermittently whilst launching a clocker swarm within brooklyn, it uses high
> CPU levels and becomes unresponsive. This was noted when testing failover by
> manally stopping some nodes with `shutdown -h`.
> [jstack 1|https://gist.github.com/drigodwin/c5946d23ed11350f393d9ba9b80a2a2d]
> [jstack 2|https://gist.github.com/drigodwin/5619b02c0c1d53ceb0c99234d8f0dd96]
> [jclouds.debug.log|https://gist.github.com/drigodwin/365d39d216e6a56c634a5020496ef8f1]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
