On Sun, May 4, 2008 at 3:04 AM, Matthew Toseland
<toad at amphibian.dyndns.org> wrote:
> On Friday 02 May 2008 11:10, j16sdiz at freenetproject.org wrote:
>
>
> > Author: j16sdiz
> > Date: 2008-05-02 10:10:12 +0000 (Fri, 02 May 2008)
> > New Revision: 19675
> >
> > Modified:
> > trunk/freenet/src/freenet/crypt/SHA256.java
> > trunk/freenet/src/freenet/node/MemoryChecker.java
> > trunk/freenet/src/freenet/node/Node.java
> > trunk/freenet/src/freenet/store/BerkeleyDBFreenetStore.java
> > trunk/freenet/src/freenet/support/OOMHandler.java
> > trunk/freenet/src/freenet/support/OOMHook.java
> > Log:
> > alternative low memory / out of memory handling
> >
> > Modified: trunk/freenet/src/freenet/crypt/SHA256.java
> > ===================================================================
> > --- trunk/freenet/src/freenet/crypt/SHA256.java 2008-05-02 10:09:13
> UTC (rev
> 19674)
> > +++ trunk/freenet/src/freenet/crypt/SHA256.java 2008-05-02 10:10:12
> UTC (rev
> 19675)
> > @@ -44,6 +44,8 @@
> > import freenet.node.Node;
> > import freenet.node.NodeInitException;
> > import freenet.support.Logger;
> > +import freenet.support.OOMHandler;
> > +import freenet.support.OOMHook;
> > import freenet.support.io.Closer;
> >
> > /**
> > @@ -105,7 +107,7 @@
> > String algo = md256.getAlgorithm();
> > if(!(algo.equals("SHA-256") || algo.equals("SHA256")))
> > throw new IllegalArgumentException("Should be
> SHA-256 but is " + algo);
> > - if (digests.size() > 16) // don't cache too many of them
> > + if (digests.size() > 16 || noCache) // don't cache too many
> of them
> > return;
> > md256.reset();
> > digests.add(md256);
> > @@ -121,4 +123,20 @@
> > public static int getDigestLength() {
> > return HASH_SIZE;
> > }
> > +
> > + private static boolean noCache = false;
> > +
> > + static {
> > + OOMHandler.addOOMHook(new OOMHook() {
> > + public void handleLowMemory() throws Exception {
> > + digests.clear();
> > + noCache = true;
> > + }
> > +
> > + public void handleOutOfMemory() throws Exception {
> > + digests.clear();
> > + noCache = true;
> > + }
> > + });
> > + }
> > }
>
> So when we reach the memory limit, we turn off caching and get more churn and
> therefore hit it more quickly ... well I suppose it's a tradeoff.
>
> >
> > Modified: trunk/freenet/src/freenet/node/MemoryChecker.java
> > ===================================================================
> > --- trunk/freenet/src/freenet/node/MemoryChecker.java 2008-05-02 10:09:13
> UTC (rev 19674)
> > +++ trunk/freenet/src/freenet/node/MemoryChecker.java 2008-05-02 10:10:12
> UTC (rev 19675)
> > @@ -4,6 +4,7 @@
> > package freenet.node;
> >
> > import freenet.support.Logger;
> > +import freenet.support.OOMHandler;
> > import freenet.support.SizeUtil;
> >
> > public class MemoryChecker implements Runnable {
> > @@ -42,6 +43,11 @@
> >
> > Logger.normal(this, "Memory in
> use: "+SizeUtil.formatSize((r.totalMemory()-r.freeMemory())));
> >
> > + if (r.freeMemory() < 4096 * 1024 * 1024) { // free memory <
> 8 MB
> > + Logger.error(this, "memory too low, trying to free
> some");
> > + OOMHandler.lowMemory();
> > + }
> > +
>
> I've changed this to do a GC and then re-check. If we are still at the limit
> for total memory and have less than 8MB free, *then* we hit the low memory
> handler.
The problem is, when OutOfMemoryError is thrown, it's too late to recover.
We can't control where OutOfMemory occur, When I say "can't control",
I really mean it.
For example, if it occur in BDBFS, the database may have corrupted,
transaction can't be aborted, etc.. what if OOM occur which Vector
trying to resize itself?
Besides these, JIT may allocate memory, finalization need memory,
even GC can take some memory.
Btw, -server vm do GC all the time, not just when it hit the limit[1].
It just try a bit harder then it's close to the limit. There are lots
of tunable for this....
[1] http://java.sun.com/javase/technologies/hotspot/gc/gc_tuning_6.html
>
> I'm not entirely convinced though. We will end up doing a lot more full GCs
> this way ... often on systems that don't have a memory problem.
>
If you read the documentation from java, you will see it's recommended
_not_ to set -Xms = -Xmx ..
It's because each full GC cost less time to finish if you do it more
often. If you set Xms=Xmx, full gc will kick in only when memory is
really full -- too much object to collect, cost lots of latency.
