Osma
Comments inline:
On 05/06/2014 10:11, "Osma Suominen" <[email protected]> wrote:
Hi all!
On 30/05/14 16:36, Mark Feblowitz wrote:
After some amount of time I see a series of messages after update posts
WARN [xxxxxx] RC = 500 : Java heap space
And I’m seeing "java.lang.OutOfMemoryError: Java heap space” errors.
I also got this error yesterday on an important machine.
My setup is this: Fuseki 1.0.1 with a single TDB (no inference) that has
grown to 13GB and an additional jena-text index of 200MB. Fuseki is
given approx. 6GB of heap (-Xmx6000M) on a machine with 16GB RAM. The
machine is a virtual machine running 64bit CentOS 6, kernel
2.6.32-431.11.2.el6.x86_64, java -version gives this:
java version "1.7.0_51"
OpenJDK Runtime Environment (rhel-2.4.4.1.el6_5-x86_64 u51-b02)
OpenJDK 64-Bit Server VM (build 24.45-b08, mixed mode)
At the time this happened there were no updates (we usually only update
during the night), just read-only SELECT and CONSTRUCT queries coming in
at around 8 queries per second on average.
As has been explained in this thread the problem is that the continuous
query load prevents the in-memory transaction journal from being fully
flushed to disk. If the read only queries continue through the night
while you do updates then they will block the journal flush and you will
eventually hit this case
JENA-703 (https://issues.apache.org/jira/browse/JENA-703) describes the
proposed fix for this issue but the side effect of that fix (as and when
it gets implemented) will be that for a system under continuous load reads
will be occasionally blocked and therefore some queries may experience
delays.
Suddenly queries stop working and CPU usage rises to around 350% (the
machine has 4 cores). Errors like this appear in the Fuseki log:
2014-06-04 11:42:52,293 WARN Fuseki :: [14739636] RC = 500
: Java heap space
java.lang.OutOfMemoryError: Java heap space
2014-06-04 11:39:06,670 WARN Fuseki :: [14739587] RC = 500
: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
2014-06-04 11:43:54,817 INFO Fuseki :: [14739587] 500 GC
overhead limit exceeded (1,032.111 s)
2014-06-04 12:24:56,868 WARN Fuseki :: [14739738] RC = 500
: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
2014-06-04 12:24:04,660 WARN Fuseki :: [14739862] RC = 500
: Java heap space
java.lang.OutOfMemoryError: Java heap space
2014-06-04 12:43:44,167 INFO Fuseki :: [14739738] 500 GC
overhead limit exceeded (2,227.717 s)
2014-06-04 12:43:44,167 INFO Fuseki :: [14739862] 500 Java
heap space (2,227.719 s)
2014-06-04 14:33:30,906 WARN Fuseki :: [14740021] RC = 500
: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
2014-06-04 14:34:21,722 INFO Fuseki :: [14740021] 500 GC
overhead limit exceeded (4,850.886 s)
The timestamps in the log entries are not always in order, as you can
see above. Sometimes it takes more than an hour for an individual query
to fail (see last entry).
Needless to say this is a bit nasty way of failing - the process is
running, consuming nearly all CPU, but responding to queries very slowly
or not at all. It would even be better if the process just died, so
something else could restart it. I am considering using a tool such as
Monit to watch the Fuseki process and restart it if it starts behaving
oddly.
This is really a JVM issue and not something we can control. The JVM
allows catching OOM errors (for better or worse) but in doing so it often
leaves applications in a state where they are extremely close to the heap
limit and so the user code hangs while the JVM furiously tries to GC
enough memory for it to continue.
Am I doing something obviously wrong here?
No, this is a known limitation of TDBs architecture
Should I just give the JVM
even more memory, or adjust some of the other JVM options?
That only prolongs the time to failure and TDB relies heavily on memory
mapped files which are off heap so increasing the heap size impacts
performance because it cause more swapping at the OS level
Is there any
way to force the GC to fail faster, or otherwise avoid futile attempts
of freeing more memory?
You can try the solution detailed at
http://stackoverflow.com/a/3878199/107591
Add -XX:OnOutOfMemoryError="kill -9 %p"
That is an Oracle JVM option so no guarantee it works on OpenJDK
Rob
Even better, could Fuseki just stick to the
memory it is given?
Thanks,
Osma
--
Osma Suominen
D.Sc. (Tech), Information Systems Specialist
National Library of Finland
P.O. Box 26 (Teollisuuskatu 23)
00014 HELSINGIN YLIOPISTO
Tel. +358 50 3199529
[email protected]
http://www.nationallibrary.fi
