Hi Hugo,
On 01/11/2023 19:43, Hugo Mills wrote:
Hi,
We’ve got an application we’ve inherited recently which uses a Fuseki
database. It was originally Fuseki 3.4.0, and has been upgraded to 4.9.0
recently. The 3.4.0 server needed regular restarts (once a day) in order
to keep working; the 4.9.0 server is even more unreliable, and has been
running out of memory and being OOM-killed multiple times a day. This
afternoon, it crashed enough times, fast enough, to make Kubernetes go
into a back-off loop, and brought the app down for some time.
We’re using OpenJDK 19. The JVM options are: “-Xmx:30g -Xms18g”, and the
container we’re running it in has a memory limit of 31 GiB.
Setting Xmx close to the container limit can cause problems.
The JVM itself takes space and the operating system needs space.
The JVM itself has a ~1G extra space for direct memory which networking
uses.
The Java heap will almost certainly grow to reach Xmx at some point
because Java delays running full garbage collections. The occasional
drops you see are likely incremental garbage collections happening.
If Xxm is very close to container limit, the heap will naturally grow
(it does not know about the container limit), then the total in-use
memory for the machine is reached and the container is killed.
30G heap looks like a very tight setting. Is there anything customized
running in Fuseki? is the server dedicated to Fuseki?
As Conal mentioned, TDB used memory mapped files - these are not part of
the heap. They are part of the OS virtual memory.
Is this a single database?
One TDB database needs about 4G RAM of heap space. Try a setting of -Xmx4G.
Only if you have a high proportion of very large literals will that
setting not work.
More is not better from TDB's point of view. Space for memory mapped
files is handled elsewhere, and that space that will expand and contract
as needed. If that space is squeezed out the system will slow down.
We tried the
“-XX:+UserSerialGC” option this evening, but it didn’t seem to help
much. We see the RAM usage of the java process rising steadily as
queries are made, with occasional small, but insufficient, drops.
The store is somewhere around 20M triples in size.
Is this a TDB database or in-memory? (I'm guessing TDB but could you
confirm that.)
Query processing can lead to a lot of memory use if the queries are
inefficient and there is a high, overlapping query load.
What is the query load on the server? Are there many overlapping requests?
Could anyone suggest any tweaks or options we could do to make this more
stable, and not leak memory? We’ve downgraded to 3.4.0 again, and it’s
not running out of space every few minutes at least, but it still has an
ever-growing memory usage.
Thanks,
Hugo.
*Dr. Hugo Mills*
Senior Data Scientist
hugo.mi...@agrimetrics.co.uk <mailto:hugo.mi...@agrimetrics.co.uk>
