Hi Hugo,
A data size of 20Mt is not big and shouldn't need anything like 30G of
heap let alone 18G of stack. As others have said you need a much bigger
container limit if you going pack both those plus the JVM plus direct
buffers in.
For fuseki in kubernetes at that scale we would typically run with a
heap more like 2G and a container limit of 4G (assuming TDB1) and no
need to increase stack defaults. For larger sizes in the 100-200Mt
region then 3-4G of heap and 8G container limit typically works for us.
We only get up to needing machine sizes on the 30G range for 600MT and
very high request rates (millions a day) and even then we keep heap
relatively small (~12G) to leave space for the OS caching. For TDB1 the
heap size, once there's enough to work in, is more critical for updates
than for query and at 20M your updates can't be that big.
The progressive jetty memory leak mentioned in the threads Martynas
referenced was solved by switching off Jetty's use of direct memory,
which you can do with configuration files in your container. I don't
think this is the cause of your problems in any case unless you are
dealing with a very high request rate.
Hope this helps.
Dave
On 03/11/2023 10:08, Hugo Mills wrote:
Thanks to all who replied. We're trying out all the recommendations (slowly),
and will update when we've got something to report.
Hugo.
Dr. Hugo Mills
Senior Data Scientist
[email protected]
NEWS: Visit our Data Marketplace to explore our agrifood data catalogue.
www.agrimetrics.co.uk
-----Original Message-----
From: Andy Seaborne <[email protected]>
Sent: Thursday, November 2, 2023 8:40 AM
To: [email protected]
Subject: Re: Ever-increasing memory usage in Fuseki
[You don't often get email from [email protected]. Learn why this is important at
https://aka.ms/LearnAboutSenderIdentification ]
Hi Hugo,
On 01/11/2023 19:43, Hugo Mills wrote:
Hi,
We've got an application we've inherited recently which uses a Fuseki
database. It was originally Fuseki 3.4.0, and has been upgraded to
4.9.0 recently. The 3.4.0 server needed regular restarts (once a day)
in order to keep working; the 4.9.0 server is even more unreliable,
and has been running out of memory and being OOM-killed multiple times
a day. This afternoon, it crashed enough times, fast enough, to make
Kubernetes go into a back-off loop, and brought the app down for some time.
We're using OpenJDK 19. The JVM options are: "-Xmx:30g -Xms18g", and
the container we're running it in has a memory limit of 31 GiB.
Setting Xmx close to the container limit can cause problems.
The JVM itself takes space and the operating system needs space.
The JVM itself has a ~1G extra space for direct memory which networking uses.
The Java heap will almost certainly grow to reach Xmx at some point because
Java delays running full garbage collections. The occasional drops you see are
likely incremental garbage collections happening.
If Xxm is very close to container limit, the heap will naturally grow (it does
not know about the container limit), then the total in-use memory for the
machine is reached and the container is killed.
30G heap looks like a very tight setting. Is there anything customized running
in Fuseki? is the server dedicated to Fuseki?
As Conal mentioned, TDB used memory mapped files - these are not part of the
heap. They are part of the OS virtual memory.
Is this a single database?
One TDB database needs about 4G RAM of heap space. Try a setting of -Xmx4G.
Only if you have a high proportion of very large literals will that setting not
work.
More is not better from TDB's point of view. Space for memory mapped files is
handled elsewhere, and that space that will expand and contract as needed. If
that space is squeezed out the system will slow down.
We tried the
"-XX:+UserSerialGC" option this evening, but it didn't seem to help
much. We see the RAM usage of the java process rising steadily as
queries are made, with occasional small, but insufficient, drops.
The store is somewhere around 20M triples in size.
Is this a TDB database or in-memory? (I'm guessing TDB but could you confirm
that.)
Query processing can lead to a lot of memory use if the queries are inefficient
and there is a high, overlapping query load.
What is the query load on the server? Are there many overlapping requests?
Could anyone suggest any tweaks or options we could do to make this
more stable, and not leak memory? We've downgraded to 3.4.0 again, and
it's not running out of space every few minutes at least, but it still
has an ever-growing memory usage.
Thanks,
Hugo.
*Dr. Hugo Mills*
Senior Data Scientist
[email protected] <mailto:[email protected]>
