Thanks to all who replied. We're trying out all the recommendations (slowly), and will update when we've got something to report.
Hugo. Dr. Hugo Mills Senior Data Scientist [email protected] NEWS: Visit our Data Marketplace to explore our agrifood data catalogue. www.agrimetrics.co.uk -----Original Message----- From: Andy Seaborne <[email protected]> Sent: Thursday, November 2, 2023 8:40 AM To: [email protected] Subject: Re: Ever-increasing memory usage in Fuseki [You don't often get email from [email protected]. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] Hi Hugo, On 01/11/2023 19:43, Hugo Mills wrote: > Hi, > > We've got an application we've inherited recently which uses a Fuseki > database. It was originally Fuseki 3.4.0, and has been upgraded to > 4.9.0 recently. The 3.4.0 server needed regular restarts (once a day) > in order to keep working; the 4.9.0 server is even more unreliable, > and has been running out of memory and being OOM-killed multiple times > a day. This afternoon, it crashed enough times, fast enough, to make > Kubernetes go into a back-off loop, and brought the app down for some time. > > We're using OpenJDK 19. The JVM options are: "-Xmx:30g -Xms18g", and > the container we're running it in has a memory limit of 31 GiB. Setting Xmx close to the container limit can cause problems. The JVM itself takes space and the operating system needs space. The JVM itself has a ~1G extra space for direct memory which networking uses. The Java heap will almost certainly grow to reach Xmx at some point because Java delays running full garbage collections. The occasional drops you see are likely incremental garbage collections happening. If Xxm is very close to container limit, the heap will naturally grow (it does not know about the container limit), then the total in-use memory for the machine is reached and the container is killed. 30G heap looks like a very tight setting. Is there anything customized running in Fuseki? is the server dedicated to Fuseki? As Conal mentioned, TDB used memory mapped files - these are not part of the heap. They are part of the OS virtual memory. Is this a single database? One TDB database needs about 4G RAM of heap space. Try a setting of -Xmx4G. Only if you have a high proportion of very large literals will that setting not work. More is not better from TDB's point of view. Space for memory mapped files is handled elsewhere, and that space that will expand and contract as needed. If that space is squeezed out the system will slow down. > We tried the > "-XX:+UserSerialGC" option this evening, but it didn't seem to help > much. We see the RAM usage of the java process rising steadily as > queries are made, with occasional small, but insufficient, drops. > The store is somewhere around 20M triples in size. Is this a TDB database or in-memory? (I'm guessing TDB but could you confirm that.) Query processing can lead to a lot of memory use if the queries are inefficient and there is a high, overlapping query load. What is the query load on the server? Are there many overlapping requests? > Could anyone suggest any tweaks or options we could do to make this > more stable, and not leak memory? We've downgraded to 3.4.0 again, and > it's not running out of space every few minutes at least, but it still > has an ever-growing memory usage. > > Thanks, > > Hugo. > > *Dr. Hugo Mills* > > Senior Data Scientist > > [email protected] <mailto:[email protected]>
