On 30/09/2022 12:39, Martynas Jusevičius wrote:
On Fri, Sep 30, 2022 at 11:40 AM Andy Seaborne <[email protected]> wrote:
Depends on what "runs out of memory means".
MEM% in docker stats reaches 100% and the container quits. Inspecting
it then shows OOMKilled: true.
So you have a 10G container that crashes out with a max 8G heap that
crashes out when the heap is at 40%. That does not add up.
The JVM takes may be 0.5G (VM, loaded classes etc).
The kernel takes what? 0.5G tops. The rest is file system cache unless
another process is involved.
TDB uses the file system cache which flexes up and down and is mapped
(virtual memory) into the address space of the JVM.
Fuseki/TDB does not use off-heap same-process memory (IIRC Lucene might
if available but this is fixed size.)
Andy
If the container is being killed by the host, then it is likely some
process in the container is asking for memory (sbrk), and there is a
container limit (or ulimit?) being exceeded, so the container runtime or
the host kills the container.
Not sure if there's another way to check, but using top shows only the
java process running inside the container.
One source of sbrk is a JVM growing the heap. It is not the only source
in the container OS.
There is also the direct memory space in a JVM - it's fixed size (it
sits below the heap).
So if heap size was lowered (without changing mem_limit) then the
off-heap memory got an increase? And the issue went away, meaning it
wasn't large enough initially.
>
Java avoids doing major GC when it can grow the heap. Java will use
heap until it approaches the heap limit rather than do a full GC so it
can use more memory that it needs. This is regardless of the memory
working set size. It sbrk's even when it does not need to - a
time/space tradeoff.
My observation in VisualVM that the heap size did not exceed ~40% of
the max size. When I had allocated 8GB the GC kicked in around 3GB,
when I allocated 4GB it kicked in around 1.5GB.
If it works at 4G Fuseki isn't leaking memory.
Yes but it would be nice to have general guidelines for use in Docker
because this seems to be a recurring issue.
Is the container limited?
Is the container runtime limited?
As in mem_limit? As mentioned initially, mem_limit was at 10GB the whole time.
Dockerfile is the default Dockerfile that ships with Fuseki.
I've committed a Dockerfile and settings that can be used to a remote
Fuseki from VisualVM using a JMX connection:
https://github.com/AtomGraph/fuseki-docker#profiling
Andy
On 30/09/2022 06:36, Lorenz Buehmann wrote:
From my understanding, a larger heap space for Fuseki should only be
necessary when doing reasoning or e.g. loading the geospatial index. A
TDB database on the other hand is backed by memory mapped files, i.e.
makes use of off-heap memory and let the OS do all the work.
Indeed, I cannot explain why assigning more heap makes the Fuseki thread
consume as much until OOM is reached.
We also have a Fuseki Docker deployment, and we assigned Fuseki way more
memory (64GB) because of generating a large scale spatial index needs
all geometry objects in memory once. But it didn't crash because of what
you describe with the query load (maybe we never had such a constant load).
Indeed, comparison is difficult, different machines, different Docker
container, different Fuseki version ...
I think Andy will have better explanations and maybe also others like
Rob or people already using Fuseki@Docker
On 29.09.22 16:45, Martynas Jusevičius wrote:
Still hasn't crashed, so less heap could be the solution in this case.
On Thu, Sep 29, 2022 at 3:12 PM Martynas Jusevičius
<[email protected]> wrote:
I've lowered the heap size to 4GB to leave more off-heap memory (6GB).
It's been an hour and OOMKilled hasn't happened yet unlike before.
MEM% in docker stats peaks around 70%.
On Thu, Sep 29, 2022 at 12:41 PM Martynas Jusevičius
<[email protected]> wrote:
OK the findings are weird so far...
Under constant query load on my local Docker, MEM% of the Fuseki
container reached 100% within 45 minutes and it got OOMKilled.
However, the Used heap "teeth" in VisualVM were below 3GB of the total
~8GB Heap size the whole time.
What does that tell us?
On Thu, Sep 29, 2022 at 11:58 AM Martynas Jusevičius
<[email protected]> wrote:
Hi Eugen,
I have the debugger working, I was trying to connect the profiler :)
Finally I managed to connect from VisualVM on Windows thanks to this
answer:
https://stackoverflow.com/questions/66222727/how-to-connect-to-jmx-server-running-inside-wsl2/71881475#71881475
I've launched an infinite curl loop to create some query load, but
what now? What should I be looking for in VisualVM?
On Thu, Sep 29, 2022 at 11:33 AM Eugen Stan
<[email protected]> wrote:
For debugging, you need to do the following:
* pass JVM options to enable debugging
* expose docker port for JVM debug you chose
https://stackoverflow.com/questions/138511/what-are-java-command-line-options-to-set-to-allow-jvm-to-be-remotely-debugged
You should be able to do all this without changing the image:
docker env
variables and docker port option.
Once container is started and port is listening, open (confirm with
docker ps) connect to it to debug.
Good luck,
On 29.09.2022 11:22, Martynas Jusevičius wrote:
On Thu, Sep 29, 2022 at 9:41 AM Lorenz Buehmann
<[email protected]> wrote:
You're working on an in-memory dataset?
No the datasets are TDB2-backed
Does it also happen with Jena 4.6.1?
Don't know :)
I wanted to run a profiler and tried connecting from VisualVM on
Windows to the Fuseki container but neither jstatd nor JMX
connections
worked...
Now I want to run VisualVM inside the container itself but this
requires changing the Docker image in a way that I haven't
figured out
yet.
On 28.09.22 20:23, Martynas Jusevičius wrote:
Hi,
We have a dockerized Fuseki 4.5.0 instance that is gradually
running
out of memory over the course of a few hours.
3 datasets, none larger than 100000 triples. The load is
negligible
(maybe a few bursts x 10 simple queries per minute), no updates.
Dockerfile:
https://github.com/AtomGraph/fuseki-docker/blob/master/Dockerfile
Memory settings:
mem_limit: 10240m
JAVA_OPTIONS=-Xmx7700m -Xms7700m
Any advice?
Martynas
--
Eugen Stan
+40770 941 271 / https://www.netdava.com
