On Fri, Sep 30, 2022 at 11:40 AM Andy Seaborne <a...@apache.org> wrote: > > Depends on what "runs out of memory means".
MEM% in docker stats reaches 100% and the container quits. Inspecting it then shows OOMKilled: true. > > If the container is being killed by the host, then it is likely some > process in the container is asking for memory (sbrk), and there is a > container limit (or ulimit?) being exceeded, so the container runtime or > the host kills the container. > Not sure if there's another way to check, but using top shows only the java process running inside the container. > One source of sbrk is a JVM growing the heap. It is not the only source > in the container OS. > > There is also the direct memory space in a JVM - it's fixed size (it > sits below the heap). > So if heap size was lowered (without changing mem_limit) then the off-heap memory got an increase? And the issue went away, meaning it wasn't large enough initially. > Java avoids doing major GC when it can grow the heap. Java will use > heap until it approaches the heap limit rather than do a full GC so it > can use more memory that it needs. This is regardless of the memory > working set size. It sbrk's even when it does not need to - a > time/space tradeoff. My observation in VisualVM that the heap size did not exceed ~40% of the max size. When I had allocated 8GB the GC kicked in around 3GB, when I allocated 4GB it kicked in around 1.5GB. > > If it works at 4G Fuseki isn't leaking memory. Yes but it would be nice to have general guidelines for use in Docker because this seems to be a recurring issue. > > Is the container limited? > Is the container runtime limited? As in mem_limit? As mentioned initially, mem_limit was at 10GB the whole time. Dockerfile is the default Dockerfile that ships with Fuseki. I've committed a Dockerfile and settings that can be used to a remote Fuseki from VisualVM using a JMX connection: https://github.com/AtomGraph/fuseki-docker#profiling > > Andy > > On 30/09/2022 06:36, Lorenz Buehmann wrote: > > From my understanding, a larger heap space for Fuseki should only be > > necessary when doing reasoning or e.g. loading the geospatial index. A > > TDB database on the other hand is backed by memory mapped files, i.e. > > makes use of off-heap memory and let the OS do all the work. > > > > Indeed, I cannot explain why assigning more heap makes the Fuseki thread > > consume as much until OOM is reached. > > > > We also have a Fuseki Docker deployment, and we assigned Fuseki way more > > memory (64GB) because of generating a large scale spatial index needs > > all geometry objects in memory once. But it didn't crash because of what > > you describe with the query load (maybe we never had such a constant load). > > > > Indeed, comparison is difficult, different machines, different Docker > > container, different Fuseki version ... > > > > > > I think Andy will have better explanations and maybe also others like > > Rob or people already using Fuseki@Docker > > > > On 29.09.22 16:45, Martynas Jusevičius wrote: > >> Still hasn't crashed, so less heap could be the solution in this case. > >> > >> On Thu, Sep 29, 2022 at 3:12 PM Martynas Jusevičius > >> <marty...@atomgraph.com> wrote: > >>> I've lowered the heap size to 4GB to leave more off-heap memory (6GB). > >>> It's been an hour and OOMKilled hasn't happened yet unlike before. > >>> MEM% in docker stats peaks around 70%. > >>> > >>> On Thu, Sep 29, 2022 at 12:41 PM Martynas Jusevičius > >>> <marty...@atomgraph.com> wrote: > >>>> OK the findings are weird so far... > >>>> > >>>> Under constant query load on my local Docker, MEM% of the Fuseki > >>>> container reached 100% within 45 minutes and it got OOMKilled. > >>>> > >>>> However, the Used heap "teeth" in VisualVM were below 3GB of the total > >>>> ~8GB Heap size the whole time. > >>>> > >>>> What does that tell us? > >>>> > >>>> > >>>> On Thu, Sep 29, 2022 at 11:58 AM Martynas Jusevičius > >>>> <marty...@atomgraph.com> wrote: > >>>>> Hi Eugen, > >>>>> > >>>>> I have the debugger working, I was trying to connect the profiler :) > >>>>> Finally I managed to connect from VisualVM on Windows thanks to this > >>>>> answer: > >>>>> https://stackoverflow.com/questions/66222727/how-to-connect-to-jmx-server-running-inside-wsl2/71881475#71881475 > >>>>> > >>>>> > >>>>> I've launched an infinite curl loop to create some query load, but > >>>>> what now? What should I be looking for in VisualVM? > >>>>> > >>>>> On Thu, Sep 29, 2022 at 11:33 AM Eugen Stan > >>>>> <eugen.s...@netdava.com> wrote: > >>>>>> For debugging, you need to do the following: > >>>>>> > >>>>>> * pass JVM options to enable debugging > >>>>>> * expose docker port for JVM debug you chose > >>>>>> > >>>>>> https://stackoverflow.com/questions/138511/what-are-java-command-line-options-to-set-to-allow-jvm-to-be-remotely-debugged > >>>>>> > >>>>>> > >>>>>> You should be able to do all this without changing the image: > >>>>>> docker env > >>>>>> variables and docker port option. > >>>>>> > >>>>>> Once container is started and port is listening, open (confirm with > >>>>>> docker ps) connect to it to debug. > >>>>>> > >>>>>> Good luck, > >>>>>> > >>>>>> On 29.09.2022 11:22, Martynas Jusevičius wrote: > >>>>>>> On Thu, Sep 29, 2022 at 9:41 AM Lorenz Buehmann > >>>>>>> <buehm...@informatik.uni-leipzig.de> wrote: > >>>>>>>> You're working on an in-memory dataset? > >>>>>>> No the datasets are TDB2-backed > >>>>>>> > >>>>>>>> Does it also happen with Jena 4.6.1? > >>>>>>> Don't know :) > >>>>>>> > >>>>>>> I wanted to run a profiler and tried connecting from VisualVM on > >>>>>>> Windows to the Fuseki container but neither jstatd nor JMX > >>>>>>> connections > >>>>>>> worked... > >>>>>>> Now I want to run VisualVM inside the container itself but this > >>>>>>> requires changing the Docker image in a way that I haven't > >>>>>>> figured out > >>>>>>> yet. > >>>>>>> > >>>>>>>> On 28.09.22 20:23, Martynas Jusevičius wrote: > >>>>>>>>> Hi, > >>>>>>>>> > >>>>>>>>> We have a dockerized Fuseki 4.5.0 instance that is gradually > >>>>>>>>> running > >>>>>>>>> out of memory over the course of a few hours. > >>>>>>>>> > >>>>>>>>> 3 datasets, none larger than 100000 triples. The load is > >>>>>>>>> negligible > >>>>>>>>> (maybe a few bursts x 10 simple queries per minute), no updates. > >>>>>>>>> > >>>>>>>>> Dockerfile: > >>>>>>>>> https://github.com/AtomGraph/fuseki-docker/blob/master/Dockerfile > >>>>>>>>> Memory settings: > >>>>>>>>> mem_limit: 10240m > >>>>>>>>> JAVA_OPTIONS=-Xmx7700m -Xms7700m > >>>>>>>>> > >>>>>>>>> Any advice? > >>>>>>>>> > >>>>>>>>> Martynas > >>>>>> -- > >>>>>> Eugen Stan > >>>>>> > >>>>>> +40770 941 271 / https://www.netdava.com
