On Fri, Sep 30, 2022 at 5:06 PM Andy Seaborne <a...@apache.org> wrote: > > On 30/09/2022 12:39, Martynas Jusevičius wrote: > > On Fri, Sep 30, 2022 at 11:40 AM Andy Seaborne <a...@apache.org> wrote: > >> > >> Depends on what "runs out of memory means". > > > > MEM% in docker stats reaches 100% and the container quits. Inspecting > > it then shows OOMKilled: true. > > So you have a 10G container that crashes out with a max 8G heap that > crashes out when the heap is at 40%. That does not add up.
I agree it's odd. This article suggests that in Docker OOMKilled happens when "the max heap size is incorrect (over the available memory limit)" and that "The java.lang.OutOfMemoryError is different. That indicates that the max heap size is not enough to hold all live objects in memory". https://dzone.com/articles/why-my-java-application-is-oomkilled But what takes up the memory then? And how can this be investigated? The profiler setup is ready. LinkedDataHub runs with mem_limit: 1872m in the same setup and never gets OOMKilled, but it uses -XX:+UseContainerSupport -XX:MaxRAMPercentage=75 instead of -Xmx/-Xms. > > The JVM takes may be 0.5G (VM, loaded classes etc). > The kernel takes what? 0.5G tops. The rest is file system cache unless > another process is involved. > > TDB uses the file system cache which flexes up and down and is mapped > (virtual memory) into the address space of the JVM. > > Fuseki/TDB does not use off-heap same-process memory (IIRC Lucene might > if available but this is fixed size.) > > Andy > > >> If the container is being killed by the host, then it is likely some > >> process in the container is asking for memory (sbrk), and there is a > >> container limit (or ulimit?) being exceeded, so the container runtime or > >> the host kills the container. > >> > > > > Not sure if there's another way to check, but using top shows only the > > java process running inside the container. > > > >> One source of sbrk is a JVM growing the heap. It is not the only source > >> in the container OS. > >> > >> There is also the direct memory space in a JVM - it's fixed size (it > >> sits below the heap). > >> > > > > So if heap size was lowered (without changing mem_limit) then the > > off-heap memory got an increase? And the issue went away, meaning it > > wasn't large enough initially. > > > >> Java avoids doing major GC when it can grow the heap. Java will use > >> heap until it approaches the heap limit rather than do a full GC so it > >> can use more memory that it needs. This is regardless of the memory > >> working set size. It sbrk's even when it does not need to - a > >> time/space tradeoff. > > > > My observation in VisualVM that the heap size did not exceed ~40% of > > the max size. When I had allocated 8GB the GC kicked in around 3GB, > > when I allocated 4GB it kicked in around 1.5GB. > > > >> > >> If it works at 4G Fuseki isn't leaking memory. > > > > Yes but it would be nice to have general guidelines for use in Docker > > because this seems to be a recurring issue. > > > >> > >> Is the container limited? > >> Is the container runtime limited? > > > > As in mem_limit? As mentioned initially, mem_limit was at 10GB the whole > > time. > > Dockerfile is the default Dockerfile that ships with Fuseki. > > > > I've committed a Dockerfile and settings that can be used to a remote > > Fuseki from VisualVM using a JMX connection: > > https://github.com/AtomGraph/fuseki-docker#profiling > > > >> > >> Andy > >> > >> On 30/09/2022 06:36, Lorenz Buehmann wrote: > >>> From my understanding, a larger heap space for Fuseki should only be > >>> necessary when doing reasoning or e.g. loading the geospatial index. A > >>> TDB database on the other hand is backed by memory mapped files, i.e. > >>> makes use of off-heap memory and let the OS do all the work. > >>> > >>> Indeed, I cannot explain why assigning more heap makes the Fuseki thread > >>> consume as much until OOM is reached. > >>> > >>> We also have a Fuseki Docker deployment, and we assigned Fuseki way more > >>> memory (64GB) because of generating a large scale spatial index needs > >>> all geometry objects in memory once. But it didn't crash because of what > >>> you describe with the query load (maybe we never had such a constant > >>> load). > >>> > >>> Indeed, comparison is difficult, different machines, different Docker > >>> container, different Fuseki version ... > >>> > >>> > >>> I think Andy will have better explanations and maybe also others like > >>> Rob or people already using Fuseki@Docker > >>> > >>> On 29.09.22 16:45, Martynas Jusevičius wrote: > >>>> Still hasn't crashed, so less heap could be the solution in this case. > >>>> > >>>> On Thu, Sep 29, 2022 at 3:12 PM Martynas Jusevičius > >>>> <marty...@atomgraph.com> wrote: > >>>>> I've lowered the heap size to 4GB to leave more off-heap memory (6GB). > >>>>> It's been an hour and OOMKilled hasn't happened yet unlike before. > >>>>> MEM% in docker stats peaks around 70%. > >>>>> > >>>>> On Thu, Sep 29, 2022 at 12:41 PM Martynas Jusevičius > >>>>> <marty...@atomgraph.com> wrote: > >>>>>> OK the findings are weird so far... > >>>>>> > >>>>>> Under constant query load on my local Docker, MEM% of the Fuseki > >>>>>> container reached 100% within 45 minutes and it got OOMKilled. > >>>>>> > >>>>>> However, the Used heap "teeth" in VisualVM were below 3GB of the total > >>>>>> ~8GB Heap size the whole time. > >>>>>> > >>>>>> What does that tell us? > >>>>>> > >>>>>> > >>>>>> On Thu, Sep 29, 2022 at 11:58 AM Martynas Jusevičius > >>>>>> <marty...@atomgraph.com> wrote: > >>>>>>> Hi Eugen, > >>>>>>> > >>>>>>> I have the debugger working, I was trying to connect the profiler :) > >>>>>>> Finally I managed to connect from VisualVM on Windows thanks to this > >>>>>>> answer: > >>>>>>> https://stackoverflow.com/questions/66222727/how-to-connect-to-jmx-server-running-inside-wsl2/71881475#71881475 > >>>>>>> > >>>>>>> > >>>>>>> I've launched an infinite curl loop to create some query load, but > >>>>>>> what now? What should I be looking for in VisualVM? > >>>>>>> > >>>>>>> On Thu, Sep 29, 2022 at 11:33 AM Eugen Stan > >>>>>>> <eugen.s...@netdava.com> wrote: > >>>>>>>> For debugging, you need to do the following: > >>>>>>>> > >>>>>>>> * pass JVM options to enable debugging > >>>>>>>> * expose docker port for JVM debug you chose > >>>>>>>> > >>>>>>>> https://stackoverflow.com/questions/138511/what-are-java-command-line-options-to-set-to-allow-jvm-to-be-remotely-debugged > >>>>>>>> > >>>>>>>> > >>>>>>>> You should be able to do all this without changing the image: > >>>>>>>> docker env > >>>>>>>> variables and docker port option. > >>>>>>>> > >>>>>>>> Once container is started and port is listening, open (confirm with > >>>>>>>> docker ps) connect to it to debug. > >>>>>>>> > >>>>>>>> Good luck, > >>>>>>>> > >>>>>>>> On 29.09.2022 11:22, Martynas Jusevičius wrote: > >>>>>>>>> On Thu, Sep 29, 2022 at 9:41 AM Lorenz Buehmann > >>>>>>>>> <buehm...@informatik.uni-leipzig.de> wrote: > >>>>>>>>>> You're working on an in-memory dataset? > >>>>>>>>> No the datasets are TDB2-backed > >>>>>>>>> > >>>>>>>>>> Does it also happen with Jena 4.6.1? > >>>>>>>>> Don't know :) > >>>>>>>>> > >>>>>>>>> I wanted to run a profiler and tried connecting from VisualVM on > >>>>>>>>> Windows to the Fuseki container but neither jstatd nor JMX > >>>>>>>>> connections > >>>>>>>>> worked... > >>>>>>>>> Now I want to run VisualVM inside the container itself but this > >>>>>>>>> requires changing the Docker image in a way that I haven't > >>>>>>>>> figured out > >>>>>>>>> yet. > >>>>>>>>> > >>>>>>>>>> On 28.09.22 20:23, Martynas Jusevičius wrote: > >>>>>>>>>>> Hi, > >>>>>>>>>>> > >>>>>>>>>>> We have a dockerized Fuseki 4.5.0 instance that is gradually > >>>>>>>>>>> running > >>>>>>>>>>> out of memory over the course of a few hours. > >>>>>>>>>>> > >>>>>>>>>>> 3 datasets, none larger than 100000 triples. The load is > >>>>>>>>>>> negligible > >>>>>>>>>>> (maybe a few bursts x 10 simple queries per minute), no updates. > >>>>>>>>>>> > >>>>>>>>>>> Dockerfile: > >>>>>>>>>>> https://github.com/AtomGraph/fuseki-docker/blob/master/Dockerfile > >>>>>>>>>>> Memory settings: > >>>>>>>>>>> mem_limit: 10240m > >>>>>>>>>>> JAVA_OPTIONS=-Xmx7700m -Xms7700m > >>>>>>>>>>> > >>>>>>>>>>> Any advice? > >>>>>>>>>>> > >>>>>>>>>>> Martynas > >>>>>>>> -- > >>>>>>>> Eugen Stan > >>>>>>>> > >>>>>>>> +40770 941 271 / https://www.netdava.com
