Re: Re: Fuseki container is OOMKilled
From my understanding, a larger heap space for Fuseki should only be necessary when doing reasoning or e.g. loading the geospatial index. A TDB database on the other hand is backed by memory mapped files, i.e. makes use of off-heap memory and let the OS do all the work. Indeed, I cannot explain why assigning more heap makes the Fuseki thread consume as much until OOM is reached. We also have a Fuseki Docker deployment, and we assigned Fuseki way more memory (64GB) because of generating a large scale spatial index needs all geometry objects in memory once. But it didn't crash because of what you describe with the query load (maybe we never had such a constant load). Indeed, comparison is difficult, different machines, different Docker container, different Fuseki version ... I think Andy will have better explanations and maybe also others like Rob or people already using Fuseki@Docker On 29.09.22 16:45, Martynas Jusevičius wrote: Still hasn't crashed, so less heap could be the solution in this case. On Thu, Sep 29, 2022 at 3:12 PM Martynas Jusevičius wrote: I've lowered the heap size to 4GB to leave more off-heap memory (6GB). It's been an hour and OOMKilled hasn't happened yet unlike before. MEM% in docker stats peaks around 70%. On Thu, Sep 29, 2022 at 12:41 PM Martynas Jusevičius wrote: OK the findings are weird so far... Under constant query load on my local Docker, MEM% of the Fuseki container reached 100% within 45 minutes and it got OOMKilled. However, the Used heap "teeth" in VisualVM were below 3GB of the total ~8GB Heap size the whole time. What does that tell us? On Thu, Sep 29, 2022 at 11:58 AM Martynas Jusevičius wrote: Hi Eugen, I have the debugger working, I was trying to connect the profiler :) Finally I managed to connect from VisualVM on Windows thanks to this answer: https://stackoverflow.com/questions/66222727/how-to-connect-to-jmx-server-running-inside-wsl2/71881475#71881475 I've launched an infinite curl loop to create some query load, but what now? What should I be looking for in VisualVM? On Thu, Sep 29, 2022 at 11:33 AM Eugen Stan wrote: For debugging, you need to do the following: * pass JVM options to enable debugging * expose docker port for JVM debug you chose https://stackoverflow.com/questions/138511/what-are-java-command-line-options-to-set-to-allow-jvm-to-be-remotely-debugged You should be able to do all this without changing the image: docker env variables and docker port option. Once container is started and port is listening, open (confirm with docker ps) connect to it to debug. Good luck, On 29.09.2022 11:22, Martynas Jusevičius wrote: On Thu, Sep 29, 2022 at 9:41 AM Lorenz Buehmann wrote: You're working on an in-memory dataset? No the datasets are TDB2-backed Does it also happen with Jena 4.6.1? Don't know :) I wanted to run a profiler and tried connecting from VisualVM on Windows to the Fuseki container but neither jstatd nor JMX connections worked... Now I want to run VisualVM inside the container itself but this requires changing the Docker image in a way that I haven't figured out yet. On 28.09.22 20:23, Martynas Jusevičius wrote: Hi, We have a dockerized Fuseki 4.5.0 instance that is gradually running out of memory over the course of a few hours. 3 datasets, none larger than 10 triples. The load is negligible (maybe a few bursts x 10 simple queries per minute), no updates. Dockerfile: https://github.com/AtomGraph/fuseki-docker/blob/master/Dockerfile Memory settings: mem_limit: 10240m JAVA_OPTIONS=-Xmx7700m -Xms7700m Any advice? Martynas -- Eugen Stan +40770 941 271 / https://www.netdava.com
Re: Fuseki container is OOMKilled
Still hasn't crashed, so less heap could be the solution in this case. On Thu, Sep 29, 2022 at 3:12 PM Martynas Jusevičius wrote: > > I've lowered the heap size to 4GB to leave more off-heap memory (6GB). > It's been an hour and OOMKilled hasn't happened yet unlike before. > MEM% in docker stats peaks around 70%. > > On Thu, Sep 29, 2022 at 12:41 PM Martynas Jusevičius > wrote: > > > > OK the findings are weird so far... > > > > Under constant query load on my local Docker, MEM% of the Fuseki > > container reached 100% within 45 minutes and it got OOMKilled. > > > > However, the Used heap "teeth" in VisualVM were below 3GB of the total > > ~8GB Heap size the whole time. > > > > What does that tell us? > > > > > > On Thu, Sep 29, 2022 at 11:58 AM Martynas Jusevičius > > wrote: > > > > > > Hi Eugen, > > > > > > I have the debugger working, I was trying to connect the profiler :) > > > Finally I managed to connect from VisualVM on Windows thanks to this > > > answer: > > > https://stackoverflow.com/questions/66222727/how-to-connect-to-jmx-server-running-inside-wsl2/71881475#71881475 > > > > > > I've launched an infinite curl loop to create some query load, but > > > what now? What should I be looking for in VisualVM? > > > > > > On Thu, Sep 29, 2022 at 11:33 AM Eugen Stan > > > wrote: > > > > > > > > For debugging, you need to do the following: > > > > > > > > * pass JVM options to enable debugging > > > > * expose docker port for JVM debug you chose > > > > > > > > https://stackoverflow.com/questions/138511/what-are-java-command-line-options-to-set-to-allow-jvm-to-be-remotely-debugged > > > > > > > > You should be able to do all this without changing the image: docker env > > > > variables and docker port option. > > > > > > > > Once container is started and port is listening, open (confirm with > > > > docker ps) connect to it to debug. > > > > > > > > Good luck, > > > > > > > > On 29.09.2022 11:22, Martynas Jusevičius wrote: > > > > > On Thu, Sep 29, 2022 at 9:41 AM Lorenz Buehmann > > > > > wrote: > > > > >> > > > > >> You're working on an in-memory dataset? > > > > > > > > > > No the datasets are TDB2-backed > > > > > > > > > >> Does it also happen with Jena 4.6.1? > > > > > > > > > > Don't know :) > > > > > > > > > > I wanted to run a profiler and tried connecting from VisualVM on > > > > > Windows to the Fuseki container but neither jstatd nor JMX connections > > > > > worked... > > > > > Now I want to run VisualVM inside the container itself but this > > > > > requires changing the Docker image in a way that I haven't figured out > > > > > yet. > > > > > > > > > >> > > > > >> On 28.09.22 20:23, Martynas Jusevičius wrote: > > > > >>> Hi, > > > > >>> > > > > >>> We have a dockerized Fuseki 4.5.0 instance that is gradually running > > > > >>> out of memory over the course of a few hours. > > > > >>> > > > > >>> 3 datasets, none larger than 10 triples. The load is negligible > > > > >>> (maybe a few bursts x 10 simple queries per minute), no updates. > > > > >>> > > > > >>> Dockerfile: > > > > >>> https://github.com/AtomGraph/fuseki-docker/blob/master/Dockerfile > > > > >>> Memory settings: > > > > >>> mem_limit: 10240m > > > > >>> JAVA_OPTIONS=-Xmx7700m -Xms7700m > > > > >>> > > > > >>> Any advice? > > > > >>> > > > > >>> Martynas > > > > > > > > -- > > > > Eugen Stan > > > > > > > > +40770 941 271 / https://www.netdava.com
Re: Fuseki container is OOMKilled
I've lowered the heap size to 4GB to leave more off-heap memory (6GB). It's been an hour and OOMKilled hasn't happened yet unlike before. MEM% in docker stats peaks around 70%. On Thu, Sep 29, 2022 at 12:41 PM Martynas Jusevičius wrote: > > OK the findings are weird so far... > > Under constant query load on my local Docker, MEM% of the Fuseki > container reached 100% within 45 minutes and it got OOMKilled. > > However, the Used heap "teeth" in VisualVM were below 3GB of the total > ~8GB Heap size the whole time. > > What does that tell us? > > > On Thu, Sep 29, 2022 at 11:58 AM Martynas Jusevičius > wrote: > > > > Hi Eugen, > > > > I have the debugger working, I was trying to connect the profiler :) > > Finally I managed to connect from VisualVM on Windows thanks to this > > answer: > > https://stackoverflow.com/questions/66222727/how-to-connect-to-jmx-server-running-inside-wsl2/71881475#71881475 > > > > I've launched an infinite curl loop to create some query load, but > > what now? What should I be looking for in VisualVM? > > > > On Thu, Sep 29, 2022 at 11:33 AM Eugen Stan wrote: > > > > > > For debugging, you need to do the following: > > > > > > * pass JVM options to enable debugging > > > * expose docker port for JVM debug you chose > > > > > > https://stackoverflow.com/questions/138511/what-are-java-command-line-options-to-set-to-allow-jvm-to-be-remotely-debugged > > > > > > You should be able to do all this without changing the image: docker env > > > variables and docker port option. > > > > > > Once container is started and port is listening, open (confirm with > > > docker ps) connect to it to debug. > > > > > > Good luck, > > > > > > On 29.09.2022 11:22, Martynas Jusevičius wrote: > > > > On Thu, Sep 29, 2022 at 9:41 AM Lorenz Buehmann > > > > wrote: > > > >> > > > >> You're working on an in-memory dataset? > > > > > > > > No the datasets are TDB2-backed > > > > > > > >> Does it also happen with Jena 4.6.1? > > > > > > > > Don't know :) > > > > > > > > I wanted to run a profiler and tried connecting from VisualVM on > > > > Windows to the Fuseki container but neither jstatd nor JMX connections > > > > worked... > > > > Now I want to run VisualVM inside the container itself but this > > > > requires changing the Docker image in a way that I haven't figured out > > > > yet. > > > > > > > >> > > > >> On 28.09.22 20:23, Martynas Jusevičius wrote: > > > >>> Hi, > > > >>> > > > >>> We have a dockerized Fuseki 4.5.0 instance that is gradually running > > > >>> out of memory over the course of a few hours. > > > >>> > > > >>> 3 datasets, none larger than 10 triples. The load is negligible > > > >>> (maybe a few bursts x 10 simple queries per minute), no updates. > > > >>> > > > >>> Dockerfile: > > > >>> https://github.com/AtomGraph/fuseki-docker/blob/master/Dockerfile > > > >>> Memory settings: > > > >>> mem_limit: 10240m > > > >>> JAVA_OPTIONS=-Xmx7700m -Xms7700m > > > >>> > > > >>> Any advice? > > > >>> > > > >>> Martynas > > > > > > -- > > > Eugen Stan > > > > > > +40770 941 271 / https://www.netdava.com
Re: Fuseki container is OOMKilled
OK the findings are weird so far... Under constant query load on my local Docker, MEM% of the Fuseki container reached 100% within 45 minutes and it got OOMKilled. However, the Used heap "teeth" in VisualVM were below 3GB of the total ~8GB Heap size the whole time. What does that tell us? On Thu, Sep 29, 2022 at 11:58 AM Martynas Jusevičius wrote: > > Hi Eugen, > > I have the debugger working, I was trying to connect the profiler :) > Finally I managed to connect from VisualVM on Windows thanks to this > answer: > https://stackoverflow.com/questions/66222727/how-to-connect-to-jmx-server-running-inside-wsl2/71881475#71881475 > > I've launched an infinite curl loop to create some query load, but > what now? What should I be looking for in VisualVM? > > On Thu, Sep 29, 2022 at 11:33 AM Eugen Stan wrote: > > > > For debugging, you need to do the following: > > > > * pass JVM options to enable debugging > > * expose docker port for JVM debug you chose > > > > https://stackoverflow.com/questions/138511/what-are-java-command-line-options-to-set-to-allow-jvm-to-be-remotely-debugged > > > > You should be able to do all this without changing the image: docker env > > variables and docker port option. > > > > Once container is started and port is listening, open (confirm with > > docker ps) connect to it to debug. > > > > Good luck, > > > > On 29.09.2022 11:22, Martynas Jusevičius wrote: > > > On Thu, Sep 29, 2022 at 9:41 AM Lorenz Buehmann > > > wrote: > > >> > > >> You're working on an in-memory dataset? > > > > > > No the datasets are TDB2-backed > > > > > >> Does it also happen with Jena 4.6.1? > > > > > > Don't know :) > > > > > > I wanted to run a profiler and tried connecting from VisualVM on > > > Windows to the Fuseki container but neither jstatd nor JMX connections > > > worked... > > > Now I want to run VisualVM inside the container itself but this > > > requires changing the Docker image in a way that I haven't figured out > > > yet. > > > > > >> > > >> On 28.09.22 20:23, Martynas Jusevičius wrote: > > >>> Hi, > > >>> > > >>> We have a dockerized Fuseki 4.5.0 instance that is gradually running > > >>> out of memory over the course of a few hours. > > >>> > > >>> 3 datasets, none larger than 10 triples. The load is negligible > > >>> (maybe a few bursts x 10 simple queries per minute), no updates. > > >>> > > >>> Dockerfile: > > >>> https://github.com/AtomGraph/fuseki-docker/blob/master/Dockerfile > > >>> Memory settings: > > >>> mem_limit: 10240m > > >>> JAVA_OPTIONS=-Xmx7700m -Xms7700m > > >>> > > >>> Any advice? > > >>> > > >>> Martynas > > > > -- > > Eugen Stan > > > > +40770 941 271 / https://www.netdava.com
Re: Fuseki container is OOMKilled
Hi Eugen, I have the debugger working, I was trying to connect the profiler :) Finally I managed to connect from VisualVM on Windows thanks to this answer: https://stackoverflow.com/questions/66222727/how-to-connect-to-jmx-server-running-inside-wsl2/71881475#71881475 I've launched an infinite curl loop to create some query load, but what now? What should I be looking for in VisualVM? On Thu, Sep 29, 2022 at 11:33 AM Eugen Stan wrote: > > For debugging, you need to do the following: > > * pass JVM options to enable debugging > * expose docker port for JVM debug you chose > > https://stackoverflow.com/questions/138511/what-are-java-command-line-options-to-set-to-allow-jvm-to-be-remotely-debugged > > You should be able to do all this without changing the image: docker env > variables and docker port option. > > Once container is started and port is listening, open (confirm with > docker ps) connect to it to debug. > > Good luck, > > On 29.09.2022 11:22, Martynas Jusevičius wrote: > > On Thu, Sep 29, 2022 at 9:41 AM Lorenz Buehmann > > wrote: > >> > >> You're working on an in-memory dataset? > > > > No the datasets are TDB2-backed > > > >> Does it also happen with Jena 4.6.1? > > > > Don't know :) > > > > I wanted to run a profiler and tried connecting from VisualVM on > > Windows to the Fuseki container but neither jstatd nor JMX connections > > worked... > > Now I want to run VisualVM inside the container itself but this > > requires changing the Docker image in a way that I haven't figured out > > yet. > > > >> > >> On 28.09.22 20:23, Martynas Jusevičius wrote: > >>> Hi, > >>> > >>> We have a dockerized Fuseki 4.5.0 instance that is gradually running > >>> out of memory over the course of a few hours. > >>> > >>> 3 datasets, none larger than 10 triples. The load is negligible > >>> (maybe a few bursts x 10 simple queries per minute), no updates. > >>> > >>> Dockerfile: > >>> https://github.com/AtomGraph/fuseki-docker/blob/master/Dockerfile > >>> Memory settings: > >>> mem_limit: 10240m > >>> JAVA_OPTIONS=-Xmx7700m -Xms7700m > >>> > >>> Any advice? > >>> > >>> Martynas > > -- > Eugen Stan > > +40770 941 271 / https://www.netdava.com
Re: Fuseki container is OOMKilled
For debugging, you need to do the following: * pass JVM options to enable debugging * expose docker port for JVM debug you chose https://stackoverflow.com/questions/138511/what-are-java-command-line-options-to-set-to-allow-jvm-to-be-remotely-debugged You should be able to do all this without changing the image: docker env variables and docker port option. Once container is started and port is listening, open (confirm with docker ps) connect to it to debug. Good luck, On 29.09.2022 11:22, Martynas Jusevičius wrote: On Thu, Sep 29, 2022 at 9:41 AM Lorenz Buehmann wrote: You're working on an in-memory dataset? No the datasets are TDB2-backed Does it also happen with Jena 4.6.1? Don't know :) I wanted to run a profiler and tried connecting from VisualVM on Windows to the Fuseki container but neither jstatd nor JMX connections worked... Now I want to run VisualVM inside the container itself but this requires changing the Docker image in a way that I haven't figured out yet. On 28.09.22 20:23, Martynas Jusevičius wrote: Hi, We have a dockerized Fuseki 4.5.0 instance that is gradually running out of memory over the course of a few hours. 3 datasets, none larger than 10 triples. The load is negligible (maybe a few bursts x 10 simple queries per minute), no updates. Dockerfile: https://github.com/AtomGraph/fuseki-docker/blob/master/Dockerfile Memory settings: mem_limit: 10240m JAVA_OPTIONS=-Xmx7700m -Xms7700m Any advice? Martynas -- Eugen Stan +40770 941 271 / https://www.netdava.com begin:vcard fn:Eugen Stan n:Stan;Eugen email;internet:eugen.s...@netdava.com tel;cell:+40720898747 x-mozilla-html:FALSE url:https://www.netdava.com version:2.1 end:vcard
Re: Fuseki container is OOMKilled
On Thu, Sep 29, 2022 at 9:41 AM Lorenz Buehmann wrote: > > You're working on an in-memory dataset? No the datasets are TDB2-backed > Does it also happen with Jena 4.6.1? Don't know :) I wanted to run a profiler and tried connecting from VisualVM on Windows to the Fuseki container but neither jstatd nor JMX connections worked... Now I want to run VisualVM inside the container itself but this requires changing the Docker image in a way that I haven't figured out yet. > > On 28.09.22 20:23, Martynas Jusevičius wrote: > > Hi, > > > > We have a dockerized Fuseki 4.5.0 instance that is gradually running > > out of memory over the course of a few hours. > > > > 3 datasets, none larger than 10 triples. The load is negligible > > (maybe a few bursts x 10 simple queries per minute), no updates. > > > > Dockerfile: > > https://github.com/AtomGraph/fuseki-docker/blob/master/Dockerfile > > Memory settings: > > mem_limit: 10240m > > JAVA_OPTIONS=-Xmx7700m -Xms7700m > > > > Any advice? > > > > Martynas
Re: Fuseki container is OOMKilled
We experienced fuseki crashes as well. For us we had LARGE data sets and had some reasoning rules. Also lots of updates increase disk size as well. I hope it helps, Eugen On 29.09.2022 10:40, Lorenz Buehmann wrote: You're working on an in-memory dataset? Does it also happen with Jena 4.6.1? On 28.09.22 20:23, Martynas Jusevičius wrote: > Hi, > > We have a dockerized Fuseki 4.5.0 instance that is gradually running > out of memory over the course of a few hours. > > 3 datasets, none larger than 10 triples. The load is negligible > (maybe a few bursts x 10 simple queries per minute), no updates. > > Dockerfile: > https://github.com/AtomGraph/fuseki-docker/blob/master/Dockerfile > Memory settings: > mem_limit: 10240m > JAVA_OPTIONS=-Xmx7700m -Xms7700m > > Any advice? > > Martynas
Re: Fuseki container is OOMKilled
You're working on an in-memory dataset? Does it also happen with Jena 4.6.1? On 28.09.22 20:23, Martynas Jusevičius wrote: Hi, We have a dockerized Fuseki 4.5.0 instance that is gradually running out of memory over the course of a few hours. 3 datasets, none larger than 10 triples. The load is negligible (maybe a few bursts x 10 simple queries per minute), no updates. Dockerfile: https://github.com/AtomGraph/fuseki-docker/blob/master/Dockerfile Memory settings: mem_limit: 10240m JAVA_OPTIONS=-Xmx7700m -Xms7700m Any advice? Martynas
