: Mystery memory leak in fuseki
Does Fuseki have direct code dependency on Jetty? Or would it be possible
to try switching to a different servlet container such as Tomcat?
JAX-RS, which I’ve advocated here multiple times, provides such a
higher-level abstraction above servlets that would enable easy
Does Fuseki have direct code dependency on Jetty? Or would it be possible
to try switching to a different servlet container such as Tomcat?
JAX-RS, which I’ve advocated here multiple times, provides such a
higher-level abstraction above servlets that would enable easy switching.
On Fri, 25 Aug
On 25/08/2023 11:44, Andy Seaborne wrote:
On 03/07/2023 14:20, Dave Reynolds wrote:
We have a very strange problem with recent fuseki versions when
running (in docker containers) on small machines. Suspect a jetty
issue but it's not clear.
From the threads here, it does seem to be Jetty
On 03/07/2023 14:20, Dave Reynolds wrote:
We have a very strange problem with recent fuseki versions when running
(in docker containers) on small machines. Suspect a jetty issue but it's
not clear.
From the threads here, it does seem to be Jetty related.
I haven't managed to reproduce the
s not
> detailed
> > > in that documentation.
> > > > >
> > > > > Best guess from digging through their code is that the
> “heuristic”
> > > is this:
> > > > >
> > > > >
> > >
> https://github.com/eclips
ty-io/src/main/java/org/eclipse/jetty/io/AbstractByteBufferPool.java#L78-L84
> > > >
> > > > i.e., ¼ of the configured max heap size.This doesn’t necessarily
> > align with the exact sizes of process growth you see but I note the
> > documentation does ex
make the process size smaller and
sufficiently predictable for your use case?
> >
> > Rob
> >
> > From: Dave Reynolds
> > Date: Tuesday, 11 July 2023 at 08:58
> > To: users@jena.apache.org
> > Subject: Re: Mystery memory leak in fuseki
>
Thanks Dave, I am not familiar with Prometheus JVM metrics but I gather
it's an open source solution that you have coupled with grafana for
visualization. I will have a look into this.
Best,
Marco
On Tue, Jul 11, 2023 at 9:32 AM Dave Reynolds
wrote:
> Hi Marco,
>
> On 11/07/2023 09:04, Marco
for your use case?
Rob
From: Dave Reynolds
Date: Tuesday, 11 July 2023 at 08:58
To: users@jena.apache.org
Subject: Re: Mystery memory leak in fuseki
For interest[*] ...
This is what the core JVM metrics look like when transitioning from a
Jetty10 to a Jetty9.4 instance. You can see the direct buffer
: Mystery memory leak in fuseki
For interest[*] ...
This is what the core JVM metrics look like when transitioning from a
Jetty10 to a Jetty9.4 instance. You can see the direct buffer cycling up
to 500MB (which happens to be the max heap setting) on Jetty 10, nothing
on Jetty 9. The drop in Mapped
Hi Marco,
On 11/07/2023 09:04, Marco Neumann wrote:
Dave, can you say a bit more about the profiling methodology? Are you using
a tool such as VisualVM to collect the data? Or do you just use the system
monitor?
The JVM metrics here are from prometheus scanning the metrics exposed by
fuseki
Dave, can you say a bit more about the profiling methodology? Are you using
a tool such as VisualVM to collect the data? Or do you just use the system
monitor?
Marco
On Tue, Jul 11, 2023 at 8:57 AM Dave Reynolds
wrote:
> For interest[*] ...
>
> This is what the core JVM metrics look like when
For interest[*] ...
This is what the core JVM metrics look like when transitioning from a
Jetty10 to a Jetty9.4 instance. You can see the direct buffer cycling up
to 500MB (which happens to be the max heap setting) on Jetty 10, nothing
on Jetty 9. The drop in Mapped buffers is just because
After a 10 hour test of 4.9.0 with Jetty 9.4 on java 17 in the
production, containerized, environment then it is indeed very stable.
Running at less that 6% of memory on 4GB machine compared to peaks of
~50% for versions with Jetty 10. RES shows as 240K with 35K shared
(presume mostly
Since this thread has got complex, I'm posting this update here at the
top level.
Thanks to folks, especially Andy and Rob for suggestions and for
investigating.
After a lot more testing at our end I believe we now have some workarounds.
First, at least on java 17, the process growth does
as visible to the JVM) are all stable.
Cheers,
Dave
Regards,
Rob
From: Dave Reynolds
Date: Friday, 7 July 2023 at 11:11
To: users@jena.apache.org
Subject: Re: Mystery memory leak in fuseki
Hi Andy,
Thanks for looking.
Good thought on some issue with stacked requests causing thread leak but
don't
anything of note, 5.19KB of memory leaks over a 3.5 hr run
so no smoking gun there.
Regards,
Rob
From: Dave Reynolds
Date: Friday, 7 July 2023 at 11:11
To: users@jena.apache.org
Subject: Re: Mystery memory leak in fuseki
Hi Andy,
Thanks for looking.
Good thought on some issue with stacked
Hi Andy,
Thanks for looking.
Good thought on some issue with stacked requests causing thread leak but
don't think that matches our data.
From the metrics the number of threads and total thread memory used is
not that great and is stable long term while the process size grows, at
least in
I tried running without any datasets. I get the same heap effect of
growing slowly then a dropping back.
Fuseki Main (fuseki-server did the same but the figures are from main -
there is less going on)
Version 4.8.0
fuseki -v --ping --empty# No datasets
4G heap.
71M allocated
4 threads
tion there.
Best regards,
Frank
Von: Dave Reynolds
Gesendet: Dienstag, 4. Juli 2023 12:16
An: users@jena.apache.org
Betreff: Re: Mystery memory leak in fuseki
Try that again:
For interest this is what the JVM metrics look like. The main
heap/non-heap ones are:
https://www.dropbox.com/s/
k situation there.
Best regards,
Frank
Von: Dave Reynolds
Gesendet: Dienstag, 4. Juli 2023 12:16
An: users@jena.apache.org
Betreff: Re: Mystery memory leak in fuseki
Try that again:
For interest this is what the JVM metrics look like. The main
heap/non-heap ones are:
https://www.dropbox
esendet: Dienstag, 4. Juli 2023 12:16
An: users@jena.apache.org
Betreff: Re: Mystery memory leak in fuseki
Try that again:
For interest this is what the JVM metrics look like. The main
heap/non-heap ones are:
https://www.dropbox.com/s/g1ih98kprnvjvxx/fusdeki-metrics-1.png?dl=0
So stable at aro
Rob
From: Dave Reynolds
Date: Tuesday, 4 July 2023 at 09:31
To: users@jena.apache.org
Subject: Re: Mystery memory leak in fuseki
Tried 4.7.0 under most up to date java 17 and it acts like 4.8.0. After
16hours it gets to about 1.6GB and by eye has nearly flatted off
somewhat but not comp
If you can reproduce it locally then attaching a profiler like
VisualVM so you can take a heap snapshot and see where the memory is
going that would be useful
Rob
From: Dave Reynolds
Date: Tuesday, 4 July 2023 at 09:31
To: users@jena.apache.org
Subject: Re: Mystery memory leak in fuseki
Tried
memory is going that would be useful
Rob
From: Dave Reynolds
Date: Tuesday, 4 July 2023 at 09:31
To: users@jena.apache.org
Subject: Re: Mystery memory leak in fuseki
Tried 4.7.0 under most up to date java 17 and it acts like 4.8.0. After
16hours it gets to about 1.6GB and by eye has nearly flatt
en attaching a profiler like VisualVM so
> you can take a heap snapshot and see where the memory is going that would
> be useful
>
> Rob
>
> From: Dave Reynolds
> Date: Tuesday, 4 July 2023 at 09:31
> To: users@jena.apache.org
> Subject: Re: Mystery memory leak in fuseki
>
2023 at 09:31
To: users@jena.apache.org
Subject: Re: Mystery memory leak in fuseki
Tried 4.7.0 under most up to date java 17 and it acts like 4.8.0. After
16hours it gets to about 1.6GB and by eye has nearly flatted off
somewhat but not completely.
For interest here's a MEM% curve on a 4GB box
Tried 4.7.0 under most up to date java 17 and it acts like 4.8.0. After
16hours it gets to about 1.6GB and by eye has nearly flatted off
somewhat but not completely.
For interest here's a MEM% curve on a 4GB box (hope the link works).
Thanks for the suggestion, that could be useful.
Not managed to make that work yet. From within the container get
permission denied, and running it on the host is no use because the
relevant so's aren't where ltrace expects and it crashes out.
Similarly strace can't attach to the process in
You might try running `ltrace` to watch the library calls and system calls
the jvm is making.
e.g.
ltrace -S -f -p
I think the `sbrk` system call is used to allocate memory. It might be
interesting to see if you can catch the jvm invoking that system call and
also see what is happening around
On 03/07/2023 14:36, Martynas Jusevičius wrote:
There have been a few similar threads:
https://www.mail-archive.com/users@jena.apache.org/msg19871.html
https://www.mail-archive.com/users@jena.apache.org/msg18825.html
Thanks, I've seen those and not sure they quite match our case but maybe
On 03/07/2023 15:07, Andy Seaborne wrote:
A possibility:
https://www.nickebbitt.com/blog/2022/01/26/the-story-of-a-java-17-native-memory-leak/
suggests workaround
-XX:-UseStringDeduplication
https://bugs.openjdk.org/browse/JDK-8277981
https://github.com/openjdk/jdk/pull/6613
which may be in
Hi Andy,
> Could you try 4.7.0?
Will do, though each test takes quite a while :)
> This is an in-memory database?
No TDB1, sorry should have said that.
Though as I say we are leaving the system to soak with absolutely no
queries arriving so it's not TDB churn and it's RSS that's filling up.
A possibility:
https://www.nickebbitt.com/blog/2022/01/26/the-story-of-a-java-17-native-memory-leak/
suggests workaround
-XX:-UseStringDeduplication
https://bugs.openjdk.org/browse/JDK-8277981
https://github.com/openjdk/jdk/pull/6613
which may be in Java 17.0.2
Andy
Hi Dave,
Could you try 4.7.0?
4.6.0 was 2022-08-20
4.7.0 was 2022-12-27
4.8.0 was 2023-04-20
This is an in-memory database?
Micrometer/Prometheus has had several upgrades but if it is not heap and
not direct memory (I though that was a hard bound set at start up), I
don't see how it can be
There have been a few similar threads:
https://www.mail-archive.com/users@jena.apache.org/msg19871.html
https://www.mail-archive.com/users@jena.apache.org/msg18825.html
On Mon, 3 Jul 2023 at 15.20, Dave Reynolds
wrote:
> We have a very strange problem with recent fuseki versions when running
36 matches
Mail list logo
