Re: Re: Mystery memory leak in fuseki

[Martynas Jusevičius]

There is one more variable in this picture: Java’s container awareness
https://developers.redhat.com/articles/2022/04/19/java-17-whats-new-openjdks-container-awareness

Whether it has an impact in this case, I have no idea :)

On Thu, 20 Jul 2023 at 11.48, Conal McLaughlin 
wrote:

> Hey Andy,
>
> Metaspace seems to be stable!
>
> We’re running this on Java 11 currently.
> I can check it out with Java17 though.
>
> We’ve currently set Xms/Xmx to 2560MB & MaxMetaspaceSize to 256MB.
>
> The ECS task is set with a ceiling of 4GB Memory & 1vcpu.
>
> Could it be more of a race condition than size of used objects, due to
> logging?
> I do see some time sensitive eviction code in Jetty -
> https://github.com/eclipse/jetty.project/blob/9e16d81cf8922c75e3d2d96c66442b896a9c69e1/jetty-io/src/main/java/org/eclipse/jetty/io/ArrayRetainableByteBufferPool.java#L374
> Not sure if the same type of thing exists in the Jena codebase.
>
> I will try to check with `—empty` also.
>
>
> Thanks,
> Conal
>
> On 2023/07/19 21:10:24 Andy Seaborne wrote:
> > Conal,
> >
> > Thanks for the information.
> > Can you see if metaspace is growing as well?
> >
> > All,
> >
> > Could someone please try running Fuseki main, with no datasets (--empty)
> > with some healthcheck ping traffic.
> >
> >  Andy
> >
> > On 19/07/2023 14:42, Conal McLaughlin wrote:
> > > Hey Dave,
> > >
> > > Thank you for providing an in depth analysis of your issues.
> > > We appear to be witnessing the same type of problems with our current
> > > Fuseki deployment.
> > > We are deploying a containerised Fuseki into a AWS ECS task alongside
> > > other containers - this may not be ideal but that’s a different story.
> > >
> > > I just wanted to add another data point to everything you have
> described.
> > > Firstly, it does seem like “idle” (or very low traffic) instances are
> > > the problem, for us (coupled with a larger heap than necessary).
> > > We witness the same increase in the ECS task memory consumption up
> until
> > > the whole thing is killed off. Which includes the Fuseki container.
> > >
> > > In an attempt to see what was going on beneath the hood, we turned up
> > > the logging to TRACE in the log4j2.xml file provided to Fuseki.
> > > This appeared to stabilise the increasing memory consumption.
> > > Even just switching the `logger.jetty.level` to TRACE alleviates the
> issue.
> >
> > Colour me confused!
> >
> > A Log4j logger that is active will use a few objects - may that's enough
> > to trigger a minor GC which in turn is enough to flush some non-heap
> > resources.
> >
> > How big is the heap?
> > This is Java17?
> >
> > > We are testing this on Fuseki 4.8.0/TDB2 with close to 0 triples and
> > > extremely low query traffic / health checks via /ping.
> > > KPk7uhH2F9Lp.png
> > > ecs-task-memory - Image on Pasteboard
> > > 
> > > pasteboard.co 
> > >
> > > 
> > >
> > >
> > > Cheers,
> > > Conal
> > >
> > > On 2023/07/11 09:31:25 Dave Reynolds wrote:
> > >  > Hi Rob,
> > >  >
> > >  > Good point. Will try to find time to experiment with that but given
> the
> > >  > testing cycle time that will take a while and can't start
> immediately.
> > >  >
> > >  > I'm a little sceptical though. As mentioned before, all the metrics
> we
> > >  > see show the direct memory pool that Jetty uses cycling up the max
> heap
> > >  > size and then being collected but with no long term growth to match
> the
> > >  > process size growth. This really feels more like a bug (though not
> sure
> > >  > where) than tuning. The fact that actual behaviour doesn't match the
> > >  > documentation isn't encouraging.
> > >  >
> > >  > It's also pretty hard to figure what the right pool configuration
> would
> > >  > be. This thing is just being asked to deliver a few metrics (12KB
> per
> > >  > request) several times a minute but manages to eat 500MB of direct
> > >  > buffer space every 5mins. So what the right pool parameters are to
> > >  > support real usage peaks is not going to be easy to figure out.
> > >  >
> > >  > None the less you are right. That's something that should be
> explored.
> > >  >
> > >  > Dave
> > >  >
> > >  >
> > >  > On 11/07/2023 09:45, Rob @ DNR wrote:
> > >  > > Dave
> > >  > >
> > >  > > Thanks for the further information.
> > >  > >
> > >  > > Have you experimented with using Jetty 10 but providing more
> > > detailed configuration?Fuseki supports providing detailed Jetty
> > > configuration if needed via the --jetty-config option
> > >  > >
> > >  > > The following section look relevant:
> > >  > >
> > >  > >
> > >
> https://eclipse.dev/jetty/documentation/jetty-10/operations-guide/index.html#og-module-bytebufferpool
> > >  > >
> > >  > > It looks like the default is that Jetty uses a heuristic to
> > > determine these values, sadly the heuristic in question is not
> detailed
> > > in that documentation.
> > >  > >
> 

Re: OOM Killed

[Laura Morales]

> Could you try Java17?

I did try Java 17 (default jre available from Debian) but I didn't notice any 
difference. I only ran 2 tests with Java 17 though, because the job takes a 
long time to fail/finish. I only noticed a significant difference when tweaking 
the GC options.

However I did another test on a 3rd PC, with the same exact software setup 
(Debian 12, Java 17, Fuseki 4.9.0) but completely different hardware (i5 
7th-gen 4C4T, 16GB RAM, NVMe PCIe-3 x4) and it completed in ~30m (instead of 
~2h) and the max RAM usage for the Java process was 11.3GB *without* tuning any 
GC or Fuseki options.

I don't know what to make of this. Maybe it's the hardware after all, and not 
Fuseki. Or maybe the GC has more time for doing its thing with a NVMe drive 
instead of SATA/USB. However 11GB still seems very high. As far as I know, each 
HTTP request should be short lived, and all the memory of every HTTP request 
should be freed (that's why it's strange that I see the memory grow over time).