Re: Ever-increasing memory usage in Fuseki

[Martynas Jusevičius]

There were several long threads about this issue in the past months. I
think the consensus was it's Jetty-related, but I don't know if the issue
is addressed.

https://lists.apache.org/thread/31ytzp3p2zg3gcsm86t1xlh4nsmdcfkc
https://lists.apache.org/thread/b64trj1c9n9rt0xjowqt4j23h9cy3v4c
https://lists.apache.org/thread/m7ypdsndjosxmdsxp9ch437305qw9mwd

On Wed, Nov 1, 2023 at 8:43 PM Hugo Mills 
wrote:

> Hi,
>
>
>
> We’ve got an application we’ve inherited recently which uses a Fuseki
> database. It was originally Fuseki 3.4.0, and has been upgraded to 4.9.0
> recently. The 3.4.0 server needed regular restarts (once a day) in order to
> keep working; the 4.9.0 server is even more unreliable, and has been
> running out of memory and being OOM-killed multiple times a day. This
> afternoon, it crashed enough times, fast enough, to make Kubernetes go into
> a back-off loop, and brought the app down for some time.
>
>
>
> We’re using OpenJDK 19. The JVM options are: “-Xmx:30g -Xms18g”, and the
> container we’re running it in has a memory limit of 31 GiB. We tried the
> “-XX:+UserSerialGC” option this evening, but it didn’t seem to help much.
> We see the RAM usage of the java process rising steadily as queries are
> made, with occasional small, but insufficient, drops.
>
> The store is somewhere around 20M triples in size.
>
>
>
> Could anyone suggest any tweaks or options we could do to make this more
> stable, and not leak memory? We’ve downgraded to 3.4.0 again, and it’s not
> running out of space every few minutes at least, but it still has an
> ever-growing memory usage.
>
>
>
> Thanks,
>
> Hugo.
>
>
>
> *Dr. Hugo Mills*
>
> Senior Data Scientist
>
> hugo.mi...@agrimetrics.co.uk
>
>
> [image: Text Description automatically generated]
>
> *NEWS: *Visit our Data Marketplace
>  to explore our agrifood data
> catalogue.
>
> www.agrimetrics.co.uk
> [image: Icon Description automatically
> generated] [image: Icon
> Description automatically generated with medium confidence]
> 
>
>
>
>
>
>
>

Re: Ever-increasing memory usage in Fuseki

[Conal McLaughlin]

Hey Hugo,

Hard to say without more information but we have experienced containers
being killed like you described and solved it by setting
-XX:MaxDirectMemorySize.

Jena/TDB storage makes use of memory mapped files
https://jena.apache.org/documentation/tdb/store-parameters.html#file-access---mapped-and-direct-modes
by default so this has an effect on overall memory usage inside the
container that is not immediately apparent when observing JVM stats.

Conal
IOTICS


On Wed, 1 Nov 2023, 19:43 Hugo Mills,  wrote:

> Hi,
>
>
>
> We’ve got an application we’ve inherited recently which uses a Fuseki
> database. It was originally Fuseki 3.4.0, and has been upgraded to 4.9.0
> recently. The 3.4.0 server needed regular restarts (once a day) in order to
> keep working; the 4.9.0 server is even more unreliable, and has been
> running out of memory and being OOM-killed multiple times a day. This
> afternoon, it crashed enough times, fast enough, to make Kubernetes go into
> a back-off loop, and brought the app down for some time.
>
>
>
> We’re using OpenJDK 19. The JVM options are: “-Xmx:30g -Xms18g”, and the
> container we’re running it in has a memory limit of 31 GiB. We tried the
> “-XX:+UserSerialGC” option this evening, but it didn’t seem to help much.
> We see the RAM usage of the java process rising steadily as queries are
> made, with occasional small, but insufficient, drops.
>
> The store is somewhere around 20M triples in size.
>
>
>
> Could anyone suggest any tweaks or options we could do to make this more
> stable, and not leak memory? We’ve downgraded to 3.4.0 again, and it’s not
> running out of space every few minutes at least, but it still has an
> ever-growing memory usage.
>
>
>
> Thanks,
>
> Hugo.
>
>
>
> *Dr. Hugo Mills*
>
> Senior Data Scientist
>
> hugo.mi...@agrimetrics.co.uk
>
>
> [image: Text Description automatically generated]
>
> *NEWS: *Visit our Data Marketplace
>  to explore our agrifood data
> catalogue.
>
> www.agrimetrics.co.uk
> [image: Icon Description automatically
> generated] [image: Icon
> Description automatically generated with medium confidence]
> 
>
>
>
>
>
>
>

Ever-increasing memory usage in Fuseki

[Hugo Mills]

Hi,

We've got an application we've inherited recently which uses a Fuseki database. 
It was originally Fuseki 3.4.0, and has been upgraded to 4.9.0 recently. The 
3.4.0 server needed regular restarts (once a day) in order to keep working; the 
4.9.0 server is even more unreliable, and has been running out of memory and 
being OOM-killed multiple times a day. This afternoon, it crashed enough times, 
fast enough, to make Kubernetes go into a back-off loop, and brought the app 
down for some time.

We're using OpenJDK 19. The JVM options are: "-Xmx:30g -Xms18g", and the 
container we're running it in has a memory limit of 31 GiB. We tried the 
"-XX:+UserSerialGC" option this evening, but it didn't seem to help much. We 
see the RAM usage of the java process rising steadily as queries are made, with 
occasional small, but insufficient, drops.
The store is somewhere around 20M triples in size.

Could anyone suggest any tweaks or options we could do to make this more 
stable, and not leak memory? We've downgraded to 3.4.0 again, and it's not 
running out of space every few minutes at least, but it still has an 
ever-growing memory usage.

Thanks,
Hugo.

Dr. Hugo Mills
Senior Data Scientist
hugo.mi...@agrimetrics.co.uk

[Text  Description automatically generated]
NEWS: Visit our Data Marketplace to 
explore our agrifood data catalogue.
www.agrimetrics.co.uk
[Icon  Description automatically 
generated][Icon  Description 
automatically generated with medium confidence]

[ANN] Apache Jena 4.10.0

[Andy Seaborne]

The Apache Jena development community is pleased to
announce the release of Apache Jena 4.10.0

In this release:

* Prepare for Jena5

  Check use of deprecated API calls
These are largely being removed in Jena5.

  Jena5 will require Java17

  jena5 Fuseki will switch from javax.servlet to jakarta.servlet
This will require use of Apache Tomcat 10 to run the WAR file.

  jena-jdbc is planned for retirement in Jena 5.0.0

See the Jena5 label in the github issues area:

https://github.com/apache/jena/issues?q=is%3Aissue+label%3Ajena5

* Development will switch to Jena5.
  The 'main' branch is now for Jena5 development.
  There is a branch 'jena4' marking the 4.10.0 release

== Notes

All issues: https://s.apache.org/jena-4.10.0-issues

There is a CHANGES.txt in the root of the repository
with the history of announcement messages.

 Contributions:

Shawn Smith
"Race condition with QueryEngineRegistry and
UpdateEngineRegistry init()"
  https://issues.apache.org/jira/browse/JENA-2356

Ali Ariff
"Labeling for Blank Nodes Across Writers"
  https://github.com/apache/jena/issues/1997

sszuev
"jena-core: add more javadocs about Graph-mem thread-safety and 
ConcurrentModificationException"

  https://github.com/apache/jena/pull/1994

sszuev
GH-1419: fix DatasetGraphMap#clear
  https://github.com/apache/jena/issue/1419

sszuev
GH-1374: add copyWithRegisties Context helper method
  https://github.com/apache/jena/issue/1374


All issues in this release:
https://s.apache.org/jena-4.10.0-issues

 Key upgrades

org.apache.lucene : 9.5.0 -> 9.7.0
org.apache.commons:commons-lang3: 3.12.0 -> 3.13.0
org.apache.sis.core:sis-referencing : 1.1 -> 1.4

== Obtaining Apache Jena 4.10.0

* Via central.maven.org

The main jars and their dependencies can used with:

  
org.apache.jena
apache-jena-libs
pom
4.10.0
  

Full details of all maven artifacts are described at:

http://jena.apache.org/download/maven.html

* As binary downloads

Apache Jena libraries are available as a binary distribution of
libraries. For details of a global mirror copy of Jena binaries please see:

http://jena.apache.org/download/

* Source code for the release

The signed source code of this release is available at:

http://www.apache.org/dist/jena/source/

and the signed master source for all Apache Jena releases is available
at: http://archive.apache.org/dist/jena/

== Contributing

If you would like to help out, a good place to look is the list of
unresolved JIRA at:

https://https://github.com/apache/jena/issuesissues-current

or review pull requests at

https://github.com/apache/jena/pulls

or drop into the dev@ list.

We use github pull requests and other ways for accepting code:
 https://github.com/apache/jena/blob/master/CONTRIBUTING.md