On 13/03/18 09:52, Adrian Gschwend wrote:
On 12.03.18 14:37, Andy Seaborne wrote:

Yes - all this was just using the command line tools.

Default heap but that's different for me and for you.  I'm running with
8G heap (25% of 32G).

4G didn't work.

right I tested it now, for the current set 7GB seems to be the barrier.
Earlier:
"""
   And I once allocated almost all I had on my system (> 8GB)
"""

I'm afraid it sounds like that didn't get set - if you can see the
commandline for the running java process via system tools, it will have
the -Xmx setting.  Or your data/query are different - I see %% stuff in
the update.

jup I said something wrong there, it was on 3g, sorry about.

No problem ("been there, done that, will do it again")

Don't over allocate - the TDB files are memory mapped files and that dos
not come out heap.

is there a rule of thumb for that? I now assigned 7GB on a 8GB machine.
But this is a VM spin up for that particular job (in Gitlab) and then
destroyed right afterwards.

The main sources of RAM usage are:

* NodeTable cache - allow 2G per database or the size of the nodes.dat file - usually only relevant for many databases,

* The transaction workspace (TDB1 only, not TBD2)

* The query workspace.

so it's hard to pin down because "workspace" depends a lot on what is done with the system. Large updates do use a lot of TDB1-heap space.

In TBD2, the transaction workspace is a fixed few 100 bytes of heap.

It's about to run out of heap.  java8 has a peculiar feature that when
heap usage grows, it tries to full GC to create space, then tries again
and again, ... to the point where the machine is only doing GCs which
are parallel hence all the CPUs go crazy.

ok that explains. Is that something I should add to the docs for others?
I've seen this pattern quite often but never made that link to memory.

It is an effect of Java8, not specific to Jena. Java8 has a differnt parallel GCs to Java7. I think the Java7-like one is -XX:-UseParallelOldGC.

And it will change again, the default GC in Java9 is G1 (Garbage First GC"). We'll find out how that manifests itself.

    Andy


regards

Adrian

Reply via email to