Re: Out of memory

[Andy Seaborne]

This case should be optimized to be the flipped join(VALUES, BGP)

(prefix ((lsr: <lsr:>))
  (sequence
    (table (vars ?s)
      (row [?s lsr:10609f75-5cf3-4544-8fc1-c361778c3bd8])
      (row [?s lsr:88d0bb8c-35d8-4051-a27d-a0d93af77985])
      (row [?s lsr:fc7b2c65-453e-469b-9c5d-8c7ee4ee6902])
      (row [?s lsr:239c6da0-4c24-4539-a277-c9756d6257ee])
      (row [?s lsr:2ef0190d-6271-447a-992f-6225fc440897])
      (row [?s lsr:6aaf601c-ccf4-4e59-9757-1a463db49fa9])
      (row [?s lsr:d7c9dc96-cd61-4a31-b466-bb2491a3ceaf])
      (row [?s lsr:6f6802cf-0336-4234-90b8-cc8780058f0d])
      (row [?s lsr:d1e2751b-4332-4d57-95e4-ca8070c16782])
      (row [?s lsr:81053775-4722-4a00-b3f7-33d4feb3629b])
    )
    (bgp (triple ?s ?p ?o))))

    Andy

On 29/01/2019 14:28, Rob Vesse wrote:

This may be partly a case of a simple looking query having unexpected execution 
semantics.  Strictly speaking your query says select all triples in the 
specific graph then join them with these list of values for ?s.  Now the 
optimiser should, and does appear, to do the right thing and flip the join 
order i.e. it uses the concrete values from the VALUES block to search for 
triples with those subjects in the specific graph.  However if the query had 
other elements involved the optimiser might not kick in, a better query would 
place the VALUES prior to using the variables defined in the VALUES block.

This sounds like memory/cache thrashing.  From what you have described, running 
variants on this query 50k times, you are basically walking over your entire 
dataset extracting it piece by piece?

Assuming the Graph URI and the URIs in your VALUES block change in each query 
then every query is looking at a different section of the database causing a 
lot of data to be cached and then evicted both in terms of on-heap memory 
structures (the node table cache) and potentially also for the off heap memory 
mapped files which may be being paged in and out as the code traverses the 
B-Tree indexes.

Is there also some other query involved that extracts the Graph URIs and 
Subject URIs of interest that is being executed in parallel with the script?  
Or has the input from the script been pre-calculated ahead of time, comes from 
elsewhere etc?

Rob

On 29/01/2019, 14:06, "Mikael Pesonen" <mikael.peso...@lingsoft.fi> wrote:

Server: /usr/bin/java

     
-Dlog4j.configuration=file:/home/text/tools/apache-jena-fuseki-3.9.0/log4j.properties
     -Xmx5600M -jar fuseki-server.jar --update --port 3030
     --loc=/home/text/tools/jena_data_test/ /ds

No custom configs, default installation package. Sparql similar to this (returns 5-10 triplets) : CONSTRUCT { ?s ?p ?o }

     FROM <https://resource.lingsoft.fi/4f13c609-48b4-4e4d-a40b-2d7946f88234/>
     WHERE
     {
              ?s ?p ?o

VALUES ?s {lsr:10609f75-5cf3-4544-8fc1-c361778c3bd8

     lsr:88d0bb8c-35d8-4051-a27d-a0d93af77985
     lsr:fc7b2c65-453e-469b-9c5d-8c7ee4ee6902
     lsr:239c6da0-4c24-4539-a277-c9756d6257ee
     lsr:2ef0190d-6271-447a-992f-6225fc440897
     lsr:6aaf601c-ccf4-4e59-9757-1a463db49fa9
     lsr:d7c9dc96-cd61-4a31-b466-bb2491a3ceaf
     lsr:6f6802cf-0336-4234-90b8-cc8780058f0d
     lsr:d1e2751b-4332-4d57-95e4-ca8070c16782
     lsr:81053775-4722-4a00-b3f7-33d4feb3629b}
     }

I solved this by adding sleep to script. So I guess it's about the java

     memory manager not getting time to free memory? Even with sleep it was
     barely doable, memory consumption changing rapidly between 1,5 gig - 6 gig.

On 29/01/2019 15:50, Andy Seaborne wrote:

     > Mikael,
     >
     > There aren't enough details except to mention the suspects like sorting.
     >
     > With all the questions on the list, I personally don't track the
     > details of each installation so please also remind me of your current
     > setup.
     >
     >     Andy
     >
     > On 29/01/2019 11:32, Mikael Pesonen wrote:
     >>
     >> I'm not able to run a basic read-only script without running out of
     >> memory on the server.
     >>
     >> Consumption goes to 7+gigs (VM 10+ gigs), then system kills Fuseki
     >> when running out of memory.
     >> All I'm running is simple sparql query getting few triples of
     >> resource. This is run for about 50k times.
     >>
     >> All settings are default, using GSP.
     >>
     >>

--

     Lingsoft - 30 years of Leading Language Management

www.lingsoft.fi Speech Applications - Language Management - Translation - Reader's and Writer's Tools - Text Tools - E-books and M-books Mikael Pesonen

     System Engineer

e-mail: mikael.peso...@lingsoft.fi

     Tel. +358 2 279 3300

Time zone: GMT+2 Helsinki Office

     Eteläranta 10
     FI-00130 Helsinki
     FINLAND

Turku Office

     Kauppiaskatu 5 A
     FI-20100 Turku
     FINLAND