Re: [Virtuoso-users] Crash of virtuoso 7 stable - caused by OOM killer
Hi Hugh,
Sorry for wasting your time. I finally got the cause down.
When I saw this for the first time in the gdb
Program terminated with signal SIGKILL, Killed.
The program no longer exists.
And this in dmesg
Out of memory: Kill process 14738 (virtuoso-t) score 763 or sacrifice child
Killed process 14738, UID 21059, (virtuoso-t) total-vm:16179784kB,
anon-rss:12910420kB, file-rss:152kB
The total-vm size in that dmesg line is surprising its is nearly 16gb of
memory use.
Yet the NumberOfBuffers is 904897 (about 8 gb assuming 9KB pages) and
MaxDirtyBuffers 50. The odd size of the NumberOfBuffers is due to
auto sizing this setting in the conf from the managing process as
virtuoso is basically running in a embedded/lite mode.
The server is in O_DIRECT mode (set to 1)
What can I set so that this does not happen? i.e. why is the memory
consumption so large during this SPARQL update? Is it because my max
result size?
Regards,
Jerven
On 13/02/14 20:57, Jerven Bolleman wrote:
Hi Hugh,
Yes, that should be the file (all those are the same)
I switched back to the stable branch because the develop/7 branch would not
build cleanly when I tried.
This happened before and you guys normally fix it and I was not in a hurry.
I finally have the debug build working and ulimit set etc… so I think I will
send the backtrace tomorrow.
Regards,
Jerven
On 13 Feb 2014, at 17:52, Hugh Williams hwilli...@openlinksw.com wrote:
Hi Jerven,
From our point of view there is no real value in building from the stable/7
branch from July last year, as it is about to be superseded in the new VOS 7
release by develop/7 when finalised, which is what we would be more interested
in knowing of any problem being encountered using ?
In your email below you mention loading the go.rdf.gz uniprot dataset, do you
have URL to what you loaded or is it one of:
resources
url type=ftp location=ch preference=100
ftp://ftp.expasy.org/databases/uniprot/current_release/rdf/go.rdf.gz
/url
url type=ftp location=uk preference=100
ftp://ftp.ebi.ac.uk/pub/databases/uniprot/current_release/rdf/go.rdf.gz
/url
url type=ftp location=us preference=100
ftp://ftp.uniprot.org/pub/databases/uniprot/current_release/rdf/go.rdf.gz
/url
/resources
and I can load here to try recreate your problem with develop/7 ?
Best Regards
Hugh Williams
Professional Services
OpenLink Software, Inc. // http://www.openlinksw.com/
Weblog -- http://www.openlinksw.com/blogs/
LinkedIn -- http://www.linkedin.com/company/openlink-software/
Twitter -- http://twitter.com/OpenLink
Google+ -- http://plus.google.com/100570109519069333827/
Facebook -- http://www.facebook.com/OpenLinkSoftware
Universal Data Access, Integration, and Management Technology Providers
On 13 Feb 2014, at 10:14, Jerven Bolleman jerven.bolle...@isb-sib.ch wrote:
That was premature, I just reran the experiment and now it crashed again :(
Will try to see where that core dump went.
Regards,
Jerven
On 13/02/14 09:25, Jerven Bolleman wrote:
Hi All,
I have managed to build a debug build.
The funny thing is it now succeeds!
I do get this message once the next time I call a checkpoint.
18:02:22 suspect to miss a flush of L=100 in cpt, line 1008
18:02:23 Buffer 0x7fc9f79516f8 occupied in cpt
Regards,
Jerven
On 11/02/14 16:06, Jerven Bolleman wrote:
Hi Hugh,
I don't understand how to build the debug release.
I can't find the lines to patch e.g.
cd /scratch/virtuoso-opensource
git pull;git status;
grep CONFIGURE_ARGS Makefile
remote: Counting objects: 298, done.
remote: Compressing objects: 100% (298/298), done.
remote: Total 298 (delta 174), reused 0 (delta 0)
Receiving objects: 100% (298/298), 1.44 MiB | 974 KiB/s, done.
Resolving deltas: 100% (174/174), done.
From git://github.com/openlink/virtuoso-opensource
74ed20c..0b71fb6 develop/7 - origin/develop/7
Already up-to-date.
# On branch stable/7
nothing to commit (working directory clean)
As you can notice I am now trying out the stable branch.
Because when I tried building the develop/7 branch yesterday I had a
failure.
I do have a better idea of what is going on to trigger the bug.
First of all I have rewritten the update statement to look like this.
PREFIX rdfs:http://www.w3.org/2000/01/rdf-schema#
INSERT { GRAPH http://beta.sparql.uniprot.org/taxonomy {
?sub rdfs:subClassOf ?super}}
WHERE {
SELECT DISTINCT ?super ?sub
WHERE {GRAPH http://beta.sparql.uniprot.org/taxonomy {
?sub rdfs:subClassOf [] .
?sub rdfs:subClassOf/rdfs:subClassOf+ ?super .
[] rdfs:subClassOf ?super . }
}
}
Which runs in about 25 minutes for the taxonomy data I have been using.
Once the TransactionAfterImageLimit is raised (in my case to about 1GB
or 10 bytes)
Then I try loading the go.rdf.gz data into a named which works fine and
run the nearly identical query.
PREFIX rdfs:http://www.w3.org/2000/01/rdf-schema#
INSERT { GRAPH
Re: [Virtuoso-users] Crash of virtuoso 7 stable - caused by OOM killer
Hi Hugh, It is run via sesame. I will try running it directly as a prepared statement via JDBC and see if that makes a difference. Regards, Jerven On 17/02/14 14:02, Hugh Williams wrote: Hi Jerven, Not a waste of time as you still have an apparent issue ... Going back to the original issue I am downloading the go.rdf.gz dataset to load and run the test SPARQL insert query you provided against the Virtuoso instance. Can you confirm if this query is executed via Sesame or can it be run via isql to see the same problem ? Best Regards Hugh Williams Professional Services OpenLink Software, Inc. // http://www.openlinksw.com/ Weblog -- http://www.openlinksw.com/blogs/ LinkedIn -- http://www.linkedin.com/company/openlink-software/ Twitter -- http://twitter.com/OpenLink Google+ -- http://plus.google.com/100570109519069333827/ Facebook -- http://www.facebook.com/OpenLinkSoftware Universal Data Access, Integration, and Management Technology Providers On 17 Feb 2014, at 08:53, Jerven Bolleman jerven.bolle...@isb-sib.ch mailto:jerven.bolle...@isb-sib.ch wrote: Hi Hugh, Sorry for wasting your time. I finally got the cause down. When I saw this for the first time in the gdb Program terminated with signal SIGKILL, Killed. The program no longer exists. And this in dmesg Out of memory: Kill process 14738 (virtuoso-t) score 763 or sacrifice child Killed process 14738, UID 21059, (virtuoso-t) total-vm:16179784kB, anon-rss:12910420kB, file-rss:152kB The total-vm size in that dmesg line is surprising its is nearly 16gb of memory use. Yet the NumberOfBuffers is 904897 (about 8 gb assuming 9KB pages) and MaxDirtyBuffers 50. The odd size of the NumberOfBuffers is due to auto sizing this setting in the conf from the managing process as virtuoso is basically running in a embedded/lite mode. The server is in O_DIRECT mode (set to 1) What can I set so that this does not happen? i.e. why is the memory consumption so large during this SPARQL update? Is it because my max result size? Regards, Jerven On 13/02/14 20:57, Jerven Bolleman wrote: Hi Hugh, Yes, that should be the file (all those are the same) I switched back to the stable branch because the develop/7 branch would not build cleanly when I tried. This happened before and you guys normally fix it and I was not in a hurry. I finally have the debug build working and ulimit set etc… so I think I will send the backtrace tomorrow. Regards, Jerven On 13 Feb 2014, at 17:52, Hugh Williams hwilli...@openlinksw.com mailto:hwilli...@openlinksw.com wrote: Hi Jerven, From our point of view there is no real value in building from the stable/7 branch from July last year, as it is about to be superseded in the new VOS 7 release by develop/7 when finalised, which is what we would be more interested in knowing of any problem being encountered using ? In your email below you mention loading the go.rdf.gz uniprot dataset, do you have URL to what you loaded or is it one of: resources url type=ftp location=ch preference=100 ftp://ftp.expasy.org/databases/uniprot/current_release/rdf/go.rdf.gz /url url type=ftp location=uk preference=100 ftp://ftp.ebi.ac.uk/pub/databases/uniprot/current_release/rdf/go.rdf.gz /url url type=ftp location=us preference=100 ftp://ftp.uniprot.org/pub/databases/uniprot/current_release/rdf/go.rdf.gz /url /resources and I can load here to try recreate your problem with develop/7 ? Best Regards Hugh Williams Professional Services OpenLink Software, Inc. // http://www.openlinksw.com/ Weblog -- http://www.openlinksw.com/blogs/ LinkedIn -- http://www.linkedin.com/company/openlink-software/ Twitter -- http://twitter.com/OpenLink Google+ -- http://plus.google.com/100570109519069333827/ Facebook -- http://www.facebook.com/OpenLinkSoftware Universal Data Access, Integration, and Management Technology Providers On 13 Feb 2014, at 10:14, Jerven Bolleman jerven.bolle...@isb-sib.ch wrote: That was premature, I just reran the experiment and now it crashed again :( Will try to see where that core dump went. Regards, Jerven On 13/02/14 09:25, Jerven Bolleman wrote: Hi All, I have managed to build a debug build. The funny thing is it now succeeds! I do get this message once the next time I call a checkpoint. 18:02:22 suspect to miss a flush of L=100 in cpt, line 1008 18:02:23 Buffer 0x7fc9f79516f8 occupied in cpt Regards, Jerven On 11/02/14 16:06, Jerven Bolleman wrote: Hi Hugh, I don't understand how to build the debug release. I can't find the lines to patch e.g. cd /scratch/virtuoso-opensource git pull;git status; grep CONFIGURE_ARGS Makefile remote: Counting objects: 298, done. remote: Compressing objects: 100% (298/298), done. remote: Total 298 (delta 174), reused 0 (delta 0) Receiving objects: 100% (298/298), 1.44 MiB | 974 KiB/s, done. Resolving deltas: 100% (174/174), done. From
Re: [Virtuoso-users] Crash of virtuoso 7 stable - caused by OOM killer
Hi Hugh,
Running it as a preparedStatement does change the outcome
I see these again
14:12:30 * Monitor: Low query memory limit, try to increase MaxQueryMem
again,
While running I had a look at status();
And it gives a very long list of
16384: IER 8
80: ISR NO OWNER
76: ISR NO OWNER
72: ISR NO OWNER
68: ISR NO OWNER
64: ISR NO OWNER
60: ISR NO OWNER
56: ISR NO OWNER
52: ISR NO OWNER
48: ISR NO OWNER
44: ISR NO OWNER
40: ISR NO OWNER
36: ISR NO OWNER
32: ISR NO OWNER
28: ISR NO OWNER
24: ISR NO OWNER
20: ISR NO OWNER
etc.. etc..
Yet in the end using a prepared statement or not does not make a difference.
Also using isql.
SQL sparql PREFIX rdfs:http://www.w3.org/2000/01/rdf-schema#
INSERT { GRAPH http://beta.sparql.uniprot.org/go/ {
?sub rdfs:subClassOf ?super}}
WHERE { SELECT DISTINCT ?super ?sub
WHERE {GRAPH http://beta.sparql.uniprot.org/go/ {
?sub rdfs:subClassOf [] .
?sub rdfs:subClassOf/rdfs:subClassOf+ ?super .
[] rdfs:subClassOf ?super .
}
}
}Type the rest of statement, end with a semicolon (;) Type the rest of
statement, end with a semicolon (;) Type the rest of statement, end
with a semicolon (;) Type the rest of statement, end with a semicolon
(;) Type the rest of statement, end with a semicolon (;) Type the rest
of statement, end with a semicolon (;) Type the rest of statement, end
with a semicolon (;) Type the rest of statement, end with a semicolon
(;) Type the rest of statement, end with a semicolon (;) Type the rest
of statement, end with a semicolon (;) ;
*** Error 08S01: [Virtuoso Driver]CL065: Lost connection to server
in lines 2-12 of Top-Level:
#line 2 (console)
sparql PREFIX rdfs:http://www.w3.org/2000/01/rdf-schema#
INSERT { GRAPH http://beta.sparql.uniprot.org/go/ {
?sub rdfs:subClassOf ?super}}
WHERE { SELECT DISTINCT ?super ?sub
WHERE {GRAPH http://beta.sparql.uniprot.org/go/ {
?sub rdfs:subClassOf [] .
?sub rdfs:subClassOf/rdfs:subClassOf+ ?super .
[] rdfs:subClassOf ?super .
}
}
}
Regards,
Jerven
On 17/02/14 14:04, Jerven Bolleman wrote:
Hi Hugh,
It is run via sesame. I will try running it directly as a prepared
statement via JDBC and see if that makes a difference.
Regards,
Jerven
On 17/02/14 14:02, Hugh Williams wrote:
Hi Jerven,
Not a waste of time as you still have an apparent issue ...
Going back to the original issue I am downloading the go.rdf.gz dataset
to load and run the test SPARQL insert query you provided against the
Virtuoso instance. Can you confirm if this query is executed via Sesame
or can it be run via isql to see the same problem ?
Best Regards
Hugh Williams
Professional Services
OpenLink Software, Inc. // http://www.openlinksw.com/
Weblog -- http://www.openlinksw.com/blogs/
LinkedIn -- http://www.linkedin.com/company/openlink-software/
Twitter -- http://twitter.com/OpenLink
Google+ -- http://plus.google.com/100570109519069333827/
Facebook -- http://www.facebook.com/OpenLinkSoftware
Universal Data Access, Integration, and Management Technology Providers
On 17 Feb 2014, at 08:53, Jerven Bolleman jerven.bolle...@isb-sib.ch
mailto:jerven.bolle...@isb-sib.ch wrote:
Hi Hugh,
Sorry for wasting your time. I finally got the cause down.
When I saw this for the first time in the gdb
Program terminated with signal SIGKILL, Killed.
The program no longer exists.
And this in dmesg
Out of memory: Kill process 14738 (virtuoso-t) score 763 or sacrifice
child
Killed process 14738, UID 21059, (virtuoso-t) total-vm:16179784kB,
anon-rss:12910420kB, file-rss:152kB
The total-vm size in that dmesg line is surprising its is nearly 16gb
of memory use.
Yet the NumberOfBuffers is 904897 (about 8 gb assuming 9KB pages) and
MaxDirtyBuffers 50. The odd size of the NumberOfBuffers is due to
auto sizing this setting in the conf from the managing process as
virtuoso is basically running in a embedded/lite mode.
The server is in O_DIRECT mode (set to 1)
What can I set so that this does not happen? i.e. why is the memory
consumption so large during this SPARQL update? Is it because my max
result size?
Regards,
Jerven
On 13/02/14 20:57, Jerven Bolleman wrote:
Hi Hugh,
Yes, that should be the file (all those are the same)
I switched back to the stable branch because the develop/7 branch
would not build cleanly when I tried.
This happened before and you guys normally fix it and I was not in a
hurry.
I finally have the debug build working and ulimit set etc… so I think
I will send the backtrace tomorrow.
Regards,
Jerven
On 13 Feb 2014, at
Re: [Virtuoso-users] Crash of virtuoso 7 stable - caused by OOM killer
I will run virtuoso in valgrind --leak-check-yes and see what comes out. Regards, Jerven On 17/02/14 15:32, Hugh Williams wrote: Hi Jerven, Ok, if you could provide a compilable Sesame or JDBC program demonstrating this issue then we can attempt the recreation with it as I have just been executing it from the isql command line tool ? Best Regards Hugh Williams Professional Services OpenLink Software, Inc. // http://www.openlinksw.com/ Weblog -- http://www.openlinksw.com/blogs/ LinkedIn -- http://www.linkedin.com/company/openlink-software/ Twitter -- http://twitter.com/OpenLink Google+ -- http://plus.google.com/100570109519069333827/ Facebook -- http://www.facebook.com/OpenLinkSoftware Universal Data Access, Integration, and Management Technology Providers On 17 Feb 2014, at 13:04, Jerven Bolleman jerven.bolle...@isb-sib.ch mailto:jerven.bolle...@isb-sib.ch wrote: Hi Hugh, It is run via sesame. I will try running it directly as a prepared statement via JDBC and see if that makes a difference. Regards, Jerven On 17/02/14 14:02, Hugh Williams wrote: Hi Jerven, Not a waste of time as you still have an apparent issue ... Going back to the original issue I am downloading the go.rdf.gz dataset to load and run the test SPARQL insert query you provided against the Virtuoso instance. Can you confirm if this query is executed via Sesame or can it be run via isql to see the same problem ? Best Regards Hugh Williams Professional Services OpenLink Software, Inc. // http://www.openlinksw.com/ Weblog -- http://www.openlinksw.com/blogs/ LinkedIn -- http://www.linkedin.com/company/openlink-software/ Twitter -- http://twitter.com/OpenLink Google+ -- http://plus.google.com/100570109519069333827/ Facebook -- http://www.facebook.com/OpenLinkSoftware Universal Data Access, Integration, and Management Technology Providers On 17 Feb 2014, at 08:53, Jerven Bolleman jerven.bolle...@isb-sib.ch mailto:jerven.bolle...@isb-sib.ch mailto:jerven.bolle...@isb-sib.ch wrote: Hi Hugh, Sorry for wasting your time. I finally got the cause down. When I saw this for the first time in the gdb Program terminated with signal SIGKILL, Killed. The program no longer exists. And this in dmesg Out of memory: Kill process 14738 (virtuoso-t) score 763 or sacrifice child Killed process 14738, UID 21059, (virtuoso-t) total-vm:16179784kB, anon-rss:12910420kB, file-rss:152kB The total-vm size in that dmesg line is surprising its is nearly 16gb of memory use. Yet the NumberOfBuffers is 904897 (about 8 gb assuming 9KB pages) and MaxDirtyBuffers 50. The odd size of the NumberOfBuffers is due to auto sizing this setting in the conf from the managing process as virtuoso is basically running in a embedded/lite mode. The server is in O_DIRECT mode (set to 1) What can I set so that this does not happen? i.e. why is the memory consumption so large during this SPARQL update? Is it because my max result size? Regards, Jerven On 13/02/14 20:57, Jerven Bolleman wrote: Hi Hugh, Yes, that should be the file (all those are the same) I switched back to the stable branch because the develop/7 branch would not build cleanly when I tried. This happened before and you guys normally fix it and I was not in a hurry. I finally have the debug build working and ulimit set etc… so I think I will send the backtrace tomorrow. Regards, Jerven On 13 Feb 2014, at 17:52, Hugh Williams hwilli...@openlinksw.com mailto:hwilli...@openlinksw.com mailto:hwilli...@openlinksw.com wrote: Hi Jerven, From our point of view there is no real value in building from the stable/7 branch from July last year, as it is about to be superseded in the new VOS 7 release by develop/7 when finalised, which is what we would be more interested in knowing of any problem being encountered using ? In your email below you mention loading the go.rdf.gz uniprot dataset, do you have URL to what you loaded or is it one of: resources url type=ftp location=ch preference=100 ftp://ftp.expasy.org/databases/uniprot/current_release/rdf/go.rdf.gz /url url type=ftp location=uk preference=100 ftp://ftp.ebi.ac.uk/pub/databases/uniprot/current_release/rdf/go.rdf.gz /url url type=ftp location=us preference=100 ftp://ftp.uniprot.org/pub/databases/uniprot/current_release/rdf/go.rdf.gz /url /resources and I can load here to try recreate your problem with develop/7 ? Best Regards Hugh Williams Professional Services OpenLink Software, Inc. // http://www.openlinksw.com/ Weblog -- http://www.openlinksw.com/blogs/ LinkedIn -- http://www.linkedin.com/company/openlink-software/ Twitter -- http://twitter.com/OpenLink Google+ -- http://plus.google.com/100570109519069333827/ Facebook -- http://www.facebook.com/OpenLinkSoftware Universal Data Access, Integration, and Management Technology Providers On 13 Feb 2014, at 10:14, Jerven Bolleman
