Hello all,
I am new to this list and to OWLIM-SE and was wondering if anyone could offer
advice for loading a large triple store. I am trying to load 670M triples into
a repository using the openrdf-sesame workbench under tomcat6 on a single linux
VM with 64-bit hardware and 64GB of memory.
My JVM has the following: -Xms32g -Xmx32g -XX:MaxPermSize=256m
Here is the log info for my repository configuration:
...
[INFO ] 2013-03-27 13:57:00,720 [repositories/BFWorks_STF] Configured parameter
'entity-id-size' to '32'
[INFO ] 2013-03-27 13:57:00,720 [repositories/BFWorks_STF] Configured parameter
'enable-context-index' to 'false'
[INFO ] 2013-03-27 13:57:00,720 [repositories/BFWorks_STF] Configured parameter
'entity-index-size' to '100000000'
[INFO ] 2013-03-27 13:57:00,720 [repositories/BFWorks_STF] Configured parameter
'tuple-index-memory' to '1600m'
[INFO ] 2013-03-27 13:57:00,721 [repositories/BFWorks_STF] Configured parameter
'cache-memory' to '3200m'
[INFO ] 2013-03-27 13:57:00,721 [repositories/BFWorks_STF] Cache pages for
tuples: 83886
[INFO ] 2013-03-27 13:57:00,721 [repositories/BFWorks_STF] Cache pages for
predicates: 0
[INFO ] 2013-03-27 13:57:00,721 [repositories/BFWorks_STF] Configured parameter
'storage-folder' to 'storage'
[INFO ] 2013-03-27 13:57:00,741 [repositories/BFWorks_STF] Configured parameter
'in-memory-literal-properties' to 'false'
[INFO ] 2013-03-27 13:57:00,742 [repositories/BFWorks_STF] Configured parameter
'repository-type' to 'file-repository'
The loading came to a standstill after 19 hours and tomcat threw an
OutOfMemoryError: GC overhead limit exceeded.
My question is what the application is doing with all this memory and whether I
configured my instance correctly for this load to finish. I also see a lot of
entries in the main log such as this:
[WARN ] 2013-03-28 08:50:59,114 [repositories/BFWorks_STF] [Rio error]
Unescaped backslash in: L\'ambassadrice (314764886, -1)
Could these "Rio errors" be contributing to my troubles? I was also wondering
if there was a way to configure logging to be able to track the application's
progress. Right now these warnings are the only way I can tell how far the
loading has progressed.
Advice from anyone who has experience successfully loading a large triplestore
is much appreciated! Thanks in advance!
- Josh
Joshua Greben
Library Systems Programmer & Analyst
Stanford University Libraries
(650) 714-1937
jgre...@stanford.edu
_______________________________________________
Owlim-discussion mailing list
Owlim-discussion@ontotext.com
http://ontomail.semdata.org/cgi-bin/mailman/listinfo/owlim-discussion
