Marvin Lugair wrote:
Hi,
I am trying to load the new dbpedia 3.2 in virtuoso. I am using the dbpedia
loading script posted on the virtuoso website
http://docs.openlinksw.com/virtuoso/rdfperformancetuning.html
In addition to the error I got in loading the infobocmappingbas file (see my
previous email), loading the other files is taking too long... the
longabstracts.nt file has been trying to load for more than 3 hours (just for
that one file).
I have 8 gigs of ram on a dual core and the ram is not even being maxed in the
process.
Hi,
Did you set NumberOfBuffers and MaxDirtyBuffers in the ini file as
indicated in the docs? How big is your process during the load? For 8GB
box the correct number of buffers is around 550000 - about 2/3 of
available RAM - you need to leave some for other processes in the
system, and the operating system's I/O buffers, etc.
Also please see that you have set the default transaction isolation mode
in [Parameters] - "DefaultIsolation = 2" (Read Committed)
If you have the hardware set up so that you have more physical disk
devices on separate buses, you may benefit from additional I/O
parallelization by setting Virtuoso to use striped DB - i.e. store the
DB in slices on different physical devices.
You should pay close attention to various system load indicators to see
if the process is I/O or CPU bound in order to see if excessive I/O, due
to swapping or the process running out of memory is causing this - when
the process has enough RAM and the system's I/O devices and read
buffering are healthy, the virtuoso-t process should hit and sustain
close to 100% usage on both cores during most of the load. Use 'top' to
see the CPU usage stats while loading. Once your DB working set grows to
exceed available memory buffers, you will see the system go I/O bound
and the CPU usage of the process drop. That's when you need a) patience,
or b) a trip to the nearest shop for another DIMM or four. Hopefully you
won't have much more data to left load at that stage, though.
2- Has anyone tried to load all of dbpedia 3.2 core in any RDF store ans
succeeded?
Well, dbpedia.org comes in mind ;)
Yrjänä
--
Yrjana Rankka | gh...@openlinksw.com
Developer, Virtuoso Team | http://www.openlinksw.com
| Making Technology Work For You
