Marvin Lugair wrote: > Hi, > I am trying to load the new dbpedia 3.2 in virtuoso. I am using the dbpedia > loading script posted on the virtuoso website > http://docs.openlinksw.com/virtuoso/rdfperformancetuning.html > > In addition to the error I got in loading the infobocmappingbas file (see my > previous email), loading the other files is taking too long... the > longabstracts.nt file has been trying to load for more than 3 hours (just for > that one file). > I have 8 gigs of ram on a dual core and the ram is not even being maxed in > the process. > > Hi,
Did you set NumberOfBuffers and MaxDirtyBuffers in the ini file as indicated in the docs? How big is your process during the load? For 8GB box the correct number of buffers is around 550000 - about 2/3 of available RAM - you need to leave some for other processes in the system, and the operating system's I/O buffers, etc. Also please see that you have set the default transaction isolation mode in [Parameters] - "DefaultIsolation = 2" (Read Committed) If you have the hardware set up so that you have more physical disk devices on separate buses, you may benefit from additional I/O parallelization by setting Virtuoso to use striped DB - i.e. store the DB in slices on different physical devices. You should pay close attention to various system load indicators to see if the process is I/O or CPU bound in order to see if excessive I/O, due to swapping or the process running out of memory is causing this - when the process has enough RAM and the system's I/O devices and read buffering are healthy, the virtuoso-t process should hit and sustain close to 100% usage on both cores during most of the load. Use 'top' to see the CPU usage stats while loading. Once your DB working set grows to exceed available memory buffers, you will see the system go I/O bound and the CPU usage of the process drop. That's when you need a) patience, or b) a trip to the nearest shop for another DIMM or four. Hopefully you won't have much more data to left load at that stage, though. > 2- Has anyone tried to load all of dbpedia 3.2 core in any RDF store ans > succeeded? > Well, dbpedia.org comes in mind ;) Yrjänä -- Yrjana Rankka | [EMAIL PROTECTED] Developer, Virtuoso Team | http://www.openlinksw.com | Making Technology Work For You ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Dbpedia-discussion mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/dbpedia-discussion
