Well I am watching my Virtuoso instance right now and even though all the rows have been in state 2 for at least an hour I am still observing a load average of 1.14 thanks to Virtuoso CPU and disk activity. So something is still going on although the load is "done". I've seen this behavior on and off in the past and I think that one contribution factor could be having more than one rdf_loader_run() running.
On Mon, Apr 21, 2014 at 10:52 PM, Hugh Williams <hwilli...@openlinksw.com> wrote: > Hi Paul, > > Using the Virtuoso RDF Bulk Loader (rdf_loader_run) as you appear to be, the > load is complete when all files in the load_list table have an "ll_state" = > 2 , 0 means still to be loaded, 1 means load is in progress. > > What does the "status()" command report when the load rate slows as I > suspect all the "buffers" (NumberOfBuffers in INI) are being consumed at > that point and will thus slow load rates. > > Note you can also monitor the load rates using the Virtuoso LDMeter > functions as detailed at: > > http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtTipsAndTricksGuideLDMeterUtility > > Best Regards > Hugh Williams > Professional Services > OpenLink Software, Inc. // http://www.openlinksw.com/ > Weblog -- http://www.openlinksw.com/blogs/ > LinkedIn -- http://www.linkedin.com/company/openlink-software/ > Twitter -- http://twitter.com/OpenLink > Google+ -- http://plus.google.com/100570109519069333827/ > Facebook -- http://www.facebook.com/OpenLinkSoftware > Universal Data Access, Integration, and Management Technology Providers > > On 21 Apr 2014, at 23:37, Paul Houle <ontolo...@gmail.com> wrote: > > The image below is a trace of the CPU consumption of a bulk load I did > in Virtuoso. > > https://raw.githubusercontent.com/paulhoule/images/master/CpuForVirtuosoLoad.PNG > > Here I am loading half a billion or so triples in Amazon EC2 using an > r3.xlarge, with Virtuoso running on the instance storage SSD. In > terms of time and cost, this is a great chart, but there is one > detail that I'm concerned about. > > At first I queued up the 'a' dataset and started one rdf_loader_run. > That used around 50% of system resources on a four core machine, so > when I started a second loader, the rate sped up until it reached > 100%. > > It falls to zero because it finished the 'a' data, then I queued the > rest of the data and the system ran at 100%. Then there is a glitch > at one point, then the CPU consumption falls in two plateaus, with > another glitch at the end. > > Definitely in the second plateau was looking at the machine and > noticed that the db.dba.load_list said the loading was done but that > the CPU was still running hard and, more to the point, I/O was still > being done to the SSD. I don't see anything in the log that > correlates with the end of this phase. > > What is Virtuoso doing in this phase after the loading is said to be > complete? Would it be safe to shut Virtuoso down at this time? I can > watch the load_list and the CPU meter to get a pretty good guess of > when it is done, but is there some other way I know Virtuoso is done > with this activity? > > > > -- > Paul Houle > Expert on Freebase, DBpedia, Hadoop and RDF > (607) 539 6254 paul.houle on Skype ontolo...@gmail.com > > ------------------------------------------------------------------------------ > Start Your Social Network Today - Download eXo Platform > Build your Enterprise Intranet with eXo Platform Software > Java Based Open Source Intranet - Social, Extensible, Cloud Ready > Get Started Now And Turn Your Intranet Into A Collaboration Platform > http://p.sf.net/sfu/ExoPlatform > _______________________________________________ > Virtuoso-users mailing list > Virtuoso-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/virtuoso-users > > -- Paul Houle Expert on Freebase, DBpedia, Hadoop and RDF (607) 539 6254 paul.houle on Skype ontolo...@gmail.com ------------------------------------------------------------------------------ Start Your Social Network Today - Download eXo Platform Build your Enterprise Intranet with eXo Platform Software Java Based Open Source Intranet - Social, Extensible, Cloud Ready Get Started Now And Turn Your Intranet Into A Collaboration Platform http://p.sf.net/sfu/ExoPlatform _______________________________________________ Virtuoso-users mailing list Virtuoso-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/virtuoso-users
