Hi Paul,
If there are still rdf_loader_run() processes running then the "status();"
would show this to be the case or whatever other process might be running, but
if all files are in ll_state=2 in the load_list table then no rdf_loader_run()
processes should be in use.
Best Regards
Hugh Williams
Professional Services
OpenLink Software, Inc. // http://www.openlinksw.com/
Weblog -- http://www.openlinksw.com/blogs/
LinkedIn -- http://www.linkedin.com/company/openlink-software/
Twitter -- http://twitter.com/OpenLink
Google+ -- http://plus.google.com/100570109519069333827/
Facebook -- http://www.facebook.com/OpenLinkSoftware
Universal Data Access, Integration, and Management Technology Providers
On 22 Apr 2014, at 03:56, Paul Houle <ontolo...@gmail.com> wrote:
> Well I am watching my Virtuoso instance right now and even though all
> the rows have been in state 2 for at least an hour I am still
> observing a load average of 1.14 thanks to Virtuoso CPU and disk
> activity. So something is still going on although the load is "done".
> I've seen this behavior on and off in the past and I think that one
> contribution factor could be having more than one rdf_loader_run()
> running.
>
> On Mon, Apr 21, 2014 at 10:52 PM, Hugh Williams
> <hwilli...@openlinksw.com> wrote:
>> Hi Paul,
>>
>> Using the Virtuoso RDF Bulk Loader (rdf_loader_run) as you appear to be, the
>> load is complete when all files in the load_list table have an "ll_state" =
>> 2 , 0 means still to be loaded, 1 means load is in progress.
>>
>> What does the "status()" command report when the load rate slows as I
>> suspect all the "buffers" (NumberOfBuffers in INI) are being consumed at
>> that point and will thus slow load rates.
>>
>> Note you can also monitor the load rates using the Virtuoso LDMeter
>> functions as detailed at:
>>
>> http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtTipsAndTricksGuideLDMeterUtility
>>
>> Best Regards
>> Hugh Williams
>> Professional Services
>> OpenLink Software, Inc. // http://www.openlinksw.com/
>> Weblog -- http://www.openlinksw.com/blogs/
>> LinkedIn -- http://www.linkedin.com/company/openlink-software/
>> Twitter -- http://twitter.com/OpenLink
>> Google+ -- http://plus.google.com/100570109519069333827/
>> Facebook -- http://www.facebook.com/OpenLinkSoftware
>> Universal Data Access, Integration, and Management Technology Providers
>>
>> On 21 Apr 2014, at 23:37, Paul Houle <ontolo...@gmail.com> wrote:
>>
>> The image below is a trace of the CPU consumption of a bulk load I did
>> in Virtuoso.
>>
>> https://raw.githubusercontent.com/paulhoule/images/master/CpuForVirtuosoLoad.PNG
>>
>> Here I am loading half a billion or so triples in Amazon EC2 using an
>> r3.xlarge, with Virtuoso running on the instance storage SSD. In
>> terms of time and cost, this is a great chart, but there is one
>> detail that I'm concerned about.
>>
>> At first I queued up the 'a' dataset and started one rdf_loader_run.
>> That used around 50% of system resources on a four core machine, so
>> when I started a second loader, the rate sped up until it reached
>> 100%.
>>
>> It falls to zero because it finished the 'a' data, then I queued the
>> rest of the data and the system ran at 100%. Then there is a glitch
>> at one point, then the CPU consumption falls in two plateaus, with
>> another glitch at the end.
>>
>> Definitely in the second plateau was looking at the machine and
>> noticed that the db.dba.load_list said the loading was done but that
>> the CPU was still running hard and, more to the point, I/O was still
>> being done to the SSD. I don't see anything in the log that
>> correlates with the end of this phase.
>>
>> What is Virtuoso doing in this phase after the loading is said to be
>> complete? Would it be safe to shut Virtuoso down at this time? I can
>> watch the load_list and the CPU meter to get a pretty good guess of
>> when it is done, but is there some other way I know Virtuoso is done
>> with this activity?
>>
>>
>>
>> --
>> Paul Houle
>> Expert on Freebase, DBpedia, Hadoop and RDF
>> (607) 539 6254 paul.houle on Skype ontolo...@gmail.com
>>
>> ------------------------------------------------------------------------------
>> Start Your Social Network Today - Download eXo Platform
>> Build your Enterprise Intranet with eXo Platform Software
>> Java Based Open Source Intranet - Social, Extensible, Cloud Ready
>> Get Started Now And Turn Your Intranet Into A Collaboration Platform
>> http://p.sf.net/sfu/ExoPlatform
>> _______________________________________________
>> Virtuoso-users mailing list
>> Virtuoso-users@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/virtuoso-users
>>
>>
>
>
>
> --
> Paul Houle
> Expert on Freebase, DBpedia, Hadoop and RDF
> (607) 539 6254 paul.houle on Skype ontolo...@gmail.com
>
> ------------------------------------------------------------------------------
> Start Your Social Network Today - Download eXo Platform
> Build your Enterprise Intranet with eXo Platform Software
> Java Based Open Source Intranet - Social, Extensible, Cloud Ready
> Get Started Now And Turn Your Intranet Into A Collaboration Platform
> http://p.sf.net/sfu/ExoPlatform
> _______________________________________________
> Virtuoso-users mailing list
> Virtuoso-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/virtuoso-users
------------------------------------------------------------------------------
Start Your Social Network Today - Download eXo Platform
Build your Enterprise Intranet with eXo Platform Software
Java Based Open Source Intranet - Social, Extensible, Cloud Ready
Get Started Now And Turn Your Intranet Into A Collaboration Platform
http://p.sf.net/sfu/ExoPlatform
_______________________________________________
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users
