Hi 

Looking at the status output all the buffers are pretty much in use:

        680000 buffers, 679997 used,

So the load is starting to write to disk and swapping in and out of memory 
which kills performance. What is the current count of triples that you have 
loaded as it would appear your server needs more memory to host that many 
triples with goo performance ?

Best Regards
Hugh Williams
Professional Services
OpenLink Software, Inc.      //              http://www.openlinksw.com/
Weblog   -- http://www.openlinksw.com/blogs/
LinkedIn -- http://www.linkedin.com/company/openlink-software/
Twitter  -- http://twitter.com/OpenLink
Google+  -- http://plus.google.com/100570109519069333827/
Facebook -- http://www.facebook.com/OpenLinkSoftware
Universal Data Access, Integration, and Management Technology Providers

On 26 Oct 2012, at 13:18, Y. Dong wrote:

> Hello there, 
> 
> I am currently loading 400GB of yago ontology data into Virtuoso. The data is 
> split into 114 rdf/xml files and each is about 3GB. I disable the logging and 
> transactions to speed up loading(log_enable(2)) and then I use the bulk load 
> functionality of virtuoso. (ld_dir() then rdf_loader_run()). 
> 
> This worked well as it took about 2 hrs to load one file and Virtuoso was 
> using almost 100% of CPU, suggesting it is busy with the task. But when it 
> began to load the 100th file, it slowed dramatically. It was already 16 hours 
> but the 100th file was still not loaded. Besides, Virtuoso was only using 
> less than 10% of CPU. So I restarted the server and resumed the loading 
> process, however it was still very slow. 
> 
> Well, although it doesn't develop any errors or exceptions but it nows seems 
> to take forever to load one file and CPU usage(10%) suggests Virtuoso is not 
> actively working on it. I'm afraid the loading will ever finish. And I don't 
> understand why it can just slow down dramatically while system resources(CPU, 
> Memory, disk) are all still abundant. Am I running into some bottleneck of 
> Virtuoso here? 
> 
> The server is running on an amazon instance with Ubuntu 12.04.01 LTS, 8GB 
> Memory and more than enough disk space. Details about Virtuoso is in the 
> status info from isql-v
> 
> OpenLink Virtuoso  Server
> Version 06.01.3127-pthreads for Linux as of Sep 14 2012 
> Started on: 2012/10/26 01:05 GMT+00
> 
> Database Status:
>  File size 576129597440, 70328320 pages, 18725442 free.
>  680000 buffers, 679997 used, 119184 dirty 2 wired down, repl age 1890649 0 
> w. io 1 w/crsr.
>  Disk Usage: 8161133 reads avg 2 msec, 80% r 6% w last  2033 s, 4768340 
> writes,
>    23476 read ahead, batch = 155.  Autocompact 4114034 in 2626277 out, 36% 
> saved.
> Gate:  15171 2nd in reads, 0 gate write waits, 0 in while read 0 busy scrap. 
> Log = /home/ubuntu/LARGEDATA/virtuoso.trx, 4461 bytes
> 51482061 pages have been changed since last backup (in checkpoint state)
> Current backup timestamp: 0x0000-0x00-0x00
> Last backup date: unknown
> Clients: 9 connects, max 2 concurrent
> RPC: 49 calls, -5 pending, 2 max until now, 0 queued, 18 burst reads (36%), 0 
> second brk=5826801664
> Checkpoint Remap 0 pages, 0 mapped back. 4471 s atomic time.
>    DB master 70328320 total 18725206 free 0 remap 0 mapped back
>   temp  256 total 251 free
> 
> Lock Status: 0 deadlocks of which 0 2r1w, 0 waits,
>   Currently 2 threads running 0 threads waiting 0 threads in vdb.
> Pending:
>  2591533: IER 1 
>      0: IER 1 
>  70112036: IER 1 
>      7: IER 1 
>  3174786: IER 1 
>      110: IER 1 
>  69942572: IER 1 
>      388: IER 1 
> 
> Client 1111:1:  Account: dba, 626 bytes in, 12373 bytes out, 1 stmts.
> PID: 29252, OS: unix, Application: unknown, IP#: 127.0.0.1
> Transaction status: PENDING, 1 threads.
> Locks: 
> 
> Client 1111:9:  Account: dba, 203 bytes in, 256 bytes out, 1 stmts.
> PID: 31667, OS: unix, Application: unknown, IP#: 127.0.0.1
> Transaction status: PENDING, 1 threads.
> Locks: 
> 
> 
> Running Statements:
> Time (msec) Text
>        3401 status()
>    38700583 rdf_loader_run()
> 
> 
> Hash indexes
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_sfd2d_oct
> _______________________________________________
> Virtuoso-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/virtuoso-users

Attachment: smime.p7s
Description: S/MIME cryptographic signature

------------------------------------------------------------------------------
The Windows 8 Center 
In partnership with Sourceforge
Your idea - your app - 30 days. Get started!
http://windows8center.sourceforge.net/
what-html-developers-need-to-know-about-coding-windows-8-metro-style-apps/
_______________________________________________
Virtuoso-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

Reply via email to