Hi Dong, As can be seen from your status output 10mins before the apparent hang the buffers ie memory is being used up:
"680000 buffers, 510574 used”
So as I suspect the system does not have enough memory for the dataset size
being loaded. So what is the size of the dataset being loaded currently in
terms of triple count, and what data is already loaded in the database as when
you started the load the status command reported more than half the buffers
already in use:
"680000 buffers, 349688 used"
Best Regards
Hugh Williams
Professional Services
OpenLink Software, Inc. // http://www.openlinksw.com/
Weblog -- http://www.openlinksw.com/blogs/
LinkedIn -- http://www.linkedin.com/company/openlink-software/
Twitter -- http://twitter.com/OpenLink
Google+ -- http://plus.google.com/100570109519069333827/
Facebook -- http://www.facebook.com/OpenLinkSoftware
Universal Data Access, Integration, and Management Technology Providers
On 28 Nov 2012, at 16:50, Y. Dong wrote:
> Hi Hugh,
>
> After many trials and errors, I find that when loading the 15 ttl files, the
> loading is always stuck at a particular file called yagoTypes.ttl, which only
> contains rdf:type triples. This file is around 800 MB, other files with sizes
> over 2GB are loaded successfully so the size shouldn't be a problem. What I
> don't understand is that isn't loading just dumping triples from ttl files to
> virtuoso database, which should be really straightforward? It seems that
> Virtuoso is trying to be clever about the actual contents in a ttl file, it's
> just really puzzling to me why it needs to stuck at file with particular
> contents. It never happened on other sql based database I used before.
>
> That being said, I think it is hardly a memory issue because of database
> size, i use the same setting to host another 500GB database without much
> issue. Anyway, i posted output from status() below. The first one is just
> beginning the loading, the second one is about 10 mins before Virtuoso stops
> responding.
>
>
> OpenLink Virtuoso Server
> Version 06.01.3127-pthreads for Linux as of Sep 14 2012
> Started on: 2012/11/28 14:07 GMT+00
>
> Database Status:
> File size 7698644992, 939776 pages, 361393 free.
> 680000 buffers, 349688 used, 0 dirty 0 wired down, repl age 0 0 w. io 0
> w/crsr.
> Disk Usage: 312538 reads avg 0 msec, 0% r 0% w last 0 s, 257747 writes,
> 2092 read ahead, batch = 140. Autocompact 202995 in 137881 out, 32%
> saved.
> Gate: 416 2nd in reads, 0 gate write waits, 0 in while read 0 busy scrap.
> Log = /home/ubuntu/LARGEDATA/yago2sOnt/virtuoso.trx, 155 bytes
> 396346 pages have been changed since last backup (in checkpoint state)
> Current backup timestamp: 0x0000-0x00-0x00
> Last backup date: unknown
> Clients: 1 connects, max 1 concurrent
> RPC: 4 calls, 1 pending, 1 max until now, 0 queued, 0 burst reads (0%), 0
> second brk=5948985344
> Checkpoint Remap 181811 pages, 0 mapped back. 43 s atomic time.
> DB master 939776 total 361393 free 181811 remap 0 mapped back
> temp 256 total 251 free
>
> Lock Status: 0 deadlocks of which 0 2r1w, 0 waits,
> Currently 1 threads running 0 threads waiting 0 threads in vdb.
> Pending:
>
> Client 1111:1: Account: dba, 203 bytes in, 256 bytes out, 1 stmts.
> PID: 1081, OS: unix, Application: unknown, IP#: 127.0.0.1
> Transaction status: PENDING, 1 threads.
> Locks:
>
>
> Running Statements:
> Time (msec) Text
> 176 status()
>
>
> Hash indexes
>
>
>
>
>
>
>
>
>
>
>
>
> OpenLink Virtuoso Server
> Version 06.01.3127-pthreads for Linux as of Sep 14 2012
> Started on: 2012/11/28 14:07 GMT+00
>
> Database Status:
> File size 11312037888, 1380864 pages, 515539 free.
> 680000 buffers, 510574 used, 7906 dirty 0 wired down, repl age 3248261 0 w.
> io 0 w/crsr.
> Disk Usage: 315439 reads avg 0 msec, 0% r 0% w last 6 s, 705064 writes,
> 2110 read ahead, batch = 139. Autocompact 597047 in 412146 out, 30%
> saved.
> Gate: 419 2nd in reads, 0 gate write waits, 0 in while read 0 busy scrap.
> Log = /home/ubuntu/LARGEDATA/yago2sOnt/virtuoso.trx, 499575222 bytes
> 546254 pages have been changed since last backup (in checkpoint state)
> Current backup timestamp: 0x0000-0x00-0x00
> Last backup date: unknown
> Clients: 14 connects, max 2 concurrent
> RPC: 140 calls, -9 pending, 1 max until now, 0 queued, 38 burst reads (27%),
> 0 second brk=5974286336
> Checkpoint Remap 148694 pages, 0 mapped back. 63 s atomic time.
> DB master 1380864 total 515539 free 148694 remap 8326 mapped back
> temp 256 total 251 free
>
> Lock Status: 0 deadlocks of which 0 2r1w, 0 waits,
> Currently 2 threads running 0 threads waiting 0 threads in vdb.
> Pending:
>
> Client 1111:3: Account: dba, 358 bytes in, 352 bytes out, 1 stmts.
> PID: 1090, OS: unix, Application: unknown, IP#: 127.0.0.1
> Transaction status: PENDING, 1 threads.
> Locks:
>
> Client 1111:14: Account: dba, 320 bytes in, 2812 bytes out, 1 stmts.
> PID: 2009, OS: unix, Application: unknown, IP#: 127.0.0.1
> Transaction status: PENDING, 1 threads.
> Locks:
>
>
> Running Statements:
> Time (msec) Text
> 208 status()
> 938348 rdf_loader_run(log_enable=>3)
>
>
> Hash indexes
>
smime.p7s
Description: S/MIME cryptographic signature
------------------------------------------------------------------------------ Keep yourself connected to Go Parallel: VERIFY Test and improve your parallel project with help from experts and peers. http://goparallel.sourceforge.net
_______________________________________________ Virtuoso-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/virtuoso-users
