On 9/16/11 9:41 AM, karel braeckman wrote:
And the Virtuoso config is a single instance:OpenLink Virtuoso version 06.01.3127, on Linux (x86_64-unknown-linux-gnu), Single Edition There are currently about 300M triples in the store. I read that 500M per 16GB or RAM should still run ok, so I don't think a clustered version should be necessary? Performing Sparql queries still is very fast, but adding / deleting triples seems to be very slow.
Thing is that via the cluster edition you get parallelism re. Insert, Update, and Delete (IUD) operations, in addition to Read oriented queries.
Since we have a new version of Virtuoso at: http://dbpedia-live.openlinksw.com/live , which is operating in similar mode to yours i.e., connecting to live.dbpedia.org we should be able to send you a patched single-server edition that would at least ensure parity re. IUD operations relative to our instance.
Patrick will make this patch available to you so that we can eliminate issues that might be version related etc..
Kingsley
On Fri, Sep 16, 2011 at 2:26 PM, karel braeckman <[email protected]> wrote:Forgot to mention the machine details: 24GB RAM 2x quadcore Xeon E5540 2.5GHz Virtuoso data is on SSD disks Regards, Karel On Fri, Sep 16, 2011 at 1:57 PM, karel braeckman<[email protected]> wrote:Hi all, We tried setting up DBpedia Live on one of our local servers and keep it in sync with the official DBpedia live, using the dbpintegrator tool. Our setup is documented here: http://medialoep.vrtmedialab.be/blog/?p=96. However, adding / deleting triples from / to Virtuoso has become extremely slow. I was pointed in the direction of http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtRDFPerformanceTuning by @kidehen and @pkleef. I changed the following parameters in Virtuoso's .ini file from their default value: [Database] MaxCheckpointRemap = 1000000 [Parameters] NumberOfBuffers = 1360000 MaxDirtyBuffers = 1000000 I can see no real difference in performance though, Virtuoso gets stuck on a DELETE query (it is busy with it for the last 125 minutes at 100% CPU :-S). As I am not at all an expert on Virtuoso tuning, all suggestions are welcome. For completeness, I have included the output of status('rhck'); in isql below. Best regards, Karel ---------------------------------------------- OpenLink Virtuoso Server Version 06.01.3127-pthreads for Linux as of Mar 16 2011 Started on: 2011/09/16 11:35 GMT+120 Database Status: File size 56727961600, 6924800 pages, 2771949 free. 1360000 buffers, 47404 used, 2 dirty 0 wired down, repl age 0 0 w. io 40 w/crsr. Disk Usage: 48335 reads avg 0 msec, 0% r 0% w last 254 s, 10448 writes, 136 read ahead, batch = 230. Autocompact 22 in 12 out, 43% saved. Gate: 391 2nd in reads, 0 gate write waits, 0 in while read 0 busy scrap. Log = virtuoso.trx, 4003 bytes 4151261 pages have been changed since last backup (in checkpoint state) Current backup timestamp: 0x0000-0x00-0x00 Last backup date: unknown Clients: 1 connects, max 1 concurrent RPC: 65 calls, 1 pending, 1 max until now, 0 queued, 0 burst reads (0%), 0 second brk=11304103936 Checkpoint Remap 1159 pages, 0 mapped back. 125 s atomic time. DB master 6924800 total 2771949 free 1159 remap 0 mapped back temp 768 total 763 free Lock Status: 0 deadlocks of which 0 2r1w, 0 waits, Currently 2 threads running 0 threads waiting 0 threads in vdb. Pending: Client 1111:1: Account: dba, 361734 bytes in, 3237 bytes out, 17 stmts. Transaction status: PENDING, 1 threads. Locks: Running Statements: Time (msec) Text 7797190 sparql DELETE FROM<http://live.dbpedia.org> { <http://dbpedia.org/resource/T Replication Status: Server db-UBUNTU. db-MUSSORGSKY db-MUSSORGSKY 0 OFF. db-UBUNTU db-UBUNTU 0 OFF. Index Usage: Table Index Touches Reads %Miss Locks Waits %W n-dead DB.DBA.RDF_PREFIX RDF_PREFIX 3 95 2375% 6 0 0% 0 DB.DBA.RDF_PREFIX DB_DBA_RDF_PREFIX_UNQC_RP_ID 2193 4 0% 3 0 0% 0 DB.DBA.RDF_IRI RDF_IRI 40 4670 11390% 80 0 0% 0 DB.DBA.RDF_IRI DB_DBA_RDF_IRI_UNQC_RI_ID 353 8 2% 40 0 0% 0 DB.DBA.RDF_QUAD RDF_QUAD 953949641 13253 0% 2445 0 0% 0 DB.DBA.RDF_QUAD RDF_QUAD_SP 5596 447 7% 465 0 0% 0 DB.DBA.RDF_QUAD RDF_QUAD_POGS 1512 13514 893% 829 0 0% 0 DB.DBA.RDF_QUAD RDF_QUAD_GS 564 500 88% 80 0 0% 0 DB.DBA.RDF_QUAD RDF_QUAD_OP 152 942 615% 152 0 0% 0 DB.DBA.RDF_OBJ RDF_OBJ 3151 4134 131% 325 0 0% 0 DB.DBA.RDF_OBJ RO_VAL 954 2319 242% 137 0 0% 0 DB.DBA.RO_START RO_START 35 64 177% 35 0 0% 0 DB.DBA.RDF_DATATYPE RDF_DATATYPE 230 1 0% 238 0 0% 0 DB.DBA.RDF_LANGUAGE RDF_LANGUAGE 840 1 0% 0 0 0% 0 DB.DBA.RDF_LANGUAGE DB_DBA_RDF_LANGUAGE_UNQC_RL_TWOBYTE 334 1 0% 344 0 0% 0 DB.DBA.RDF_OBJ_FT_RULES RDF_OBJ_FT_RULES 11 1 8% 14 0 0% 0 DB.DBA.RDF_GRAPH_GROUP RDF_GRAPH_GROUP_IRI 2 1 33% 1 0 0% 0 WS.WS.SYS_DAV_COL SYS_DAV_COL 13862 1 0% 20890 0 0% 0 WS.WS.SYS_DAV_COL SYS_DAV_COL_ID 221 1 0% 46 0 0% 0 WS.WS.SYS_DAV_RES SYS_DAV_RES 17368 164 0% 4409 0 0% 0 WS.WS.SYS_DAV_RES SYS_DAV_RES_COL 3242 4 0% 3609 0 0% 0 WS.WS.SYS_DAV_RES SYS_DAV_RES_FULL_PATH 7484 4 0% 4961 0 0% 0 WS.WS.SYS_DAV_RES SYS_DAV_RES_IID 3 1 25% 3 0 0% 0 WS.WS.SYS_DAV_RES_TYPES SYS_DAV_RES_TYPES 5 1 16% 10 0 0% 0 DB.DBA.HTTP_PATH HTTP_PATH 10 1 9% 11 0 0% 0 DB.DBA.VSPX_SESSION VSPX_SESSION 484 1 0% 522 0 0% 0 VAD.DBA.VAD_REGISTRY VAD_REGISTRY 1664 3 0% 356 0 0% 0 VAD.DBA.VAD_REGISTRY VAD_REGISTRY_CHDIR 928 1 0% 357 0 0% 0 VAD.DBA.VAD_REGISTRY VAD_REGISTRY_KEY 1 2 100% 1 0 0% 0 VAD.DBA.VAD_HELP VAD_HELP 23 1 4% 46 0 0% 0 Hash indexes ----------------------------------------------------------------------------------------------------------------- BlackBerry® DevCon Americas, Oct. 18-20, San Francisco, CA http://p.sf.net/sfu/rim-devcon-copy2 _______________________________________________ Dbpedia-discussion mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/dbpedia-discussion
-- Regards, Kingsley Idehen President& CEO OpenLink Software Web: http://www.openlinksw.com Weblog: http://www.openlinksw.com/blog/~kidehen Twitter/Identi.ca: kidehen
smime.p7s
Description: S/MIME Cryptographic Signature
------------------------------------------------------------------------------ BlackBerry® DevCon Americas, Oct. 18-20, San Francisco, CA http://p.sf.net/sfu/rim-devcon-copy2
_______________________________________________ Dbpedia-discussion mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/dbpedia-discussion
