Hi Hugh,
That saved me precious time. In the meantime I have upgraded to VOS7 and
try to load the large set using Bulk Loader.
But it is not working. See log below, this is what I get and Virtuoso
slowly dies after accumulating about 1GB in db, the service is not
available any more, I have to kill the process.
After killing the process I restore the initial situation, just
virtuoso.ini and restart the server that creates a new db. Do I have to
restore some more, or better; how can I get it to stop waiting and
proceed loading?
Thanks,
Roland
SQL> ld_dir ('/home/roland/Documents/metamatter/bieb/datalab/',
'output.nt', 'http://viaf.org/');
Done. -- 4 msec.
SQL> rdf_loader_run();
23:26:24 PL LOG: Loader started
23:26:44 Write wait on column page 23801. Waits should be on the index
leaf page, except when col page is held for read by background write
23:26:44 Write wait on column page 30567. Waits should be on the index
leaf page, except when col page is held for read by background write
23:26:44 Write wait on column page 30605. Waits should be on the index
leaf page, except when col page is held for read by background write
23:26:45 Write wait on column page 30606. Waits should be on the index
leaf page, except when col page is held for read by background write
23:26:45 Write wait on column page 31262. Waits should be on the index
leaf page, except when col page is held for read by background write
23:26:48 Write wait on column page 2051. Waits should be on the index
leaf page, except when col page is held for read by background write
23:26:54 Write wait on column page 2823. Waits should be on the index
leaf page, except when col page is held for read by background write
23:26:57 Write wait on column page 39246. Waits should be on the index
leaf page, except when col page is held for read by background write
23:27:06 Write wait on column page 4867. Waits should be on the index
leaf page, except when col page is held for read by background write
23:27:09 missed delete of name id cache
http://viaf.org/viaf/sourceID/LC%7Cn+2001057591# 0d
23:27:14 Write wait on column page 37357. Waits should be on the index
leaf page, except when col page is held for read by background write
23:27:14 Write wait on column page 50069. Waits should be on the index
leaf page, except when col page is held for read by background write
23:27:15 Write wait on column page 30589. Waits should be on the index
leaf page, except when col page is held for read by background write
23:27:15 Write wait on column page 50730. Waits should be on the index
leaf page, except when col page is held for read by background write
23:27:22 Write wait on column page 50807. Waits should be on the index
leaf page, except when col page is held for read by background write
23:27:26 Write wait on column page 50751. Waits should be on the index
leaf page, except when col page is held for read by background write
23:27:28 Write wait on column page 2414. Waits should be on the index
leaf page, except when col page is held for read by background write
23:27:46 Write wait on column page 2819. Waits should be on the index
leaf page, except when col page is held for read by background write
23:27:46 Write wait on column page 50090. Waits should be on the index
leaf page, except when col page is held for read by background write
23:27:46 Write wait on column page 79639. Waits should be on the index
leaf page, except when col page is held for read by background write
23:27:59 Write wait on column page 97536. Waits should be on the index
leaf page, except when col page is held for read by background write
23:28:02 Write wait on column page 86017. Waits should be on the index
leaf page, except when col page is held for read by background write
23:28:06 Write wait on column page 95255. Waits should be on the index
leaf page, except when col page is held for read by background write
23:28:19 missed delete of name id cache 0d
23:28:20 Write wait on column page 30626. Waits should be on the index
leaf page, except when col page is held for read by background write
23:28:23 Write wait on column page 2568. Waits should be on the index
leaf page, except when col page is held for read by background write
On 04/22/2013 02:33 PM, Hugh Williams wrote:
Hi Roland,
You should use the Virtuoso RDF Bulik Loader for loading such large
datasets as detailed at:
http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtBulkRDFLoader
From the error you report it would appear a checkpoint is being
performed in the middle of your load, so checking should be disabled
which is what the bulkloader does automatically and also turns of
transaction logging with log_enable(3) to prevent the log from running
out of memory, which once again the bulk loader does automatically and
is probably the next error you would hit ...
Best Regards
Hugh Williams
Professional Services
OpenLink Software, Inc. // http://www.openlinksw.com/
Weblog -- http://www.openlinksw.com/blogs/
LinkedIn -- http://www.linkedin.com/company/openlink-software/
Twitter -- http://twitter.com/OpenLink
Google+ -- http://plus.google.com/100570109519069333827/
Facebook -- http://www.facebook.com/OpenLinkSoftware
Universal Data Access, Integration, and Management Technology Providers
On 22 Apr 2013, at 13:07, Roland Cornelissen <metamatter...@gmail.com
<mailto:metamatter...@gmail.com>> wrote:
Hi,
I am trying to load a large dataset (~50GB) and bumped into:
*** Error 40001: [Virtuoso Driver][Virtuoso Server]SR325: Transaction
aborted due to a database checkpoint or database-wide atomic operation.
Please retry transaction
at line 10 of Top-Level:
ttlp_mt (file_to_string_output ('/usr/viaf/viaf_fixed.nt'), '',
'http://viaf.org/' <http://viaf.org/%27>, 255)
I have 16GB memory in my box, and set:
NumberOfBuffers = 680000
MaxDirtyBuffers = 500000
and:
MaxCheckpointRemap = 2000
So i guess the MaxCheckpointRemap is the problem. At what value should I
set it?
Here [1] it says 1/4 of the database size in pages, 8K per page.
So when I calculate with a DB of 50GB that would be 1562500, is that
correct?
Am I missing something else I should check before kicking of another run?
Thanks,
Roland
[1]
http://www.openlinksw.com/OdbcRails/main/Main/VirtRDFPerformanceTuning
------------------------------------------------------------------------------
Precog is a next-generation analytics platform capable of advanced
analytics on semi-structured data. The platform includes APIs for
building
apps and a phenomenal toolset for data science. Developers can use
our toolset for easy data analysis & visualization. Get a free account!
http://www2.precog.com/precogplatform/slashdotnewsletter
_______________________________________________
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users
------------------------------------------------------------------------------
Try New Relic Now & We'll Send You this Cool Shirt
New Relic is the only SaaS-based application performance monitoring service
that delivers powerful full stack analytics. Optimize and monitor your
browser, app, & servers with just a few lines of code. Try New Relic
and get this awesome Nerd Life shirt! http://p.sf.net/sfu/newrelic_d2d_apr
_______________________________________________
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users
