Hi Roland, Are any error reported in the "virtuoso.log" file when the load is in progress ? Also, is this a publicly available datasets your are loading that we could try locally ?
Best Regards Hugh Williams Professional Services OpenLink Software, Inc. // http://www.openlinksw.com/ Weblog -- http://www.openlinksw.com/blogs/ LinkedIn -- http://www.linkedin.com/company/openlink-software/ Twitter -- http://twitter.com/OpenLink Google+ -- http://plus.google.com/100570109519069333827/ Facebook -- http://www.facebook.com/OpenLinkSoftware Universal Data Access, Integration, and Management Technology Providers On 23 Apr 2013, at 22:39, Roland <metamatter...@gmail.com> wrote: > Hi Hugh, > > That saved me precious time. In the meantime I have upgraded to VOS7 and try > to load the large set using Bulk Loader. > But it is not working. See log below, this is what I get and Virtuoso slowly > dies after accumulating about 1GB in db, the service is not available any > more, I have to kill the process. > > After killing the process I restore the initial situation, just virtuoso.ini > and restart the server that creates a new db. Do I have to restore some more, > or better; how can I get it to stop waiting and proceed loading? > > Thanks, > Roland > > > SQL> ld_dir ('/home/roland/Documents/metamatter/bieb/datalab/', 'output.nt', > 'http://viaf.org/'); > > Done. -- 4 msec. > SQL> rdf_loader_run(); > 23:26:24 PL LOG: Loader started > 23:26:44 Write wait on column page 23801. Waits should be on the index leaf > page, except when col page is held for read by background write > 23:26:44 Write wait on column page 30567. Waits should be on the index leaf > page, except when col page is held for read by background write > 23:26:44 Write wait on column page 30605. Waits should be on the index leaf > page, except when col page is held for read by background write > 23:26:45 Write wait on column page 30606. Waits should be on the index leaf > page, except when col page is held for read by background write > 23:26:45 Write wait on column page 31262. Waits should be on the index leaf > page, except when col page is held for read by background write > 23:26:48 Write wait on column page 2051. Waits should be on the index leaf > page, except when col page is held for read by background write > 23:26:54 Write wait on column page 2823. Waits should be on the index leaf > page, except when col page is held for read by background write > 23:26:57 Write wait on column page 39246. Waits should be on the index leaf > page, except when col page is held for read by background write > 23:27:06 Write wait on column page 4867. Waits should be on the index leaf > page, except when col page is held for read by background write > 23:27:09 missed delete of name id cache > http://viaf.org/viaf/sourceID/LC%7Cn+2001057591# 0d > 23:27:14 Write wait on column page 37357. Waits should be on the index leaf > page, except when col page is held for read by background write > 23:27:14 Write wait on column page 50069. Waits should be on the index leaf > page, except when col page is held for read by background write > 23:27:15 Write wait on column page 30589. Waits should be on the index leaf > page, except when col page is held for read by background write > 23:27:15 Write wait on column page 50730. Waits should be on the index leaf > page, except when col page is held for read by background write > 23:27:22 Write wait on column page 50807. Waits should be on the index leaf > page, except when col page is held for read by background write > 23:27:26 Write wait on column page 50751. Waits should be on the index leaf > page, except when col page is held for read by background write > 23:27:28 Write wait on column page 2414. Waits should be on the index leaf > page, except when col page is held for read by background write > 23:27:46 Write wait on column page 2819. Waits should be on the index leaf > page, except when col page is held for read by background write > 23:27:46 Write wait on column page 50090. Waits should be on the index leaf > page, except when col page is held for read by background write > 23:27:46 Write wait on column page 79639. Waits should be on the index leaf > page, except when col page is held for read by background write > 23:27:59 Write wait on column page 97536. Waits should be on the index leaf > page, except when col page is held for read by background write > 23:28:02 Write wait on column page 86017. Waits should be on the index leaf > page, except when col page is held for read by background write > 23:28:06 Write wait on column page 95255. Waits should be on the index leaf > page, except when col page is held for read by background write > 23:28:19 missed delete of name id cache 0d > 23:28:20 Write wait on column page 30626. Waits should be on the index leaf > page, except when col page is held for read by background write > 23:28:23 Write wait on column page 2568. Waits should be on the index leaf > page, except when col page is held for read by background write > > > > On 04/22/2013 02:33 PM, Hugh Williams wrote: >> Hi Roland, >> >> You should use the Virtuoso RDF Bulik Loader for loading such large datasets >> as detailed at: >> >> http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtBulkRDFLoader >> >> From the error you report it would appear a checkpoint is being performed in >> the middle of your load, so checking should be disabled which is what the >> bulkloader does automatically and also turns of transaction logging with >> log_enable(3) to prevent the log from running out of memory, which once >> again the bulk loader does automatically and is probably the next error you >> would hit ... >> >> Best Regards >> Hugh Williams >> Professional Services >> OpenLink Software, Inc. // http://www.openlinksw.com/ >> Weblog -- http://www.openlinksw.com/blogs/ >> LinkedIn -- http://www.linkedin.com/company/openlink-software/ >> Twitter -- http://twitter.com/OpenLink >> Google+ -- http://plus.google.com/100570109519069333827/ >> Facebook -- http://www.facebook.com/OpenLinkSoftware >> Universal Data Access, Integration, and Management Technology Providers >> >> On 22 Apr 2013, at 13:07, Roland Cornelissen <metamatter...@gmail.com> wrote: >> >>> Hi, >>> >>> I am trying to load a large dataset (~50GB) and bumped into: >>> >>> *** Error 40001: [Virtuoso Driver][Virtuoso Server]SR325: Transaction >>> aborted due to a database checkpoint or database-wide atomic operation. >>> Please retry transaction >>> at line 10 of Top-Level: >>> ttlp_mt (file_to_string_output ('/usr/viaf/viaf_fixed.nt'), '', >>> 'http://viaf.org/', 255) >>> >>> I have 16GB memory in my box, and set: >>> NumberOfBuffers = 680000 >>> MaxDirtyBuffers = 500000 >>> and: >>> MaxCheckpointRemap = 2000 >>> >>> So i guess the MaxCheckpointRemap is the problem. At what value should I >>> set it? >>> Here [1] it says 1/4 of the database size in pages, 8K per page. >>> So when I calculate with a DB of 50GB that would be 1562500, is that >>> correct? >>> >>> Am I missing something else I should check before kicking of another run? >>> >>> Thanks, >>> Roland >>> >>> [1] http://www.openlinksw.com/OdbcRails/main/Main/VirtRDFPerformanceTuning >>> >>> >>> ------------------------------------------------------------------------------ >>> Precog is a next-generation analytics platform capable of advanced >>> analytics on semi-structured data. The platform includes APIs for building >>> apps and a phenomenal toolset for data science. Developers can use >>> our toolset for easy data analysis & visualization. Get a free account! >>> http://www2.precog.com/precogplatform/slashdotnewsletter >>> _______________________________________________ >>> Virtuoso-users mailing list >>> Virtuoso-users@lists.sourceforge.net >>> https://lists.sourceforge.net/lists/listinfo/virtuoso-users >> > > ------------------------------------------------------------------------------ > Try New Relic Now & We'll Send You this Cool Shirt > New Relic is the only SaaS-based application performance monitoring service > that delivers powerful full stack analytics. Optimize and monitor your > browser, app, & servers with just a few lines of code. Try New Relic > and get this awesome Nerd Life shirt! > http://p.sf.net/sfu/newrelic_d2d_apr_______________________________________________ > Virtuoso-users mailing list > Virtuoso-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/virtuoso-users
smime.p7s
Description: S/MIME cryptographic signature
------------------------------------------------------------------------------ Try New Relic Now & We'll Send You this Cool Shirt New Relic is the only SaaS-based application performance monitoring service that delivers powerful full stack analytics. Optimize and monitor your browser, app, & servers with just a few lines of code. Try New Relic and get this awesome Nerd Life shirt! http://p.sf.net/sfu/newrelic_d2d_apr
_______________________________________________ Virtuoso-users mailing list Virtuoso-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/virtuoso-users
