Hi Pantelis, 152M triple graphs is quite large to be performing such an operation on which will require a significant amount of memory to complete. What is the total number of triples in Virtuoso and how many of the buffers are in use when the database is in use, which can be seen by running the status(); command from isql. As you may be able to reduce the “NumberOfBuffers” param for database workig set to make more memory available to the system for performing such large insert queries.
You should also ensure the following params are set in the INI file to ensure unused resource/threads are cleaned up immediately to maximise available memory: [Parameters] .... ThreadCleanupInterval = 1 ResourcesCleanupInterval = 1 ... See, http://docs.openlinksw.com/virtuoso/dbadm/ The server log does not show anything useful … Best Regards Hugh Williams Professional Services OpenLink Software, Inc. // http://www.openlinksw.com/ Weblog -- http://www.openlinksw.com/blogs/ LinkedIn -- http://www.linkedin.com/company/openlink-software/ Twitter -- http://twitter.com/OpenLink Google+ -- http://plus.google.com/100570109519069333827/ Facebook -- http://www.facebook.com/OpenLinkSoftware Universal Data Access, Integration, and Management Technology Providers > On 10 Aug 2016, at 09:44, Pantelis Natsiavas <natsia...@gmail.com> wrote: > > Thank you Hugh for your answer. > > I have actually already used the respective pragma before the SPARUL query. > > SPARQL DEFINE sql:log-enable 3 > INSERT .... > > The documentation of the virtuoso functions is not clear for me. However, I > think I have already done what you suggest. Right? > > The graph the SPARUL is run against is rather big. It is the > "ClinicalTrials.gov" graph of the Bio2RDF, namely 152.725.529 triplets. My VM > has 32GBs RAM and the memory consumption of the virtuoso process starts from > 40% and constantly increases until the break down (I have seen 70% memory > consumption but I cannot be certain for the peak). When the operation breaks > down, the memory is not fully released. It drops down to around 40%. Even > though in the log file it is shown that virtuoso restarts, I have to restart > the VM in order to get my full memory back. > > I have repeated the process today and I am attaching the respective log file > part (I though it might be useful). Perhaps you can understand more from the > log, or you could give me instructions for more detailed logging. > > Please note that I have already followed in the instructions regarding the > buffering (General Memory Usage Settings) and the swapping (Linux-only -- > "swappiness") on the > http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtRDFPerformanceTuning. > > I really appreciate your help. > > Kind regards, > Pantelis Natsiavas > > 2016-08-10 4:09 GMT+03:00 Hugh Williams <hwilli...@openlinksw.com>: > Hi Pantelis, > > What is the memory consumption of Virtuoso whilst running the insert query > and what is the size of the graph the insert is being performed against ? As > note for SPARUL operation against graph with large amounts of data it is > recommended queries are performed in row-wise auto commit mode to reduce > memory consumption which case otherwise be depleted causing hangs or the > server crash if memory cannot be allocated, with the Virtuoso log_enable > function as detailed at: > > http://docs.openlinksw.com/virtuoso/fn_log_enable.html > > http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtTipsAndTricksGuideDeleteLargeGraphs > > Best Regards > Hugh Williams > Professional Services > OpenLink Software, Inc. // http://www.openlinksw.com/ > Weblog -- http://www.openlinksw.com/blogs/ > LinkedIn -- http://www.linkedin.com/company/openlink-software/ > Twitter -- http://twitter.com/OpenLink > Google+ -- http://plus.google.com/100570109519069333827/ > Facebook -- http://www.facebook.com/OpenLinkSoftware > Universal Data Access, Integration, and Management Technology Providers > >> On 9 Aug 2016, at 12:11, Pantelis Natsiavas <natsia...@gmail.com> wrote: >> >> I have just retried the above long-time query and checked the virtuoso.log. >> I see that after 45 minutes I get the following log entries (I executed the >> query around 10.00): >> >> 10:24:23 * Monitor: High disk read (2) >> 10:41:23 * Monitor: High disk read (2) >> 10:43:24 * Monitor: High disk read (2) >> 10:43:41 mmap failed with 12 >> 10:43:45 mmap failed with 12 >> 10:43:45 mmap failed with 12 >> 10:43:45 mmap failed with 12 >> 10:43:46 mmap failed with 12 >> 10:43:46 mmap failed with 12 >> 10:43:46 GPF: Dkpool.c:1634 could not allocate memory with mmap >> >> The same messages (more or less) had in the previous "lost connection" >> incident. >> >> Could somebody provide some hints? >> >> Kind regards, >> Pantelis Natsiavas >> >> >> 2016-08-09 9:58 GMT+03:00 Pantelis Natsiavas <natsia...@gmail.com>: >> Hi. >> >> I am trying to execute a SPARQL INSERT-WHERE taking a lot time to complete. >> >> INSERT { >> GRAPH <a> { >> ?s <hasLabelAfterManualConversion> ?newLabel >> } >> } >> WHERE { >> GRAPH <a> { >> ?s ?p ?o . >> FILTER (REGEX(STR(?o),"abc","i")) . >> BIND(REPLACE(?o, "abc", "", "i") AS ?newLabel) >> } >> }; >> >> Trying to execute it through the web environment, I get a timeout. I thought >> that the most appropriate way to do it would be through the isql-v command >> line environment. When I try to do it, after some hours I get >> >> *** Error 08S01: [Virtuoso Driver]CL065: Lost connection to server >> >> This is rather peculiar as the isql-v runs on the same VM as the virtuoso. >> It should also be noted that the virtuoso instance responds and has not >> collapsed. >> >> I have three questions: >> 1. Is this "lost connection" behavior in the isql-v environment normal? >> 2. Does this "lost connection" message mean that the transaction has >> rolled-back or is the command still running? >> 3. Is there a better way to run long-time queries against virtuoso? >> >> Kind regards, >> Pantelis Natsiavas >> >> ------------------------------------------------------------------------------ >> What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic >> patterns at an interface-level. Reveals which users, apps, and protocols are >> consuming the most bandwidth. Provides multi-vendor support for NetFlow, >> J-Flow, sFlow and other flows. Make informed decisions using capacity >> planning reports. >> http://sdm.link/zohodev2dev_______________________________________________ >> Virtuoso-users mailing list >> Virtuoso-users@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/virtuoso-users > > > <virtuoso.log>
smime.p7s
Description: S/MIME cryptographic signature
------------------------------------------------------------------------------ What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic patterns at an interface-level. Reveals which users, apps, and protocols are consuming the most bandwidth. Provides multi-vendor support for NetFlow, J-Flow, sFlow and other flows. Make informed decisions using capacity planning reports. http://sdm.link/zohodev2dev
_______________________________________________ Virtuoso-users mailing list Virtuoso-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/virtuoso-users
