Hi Piret,
If you have 10.6 billion trips and only 599GB RAM, then given that Virtuoso 7
typically requires 10GB RAM per billion triples, I suspect all the memory is
being consumed and there will be a lot of swapping to disk, which is why I
wanted to see the output of running the “status();” command when the database
is “warm” which will show resource usage. The output of the Linux “top”
command would also be useful to see.
I see you use the “isql-vt” command which implies you are using a Virtuoso
installation supplied with one of the Linux distro’s and as said perviously is
rather old (3212) and thus would recommend updating to a latest 3214 stable or
develop 7 build from git as there have been a number of memory related and
other fixes in these later versions.
If your database integrity check is failing on this other instance then you
must to a crash-dump and recovery or a crash recovery when the normal crash
dump recovery fails, all of which are detailed at:
http://docs.openlinksw.com/virtuoso/databaseadmsrv.html#backup
Best Regards
Hugh Williams
Professional Services
OpenLink Software, Inc. // http://www.openlinksw.com/
Weblog -- http://www.openlinksw.com/blogs/
LinkedIn -- http://www.linkedin.com/company/openlink-software/
Twitter -- http://twitter.com/OpenLink
Google+ -- http://plus.google.com/100570109519069333827/
Facebook -- http://www.facebook.com/OpenLinkSoftware
Universal Data Access, Integration, and Management Technology Providers
> On 20 Oct 2015, at 12:44, Piret Lattikas <piret.latti...@zerotech.ee> wrote:
>
> Hi, Hugh.
>
> To answer some of your questions. The total count of triples in our Quad
> Store is around 10.6billion. The server machine has 6 cores and 59GB RAM.
>
> The actual .sql script run is this:
>
> log_enable(2);
>
> SPARQL DELETE { GRAPH ?graph { ?st
> <http://www.w3.org/1999/02/22-rdf-syntax-ns#subject> ?subject . }} INSERT {
> GRAPH ?graph { ?st <http://purl.org/vocab/changeset/schema#subject> ?subject
> . }} WHERE { GRAPH ?graph { ?st
> <http://www.w3.org/1999/02/22-rdf-syntax-ns#subject> ?subject . }} LIMIT
> 10000000;
>
> checkpoint;
>
> log_enable(1);
>
> And the script is called out like this:
>
> isql-vt 1111 dba dba update.sql
>
> The script was called out 5 times in a row and then another checkpoint was
> made like this:
>
> isql-vt 1111 dba dba exec="checkpoint;"
>
> No other process was using virtuoso when we were executing this script and
> when the crash happened.
>
> In addition we have a second instance of virtuoso, which often is making many
> concurrent insertions. We noticed that making integrity check in that
> instance fails after some time with the following error:
> ...
> 09:42:19 /usr/bin/virtuoso-t() [0x5db66c]
> 09:42:19 /usr/bin/virtuoso-t(sf_sql_execute_w+0x74) [0x5db884]
> 09:42:19 /usr/bin/virtuoso-t() [0x8d8f38]
> 09:42:19 /usr/bin/virtuoso-t() [0x8df6fc]
> 09:42:19 /lib/x86_64-linux-gnu/libpthread.so.0(+0x8182) [0x7ff65fd23182]
> 09:42:19 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7ff65f32d47d]
> 09:42:19 GPF: page.c:899 out of order
>
> Is it a problem if we keep using that instance, when the integrity check
> fails with this error? Should we definitely do a crash dump and restore from
> that? The reason I am asking is that this problem occurs almost every week
> and restoring from crash dump takes some time. So processes using that
> virtuoso instance are down during that time.
>
> Best Regards.
>
> Piret Lattikas
>
>
> On 19.10.15 17:41, Hugh Williams wrote:
>> Hi Piret,
>>
>> I note you are using an older Virtuoso 7.2.3212 build from Apr 2015 where as
>> the current stable release build from git or source-forge is 7.2.3214, thus
>> I would suggest updating to this later release or even the current git
>> develop/7 head to get all the latest fixes and see if the problem persists.
>>
>> The “??”’s in gdb back trace output would indicate the Virtuoso binary is
>> stripped of symbols thus to get a meaningful back trace a debug build with
>> symbols intact needs to be compiled as follows:
>>
>> 1. Configure using the —with debug option:
>>
>> ./configure —with-debug
>>
>> 2. In the Makefile, check to ensure CFLAGS has the “-g” option set which
>> generates debug information:
>>
>> CFLAGS = -g -O2
>>
>> 3. Then do "make clean all deinstall reinstall” to build a new debug
>> unstripped binary (virtuoso-t)
>>
>> 4. Start database with this new binary and force the crash condition again
>> to generate a new core file
>>
>> 5. Use gdb to load core file
>>
>> gdb virtuoso-t core
>>
>> 6. At the prompt, type "bt" or “backtrace” to backtrace through stack and
>> provide the output when top of stack is reached.
>>
>> You indicate having 1.7billion triples to be updated of which about 500
>> million are still to be updated, but what is the total count of triples in
>> the Quad Store ( select * where {?s ?p ?o} ) ? Also what are the specs of
>> the server machine being used in terms of number of cores and memory in
>> particular and what does the output of the following command run from the
>> “isql” command line tool, report when the database is “warm” :
>>
>> status();
>>
>> Can you provide the actual content of the SQL script being run ?
>>
>> Best Regards
>> Hugh Williams
>> Professional Services
>> OpenLink Software, Inc. // http://www.openlinksw.com/
>> Weblog -- http://www.openlinksw.com/blogs/
>> LinkedIn -- http://www.linkedin.com/company/openlink-software/
>> Twitter -- http://twitter.com/OpenLink
>> Google+ -- http://plus.google.com/100570109519069333827/
>> Facebook -- http://www.facebook.com/OpenLinkSoftware
>> Universal Data Access, Integration, and Management Technology Providers
>>
>>> On 19 Oct 2015, at 11:37, Piret Lattikas <piret.latti...@zerotech.ee> wrote:
>>>
>>> Hi,
>>>
>>> We were running a simple update SPARQL query for a large set of data. The
>>> update query for replacing one predicate with another, itself is this:
>>>
>>> SPARQL DELETE { GRAPH ?graph { ?st
>>> <http://www.w3.org/1999/02/22-rdf-syntax-ns#subject> ?subject . }} INSERT {
>>> GRAPH ?graph { ?st <http://purl.org/vocab/changeset/schema#subject>
>>> ?subject . }} WHERE { GRAPH ?graph { ?st
>>> <http://www.w3.org/1999/02/22-rdf-syntax-ns#subject> ?subject . }} LIMIT
>>> 10000000;
>>>
>>> The initial count of triples that needed to be updated was ~1 700 000 000.
>>> The .sql script used for update set the log_enable(2) at the beginning,
>>> before executing the actual update. And a checkpoint was called out in the
>>> same .sql script after the update query finished. We were able to run that
>>> script for quite a while without any problems. Suddenly running the script
>>> made virtuoso crash, the remaining triples count that needed updating was
>>> around 500 000 000. The error in virtuoso.log file shows the following:
>>>
>>> 09:13:37 In delete or rb of insert on RDF_QUAD dp 38144031 seg 6, col 3
>>> is 10909 values and next 8276
>>> 09:13:37 /usr/bin/virtuoso-t() [0x8d4afd]
>>> ...
>>> 09:13:37 /usr/bin/virtuoso-t() [0x53f19b]
>>> 09:13:37 /usr/bin/virtuoso-t(delete_node_input+0x13b) [0x5cbb3b]
>>> 09:13:37 /usr/bin/virtuoso-t() [0x5c7fed]
>>> 09:13:37 /usr/bin/virtuoso-t() [0x5c8416]
>>> 09:13:37 /usr/bin/virtuoso-t(table_source_input_unique+0x2a9) [0x5cfe69]
>>> 09:13:37 /usr/bin/virtuoso-t() [0x5c7fed]
>>> ...
>>> 09:13:37 /usr/bin/virtuoso-t() [0x5c826e]
>>> 09:13:37 /usr/bin/virtuoso-t(skip_node_input+0x466) [0x5cc9e6]
>>> 09:13:37 /usr/bin/virtuoso-t() [0x5c7fed]
>>> ...
>>> 09:13:37 /usr/bin/virtuoso-t() [0x5db26a]
>>> 09:13:37 /usr/bin/virtuoso-t(sf_sql_execute_w+0x74) [0x5db884]
>>> 09:13:37 /usr/bin/virtuoso-t() [0x8d8f38]
>>> 09:13:37 /usr/bin/virtuoso-t() [0x8df6fc]
>>> 09:13:37 /lib/x86_64-linux-gnu/libpthread.so.0(+0x8182) [0x7f69a6f65182]
>>> 09:13:37 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f69a656f47d]
>>> 09:13:37 GPF: collock.c:1410 uneven length cols after delete, update or
>>> rollback of insert
>>>
>>> Do get the core dump of the crash set 'ulimit -c unlimited' and started the
>>> script again. The crash is reproducable with every run. The core dump shows
>>> the following:
>>>
>>> # gdb virtuoso-t core
>>> ...
>>> [New LWP 15882]
>>> [New LWP 15879]
>>> [Thread debugging using libthread_db enabled]
>>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>> Core was generated by `/usr/bin/virtuoso-t +wait +configfile
>>> /etc/virtuoso-opensource-7/virtuoso.ini'.
>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>
>>> (gdb) bt
>>>
>>> #0 0x00000000008d4bde in ?? ()
>>> ...
>>> #6 0x000000000053f19b in ?? ()
>>> #7 0x00000000005cbb3b in delete_node_input ()
>>> #8 0x00000000005c7fed in ?? ()
>>> #9 0x00000000005c8416 in ?? ()
>>> #10 0x00000000005cfe69 in table_source_input_unique ()
>>> #11 0x00000000005c7fed in ?? ()
>>> ...
>>> #30 0x00000000005c826e in ?? ()
>>> #31 0x00000000005cc9e6 in skip_node_input ()
>>> #32 0x00000000005c7fed in ?? ()
>>> ...
>>> #43 0x00000000005db26a in ?? ()
>>> #44 0x00000000005db884 in sf_sql_execute_w ()
>>> #45 0x00000000008d8f38 in ?? ()
>>> #46 0x00000000008df6fc in ?? ()
>>> #47 0x00007f69a6f65182 in start_thread (arg=0x7f691a3bd700) at
>>> pthread_create.c:312
>>> #48 0x00007f69a656f47d in clone () at
>>> ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
>>>
>>> We run the integrity check of virtuoso with "backup '/dev/null'" and that
>>> passed successfully, so this indicated for us that the database itself is
>>> intact.
>>>
>>> The version of virtuoso we are using is:
>>>
>>> Virtuoso Open Source Edition (Column Store) (multi threaded)
>>> Version 7.2.0.3212-pthreads as of Apr 8 2015
>>> Compiled for Linux (x86_64-pc-linux-gnu)
>>> Copyright (C) 1998-2015 OpenLink Software
>>>
>>> Do you have any suggestions why this happened and how can it be fixed?
>>>
>>> Best regards,
>>>
>>> Piret Lattikas
>>> ------------------------------------------------------------------------------
>>> _______________________________________________
>>> Virtuoso-users mailing list
>>> Virtuoso-users@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/virtuoso-users
>>
>
------------------------------------------------------------------------------
_______________________________________________
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users
