Hi Thomas, What is this JDBC Connector from Oracle that is being used for the inserts in RDF/XML form ?
What is the ORCID dataset being used as the only one I see is in N-Triple
format from 2014 at:
https://datahub.io/dataset/orcid_2014_dataset
<https://datahub.io/dataset/orcid_2014_dataset>
Performing inserts with transaction would consume more memory maintaining the
transaction than with log_enable(2) which auto commits without transaction
logging in memory.
The O_DIRECT param set in your INI file is an old param for which no real
benefit has been seen on current OS’es and on a Linux system setting swappiness
as detailed at:
https://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtRDFPerformanceTuning#Linux-only%20--%20”swappiness";
Would give better results.
There is also no real need to set ColumnStore = 1 as for as the RDF_QUAD tables
is column store by default in Virtuoso 7 , so that setting would only have
effect on default SQL table creation
If you still have problems, can you provide a copy of your virtuoso.log file
and the output of the “status();” command for review ...
Best Regards
Hugh Williams
Professional Services
OpenLink Software, Inc. // http://www.openlinksw.com/
Weblog -- http://www.openlinksw.com/blogs/
LinkedIn -- http://www.linkedin.com/company/openlink-software/
Twitter -- http://twitter.com/OpenLink
Google+ -- http://plus.google.com/100570109519069333827/
Facebook -- http://www.facebook.com/OpenLinkSoftware
Universal Data Access, Integration, and Management Technology Providers
> On 9 Mar 2017, at 17:28, Thomas Michaux <mich...@abes.fr> wrote:
>
> Hello,
>
> We are loading ORCID 2016 in a V7 instance (Version 07.20.3217-pthreads for
> Linux as of Feb 10 2017), we DO NOT want to use the bulk loader, instead we
> are providing SPARQL inserts of RDF/XML files via JDBC connector from Oracle.
>
> Virtuoso is hosted on 8 cores, 32Gb platform.
>
> We successfully inserted 75 633 079 triples until virtuoso.log signals
> performances problems on "disk write throughput", is there something else to
> optimize in the virtuoso.ini while we are in this "loading" phase (no SPARQL
> "read" query from clients at the moment ) ?
>
> We've already done :
>
> - full text indexation has been delayed ( DB.DBA.VT_BATCH_UPDATE (
> 'DB.DBA.RDF_OBJ', 'ON', 8640 ); )
> - MaxCheckpointRemap = 505856 ( it's larger than 25% of total pages)
> - UnremapQuota = 0
> - DefaultIsolation = 2
> - O_DIRECT = 1 (we are on XFS filesystem)
> - ColumnStore = 1 (we started from a new, fresh .db, deleted all
> previous existing .db, .trx)
>
> Can we do something at transaction level ? We commit each JDBC insert as
> short as possible (1 insert-> 1 commit), query is :
>
> "'sparql DEFINE sql:log-enable 2 INSERT DATA INTO GRAPH '||graphe ||' { '||
> var_clob_line|| ' }'"
>
> I can see that free memory slowly decrease, and finally the server hang.
>
> Thanks for your help ! (Attached is virtuoso.ini)
>
> Thomas
> <virtuoso.ini>------------------------------------------------------------------------------
> Announcing the Oxford Dictionaries API! The API offers world-renowned
> dictionary content that is easy and intuitive to access. Sign up for an
> account today to start using our lexical data to power your apps and
> projects. Get started today and enter our developer competition.
> http://sdm.link/oxford_______________________________________________
> Virtuoso-users mailing list
> Virtuoso-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/virtuoso-users
smime.p7s
Description: S/MIME cryptographic signature
------------------------------------------------------------------------------ Announcing the Oxford Dictionaries API! The API offers world-renowned dictionary content that is easy and intuitive to access. Sign up for an account today to start using our lexical data to power your apps and projects. Get started today and enter our developer competition. http://sdm.link/oxford
_______________________________________________ Virtuoso-users mailing list Virtuoso-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/virtuoso-users
