Hi Thomas, Looking at your INI file I note in the “[Parameters]” section you have "MaxClientConnections = 10” set , which should be set to something like 100 I would recommend as this param sets the number of ServerThreads available to the SQL database for processing requests, which includes SQL connection and threads used by the server for internal processes/tasks, which are allocated on demand. The "MaxClientConnections = 10” setting in the “[HTTP Server]” section can be left as is as this is used for the HTTP Web Server thread pool and are pre-allocated on server startup for use in the thread pool.
Best Regards Hugh Williams Professional Services OpenLink Software, Inc. // http://www.openlinksw.com/ <http://www.openlinksw.com/> Weblog -- http://www.openlinksw.com/blogs/ <http://www.openlinksw.com/blogs/> LinkedIn -- http://www.linkedin.com/company/openlink-software/ <http://www.linkedin.com/company/openlink-software/> Twitter -- http://twitter.com/OpenLink <http://twitter.com/OpenLink> Google+ -- http://plus.google.com/100570109519069333827/ <http://plus.google.com/100570109519069333827/> Facebook -- http://www.facebook.com/OpenLinkSoftware <http://www.facebook.com/OpenLinkSoftware> Universal Data Access, Integration, and Management Technology Providers > On 20 Mar 2017, at 11:02, Thomas Michaux <mich...@abes.fr > <mailto:mich...@abes.fr>> wrote: > > ...sorry, sent the INDEX information tab instead of "SPACE" info tab : > > Table Index name Rows Pages Row pages Blob pages Size > DB.DBA.RDF_OBJ DB.DBA.RDF_OBJ 30443064 255445 255445 0 > 1833.25 MB > DB.DBA.RDF_IRI DB.DBA.RDF_IRI 59063426 206412 206412 0 > 1247.19 MB > DB.DBA.RDF_IRI DB_DBA_RDF_IRI_UNQC_RI_ID 59063426 183541 > 183541 0 1282.95 MB > DB.DBA.RDF_OBJ_RO_FLAGS_WORDS DB.DBA.RDF_OBJ_RO_FLAGS_WORDS 8341023 179875 > 179875 0 1332.22 MB > DB.DBA.RDF_OBJ RO_VAL 30443064 117863 117863 0 773.81 > MB > DB.DBA.RO_START DB.DBA.RO_START 30443064 85319 85319 0 > 475.31 MB > Le 20/03/2017 à 11:34, Thomas Michaux a écrit : >> >> >> Le 19/03/2017 à 16:15, Hugh Williams a écrit : >>> Hi Thomas, >> Hi, >>> Is the loading of the dataset now complete or it is still in progress as >>> you opening statement is not clear ? >>> You should not need 40GB RAM for inserting and hosting 240 million triples, >>> which should require less then 10GB depending on how well they can be >>> compressed for storage in the database. >> loading is complete, we finished at 243 188 427 triples , hosting now >> requires 25GB ram, 15Gb disk, details : >> >> void:triples 243188427 ; >> void:classes 13 ; >> void:entities 58523487 ; >> void:distinctSubjects 58523514 ; >> void:properties 32 ; >> void:distinctObjects 73171603 . >> >> Total pages 1925120 >> Free pages 607377 >> Buffers 2720000 >> Buffers used 244554 >> Dirty buffers 3 >> Wired down buffers 0 >> Table Index name Touches Reads Read % >> DB.DBA.RDF_QUAD RDF_QUAD 1562356553 36371 0 >> DB.DBA.RDF_QUAD RDF_QUAD_POGS 609423455 16989 0 >> DB.DBA.RDF_QUAD RDF_QUAD_SP 378769255 35822 0 >> DB.DBA.RDF_QUAD RDF_QUAD_GS 340377017 1634 0 >>> I assume you have set the swappiness as suggested previously ? >> yes, done, $ sysctl vm.swappiness >> vm.swappiness = 10 >> >>> When you recompiled your Virtuoso was this done from the git stable/7 or >>> develop/7 branch , as I latter has a number of memory consumption fixes >>> that would not be in stable/7, thus I would suggest building from develop/7. >> will investigate. >> >> The two main problems we encountered while loading were : >> >> - logs messages indicating "Flushing at 5.7 MB/s while application is making >> dirty pages at 1.7 MB/s." which we interpreted as not enough write speed >> while receiving lots of JDBC INSERTs (disk issue ? buffer issue ? ...) >> >> - high memory consumption (40GB RAM), virtuoso process never releasing >> memory while loading, free RAM always going down... >> >>> Have you provided a copy of your INI file previously, if not can you >>> provide a copy ? >> see attached (FYI QueryLog= was not active while loading) >>> Do ensure the following params are set to 1 in order to clean up unused >>> threads/resources and reduce memory consumption of the Virtuoso server, >>> which can otherwise be construed as memory leaks.: >>> >>> ThreadCleanupInterval = 1 >>> ResourcesCleanupInterval = 1 >> we have theses settings right. >> >> Thanks for your help, >> >> Thomas >> >> if needed we model ORCID 2016 dataset using : >> c1 c2 >> http://xmlns.com/foaf/0.1/Person <http://xmlns.com/foaf/0.1/Person> >> 28021451 >> http://purl.org/ontology/bibo/Document >> <http://purl.org/ontology/bibo/Document> >> 14283692 >> http://purl.org/ontology/bibo/Journal >> <http://purl.org/ontology/bibo/Journal> >> 9104659 >> http://xmlns.com/foaf/0.1/PersonalProfileDocument >> <http://xmlns.com/foaf/0.1/PersonalProfileDocument> >> 2527333 >> http://xmlns.com/foaf/0.1/Article <http://xmlns.com/foaf/0.1/Article> >> 974945 >> http://www.w3.org/ns/org#Membership <http://www.w3.org/ns/org#Membership> >> 807465 >> http://www.w3.org/2006/vcard/ns#Address >> <http://www.w3.org/2006/vcard/ns#Address> >> 807423 >> http://www.w3.org/ns/org#Organization >> <http://www.w3.org/ns/org#Organization> >> 807418 >> http://purl.org/ontology/bibo/Conference >> <http://purl.org/ontology/bibo/Conference> >> 769451 >> http://www.w3.org/ns/org#OrganizationalUnit >> <http://www.w3.org/ns/org#OrganizationalUnit> >> 649291 >> http://www.w3.org/2004/02/skos/core#Concept >> <http://www.w3.org/2004/02/skos/core#Concept> >> 371731 >> http://purl.org/ontology/bibo/Book <http://purl.org/ontology/bibo/Book> >> 205493 >> http://www.w3.org/ns/org#Role <http://www.w3.org/ns/org#Role> >> 168423 >> http://www.w3.org/1999/02/22-rdf-syntax-ns#Property >> <http://www.w3.org/1999/02/22-rdf-syntax-ns#Property> >> 170 >> http://www.openlinksw.com/schemas/virtrdf#QuadMapFormat >> <http://www.openlinksw.com/schemas/virtrdf#QuadMapFormat> >> 130 >> http://www.openlinksw.com/schemas/virtrdf#array-of-QuadMapFormat >> <http://www.openlinksw.com/schemas/virtrdf#array-of-QuadMapFormat> >> 98 >> http://www.w3.org/2000/01/rdf-schema#Class >> <http://www.w3.org/2000/01/rdf-schema#Class> >> 56 >> http://www.openlinksw.com/schemas/virtrdf#QuadMapValue >> <http://www.openlinksw.com/schemas/virtrdf#QuadMapValue> >> 8 >> http://www.openlinksw.com/schemas/virtrdf#array-of-QuadMapColumn >> <http://www.openlinksw.com/schemas/virtrdf#array-of-QuadMapColumn> >> 8 >> http://www.openlinksw.com/schemas/virtrdf#QuadMapColumn >> <http://www.openlinksw.com/schemas/virtrdf#QuadMapColumn> >> 8 >> >> >> >>> Best Regards >>> Hugh Williams >>> Professional Services >>> OpenLink Software, Inc. // http://www.openlinksw.com/ >>> <http://www.openlinksw.com/> >>> Weblog -- http://www.openlinksw.com/blogs/ >>> <http://www.openlinksw.com/blogs/> >>> LinkedIn -- http://www.linkedin.com/company/openlink-software/ >>> <http://www.linkedin.com/company/openlink-software/> >>> Twitter -- http://twitter.com/OpenLink <http://twitter.com/OpenLink> >>> Google+ -- http://plus.google.com/100570109519069333827/ >>> <http://plus.google.com/100570109519069333827/> >>> Facebook -- http://www.facebook.com/OpenLinkSoftware >>> <http://www.facebook.com/OpenLinkSoftware> >>> Universal Data Access, Integration, and Management Technology Providers >>> >>> >>> >>>> On 15 Mar 2017, at 17:08, Thomas Michaux <mich...@abes.fr> >>>> <mailto:mich...@abes.fr> wrote: >>>> >>>> Hello, >>>> >>>> FYI, virtuoso still loading but we needed to increase memory ressources, >>>> now the process use almost 40GB of ram : >>>> >>>> [devel@tulipe-test2 ~]$ ./memcheck-virtuoso.sh >>>> 2017-03-15T17:54 VmSize: 41273424kB 5883 >>>> >>>> stats for the graph <http://hub.abes.fr/referentiel/ORCID/2016> >>>> <http://hub.abes.fr/referentiel/ORCID/2016> (forget >>>> to mention, it's the only graph in db) : >>>> >>>> 239 451 028 triples >>>> >>>> >>>> this:Dataset a void:Dataset ; >>>> rdfs:seeAlso <http://hub.abes.fr/referentiel/ORCID/2016> >>>> <http://hub.abes.fr/referentiel/ORCID/2016> ; >>>> rdfs:label "" ; >>>> void:sparqlEndpoint <http://idrefplus.v102.abes.fr:8890/sparql> >>>> <http://idrefplus.v102.abes.fr:8890/sparql> ; >>>> void:triples 239451028 ; >>>> void:classes 13 ; >>>> void:entities 57692917 ; >>>> void:distinctSubjects 57650847 ; >>>> void:properties 32 ; >>>> void:distinctObjects 72219514 . >>>> >>>> this:sameAsLinks a void:Linkset ; >>>> void:inDataset this:Dataset ; >>>> void:triples 997389 ; >>>> void:linkPredicate owl:sameAs . >>>> >>>> >>>> Le 14/03/2017 à 10:05, Thomas Michaux a écrit : >> >> >> >> ------------------------------------------------------------------------------ >> Check out the vibrant tech community on one of the world's most >> engaging tech sites, Slashdot.org <http://slashdot.org/>! >> http://sdm.link/slashdot <http://sdm.link/slashdot> >> >> _______________________________________________ >> Virtuoso-users mailing list >> Virtuoso-users@lists.sourceforge.net >> <mailto:Virtuoso-users@lists.sourceforge.net> >> https://lists.sourceforge.net/lists/listinfo/virtuoso-users >> <https://lists.sourceforge.net/lists/listinfo/virtuoso-users> >
smime.p7s
Description: S/MIME cryptographic signature
------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________ Virtuoso-users mailing list Virtuoso-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/virtuoso-users
