Hi Sumit, What you need is --
* Fewer - larger sized .ttl files to alleviate the IO overhead, and * Multiple invocations of rdf_loader_run() in order to maximise CPU core usage There is lots of additional information contained in this doc -- https://docs.google.com/document/d/1yLOBmP4xkYvdTzga6tF2vTKHRCgo0GacmBIbW_Sf wJQ/edit?pli=1 Additionally, Virtuoso 7 (if you are not already using it) shows significant performance gains over Virtuoso 6 when loading data... I hope this helps? Regards, Garry M. Biggs Technical Support OpenLink Software, Inc. // http://www.openlinksw.com/ 10 Burlington Mall Road, Suite 265, Burlington MA 01803 Weblog -- http://www.openlinksw.com/blogs/ LinkedIn -- http://www.linkedin.com/company/openlink-software/ Twitter -- http://twitter.com/OpenLink Google+ -- http://plus.google.com/100570109519069333827/ Facebook -- http://www.facebook.com/OpenLinkSoftware Universal Data Access, Integration, and Management Technology Providers > -----Original Message----- > From: Purohit, Sumit [mailto:[email protected]] > Sent: 16 April 2013 19:07 > To: [email protected] > Subject: [Virtuoso-users] Impact of number of files on bulk RDF load > > Hi All, > > I bulk loaded 11M rdf file (.ttl) into virtuoso graph "A" using "ld_dir" and > "rdf_loader_run" and it took 8.5 hours which seems a lot slower considering > virtuoso can load data at 36K per second on a single 4 core machine as per > virtuoso documentation. ( http://goo.gl/D5zw1) Considering high number of > File IO involved in 11M files i combined all of them and created a single file of > size 37GB. But re-upload took more then 12 hours into graph "B" on same > server. > > > Does anyone has any suggestion about optimal order of number of files for > bulk upload ? > > Here is more information about my server configuration. > > vm.swappiness = 60 > system RAM : 96GB with 16 CPU > > > =====from virtuoso.ini file ===== > > ServerThreads = 10 > CheckpointInterval = 60 > O_DIRECT = 0 > NumberOfBuffers = 4000000 > MaxDirtyBuffers = 3000000 > CaseMode = 2 > MaxStaticCursorRows = 5000 > CheckpointAuditTrail = 0 > AllowOSCalls = 0 > SchedulerInterval = 10 > > > FreeTextBatchSize = 100000 > SingleCPU = 0 > > > [Client] > SQL_PREFETCH_ROWS = 100 > SQL_PREFETCH_BYTES = 16000 > SQL_QUERY_TIMEOUT = 0 > SQL_TXN_TIMEOUT = 0 > > [VDB] > ArrayOptimization = 0 > NumArrayParameters = 10 > VDBDisconnectTimeout = 1000 > KeepConnectionOnFixedThread = 0 > > [Replication] > ServerName = db-RDESC > ServerEnable = 1 > QueueMax = 50000 > > > [Striping] > Segment1 = 100M, db-seg1-1.db, db-seg1-2.db > Segment2 = 100M, db-seg2-1.db > > > > Thanks > Sumit Purohit > ---------------------------------------------------------------------------- -- > Precog is a next-generation analytics platform capable of advanced analytics > on semi-structured data. The platform includes APIs for building apps and a > phenomenal toolset for data science. Developers can use our toolset for > easy data analysis & visualization. Get a free account! > http://www2.precog.com/precogplatform/slashdotnewsletter > _______________________________________________ > Virtuoso-users mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/virtuoso-users
smime.p7s
Description: S/MIME cryptographic signature
------------------------------------------------------------------------------ Precog is a next-generation analytics platform capable of advanced analytics on semi-structured data. The platform includes APIs for building apps and a phenomenal toolset for data science. Developers can use our toolset for easy data analysis & visualization. Get a free account! http://www2.precog.com/precogplatform/slashdotnewsletter
_______________________________________________ Virtuoso-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/virtuoso-users
