Hi Sumit,

What you need is --

* Fewer - larger sized .ttl files to alleviate the IO overhead, and

* Multiple invocations of rdf_loader_run() in order to maximise CPU core
usage

There is lots of additional information contained in this doc --

 
https://docs.google.com/document/d/1yLOBmP4xkYvdTzga6tF2vTKHRCgo0GacmBIbW_Sf
wJQ/edit?pli=1


Additionally, Virtuoso 7 (if you are not already using it) shows significant
performance gains over Virtuoso 6 when loading data...

I hope this helps?

Regards,

Garry M. Biggs
Technical Support
OpenLink Software, Inc.      //              http://www.openlinksw.com/
10 Burlington Mall Road, Suite 265, Burlington MA 01803
Weblog   -- http://www.openlinksw.com/blogs/
LinkedIn -- http://www.linkedin.com/company/openlink-software/
Twitter  -- http://twitter.com/OpenLink
Google+  -- http://plus.google.com/100570109519069333827/
Facebook -- http://www.facebook.com/OpenLinkSoftware
Universal Data Access, Integration, and Management Technology Providers

> -----Original Message-----
> From: Purohit, Sumit [mailto:[email protected]]
> Sent: 16 April 2013 19:07
> To: [email protected]
> Subject: [Virtuoso-users] Impact of number of files on bulk RDF load
> 
> Hi All,
> 
> I bulk loaded 11M rdf file (.ttl) into virtuoso graph "A" using "ld_dir"
and
> "rdf_loader_run" and it took 8.5 hours which seems a lot slower
considering
> virtuoso can load data at 36K per second  on a single 4 core machine as
per
> virtuoso documentation. ( http://goo.gl/D5zw1) Considering high number of
> File IO involved in 11M files i combined all of them and created a single
file of
> size 37GB. But re-upload took more then 12 hours into graph "B" on same
> server.
> 
> 
> Does anyone has any suggestion about optimal order of number of files for
> bulk upload ?
> 
> Here is more information about my server configuration.
> 
> vm.swappiness = 60
> system RAM : 96GB with 16 CPU
> 
> 
> =====from virtuoso.ini file =====
> 
> ServerThreads            = 10
> CheckpointInterval       = 60
> O_DIRECT                 = 0
> NumberOfBuffers          = 4000000
> MaxDirtyBuffers          = 3000000
> CaseMode                 = 2
> MaxStaticCursorRows      = 5000
> CheckpointAuditTrail     = 0
> AllowOSCalls             = 0
> SchedulerInterval        = 10
> 
> 
> FreeTextBatchSize        = 100000
> SingleCPU                = 0
> 
> 
> [Client]
> SQL_PREFETCH_ROWS  = 100
> SQL_PREFETCH_BYTES = 16000
> SQL_QUERY_TIMEOUT  = 0
> SQL_TXN_TIMEOUT    = 0
> 
> [VDB]
> ArrayOptimization           = 0
> NumArrayParameters          = 10
> VDBDisconnectTimeout        = 1000
> KeepConnectionOnFixedThread = 0
> 
> [Replication]
> ServerName   = db-RDESC
> ServerEnable = 1
> QueueMax     = 50000
> 
> 
> [Striping]
> Segment1 = 100M, db-seg1-1.db, db-seg1-2.db
> Segment2 = 100M, db-seg2-1.db
> 
> 
> 
> Thanks
> Sumit Purohit
>
----------------------------------------------------------------------------
--
> Precog is a next-generation analytics platform capable of advanced
analytics
> on semi-structured data. The platform includes APIs for building apps and
a
> phenomenal toolset for data science. Developers can use our toolset for
> easy data analysis & visualization. Get a free account!
> http://www2.precog.com/precogplatform/slashdotnewsletter
> _______________________________________________
> Virtuoso-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/virtuoso-users

Attachment: smime.p7s
Description: S/MIME cryptographic signature

------------------------------------------------------------------------------
Precog is a next-generation analytics platform capable of advanced
analytics on semi-structured data. The platform includes APIs for building
apps and a phenomenal toolset for data science. Developers can use
our toolset for easy data analysis & visualization. Get a free account!
http://www2.precog.com/precogplatform/slashdotnewsletter
_______________________________________________
Virtuoso-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

Reply via email to