Hi All,

I bulk loaded 11M rdf file (.ttl) into virtuoso graph "A" using "ld_dir" and 
"rdf_loader_run" and it took 8.5 hours which seems a lot slower considering 
virtuoso can load data at 36K per second  on a single 4 core machine as per 
virtuoso documentation. ( http://goo.gl/D5zw1)
Considering high number of File IO involved in 11M files i combined all of them 
and created a single file of size 37GB. But re-upload took more then 12 hours 
into graph "B" on same server.


Does anyone has any suggestion about optimal order of number of files for bulk 
upload ?   

Here is more information about my server configuration.

vm.swappiness = 60
system RAM : 96GB with 16 CPU


=====from virtuoso.ini file =====

ServerThreads            = 10
CheckpointInterval       = 60
O_DIRECT                 = 0
NumberOfBuffers          = 4000000
MaxDirtyBuffers          = 3000000
CaseMode                 = 2
MaxStaticCursorRows      = 5000
CheckpointAuditTrail     = 0
AllowOSCalls             = 0
SchedulerInterval        = 10


FreeTextBatchSize        = 100000
SingleCPU                = 0


[Client]
SQL_PREFETCH_ROWS  = 100
SQL_PREFETCH_BYTES = 16000
SQL_QUERY_TIMEOUT  = 0
SQL_TXN_TIMEOUT    = 0

[VDB]
ArrayOptimization           = 0
NumArrayParameters          = 10
VDBDisconnectTimeout        = 1000
KeepConnectionOnFixedThread = 0

[Replication]
ServerName   = db-RDESC
ServerEnable = 1
QueueMax     = 50000


[Striping]
Segment1 = 100M, db-seg1-1.db, db-seg1-2.db
Segment2 = 100M, db-seg2-1.db



Thanks
Sumit Purohit
------------------------------------------------------------------------------
Precog is a next-generation analytics platform capable of advanced
analytics on semi-structured data. The platform includes APIs for building
apps and a phenomenal toolset for data science. Developers can use
our toolset for easy data analysis & visualization. Get a free account!
http://www2.precog.com/precogplatform/slashdotnewsletter
_______________________________________________
Virtuoso-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

Reply via email to