Hi All, I bulk loaded 11M rdf file (.ttl) into virtuoso graph "A" using "ld_dir" and "rdf_loader_run" and it took 8.5 hours which seems a lot slower considering virtuoso can load data at 36K per second on a single 4 core machine as per virtuoso documentation. ( http://goo.gl/D5zw1) Considering high number of File IO involved in 11M files i combined all of them and created a single file of size 37GB. But re-upload took more then 12 hours into graph "B" on same server.
Does anyone has any suggestion about optimal order of number of files for bulk upload ? Here is more information about my server configuration. vm.swappiness = 60 system RAM : 96GB with 16 CPU =====from virtuoso.ini file ===== ServerThreads = 10 CheckpointInterval = 60 O_DIRECT = 0 NumberOfBuffers = 4000000 MaxDirtyBuffers = 3000000 CaseMode = 2 MaxStaticCursorRows = 5000 CheckpointAuditTrail = 0 AllowOSCalls = 0 SchedulerInterval = 10 FreeTextBatchSize = 100000 SingleCPU = 0 [Client] SQL_PREFETCH_ROWS = 100 SQL_PREFETCH_BYTES = 16000 SQL_QUERY_TIMEOUT = 0 SQL_TXN_TIMEOUT = 0 [VDB] ArrayOptimization = 0 NumArrayParameters = 10 VDBDisconnectTimeout = 1000 KeepConnectionOnFixedThread = 0 [Replication] ServerName = db-RDESC ServerEnable = 1 QueueMax = 50000 [Striping] Segment1 = 100M, db-seg1-1.db, db-seg1-2.db Segment2 = 100M, db-seg2-1.db Thanks Sumit Purohit ------------------------------------------------------------------------------ Precog is a next-generation analytics platform capable of advanced analytics on semi-structured data. The platform includes APIs for building apps and a phenomenal toolset for data science. Developers can use our toolset for easy data analysis & visualization. Get a free account! http://www2.precog.com/precogplatform/slashdotnewsletter _______________________________________________ Virtuoso-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/virtuoso-users
