Hi Alexander,
I presume 50kk in 50 million ?
You indicated having allocate all you memory for use by the Virtuoso Server
whereas it is recommended about 60% of memory is allocated as detailed at:
http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtRDFPerformanceTuning
Also, are you running the Virtuoso RDF Bulk loader scripts detailed at:
http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtBulkRDFLoaderScript
as these are what we used for loading the billion plus datasets loading into
some of the endpoints we use. These script auto set log_enable mode prior to
upload. Note you can also run multiple rdf_runloader() scripts to perform
parallel loading of the datasets for better performance. I’ve had upwards for
300 million triple uploads per hour on an 8 core machine with 72GB of memory
running 8 instances of the rdf_runloader() scripts in parallel ( one for each
core).
Best Regards
Hugh Williams
Professional Services
OpenLink Software
Web: http://www.openlinksw.com
Support: http://support.openlinksw.com
Forums: http://boards.openlinksw.com/support
Twitter: http://twitter.com/OpenLink
On 26 Jun 2011, at 17:00, Alexander Sidorov wrote:
> Hello!
>
> What does influence on the bulk dataset uploading performance? I need to
> upload about 50kk triples (quads) to the server with outdated hardware and
> 1gb of memory. I have set MaxCheckpointRemap, NumberOfBuffers and
> MaxDirtyBuffers to take all available memory but uploading goes very slow,
> about 100k triples per hour (I am not even sure whether memory size plays
> significant role in dataset uploading process). Are there any other points to
> improve uploading performance? Does log_enable play role here (like in graph
> deletion)?
>
> Regards,
> Alexander
> ------------------------------------------------------------------------------
> All of the data generated in your IT infrastructure is seriously valuable.
> Why? It contains a definitive record of application performance, security
> threats, fraudulent activity, and more. Splunk takes this data and makes
> sense of it. IT sense. And common sense.
> http://p.sf.net/sfu/splunk-d2d-c2_______________________________________________
> Virtuoso-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/virtuoso-users
