Hello,
I tried xloader and the upload was completed correctly in about 14
hours. It's perfect ! thanks for your help.
How can we know when xloader is more efficient than tdbloader?
Is there a ratio of Ram/ntriples to push to be done?
regards,
Steven
Le mar., mars 14 2023 at 10:08:02 +0000, Andy Seaborne
<[email protected]> a écrit :
On 14/03/2023 08:56, Steven Blanchard wrote:
Hello,
Yes, I do a one time operation on a non-existant folder. It's faster
to do this than split input file and do multiple upload ?
Yes, the loading is to a disk.
Why does the performance of indexing decrease over time?
As the indexes grow in size, the efficiency of the OS file system
cache drops.
In loader=phased, one index is created as the loaded, then the phases
create the other indexes by copying the first index to the others ...
which is in effect a sort, and when the OS file system cache becomes
less effective, there is more disk I/O.
The xloader are designed to avoid (reduce) this. It uses an external
sort program (Linux sort(1) - batching sorting) to get the
information to load sequentially.
Andy
Thanks,
Steven
Le lun., mars 13 2023 at 21:26:55 +0000, Andy Seaborne
<[email protected] <mailto:[email protected]>> a écrit :
On 13/03/2023 18:35, Simon Bin wrote:
if you do a 1 time operation of non-existant => tdb, try xloader.
xloader is good when loading to disk, rather than on to an SSD.
Steven - is that what you are using?
Andy
On Mon, 2023-03-13 at 16:08 +0100, Steven Blanchard wrote:
Hello,
I am currently loading data in ttl format with the command line
tdb2.tdbloader (option --loader=phased ). This data consists of
1.25
billion tuples. The first step of loading is finish and the
indexing
begin. Since then, the indexing speed has been steadily decreasing
until reaching an average speed of 651 while only 474 million
triples
have been indexed. With this speed, the indexing take several
month.
Do you have tips and solution to speed up the indexing speed?
Could
this be due to the input files?
Thanks,
Steven
