Stefan Scheffler wrote: > Hey Paolo. > Thanks for your reply. > I used tdbloader2 with an own Tokenizer / Errorhandler (which just > catches / skips errors and writes them into a file). > the command was /.tdbloader2 --loc=<store> <srcpath>/* > > Is there a possibility to do incremental loads with the script files or > do i have to write a own program?
Hi Stefan, if you want to run an incremental load you should use tdbloader, not tdbloader2. tdbloader supports incremental loads, tdbloader2 not. If you are loading large datasets make sure you have enough RAM (you can load data on a machine with a lot of RAM and move indexes elsewhere). Paolo > > Regards, > Stefan > > Am 24.06.2012 10:42, schrieb Paolo Castagna: >> Hi Stefan, >> as Rob said, loading data into an empty TDB store is a different from >> loading >> data into an existing TDB store. >> >> I assume that for your second data load you used tdbloader not >> tdbloader2. >> >> tdbloader2 does not even support incremental data loads (i.e. it will >> overwrite >> your existing data). I suspect this is what is going on. >> >> Can you share the exact commands you used as well as links to the RDF >> data? >> (this way others can replicate your experiments). >> >> Regards, >> Paolo >> >> Stefan Scheffler wrote: >>> Hello, >>> At the moment i am doing some performance checks on tdb. The first i >>> checked was the import of the tdbloader2 and i got some weird results. >>> Maybe someone can help me out. Here are my testbase and the results. >>> >>> The first test was to store 12 GB of triples into an empty store (i used >>> the german dbpedia). >>> >>> Load time: 16 minutes >>> average loading: ca 81.000 triple / second >>> index time: 40 minutes >>> store size: 9,3GB >>> >>> >>> The second test was to store the same data into an allready filled store >>> As i started the import i created a store with 348.398.593 Triples from >>> DNB and HBZ (which are german libraries, store size: 33 GB). >>> Then i started to load the german dbpedia in. >>> >>> Load time: 3 hours and 4 minutes >>> average loading: ca 7200 / second >>> index time: 38 minutes >>> store size: 19 GB!!!!! >>> >>> Why does the loading time increases that immense? My expectation was, >>> that the index time increases. But it does not. There where no other big >>> proccesses running nearby. And why does the store size shrink to 19GB? I >>> am totally confused about that point. >>> >>> With friendly regards >>> Stefan >>> > >
