`
On 30 Sep 2011, at 10:52, Shri :) wrote:
> Hi All,
>
>
> @Damian thanks for the link, I will now try increasing the buffer_pool_size
> and carry out the loading..Will let you know how it goes.
Thanks Shri.
> @ Andy: Are you using the sdb bulk loader or loading via your own code?What
> format is the data in?
> I am using the following code, which I don't think it is very different from
> the one that you suggested, *my data is in .TTL format*
> Here is the snippet of my code:
> //read data into the database
> InputStream inn= new FileInputStream ("dataset_70000.nt"); long start =
> System.currentTimeMillis(); model.read(inn, "localhost", "TTL");
> loadtime=ext.elapsedTime(start); // Close the database connection
> store.close(); System.out.println("Loading time: " + loadtime);
The formatting went wonky for me, but that looks fine to me.
Sanity check: try sdbload. If that's faster we have a bug somewhere.
> @Dave I think I followed the pattern suggested in the link that you gave me
> (http://openjena.org/wiki/SDB/Loading_data), the above is the snippet of my
> source code.
> And one more thing, I didn't get the idea of "Are you wrapping the load in a
> transaction to avoid auto-commit costs?", can you please elaborate a bit on
> this?? Sorry, I am relatively a novice..
Short answer: don't worry, you're not being bitten by this.
Long answer: if you were loading the data triple by triple, e.g.
for (Statement s: statementsFromFile) {
// Do something with s
model.add(s);
}
then that would be really slow. Each add is a distinct database action, and
costly. Adding everything in one API call (as you have) should be fine.
Damian
