|
I’ve finally managed to load the Reuters Corpus into KIM (though I have not yet tried to access it – wait for another mail on that).
Whatever the problem I was having with the features XML files, it went away of its own accord, which was nice.
I ran this under Windows 2003 Server x64 with 64 bit Java, allocating 12 GB (yes GB) to KIM and 8GB to SESAME. The KIM console reports it used a tad over 3.1 GB of memory, but task manager reports about 7.5 GB of VM used.
The trick seems to be to split the files out into directories – I put 1000 articles in each of 800 odd directories. Without this there seems to be some issue with the Document Store tool initialising. Also, directories with 1.5 million files in are just a tad unwieldy!
For those interested, it took 3 days and 2 hours to complete the task, processing a little over 3 per second.
The statistics showed (times in seconds)
Doc count 806791 Doc size 1,168,948K Create time 23,928 Parse Features 197,012 Annotation time 146,348 Store time 43,028 Index time 0
We appear to have 1.36 million ‘new instances’ with new-tem-triples being 1,751,486 KB and 14 MB of trusted entities in the cache. I don’t know what that means, but it certainly sounds impressive.
Finally, the Index and store files are around 45GB.
Now to get Tomcat 5.5 working so I can SEE what was in there!
Iain --------------- Iain Downs (Microsoft MVP) Commercial Software Therapist E: [EMAIL PROTECTED] T:+44 (0) 1423 872988 http://mvp.support.microsoft.com
|
_______________________________________________ NOTE: Please REPLY TO ALL to ensure that your reply reaches all members of this mailing list.
KIM-discussion mailing list [email protected] http://ontotext.com/mailman/listinfo/kim-discussion_ontotext.com
