I’ve finally managed to load the Reuters Corpus into KIM (though I have not yet tried to access it – wait for another mail on that).

 

Whatever the problem I was having with the features XML files, it went away of its own accord, which was nice.

 

I ran this under Windows 2003 Server x64 with 64 bit Java, allocating 12 GB (yes GB) to KIM and 8GB to SESAME.  The KIM console reports it used a tad over 3.1 GB of memory, but task manager reports about 7.5 GB of VM used.

 

The trick seems to be to split the files out into directories – I put 1000 articles in each of 800 odd directories.  Without this there seems to be some issue with the Document Store tool initialising.  Also, directories with 1.5 million files in are just a tad unwieldy!

 

For those interested, it took 3 days and 2 hours to complete the task, processing a little over 3 per second.

 

The statistics showed (times in seconds)

 

Doc count         806791

Doc size           1,168,948K

Create time       23,928 

Parse Features  197,012

Annotation time 146,348

Store time         43,028

Index time         0

 

 

We appear to have 1.36 million ‘new instances’ with new-tem-triples being 1,751,486 KB and 14 MB of trusted entities in the cache.  I don’t know what that means, but it certainly sounds impressive.

 

Finally, the Index and store files are around 45GB.

 

Now to get Tomcat 5.5 working so I can SEE what was in there!

 

 

Iain

---------------

Iain Downs (Microsoft MVP)

Commercial Software Therapist

E:  [EMAIL PROTECTED]     T:+44 (0) 1423 872988

W: www.idcl.co.uk

http://mvp.support.microsoft.com

 

_______________________________________________
NOTE: Please REPLY TO ALL to ensure that your reply reaches all members of this 
mailing list.

KIM-discussion mailing list
[email protected]
http://ontotext.com/mailman/listinfo/kim-discussion_ontotext.com

Reply via email to