Re: Stress/scalability testing Lucene

[Karl �ie]

I have a index that is compiled each night that indexes 1,3gb with XML data that results into a 1,4gb index. The index takes about 11 hours to build on a dual 700mhz xeon processor with 768mb of ram. The index contains 4.388.730 documents and 953.632 terms.

Mvh karl �ie


On Thursday, Nov 21, 2002, at 15:04 Europe/Oslo, Ian Lea wrote:

Otis


I could do some tests on a solaris and/or intel linux server
that have got about 10 to 15Gb disk space to play with.
Neither has huge amounts of RAM (512 and 256Mb respectively)
but that is probably enough.  Neither server is doing much
else most of the time.

If that would help would you like me to run your test code
to try and get consistent results, or shall I knock up my
own code?



--
Ian.
[EMAIL PROTECTED]

[EMAIL PROTECTED] (Otis Gospodnetic) wrote

Hello,

Has anyone tested Lucene for scalability?
I know that some peple have indices with 10M+ documents in it, but has
anyone tried going beyond there, to 50M, 100M, 500M or more documents?
(I know the size of the index and performance of searches depends on
documents, number of fields, field types, query complexity, etc.)

Last night I wrote a simple class that creates a Lucene index of
specified size with documents containing 2 fields, one Text with about
24 bytes, and one UnStored without about 16000 bytes.
It took about 8 hours to index 100K documents, resulting in an index of
578 MB (optimized). This was on 400MHz machine with about 384MB RAM,
doing nothing else.

I then realized that I can't build a relaly big index to test Lucene's
scalability properly, simply because I don't have a big enough disk :)

So my question is:
Has anyone done this type of testing and can you share the results?
Does anyone have a machine with sufficient amount of RAM and disk and
wants to do this?

Thanks,
Otis
P.S.
If anyone is wondering about those 8 hours - this was with a plain
IndexWriter and mergeFactor set to 1000, and java -Xms50M and -Xmx80MB

----------------------------------------------------------------------
Searchable personal storage and archiving from http://www.digimem.net/

--
To unsubscribe, e-mail: <mailto:[EMAIL PROTECTED]>
For additional commands, e-mail: <mailto:[EMAIL PROTECTED]>

--
To unsubscribe, e-mail:   <mailto:[EMAIL PROTECTED]>
For additional commands, e-mail: <mailto:[EMAIL PROTECTED]>