large index - slow optimize()

2002-11-22 Thread Otis Gospodnetic
Hello,

I am building an index with a few 1M documents, and every X documents
added to the index I call optimize() on the IndexWriter.
I have noticed that as the index grows this calls takes more and more
time, even though the number of new segments that need to be merged is
the same between every optimize() call.
I suspect this is normal and not a bug, but is there no way around
that?  Do you know which part is the part that takes longer and longer
as the index grows?

Thanks,
Otis


__
Do you Yahoo!?
Yahoo! Mail Plus – Powerful. Affordable. Sign up now.
http://mailplus.yahoo.com

--
To unsubscribe, e-mail:   mailto:[EMAIL PROTECTED]
For additional commands, e-mail: mailto:[EMAIL PROTECTED]




RE: large index - slow optimize()

2002-11-22 Thread Armbrust, Daniel C.
Note - this is not a fact, this is what I think I know about how it works.

My working assumption has been its just a matter of disk speed, since during optimize, 
the entire index is copied into new files, and then at the end, the old one is 
removed.  So the more GB you have to copy, the longer it takes.

This is also the reason that you need double the size of your index available on the 
drive in order to perform an optimize, correct?  Or does this only apply when you are 
merging indexes?


Dan



-Original Message-
From: Otis Gospodnetic [mailto:[EMAIL PROTECTED]] 
Sent: Friday, November 22, 2002 12:52 PM
To: [EMAIL PROTECTED]
Subject: large index - slow optimize()


Hello,

I am building an index with a few 1M documents, and every X documents
added to the index I call optimize() on the IndexWriter.
I have noticed that as the index grows this calls takes more and more
time, even though the number of new segments that need to be merged is
the same between every optimize() call.
I suspect this is normal and not a bug, but is there no way around
that?  Do you know which part is the part that takes longer and longer
as the index grows?

Thanks,
Otis


__
Do you Yahoo!?
Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
http://mailplus.yahoo.com

--
To unsubscribe, e-mail:   mailto:[EMAIL PROTECTED]
For additional commands, e-mail: mailto:[EMAIL PROTECTED]

--
To unsubscribe, e-mail:   mailto:[EMAIL PROTECTED]
For additional commands, e-mail: mailto:[EMAIL PROTECTED]