If using a single shared IndexSearcher instance doesn't help, it may be beneficial to port your code to Java and try it there.
I'm just now getting into PyLucene myself - building a demo for a Unix User's Group presentation I'm giving.
Erik
On Feb 16, 2005, at 3:04 PM, Yura Smolsky wrote:
Hello.
I use PyLucene, python port of Lucene.
I have problem about using big index (50Gb) with IndexSearcher from many threads. I use IndexSearcher from PyLucene's PythonThread. It's really a wrapper around a Java/libgcj thread that python is tricked into thinking it's one of its own.
The core of problem:
When I have many threads (more than 5) I receive this exception:
File "/usr/lib/python2.4/site-packages/PyLucene.py", line 2241, in search
def search(*args): return _PyLucene.Searcher_search(*args)
ValueError: java.lang.OutOfMemoryError
<<No stacktrace available>>
When I decrease number of threads to 3 or even 1 then search works. How do many threads can affect to this exception?..
I have 2 Gb of memory. So with one thread the process takes like 1200-1300Mb.
Andi Vajda suggested that "There may be overhead involved in having multiple threads against a given index."
Does anyone here have experience in handling big indexes with many threads?
Any ideas are appreciated.
Yura Smolsky.
--------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
--------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
