I am working on fixing that error GC Warning: Out of Memory! Returning NIL!
I have following structure of program:
I am trying to create 10 running threads of LuceneWorkerThread from main thread.
When I set number of threads to 4 or less then it runs without exception. Number of PythonThreads affect to this exception!
Any ideas?
I don't know what the memory requirements are for a given index size. That is a question for the lucene-user or lucene-dev mailing lists. If you reach the libgcj memory limits, there may be some environment variables you can set to change them too. That is a question for the [EMAIL PROTECTED] mailing list. I do not know what they are either, but I've seen them posted and discussed before there.
Here is what I'd do resolve this issue :
1. Try it under Java, if it fails the same way, solve it there first, with
help from the lucene mailing lists. To open up a huge index with Java
Lucene, you actually don't need to write a single line of java code, the
src/demo/org/apache/lucene/demo/SearchFiles.java should be enough, it
takes a command line argument to an index directory.
If there are patches that can be applied to the Java Lucene code base
before it gets compiled into PyLucene, that help with this, I'd be happy
to apply them. 2. If it works fine under Java yet fails under python/PyLucene, investigate
what the libgcj memory limits are, with help from the [EMAIL PROTECTED]
mailing list. 3. If all fails then we have a harder problem that will require some
rethinking and some fixing of the underlying software packages used.By the way, you say that you have a 51 Gb index and a 20 Gb index. What is the size of the biggest single index files in these index directories ? There used to be a bug in libgcj that it couldn't support files bigger than 2 or 4 Gb (I don't remember). I know this bug is fixed in gcj 4.0, a PyLucene user actually verified that. I do not know that this bug has been fixed in the version of gcj we're using, gcj 3.4.3.
Andi.. _______________________________________________ pylucene-dev mailing list [email protected] http://lists.osafoundation.org/mailman/listinfo/pylucene-dev
