Hello, Andi. >> I can open index 20gb or 50gb with IndexSearcher without problem. AV> That's not what you said yesterday.
Im sorry, that I have messed things. >> And when I have decreased number of threads to 2 or 3 threads then >> this error has gone. >> So, the question is: How can PythonThread affect to this? Why the less >> amount of them does not produce exceptions of this kind? AV> There may be overhead involved in having multiple threads against a given AV> index. Have you tried this under Java yet ? No b/c I thought that its a problem of gcj's gc... AV> By the way, you say that you have a 51 Gb index and a 20 Gb index. What is the AV> size of the biggest single index files in these index directories ? There used AV> to be a bug in libgcj that it couldn't support files bigger than 2 or 4 Gb (I AV> don't remember). I know this bug is fixed in gcj 4.0, a PyLucene user actually AV> verified that. I do not know that this bug has been fixed in the version of AV> gcj we're using, gcj 3.4.3. total 51183104 -rw-rw-r-- 1 icerocket icerocket 11938376 Feb 2 12:13 _3z3mu.f1 -rw-rw-r-- 1 icerocket icerocket 11938376 Feb 2 12:13 _3z3mu.f2 -rw-rw-r-- 1 icerocket icerocket 11938376 Feb 2 12:13 _3z3mu.f3 -rw-rw-r-- 1 icerocket icerocket 11938376 Feb 2 12:13 _3z3mu.f4 -rw-rw-r-- 1 icerocket icerocket 11938376 Feb 2 12:13 _3z3mu.f5 -rw-rw-r-- 1 icerocket icerocket 11938376 Feb 2 12:13 _3z3mu.f6 -rw-rw-r-- 1 icerocket icerocket 11938376 Feb 2 12:14 _3z3mu.f7 -rw-rw-r-- 1 icerocket icerocket 11938376 Feb 2 12:14 _3z3mu.f8 -rw-rw-r-- 1 icerocket icerocket 14437452892 Feb 2 09:53 _3z3mu.fdt -rw-rw-r-- 1 icerocket icerocket 95507008 Feb 2 09:56 _3z3mu.fdx -rw-rw-r-- 1 icerocket icerocket 69 Feb 2 06:29 _3z3mu.fnm -rw-rw-r-- 1 icerocket icerocket 4312089378 Feb 2 11:03 _3z3mu.frq -rw-rw-r-- 1 icerocket icerocket 4747749584 Feb 2 12:05 _3z3mu.prx -rw-rw-r-- 1 icerocket icerocket 7632774 Feb 2 12:13 _3z3mu.tii -rw-rw-r-- 1 icerocket icerocket 524378115 Feb 2 12:13 _3z3mu.tis I have resolved 2Gb limit problem much time ago :) AV> Also, I just learned that using an unoptimized index is going to require more AV> memory. How much more is a question for lucene-user as well. Optimizing your AV> index is likely to push you over the 4Gb per file limit in gcj < 4.0 though. AV> Have you tried it ? (backing up your existing index first). My index is created using merge factor of 2. They are almost optimized :) Yura Smolsky. _______________________________________________ pylucene-dev mailing list [email protected] http://lists.osafoundation.org/mailman/listinfo/pylucene-dev
