On Tue, 22 Jan 2008, anurag uniyal wrote:
Here are the results of dumprefs for few failed runs
(calling finalize for filter after every invocation do solves the problem)
Ok, so you still have a workaround. I'm about to go offline for two weeks
and intend to get back to this upon my return.
My observation: It happens at different times but for 1m heap memory, it
happens when MyFilter count (which i keep tracking using __del__) reaches
about 49.
dumpRefs also shows almost same count for LowerCaseFilter class but
MyFilter/PythonTokenStream is not listed there?
Also in first case there is no filter class at all in dumpRefs output?
Another thing that could help - in theory - is calling gc.collect() and
System.gc() periodically.
Andi..
class org.apache.lucene.index.IndexReader$FieldOption 10
class [Ljava.lang.String; 10
class java.lang.String 11
class java.util.Locale 21
class java.lang.Class 114
runCount=602 totalRefs 244
class [Ljava.lang.String; 10
class java.lang.String 11
class java.util.Locale 21
class org.apache.lucene.analysis.LowerCaseFilter 48
class java.lang.Class 114
runCount=307 totalRefs 291
class [Ljava.lang.String; 10
class java.lang.String 11
class java.util.Locale 21
class org.apache.lucene.analysis.LowerCaseFilter 49
class java.lang.Class 114
runCount=497 totalRefs 292
class [Ljava.lang.String; 10
class java.lang.String 11
class java.util.Locale 21
class org.apache.lucene.analysis.LowerCaseFilter 49
class java.lang.Class 114
runCount=557 totalRefs 292
rgds
Anurag
----- Original Message ----
From: Andi Vajda <[EMAIL PROTECTED]>
To: anurag uniyal <[EMAIL PROTECTED]>
Cc: [email protected]
Sent: Tuesday, 22 January, 2008 12:49:01 AM
Subject: Re: [pylucene-dev] finalizing the deadly embrace
On Mon, 21 Jan 2008, anurag uniyal wrote:
This still fails *here* after removing __del__
I did rebuild JCC thats why now custom analyzer/parser works without
finalize.
test_PythonDirectory.py is also passing.
When run with -loop ?
src : trunk rev 381
lucene.VERSION = 2.2.0-613493
Python 2.4.3
Ubuntu 6.06.1 LTS
java 1.5.0_06-b05
Ok, I have such a system available to me, I'm going to give your test a try
there.
Still, could you in the meantime try to find out what is leaking ?
If Java is running out of memory, one way to track down what is causing it
is to look at the global refs PyLucene keeps to Java objects.
First, after each test cycle, print out the size of env._dumpRefs().
env is the return value from initVM()
print len(env._dumpRefs)
if that size is growing, next thing is to get the classes of these objects:
refs = sorted(env._dumpRefs(classes=True).items(),
key=lambda x: x[1], reverse=True)
print refs[0:4]
This would print out the classes of the 5 most important Java object leaks
via global refs and their counts. (the code above is used in
test/test_Sort.py, for example).
Andi..
Save all your chat conversations. Find them online at
http://in..messenger.yahoo.com/webmessengerpromo.php
_______________________________________________
pylucene-dev mailing list
[email protected]
http://lists.osafoundation.org/mailman/listinfo/pylucene-dev
