On 7. aug. 2005, at 02.29, [EMAIL PROTECTED] wrote:
Hi all,
I just coded a twisted + pylucene server. After having live query
feeds to the daemon, I notice the python server's memory just goes
up and up.
After it reaches around 1.3GB of memory, I start to get
"GC Warning: Repeated allocation of very large block""
After a minute or two over the 2GB memory mark, I have 4GB memory
rhel 4 with zero swap in use so memory is not the problem, the
server dies with
"Too many heap sections: Increase MAXHINCR or MAX_HEAP_SECTS"
I'm new to python. I tried adding gc.collect() at the end of the
searches and adding "del myvars" at the end as well before the
gc.collect(). Still memory goes up and up with no end in sight. The
code base is very small so not sure where the memory leaks are coming.
Thanks for any help.
Xing Li
Hi Xing Li,
I 'd like to say - "What you'd expect? It's Java!", but that would
just be me, venting my Java frustrations on an unsuspecting by-stander.
There are tips and tricks to make Java use more memory and compiler
options to gcj and so on. I've never been successful with that, maybe
someone else on the list has.
But, that's not the point. My experience is that (Py)Lucene will
continue to use memory into kingdom come. 8GB will just make it crash
a little later.
I have a Indexer/Searcher coded in pure Java running in tomcat. It is
quite a bit faster than my PyLucene implementation but it crashes
with "java.lang.OutOfMemoryError: Java heap space" at least once
every three days. Running under Jetty I can get it to behave for a
couple of days more, but it crashes. I suspect the "singel
threadedness" of my application is what's keeping it alive for such a
long time anyway(!).
I also has the same implementation coded in Apache/mod_python. Apache
is running in pre-fork mode and I reap Apaches children after 125
requests. At that point the memory usage of each child is ~100mb and
it's time to die. This never, ever crashes (knock on wood), but it's
not nearly as fast as the Lucene/tomcat implementation.
Unfortunately I don't know Twisted very well. Twisted is async, not
necessarily forked but never threaded? Anyway, if you can make
Twisted fork off a request and kill it dead afterwards you'd be home
free but I suspect your performance may suffer badly.
I would be very interested in taking a look at your application if
you manage to "pull it off".
best regards
/rune
Happy those, who can remain at Highbury!
Jane Austen (1775-1817)
_______________________________________________
pylucene-dev mailing list
[email protected]
http://lists.osafoundation.org/mailman/listinfo/pylucene-dev
