> I have never done anything with gcj or PyLucene on Solaris. I
> don't have access to a Solaris machine either. There are ways
> to tune the libgcj garbage collector via environment
> variables that I'm not too familiar with. I'm unsure about
> the status of the Solaris port of gcj, it 'may' be incomplete
> or in some state of disrepair. I simply don't know.
>
In general GNU ports to Solaris are quite stable - don't know about GCJ
either, though they explicitly mention Solaris support on
http://gcc.gnu.org/java/faq.html#1_5
> The [EMAIL PROTECTED] mailing list (about all things gcj) may
> have more information about the Solaris port. I remember
> someone asking about it recently.
Having checked the mailing list I mainly found Solaris specific build
problems. The GC memory leak issue is also discussed on this list, but
doesn't seem to be a platform specific issue.
What I found regarding tuning of the libgcj garbage collector via
environment variables mainly relates to debug info. For example you may set
GC_LARGE_ALLOC_WARN_INTERVAL to some value to only get every n-th warning
message of type "Repeated allocation of very large block may lead to poor GC
performance and memory leak." Don't think this is a real fix .-(
For testing purpose I did some "tuning" and set
GC_MAXIMUM_HEAP_SIZE 64000000 (64 MB)
GC_PRINT_STATS 1
GC_LARGE_ALLOC_WARN_INTERVAL 1
Running the PyLucene indexing code again now results in following GC
messages:
GC Warning: Out of Memory! Trying to continue ...
GC Warning: Out of Memory! Trying to continue ...
GC Warning: Out of Memory! Trying to continue ...
GC Warning: Out of Memory! Returning NIL!
Abort
The GC is consuming much heap size - so the process quickly consumes 60 MB
heap space (total memory usage at that time: 131 MB). This happens after
about 20mins indexing - when 28.000 objects have been visited/indexed (the
whole batch job mainly consists of a loop calling writer.addDocument(doc)).
GC stats show this consumption:
--%<--
Initiating full world-stop collection 300 after 14331640 allocd bytes
--> Marking for collection 300 after 14331640 allocd bytes + 386496 wasted
bytes
Collection 300 finished ---> heapsize = 58777600 bytes
World-stopped marking took 150 msecs
Complete collection took 180 msecs
--%<--
However it's unclear how far this info really helps to get around the
initial (memory leak) problem.
The main problem for poor [Py]Lucene users I see is that it's difficult to
find out what's going on "behind the scenes" - and how to influence this. I
understand that PyLucene is mainly a "Python wrapper" for the Java Lucene
code which is GCJ-compiled. So that PyLucene will "inherit" any problems
with memory allocation from the Java Code base. Furthermore it obviously
suffers from any performance problem that the GCJ compilation may include.
I guess the main "magic" of Lucene is in the way the automatic merging of
segments is handled. This code probably maintains large data structures and
may create/dispose quite a number of objects. On the GCJ mailing list I read
that one problem with the libgcj GC is a performance and memory penalty
compared to the Java 'builtin' GC. So the main question now is if the
reported (memory leak) problem in PyLucene is caused by the libgcj-GC (i.e.
is a GCJ bug) or is caused by the Lucene Java Code base...
> For problems with the
> garbage collector you also may want to ask the
> [EMAIL PROTECTED] mailing list.
>
Thanks will do.
Regards
Thomas
_______________________________________________
pylucene-dev mailing list
[email protected]
http://lists.osafoundation.org/mailman/listinfo/pylucene-dev
