Hi Andi,I think I've found the root of the problem and have attached a patch which, at least, gets rid of the segfault when running PyLucene in my tests. Read below for more info on how I got this far. ;-)
gcc_static_gcj_fix.patch
Description: Binary data
On Apr 5, 2006, at 11:45 AM, Andi Vajda wrote:
On Wed, 5 Apr 2006, Kevin Ollivier wrote:BTW, do you know any way to get the size of libgcj down? My build is weighing in at 41MB, while RedHat's gcc3.3 version weighs in at < 9MB. I checked configure options for optimization flags, etc. but couldn't find anything, so I was wondering if you knew a way around this. The size is acceptable so long as the thing works :-), but I'd much prefer to keep the size down especially since limits on web server space can sometimes be pretty tight.Well, hmmm, it's not getting any better. libgcj, the gcj java runtime, is based on the GNU classpath java runtime environment clean room implementation. This is work in progress. As Sun Java keeps getting bigger and as the GNU classpath project keeps getting better and more up to date, the library is not getting smaller but bigger. With gcj 4.1.0, libgcj.so.7 on my gentoo linux system is around 45 Mb. (and 68 Mb on PPC Mac OS X).Java Lucene, and hence PyLucene, uses very little of java beyond java.lang, java.util and java.io. In particular it doesn't use *any* of java.awt, javax.swing, which are big indeed....I've been hoping to be able to do static builds of PyLucene, that is, statically linking PyLucene and libgcj.a into single shared library. This is sort of tricky because of Java's dynamic nature (dynamic class comes to mind). The size gains are considerable, though. On Windows, where this is the only way I've been able to build PyLucene, the resulting _PyLucene.pyd DLL is only 4.6 Mb in size. Sadly, I've not yet been able to get a similar static build to be functional on either Linux or Mac OS X. At some point, I intend to look into this again. In the meantime any help with this is welcome.
When I tried it myself, I got similar results - specifically a segfault on startup. Checking gdb, it does seem to deal with dynamic loading, as it's crashing in natClassLoader.cc:507 (it shows a sys- >loadClass(sname, false) line here). By walking up the call stack, I tracked things down to a call in gnu.gcj.convert.UnicodeToBytes (which is in <gcc_root>/libjava/gnu/gcj/convert/UnicodeToBytes.java, line 49) that does a Class.forName using a dynamically generated encoding class name. All the possible encodings are in that same folder, with filenames of Output_*.java.
This got me to searching the gcc sources for "forName" references. In one file, <gcc_root>/libjava/gnu/gcj/runtime/FirstThread.java, I found this at the bottom of the file:
""" // If the user links statically then we need to ensure that these // classes are linked in. Otherwise bootstrapping fails. Theses // classes are only referred to via Class.forName(), so we add an // explicit mention of them here. static final Class Kcert = java.security.cert.Certificate.class; <... more similar definitions...> """So, I tried this same approach with <gcc_root>/libjava/gnu/gcj/ convert/UnicodeToBytes.java (see the patch), and it appears to have worked! :-) I haven't given it a thorough workout, but I did run <pylucene_root>/tests/test_pylucene.py, which gave no errors, and my "minimal" sample Python script that imports PyLucene did run fine on my web server. I'm not sure this would cover all run-time cases, though... Is there an easy way to find out what Java classes are used by PyLucene (e.g. a spitout equivalent to python -v), so we can match them against forName calls and only worry about patching the bits that affect us? (In all the gcc sources, I found 74 instances, some of which were in comments and the like, of course.)
BTW, would it be possible to put these definitions somewhere on the PyLucene end of things, so that gcc doesn't need to be patched and built from source to get this to work? This way, too, if we come across more symbols we can just add them to the file and rebuild PyLucene, which would save a lot of time.
In any case, after stripping the library, I was able to get _PyLucene.so down to ~6.2MB. So I think I'm going to keep working using this approach until I run into a problem that would be very difficult to resolve.
BTW, the Lucene sources only showed two .forName calls, and both were hardcoded calls that looked to be for classes within Lucene, and one of them IIRC was FSDirectory so I don't think there's much to worry about at this end.
Another option is to tweak the gcj build to leave out large portions of libgcj. This may be easier...
Worse comes to worse, I'd rather just take the size hit than really have to muck with gcc's build process. But, even better would be to get this fixed in a way that requires no mods, or minimal mods, to the gcc build process. :-)
Thanks, Kevin
Andi..
_______________________________________________ pylucene-dev mailing list [email protected] http://lists.osafoundation.org/mailman/listinfo/pylucene-dev
