Hi, We've been utilising JCC frequently recently during development and often have a need to recompile our JCC-based extensions due to changes in the wrapped code - Doing so incurs a reasonably lengthy re-compilation effort due to the serial nature of the extension building via distutils.
To help with this we currently have a potential patch that would add parallel building support to JCC within Linux (based partially off public domain code for distutils, via monkey patching.) Would this be something that might be of interest to be integrated into the mainline trunk? The following are some quick and dirty statistics for building the jcc pylucene itself (incl. java lucene which accounts for about 30-ish seconds upfront) - The JCC files are split using --files 8, and each build is preceded with a make clean: Serial (unpatched): real 5m1.502s user 5m22.887s sys 0m7.749s Parallel (patched, 4 physical cores, 8 hyperthreads, 8 parallel jobs): real 1m37.382s user 7m16.658s sys 0m8.697s Furthermore, some additional changes were made to the wrapped file generation to make the generated code more ccache friendly (additional deterministic sorting for methods and some usage of an ordered set). With these in place and the CC and CCACHE_COMPILERCHECK environment variables set to "ccache gcc" and "content" respectively, and ensuring ccache is installed, subsequent compilation time is reduced again as follows: Parallel (patched, 4 physical cores, 8 hyperthreads, 8 parallel jobs, ccache enabled): real 0m43.051s user 1m10.392s sys 0m4.547s This was a run in which nothing changed between runs, so a realistic run in which changes occur it'll be a figure between 0m43.051s and 1m37.382s, depending on how drastic the change was. If many changes are expected and you want to keep it more cache friendly then using a higher --files would probably work (to an extent), or ideally use --files separate, although it doesn't currently work for me (need to investigate). Any thoughts appreciated. Cheers, Lee -- Lee Skillen Vulcan Financial Technologies 1st Floor, 47 Malone Road, Belfast, BT9 6RY Office: +44 (0)28 95 817888 Web: www.vulcanft.com
