Hi Lee, > On Jul 14, 2014, at 19:01, Lee Skillen <lskil...@vulcanft.com> wrote: > > Hi, > > We've been utilising JCC frequently recently during development and > often have a need to recompile our JCC-based extensions due to changes > in the wrapped code - Doing so incurs a reasonably lengthy > re-compilation effort due to the serial nature of the extension > building via distutils. > > To help with this we currently have a potential patch that would add > parallel building support to JCC within Linux (based partially off > public domain code for distutils, via monkey patching.) Would this be > something that might be of interest to be integrated into the mainline > trunk? > > The following are some quick and dirty statistics for building the jcc > pylucene itself (incl. java lucene which accounts for about 30-ish > seconds upfront) - The JCC files are split using --files 8, and each > build is preceded with a make clean: > > Serial (unpatched): > > real 5m1.502s > user 5m22.887s > sys 0m7.749s > > Parallel (patched, 4 physical cores, 8 hyperthreads, 8 parallel jobs): > > real 1m37.382s > user 7m16.658s > sys 0m8.697s > > Furthermore, some additional changes were made to the wrapped file > generation to make the generated code more ccache friendly (additional > deterministic sorting for methods and some usage of an ordered set). > With these in place and the CC and CCACHE_COMPILERCHECK environment > variables set to "ccache gcc" and "content" respectively, and ensuring > ccache is installed, subsequent compilation time is reduced again as > follows: > > Parallel (patched, 4 physical cores, 8 hyperthreads, 8 parallel jobs, > ccache enabled): > > real 0m43.051s > user 1m10.392s > sys 0m4.547s > > This was a run in which nothing changed between runs, so a realistic > run in which changes occur it'll be a figure between 0m43.051s and > 1m37.382s, depending on how drastic the change was. If many changes > are expected and you want to keep it more cache friendly then using a > higher --files would probably work (to an extent), or ideally use > --files separate, although it doesn't currently work for me (need to > investigate). > > Any thoughts appreciated.
This is a pretty cool feature ! The scary part is depending on monkey patching distutils... When that breaks, it's a pain to maintain: I've been having to make my workaround for now years-old setuptools issue-43 via monkey patching more complicated as more and more branches and versions of setuptools have appeared over time. Any chance these distutils patches could be merged into distutils upstream first ? If you're using JCC only, there is no need to build PyLucene but I assume you know that already. Building JCC by itself is pretty fast. Andi.. > > Cheers, > Lee > > -- > Lee Skillen > > Vulcan Financial Technologies > 1st Floor, 47 Malone Road, Belfast, BT9 6RY > > Office: +44 (0)28 95 817888 > Web: www.vulcanft.com
