On Mon, May 16, 2016 at 3:24 AM, Nick Wellnhofer <[email protected]> wrote: > On 15/05/2016 20:38, Nick Wellnhofer wrote: >> >> Lucifers, >> >> I'd like to tackle an issue that's been nagging me for quite some time and >> build separate binaries for our test suites. Then the test binaries can be >> skipped when installing a Clownfish project.
This has been a TODO since the start. It was always a kludge to link the C test object files into the production library. If I recall correctly, I ran into problems like: * which clownfish package the tests belonged to * which prefix the test code should use * assumptions about a single source dir * access to internals from the test files Nothing insurmountable, but stuff that was hard to solve on the first iteration. It's been a while and a lot has changed -- maybe some of that is easier now. > I like the general idea that we have a > language-independent system to build static libraries which the host > language's build system can link with. Excellent. It would also be great if we set our own flags for compiling the core code and kept them the same for all hosts -- instead of compiling the core code using per-host flags as we do right now. > The main problem I see is that for now, charmonizer.exe needs a lot of > knowledge about the locations of different source files, autogenerated or > not, leading to code like the following that must be repeated for every > Clownfish project: > > https://github.com/apache/lucy-clownfish/blob/615d033e45b6fc768e62370780e322aaa68339ac/runtime/common/charmonizer.main#L276 > > This only gets worse if we build multiple binaries. > > There is an obvious improvement: Use a consistent name for the directory > that contains host-specific C code. Currently, we have "cfext", "ext, "xs", > and "src". I think Marvin already proposed to standardize on "cfext". Exactly -- `cfext` is unusual enough that it shouldn't collide with a standard dir for any host build system. > Regarding the autogenerated C files, maybe we should create only a single > file per parcel named after the host language. So for Perl instead of > > boot.c > callbacks.c > > we have > > cfish_perl.c // Boot and callback code for the > // Clownfish parcel. > testcfish_perl.c // Boot and callback code for the tests. Combining them is great. FWIW in the Python bindings I dumped the boot, callbacks and binding code into a single autogenerated file, named after the parcel (e.g. `python/_clownfish.c`, `python/_lucy.c`). In the Go bindings, all that stuff is in `go/PACKAGE/cfbind.go`, e.g. `go/lucy/cfbind.go`. The tricky bit here is that these files have per-host code, as opposed to the core code which is host-agnostic. So I actually think we should rely on the host to build them rather than the Charmonizer-generated `Makefile`. > For languages that don't need extra code, we can create empty files: > > cfish_c.c // Empty. > testcfish_c.c // Empty. > > This leaves the question of how to find the source files of a parcel. Maybe > we should stick to the rule to have a single source directory for each > parcel? Or even enforce a directory layout involving parcel names? Then > charmonizer.exe only needs the parcel (nick)name and the source directory to > work out the source files for each parcel and host language: > > $source_dir/*.c // Or even core/$parcel_nick/*.c > $host_lang/cfext/*.c // Must somehow handle parcels > autogen/source/${parcel_nick}_parcel.c > autogen/source/${parcel_nick}_${host_lang}.c If we accept that there must be a 1:1 relationship between Clownfish parcels and host extensions, this problem gets a lot easier. :) > The remaining problem is that that the host-specific, non-generated .c files > in "cfext" must also be assigned to the correct parcel. > > Speaking more generally, the .c files can be categorized as static or > generated, and as core or host-specific, resulting in four overall > categories: > > - static core > - static host-specific > - generated core > - generated host-specific I would add these as separate categories: - static test // ideally in `$parceldir/test` next to `$parceldir/core` - static host-specific test // maybe `$host_lang/cftest` (I have a slight preference for `test` for the dir where the core tests file live, as opposed to `tests`.) > This again is multiplied by the number of parcels. Not an issue if there can only be one parcel. :) Then the only problem is to build with or without including the object code for the compiled tests. Marvin Humphrey
