On Tue, Dec 29, 2015 at 04:37:16PM +0100, Ludovic Courtès wrote: > The ‘makeLibrary’, ‘compileC’, and other functions used as illustrations > in the paper would be quite easy to write using our current APIs.
I am thinking about this and maybe I'll give it a go at some point. A functional approach to buildtools incorporating Hashes makes a lot of sense. Guile can do composition, so that ticks a box. Guile has futures, that is another box ticked. Rather than calculating a dependency graph explitely I would like to use futures to have the Guile compiler implicitly create a graph which gets executed in parallel. I am not sure Guile can handle that type of parallelism at a large scale, even though I find futures in the docs https://www.gnu.org/software/guile/manual/html_node/Futures.html Eelco's Maak paper, interestingly, also mentions the 1 second limitation of make as a potential source of race conditions. I would like to have the option of using either time stamps or hash values to decide whether a compilation step has completed succesfully. SHA values arguably take longer to compute (actually pfff is a fast alternative for large data https://github.com/pfff/pfff), but I want Hash values for the next thing: I would like to have cluster compilation where jobs are put out in an opportunistic fashion. I.e., we assume semi-unlimited resources and place jobs multiple times. The one job that completes first is the one we take (the others may get killed). This would allow for longer running jobs to be put out to a heterogeneous environment (think cluster w. NFS, think cloud, think grid, think phone). For build environents this could already be of interest, for bioinformatics large data compute it could be brilliant. It may look wasteful, but truth is most compute environments are happily idle and have differing bottlenecks which would help speed up compilation (fastest one wins). Anyone interested in this project? You know, there are quite a few workflow initiatives (recent one the CWL) and build tool initiatives (CMake and sbt, for example, are fairly recent), but I believe they miss out on real software engineering solutions! Maybe we can get it right. My question, is any really talented programmer here interested to work on this and write a prototype? If you are a student we can even try and get some of the work funded.
