I'm also a bit confused about "abstracting away the build system". Do you have a small representative example that you can share? That might help to ensure we're all talking about the same thing.
The main stages of tup look like this: 1) Scan the filesystem and mark changed files in the .tup/db database (the inotify file monitor on Linux allows tup to skip this step). Note that no rules or dependencies are processed here - this is only about getting the database in sync with the filesystem. So there isn't a way to hook into the found-a-new-file logic to generate a new rule on the fly in this stage. 2) Based on the changes marked in the database, the necessary Tupfiles are parsed. For example, if you change a Tupfile directly, or add a new file in a directory, tup will re-parse that directory. Or if you remove a Tupfile, tup will delete the rules and outputs that were generated by it. In this sense, tup already only parses the changed build logic, and processes only the added/removed rules & dependencies. I'd suggest re-using this logic as much as possible. 3) Now tup can build the partial DAG and execute sub-commands, checking dependencies as it goes. >From your description, it sounds like you might be adding a new step before all this. Something like: 0) Before running tup, scan the file-system and generate Tupfiles. Is that correct? How do you decide what to put in the Tupfile? Are you parsing some sort of generic build description and then outputting rules? If so, I'd suggest trying to avoid this step, and only run your rule-generation script as part of step 2) when tup decides to re-parse a Tupfile. If you don't want to have actual Tupfiles in the subdirectories and third-party dependencies, you might be able to use Tupdefault (or Tupdefault.lua, and some day Tupdefault.py) to act as your main script, which can reside in the top-level. Tup will use this as the default Tupfile for any new directory that gets created, so new rules can be generated on the fly in that sense. For example, here is an extremely dumbed-down set of files to compile and link *.c in an arbitrary directory hierarchy: https://groups.google.com/forum/#!searchin/tup-users/tupdefault/tup-users/sxTax83epLY/rTAjp5Vuyu0J You would obviously have to expand Tupdefault.lua to handle your custom build logic. But in this way, your logic gets triggered on a per-directory basis as directories & files are added and removed. In essence you're still generating a Tupfile, but it goes straight into the database rather than using intermediate files in the filesystem. -Mike On Mon, Apr 25, 2016 at 4:59 PM, Erik Brinkman <[email protected]> wrote: > I'm not entirely sure what you mean by "abstracting away the build system", > but it seems like you don't want to use tup so much as fork it. I imagine > you could modify tups filesystem scan portion, and use that to generate the > rules you want (Note, I haven't looked at this portion of tup at all). > However, I'm not sure how much good it will since for large projects it > seems like the filesystem scan is the largest bottleneck for tup as well, > and since you're not only supporting linux, you can't use the filesystem > monitor. > > On Sun, Apr 24, 2016 at 4:00 PM Cheng Lou <[email protected]> wrote: >> >> Hi! I've only started using tup in the last two days, but the great dev >> experience has convinced me to give it a harder try for our problem. In our >> project we're experimenting with abstracting away the build system >> (generating the tupfile under the hood as an implementation detail). >> >> The end dev might add/remove files from nested directories (so a single >> flat wildcard won't do). The corresponding rules should be added/removed >> from tupfile automatically. >> >> Third party dependencies might also be added/removed (they resize in their >> respective directory, inside a known directory at the current project's top >> level), rules for each dep are recursively generated in the above fashion >> and inserted into the one top level tupfile. >> >> We're currently doing a naive `find` to discover new files inside the top >> level project & new deps, and regenerating the tupfile. The scan becomes the >> bottleneck. Is there a way to hook into tup so that only added/removed top >> project rules/deps rules are generated? We can't use the watcher because >> this needs to work on OS X & Windows. Also, ideally the only command to run >> is "tup", but since we're abstracting away we can compromise here. >> >> Any help would be appreciated. Thanks! >> >> -- >> -- >> tup-users mailing list >> email: [email protected] >> unsubscribe: [email protected] >> options: http://groups.google.com/group/tup-users?hl=en >> --- >> You received this message because you are subscribed to the Google Groups >> "tup-users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/d/optout. > > -- > -- > tup-users mailing list > email: [email protected] > unsubscribe: [email protected] > options: http://groups.google.com/group/tup-users?hl=en > --- > You received this message because you are subscribed to the Google Groups > "tup-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. -- -- tup-users mailing list email: [email protected] unsubscribe: [email protected] options: http://groups.google.com/group/tup-users?hl=en --- You received this message because you are subscribed to the Google Groups "tup-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
