Hi Mike, You got my use case perfectly well, I’m trying to avoid re-running some time consuming tasks by storing their targets on a distant volume. After checking out my pipeline source code (including the tup file) from a version system, I perform a tup init and an rsync to fetch some of these large targets; and I was wondering if tup could continue from here.
I definitely understand that recognising those pre-existing target would defeat tup core features, so I’ll just stick with some more standard use cases :) (Still just for fun, I’ll try to rsync the generated files and the tup database, I’ll see how it goes) @mike: In all cases, thanks a lot for your work, I got several build systems running under tup after just browsing the documentation a couple of times, that’s really a well thought software. > On 19 May 2021, at 19:28, Mike Shal <[email protected]> wrote: > > On Tue, May 18, 2021 at 7:20 AM Sam Sam <[email protected] > <mailto:[email protected]>> wrote: > HI everyone, > > I'm trying to use tup to model a data-oriented pipeline where several > programs are pipelined to generate large datasets. So far tup looked great, > but I was wondering how could I init a project where some target are already > generated. > > Simply doing a tup init then running tup expectedly fails with tup > complaining that a generated node already exists. > > Is this a use case that is definitely out of scope for tup ? > > Hi Sam, > > I'm not sure I fully understand your workflow. Are you looking for a way of > turning the "generated node already exists" error into a warning message and > allow tup to overwrite the file? I think I had plans to add a flag for that, > but it doesn't exist yet. The workaround for this case is to remove the files > yourself (or start with a fresh checkout of your tree before experimenting > with tup). Since the file is removed this way, tup then needs to re-generate > the output, of course. Adding the flag should be pretty straightforward to > help with cases where it's impractical to remove a large number of files. > > If you're trying to find a way to get tup to recognize a file as an output in > the database *and* avoid having to run the command with tup to generate it, I > think that would be problematic. Without running the command in tup, you > wouldn't allow discovery of all the normal inputs to the command that are > determined from the FUSE or DLL/shared library wrappers. Tup then wouldn't > know when it needs to re-run the command. > > Some background on sticky inputs (those you list in Tupfiles) and normal > inputs (those discovered by wrapping and running the program) are here: > http://gittup.org/tup/ex_dependencies.html > <http://gittup.org/tup/ex_dependencies.html> and > http://gittup.org/tup/ex_generated_header.html > <http://gittup.org/tup/ex_generated_header.html> > > Let me know if I misunderstood your use case. If you can show it with a small > example Tupfile that could help, too. > > Thanks, > -Mike > > -- > -- > tup-users mailing list > email: [email protected] > unsubscribe: [email protected] > options: http://groups.google.com/group/tup-users?hl=en > <http://groups.google.com/group/tup-users?hl=en> > --- > You received this message because you are subscribed to the Google Groups > "tup-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/tup-users/CA%2B6x0LWeNtkSvFJYiKRKp%3D3fA9ru%2BY30aRUrVCeCdfaPSc8mnw%40mail.gmail.com > > <https://groups.google.com/d/msgid/tup-users/CA%2B6x0LWeNtkSvFJYiKRKp%3D3fA9ru%2BY30aRUrVCeCdfaPSc8mnw%40mail.gmail.com?utm_medium=email&utm_source=footer>. -- -- tup-users mailing list email: [email protected] unsubscribe: [email protected] options: http://groups.google.com/group/tup-users?hl=en --- You received this message because you are subscribed to the Google Groups "tup-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/tup-users/2911C5C7-8DA2-4E46-BB37-E0C41695AF4D%40gmail.com.
