Re: [O] Not overwriting unchanged source code files when tangling
Hi everyone, I wanted to thank everyone for their helpful suggestions and wanted to share the best solutions I heard of and found. One solution is to include a rule in the makefile for every sourcecode file that that copies it and only updates the copy if something has changed (see Nick's email below). Another one is the use of a non-standard make program like makepp, that allows for using md5 checksums for files instead of timestamp in order to derive which files have to be rebuild. Thanks again! Holger On Sat, Nov 19, 2011 at 7:58 AM, Holger Hoefling wrote: > Hey Nick, > > thank you very much. That sounds like a very good solution to my problem > that does not require changes to org-mode. > > Best > > Holger > > On Sat, Nov 19, 2011 at 5:00 AM, Nick Dokos wrote: > >> Holger Hoefling wrote: >> >> > I think you misunderstood me there - I am actually not worried about how >> > computationally intensive the tangling process is. This always works >> very >> > quickly, so even if they have to be copied around and take a bit >> longer, I >> > would not mind. >> > >> >> Ah, ok - so you are talking about >> >> tanglecompile >> org ---> bunch of files -> output >> >> The tangling step produces a bunch of files that are (re)compiled (or in >> any case require some sort of lengthy processing) to produce some output >> file. >> >> >> IMO, the best way to deal with it is still make: let's say >> >> foo.org > a.x b.x c.x --> foo.out >> >> where the first arrow is the tangle and the second arrow is some >> processor, call it X. >> The standard way to set up a makefile is schematically: >> >> --8<---cut here---start->8--- >> foo.out: a.x b.x c.x >> X a.x b.c c.x -o foo.out >> >> a.x b.x c.x: foo.org >> tangle foo.org >> >> --8<---cut here---end--->8--- >> >> >> Rewrite the make file as follows: >> >> >> --8<---cut here---start->8--- >> foo.out: a.y b.y c.y >> X a.y b.y c.y -o foo.out >> >> a.y: a.x >> cmp --silent a.x a.y || cp a.x a.y >> >> b.y: b.x >> cmp --silent b.x b.y || cp b.x b.y >> >> c.y: c.x >> cmp --silent c.x c.y || cp c.x c.y >> >> a.x b.x c.x: foo.org >> tangle foo.org >> --8<---cut here---end--->8--- >> >> >> So if the *contents* of (say) a.x have not changed by the tangling, it >> compares >> equal to a.y and the copy is skipped. That leaves a.y untouched. >> >> OTOH, if the contents of a.x change (or a.y does not exist in the first >> place), the comparison fails and we copy a.x to a.y. That updates a.y >> and forces further updates on anything that depends on it. >> >> Using some make fu (works for GNU make, but not necessarily for other >> makes), >> you can write it more compactly: >> >> >> --8<---cut here---start->8--- >> foo.out: a.y b.y c.y >> X a.y b.y c.y -o foo.out >> >> %.y: %.x >> -cmp --silent $< $@ || cp $< $@ >> >> a.x b.x c.x: foo.org >> tangle foo.org >> --8<---cut here---end--->8--- >> >> HTH, >> Nick >> >> >> >
Re: [O] Not overwriting unchanged source code files when tangling
Hey Nick, thank you very much. That sounds like a very good solution to my problem that does not require changes to org-mode. Best Holger On Sat, Nov 19, 2011 at 5:00 AM, Nick Dokos wrote: > Holger Hoefling wrote: > > > I think you misunderstood me there - I am actually not worried about how > > computationally intensive the tangling process is. This always works very > > quickly, so even if they have to be copied around and take a bit longer, > I > > would not mind. > > > > Ah, ok - so you are talking about > > tanglecompile > org ---> bunch of files -> output > > The tangling step produces a bunch of files that are (re)compiled (or in > any case require some sort of lengthy processing) to produce some output > file. > > > IMO, the best way to deal with it is still make: let's say > > foo.org > a.x b.x c.x --> foo.out > > where the first arrow is the tangle and the second arrow is some > processor, call it X. > The standard way to set up a makefile is schematically: > > --8<---cut here---start->8--- > foo.out: a.x b.x c.x > X a.x b.c c.x -o foo.out > > a.x b.x c.x: foo.org > tangle foo.org > > --8<---cut here---end--->8--- > > > Rewrite the make file as follows: > > > --8<---cut here---start->8--- > foo.out: a.y b.y c.y > X a.y b.y c.y -o foo.out > > a.y: a.x > cmp --silent a.x a.y || cp a.x a.y > > b.y: b.x > cmp --silent b.x b.y || cp b.x b.y > > c.y: c.x > cmp --silent c.x c.y || cp c.x c.y > > a.x b.x c.x: foo.org > tangle foo.org > --8<---cut here---end--->8--- > > > So if the *contents* of (say) a.x have not changed by the tangling, it > compares > equal to a.y and the copy is skipped. That leaves a.y untouched. > > OTOH, if the contents of a.x change (or a.y does not exist in the first > place), the comparison fails and we copy a.x to a.y. That updates a.y > and forces further updates on anything that depends on it. > > Using some make fu (works for GNU make, but not necessarily for other > makes), > you can write it more compactly: > > > --8<---cut here---start->8--- > foo.out: a.y b.y c.y > X a.y b.y c.y -o foo.out > > %.y: %.x > -cmp --silent $< $@ || cp $< $@ > > a.x b.x c.x: foo.org > tangle foo.org > --8<---cut here---end--->8--- > > HTH, > Nick > > >
Re: [O] Not overwriting unchanged source code files when tangling
Hi Nick, I think you misunderstood me there - I am actually not worried about how computationally intensive the tangling process is. This always works very quickly, so even if they have to be copied around and take a bit longer, I would not mind. Thanks Holger On Fri, Nov 18, 2011 at 8:32 PM, Nick Dokos wrote: > Holger Hoefling wrote: > > > Hi Carsten, > > > > thanks for the suggestion, but as I agree with Brian. If there is more > > than one source file in the org-file, then the whole project would > > still be recompiled, not just the updated file. > > > > To be more exact, I actually don't want to compile things, but run R > > scripts using make. So the waiting time if a computationally intensive > > step is repeated although it is not necessary can be substantial. > > > > I wonder how difficult the following change would be (no emacs lisp > experience, also do not know the org source code): > > > > - would it be possible to write out the source files when tangling > > - into a temporary directory, then compare to the actual target files > > - and overwrite only if something has changed? Then the time stamps > > - would stay fixed. Hopefully, this would not involve too much work: > > You've lost right there unless there is a method to select *which* source > blocks to tangle. IOW, the problem is not the *comparison* of the temp and > actual > target files, it is the *production* of the temp files themselves: that's > the computationally expensive step and this method does nothing to > alleviate > that. Unless I'm missing something. > > Nick > > > - creating temporary files and remembering the mapping to true files > > > - tangling out as usual into temporary files (so probably little > > - change there) > > > - compare temporary file to true file (does emacs already have a diff > > - utility that could be used?) > > > - overwrite true file if any changes > > > - delete temporary files > >
Re: [O] Not overwriting unchanged source code files when tangling
Hi Eric, sounds like the problem may after all not be that simple.Could the code blocks be written incrementally to the buffer (or a temporary file on disk) and only after everything has been tangled out all temporary buffers or files checked against the ones on disk? Unfortunately, I do not think that breaking up the org-file into smaller org-files is a solution to my problem. I am running longer analyses (20 script files or more) each doing various things and files in parts depending on each other. If I break up the org-file into so many small parts, a large part of the advantage of using org (i.e. complete analysis in a single file with extensive documentation) will be lost. True, I can still include the subfiles during export - but i hardly ever export code blocks at all (I instead hide the code from export and produce a beamer presentation or latex article using the output from my earlier code). So overall, I would really like to keep everything together in one file. Someone else earlier suggested just tangling out to a temporary directory myself and synching in the makefile. This suggestion is close to what I would like to do and I may use it. However, it is still not quite satisfactory as this brings other problems. Source files sometimes call each other -> when they are in a different directory after tangling than they are during the make file execution (due to synching), then either execution from within org may break, or the execution from the make file. If I knew more about e-lisp, I would just hack up a solution myself. I think org-mode is a really awesome tool. Thanks for your help and suggestions Holger On Fri, Nov 18, 2011 at 8:42 PM, Eric Schulte wrote: > Brian Wightman writes: > > > On Fri, Nov 18, 2011 at 11:02 AM, Carsten Dominik > > wrote: > >> How about changing the make file so that the dependence is on the Org > file, not on the source file? > >> You could then arrange for make to call emacs in batch-mode to tangle > the source file and then compile it? > > > > The original question was trying to avoid recompiling everything > > generated from a tangle if the content didn't actually change. > > Because retangling the source rewrites /all/ of the files, and resets > > the dates, even if nothing has changed, make will then rebuild > > everything that was tangled, not just the partial set of tangled files > > that actually changed. > > > > I think the best approach in this case would be to tangle each file out > to a temporary buffer, and then just before exiting the tangle function > the content of these temporary buffers could be checked against the > files on disk, and only those buffers which differ from disk would be > written. See ob-tangle.el around line 240 for the relevant code. > Unfortunately this would not be trivial, as currently content is written > to the target files incrementally block by block. > > If the code in the .org file is grouped by subtree it may be possible to > place calls to org-narrow-to-subtree in the Makefile before tangling, so > that only part of the file is tangled. > > Finally, it may be easiest simply to play make's game as it were and > break up the Org-mode file into multiple files. These multiple files > could still be combined during export using #+INCLUDE lines from a > single master Org-mode file. > > Best -- Eric > > > > > Brian > > > > -- > Eric Schulte > http://cs.unm.edu/~eschulte/ >
Re: [O] Not overwriting unchanged source code files when tangling
Hi Carsten, thanks for the suggestion, but as I agree with Brian. If there is more than one source file in the org-file, then the whole project would still be recompiled, not just the updated file. To be more exact, I actually don't want to compile things, but run R scripts using make. So the waiting time if a computationally intensive step is repeated although it is not necessary can be substantial. I wonder how difficult the following change would be (no emacs lisp experience, also do not know the org source code): - would it be possible to write out the source files when tangling into a temporary directory, then compare to the actual target files and overwrite only if something has changed? Then the time stamps would stay fixed. Hopefully, this would not involve too much work: - creating temporary files and remembering the mapping to true files - tangling out as usual into temporary files (so probably little change there) - compare temporary file to true file (does emacs already have a diff utility that could be used?) - overwrite true file if any changes - delete temporary files Especially, with this method no dependencies would be necessary and it would not be necessary to keep track in the org files which source blocks have been changed since the last tangling Thank you for your suggestion Holger On Fri, Nov 18, 2011 at 6:02 PM, Carsten Dominik wrote: > > On 18.11.2011, at 14:17, Holger Hoefling wrote: > > > Hi, > > > > I have a problem/request for org-mode and was looking for help. I am > using org-mode to write source code files and tangle them out. I want to > compile them using make. My problem now is that org-mode overwrites the old > files every time I tangle them out, therefore also updating the time stamp > - even if nothing has changed. Subsequently, when I run make, everything > gets recompiled, not just the changed source code files as all time stamps > have changed. > > > > Is there an option for org-mode to only overwrite source code files that > get tangled out if they have truly changed? > > How about changing the make file so that the dependence is on the Org > file, not on the source file? > You could then arrange for make to call emacs in batch-mode to tangle the > source file and then compile it? > > Something along the lines of (untested, and probably wrong in this > way...) > > file.o: somefile.org > emacs -batch --eval '(org-babel-tangle-file "somefile.org")' > cc file.o > > - Carsten
[O] Not overwriting unchanged source code files when tangling
Hi, I have a problem/request for org-mode and was looking for help. I am using org-mode to write source code files and tangle them out. I want to compile them using make. My problem now is that org-mode overwrites the old files every time I tangle them out, therefore also updating the time stamp - even if nothing has changed. Subsequently, when I run make, everything gets recompiled, not just the changed source code files as all time stamps have changed. Is there an option for org-mode to only overwrite source code files that get tangled out if they have truly changed? Thank you very much for your help! Holger
