Re: 'withfile' function implementation
First, I think there may be a bug. Did you intend that appending empty data would always append a newline, such that the file grows by one newline every time? It's consistent with the new documentation but not with what you wrote above (If the file does exist and it's opened for append then the modtime on the file will NOT be updated.). I think it makes more sense to append a trailing newline only when new data was added, and the attached patch implements this (along with the rest) and documents it. On Tue, Jan 31, 2012 at 1:16 AM, Paul Smith psm...@gnu.org wrote: The problem, as I alluded to before, is that there's no way to get the exact semantics of updating a file without actually updating the file. In order to change the modtime on a file without changing the file you have to use utime()... Yes, this is something you and I discussed a while ago in a different context. What I'm thinking of doesn't involve utime and is illustrated by the attached patch. In the special case where $(file) is appending an empty string to an existing file, it reads one byte and writes it back. This gives timestamp semantics identical to an actual write, because it is an actual write. And it's safely atomic because the file has the same contents whether the write succeeds or fails. A proof-of-concept patch is attached along with a demo makefile. -David Boyce Makefile Description: Binary data func_file-dsb.diff Description: Binary data ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: 'withfile' function implementation
Thanks, this will make a lot of people happy. One nit: as mentioned earlier I think it would be good to document the relationship of $(file ...) with timestamps. Assuming nothing special is done (I haven't looked at the code) then writing a null string will not update the timestamp. I tried to make a case in favor of doing something special but either way it would be good to get the behavior on the record. David Boyce On Sun, Jan 29, 2012 at 3:02 PM, Paul Smith psm...@gnu.org wrote: On Mon, 2012-01-23 at 09:52 -0800, Lawrence Ibarria wrote: I do like this suggestion, feels quite clean! I implemented the write side of this proposal and committed it to CVS, along with regression tests and documentation. The read side is slightly more work but I can do this one too if people want it. I'm not sure there's much advantage to it over $(shell cat file) as this is unlikely to be a performance bottleneck. Although from a portability and implementation alignment point of view it would be useful. -- --- Paul D. Smith psm...@gnu.org Find some GNU make tips at: http://www.gnu.org http://make.mad-scientist.net Please remain calm...I may be mad, but I am a professional. --Mad Scientist ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: 'withfile' function implementation
On Mon, 2012-01-30 at 17:44 -0500, David Boyce wrote: Thanks, this will make a lot of people happy. One nit: as mentioned earlier I think it would be good to document the relationship of $(file ...) with timestamps. Assuming nothing special is done (I haven't looked at the code) then writing a null string will not update the timestamp. I tried to make a case in favor of doing something special but either way it would be good to get the behavior on the record. You're right that this should be documented, I'll add it to the manual. As currently implemented, if the text argument is empty then the file will be opened (either for truncate or append as specified), but it will not be written to. This means the following, at least on POSIX systems: * If the file doesn't exist it will be created and the modtime set appropriately (to now, where now is defined by the filesystem server--note this might be different than now on the local system, for builds on shared partitions!) * If the file does exist and it's opened for truncate () then the filesystem will update the modtime on it (even if it's already empty), again set to now on the filesystem server. * If the file does exist and it's opened for append () then the modtime on the file will NOT be updated. Is there something special you would prefer beyond this? Cheers! ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: 'withfile' function implementation
On Mon, Jan 30, 2012 at 6:19 PM, Paul Smith psm...@gnu.org wrote: Is there something special you would prefer beyond this? Well ... it's more of a thought than an actual request or preference, but I'm suggesting that make might want to take positive steps in your last case (empty text argument, append to existing file) to force an update of the mod time. Reasons: - Something like $(info touch $@$(file $@)) could become an efficient, portable, in-process pattern for touching a file. - Make is inherently timestamp based and clear rules are important. It could create a new class of mysterious, hard-to-reproduce bugs if this function sometimes updates the timestamp and other times not depending on the value of the text string. - As you've noted, the current semantics just fall out from POSIX behavior. To the degree practical I think it's a good design principle that timestamp semantics are specified by make (and thus portable), not deferred to the platform or filesystem. Again this is not a request per se, just something to think about before semantics are set in stone. -David Boyce ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: 'withfile' function implementation
On Mon, 2012-01-30 at 21:33 -0500, David Boyce wrote: On Mon, Jan 30, 2012 at 6:19 PM, Paul Smith psm...@gnu.org wrote: Is there something special you would prefer beyond this? Well ... it's more of a thought than an actual request or preference, but I'm suggesting that make might want to take positive steps in your last case (empty text argument, append to existing file) to force an update of the mod time. Reasons: The problem, as I alluded to before, is that there's no way to get the exact semantics of updating a file without actually updating the file. In order to change the modtime on a file without changing the file you have to use utime(), which sets the file to an explicit time... in short, the time on the current system. But modifying a file causes the timestamp to be set on the file by the system where the file resides. If the filesystem is local they're basically equivalent, but if the filesystem is remote then you can easily get two different timestamps by having the file updated in these two different ways. Maybe this is not something we should worry about. Certainly it's true that touch(1) has the same problem. I did need to make changes to the regression tests, though, to take this potential issue into account. -- --- Paul D. Smith psm...@gnu.org Find some GNU make tips at: http://www.gnu.org http://make.mad-scientist.net Please remain calm...I may be mad, but I am a professional. --Mad Scientist ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
RE: 'withfile' function implementation
On Mon, 2012-01-23 at 09:52 -0800, Lawrence Ibarria wrote: I do like this suggestion, feels quite clean! I implemented the write side of this proposal and committed it to CVS, along with regression tests and documentation. The read side is slightly more work but I can do this one too if people want it. I'm not sure there's much advantage to it over $(shell cat file) as this is unlikely to be a performance bottleneck. Although from a portability and implementation alignment point of view it would be useful. -- --- Paul D. Smith psm...@gnu.org Find some GNU make tips at: http://www.gnu.org http://make.mad-scientist.net Please remain calm...I may be mad, but I am a professional. --Mad Scientist ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
RE: 'withfile' function implementation
I do like this suggestion, feels quite clean! -- Lawrence -Original Message- From: Paul Smith [mailto:psm...@gnu.org] Sent: Monday, January 16, 2012 5:15 AM To: Lawrence Ibarria Cc: bug-make@gnu.org; Tim Murphy (tnmur...@gmail.com) Subject: Re: 'withfile' function implementation On Thu, 2011-12-15 at 15:13 -0800, Lawrence Ibarria wrote: This is a rather simple path that implements a very simplified version of what Tim suggested in his message of Sept 25th (https://lists.gnu.org/archive/html/bug-make/2011-09/msg00044.html ). Paul, what do you think? I’d rather not try to do everything, just focus on one thing. I am not sure how much safety checks play a role inside make. I was looking at this (and https://savannah.gnu.org/bugs/index.php?35147) I'm still not in love with it. I have the following concerns: 1. I still think the a, w mode thing is clunky. 2. Unclear what to do about newlines. The code doesn't add any but getting a newline into a make variable is tricky. 3. The suggested patch is really only about 20% of the work: we also need documentation, regression tests, etc. Although I guess doing this before the interface is locked down makes extra work (but not for me! :-)). Here's an alternative suggestion, modeled more on the shell's file redirection rather than the C runtime's stdio: One new function, named file (?). The function takes one or two arguments. The first argument is a filename, preceded by a token that specifies how the file is to be treated, with optional whitespace between them. The tokens are: '' : open the file with mode w (truncate) and write the second argument to the file, plus a newline. If no second argument is given, nothing is written (the file will be empty). The expansion of the function is the empty string. '' : Open the file with mode a (append) and write the second argument to the file, plus a newline. If no second argument is given, nothing is written (if the file didn't exist it will be created but empty; if the file did exist it will be unchanged). The expansion of the function is the empty string. '' : Open the file with mode r and expand to the full contents of the file. No conversion is done (the resulting text may have embedded newlines). Use $(strip ...) to remove them if desired. So, some examples might be: $(file t.out,this is the first line) $(file t.out,this is the second line) $(file t.out,this is the third line) FILE := $(file t.out) Do people think this would be useable? I thought about defining three functions , , and , rather than file, but this would require a space between the function name and the file name; if you forgot the space (as you can do in the shell) then make would be looking to expand a variable/function named t.out rather than the function with the first argument of t.out. I just wonder if it would create more issues than it's worth to be cute like that. Still, it's an option. -- --- Paul D. Smith psm...@gnu.org Find some GNU make tips at: http://www.gnu.org http://make.mad-scientist.net Please remain calm...I may be mad, but I am a professional. --Mad Scientist ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: 'withfile' function implementation
On Thu, 2011-12-15 at 15:13 -0800, Lawrence Ibarria wrote: This is a rather simple path that implements a very simplified version of what Tim suggested in his message of Sept 25th (https://lists.gnu.org/archive/html/bug-make/2011-09/msg00044.html ). Paul, what do you think? I’d rather not try to do everything, just focus on one thing. I am not sure how much safety checks play a role inside make. I was looking at this (and https://savannah.gnu.org/bugs/index.php?35147) I'm still not in love with it. I have the following concerns: 1. I still think the a, w mode thing is clunky. 2. Unclear what to do about newlines. The code doesn't add any but getting a newline into a make variable is tricky. 3. The suggested patch is really only about 20% of the work: we also need documentation, regression tests, etc. Although I guess doing this before the interface is locked down makes extra work (but not for me! :-)). Here's an alternative suggestion, modeled more on the shell's file redirection rather than the C runtime's stdio: One new function, named file (?). The function takes one or two arguments. The first argument is a filename, preceded by a token that specifies how the file is to be treated, with optional whitespace between them. The tokens are: '' : open the file with mode w (truncate) and write the second argument to the file, plus a newline. If no second argument is given, nothing is written (the file will be empty). The expansion of the function is the empty string. '' : Open the file with mode a (append) and write the second argument to the file, plus a newline. If no second argument is given, nothing is written (if the file didn't exist it will be created but empty; if the file did exist it will be unchanged). The expansion of the function is the empty string. '' : Open the file with mode r and expand to the full contents of the file. No conversion is done (the resulting text may have embedded newlines). Use $(strip ...) to remove them if desired. So, some examples might be: $(file t.out,this is the first line) $(file t.out,this is the second line) $(file t.out,this is the third line) FILE := $(file t.out) Do people think this would be useable? I thought about defining three functions , , and , rather than file, but this would require a space between the function name and the file name; if you forgot the space (as you can do in the shell) then make would be looking to expand a variable/function named t.out rather than the function with the first argument of t.out. I just wonder if it would create more issues than it's worth to be cute like that. Still, it's an option. -- --- Paul D. Smith psm...@gnu.org Find some GNU make tips at: http://www.gnu.org http://make.mad-scientist.net Please remain calm...I may be mad, but I am a professional. --Mad Scientist ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: 'withfile' function implementation
Just a quick comment on this. I looked at the implementation and it seems like a different name would suit this function - e.g. writefile. It basically writes text from it's arguments out to a file. The idea of a function called withfile seems hard to actually implement - I'm not sure how one might even do it. It would be like a special kind of $(eval), I think. The output file would be open while the contents were being evaluated and closed when they finished. Within this evaluation, any calls to $(write filehandle,) would happen to the file. I think this is a pretty hard thing to do and although I love the (to me) elegance of it, it's a bit harder to implement and the function you have provided is more than good enough for the situations that I've needed so far. I think that a withfile would come into it's own when you could put it around a giant makefile: $(withffile fh,log.out, include hugemakefile.mk ) Since you would open and close the file only once rather than repeatedly and this might be a good thing for parse speed. But I suspect that this form could have performance, memory etc implications that might be undesirable because of the need to load and store a complete copy of the makefile in memory and then evaluate it. Perhaps I will get a chance to submit some actual code one day when I can sort out open source contributions with my employer. In the meantime: $(writefile filename,value) $(appendfile filename,value) would probably be quite nice and they don't absolutely demand that one use the C library mode flags (w,w+,a etc). This is one thing that Paul didn't like from the previous suggestions. Regards, Tim On 15 December 2011 23:13, Lawrence Ibarria libar...@nvidia.com wrote: This is a rather simple path that implements a very simplified version of what Tim suggested in his message of Sept 25th (https://lists.gnu.org/archive/html/bug-make/2011-09/msg00044.html ). Paul, what do you think? I’d rather not try to do everything, just focus on one thing. I am not sure how much safety checks play a role inside make. -- Lawrence This email message is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. -- You could help some brave and decent people to have access to uncensored news by making a donation at: http://www.thezimbabwean.co.uk/friends/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: 'withfile' function implementation
On Fri, 2011-12-16 at 12:41 +, Tim Murphy wrote: I looked at the implementation and it seems like a different name would suit this function - e.g. writefile. It basically writes text from it's arguments out to a file. Thanks all; I'll take a look. -- --- Paul D. Smith psm...@gnu.org Find some GNU make tips at: http://www.gnu.org http://make.mad-scientist.net Please remain calm...I may be mad, but I am a professional. --Mad Scientist ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: 'withfile' function implementation
On Fri, Dec 16, 2011 at 7:41 AM, Tim Murphy tnmur...@gmail.com wrote: would probably be quite nice and they don't absolutely demand that one use the C library mode flags (w,w+,a etc). This is one thing that Paul didn't like from the previous suggestions. A bit of a sidetrack, but I don't agree with the logic here. The notion that r, w, and a stand for read, write, and append did not originate with the C stdio library. One could make a case against these mnemonics on grounds of English-centrism but realistically English is the lingua franca and GNU make (patsubst, --keep-going) is already thoroughly Anglicized, for better or worse. And + has a truly international mnemonic value. So: I have no particular attitude about which implementation is chosen, but being skeptical of [rwa+] simply because fopen borrowed from the same dictionary makes no sense to me. People familiar with fopen will enjoy the symmetry, others will still see the common sense in it. -David Boyce ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
