Re: possible solution for -Otarget recurse (was: Re: Some serious issues with the new -O option)
Date: Sun, 05 May 2013 20:17:50 +0300 From: Eli Zaretskii e...@gnu.org Cc: bug-make@gnu.org From: Paul Smith psm...@gnu.org Cc: bug-make@gnu.org Date: Sun, 05 May 2013 12:56:48 -0400 On Sun, 2013-05-05 at 19:36 +0300, Eli Zaretskii wrote: However, I wonder what was the reason for splitting the definition of GMK_EXPORT in two, and putting each part in a different file. Well, because the gnumake.h file is intended to be installed into the user's /usr/include or similar, and included in a binary package build of make such as RPM or DEB or whatever, and be included by the user's code, and when it's included there it should NEVER (IIUC) be using the in-make variant. That's true, but it's common to have the same header be included when building the application or library which provides the exported services. However, if you really want it back the way it was please do choose a more unique name than MAIN. Something prefixed with GMK_ at least. OK, will do. Done. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: possible solution for -Otarget recurse (was: Re: Some serious issues with the new -O option)
From: Paul Smith p...@mad-scientist.net Cc: bug-make@gnu.org Date: Sat, 04 May 2013 17:51:05 -0400 Eli, I did some cleanup in job.c to try to make things simpler and reduce duplication. I tried to be careful but it's quite possible I did something to disrupt the Windows version again. It's up to you if you want to fix any problems now or wait until I solve the above two issues and look at it all at the same time. There's nothing to fix, as things still seem to work. Thanks! However, I wonder what was the reason for splitting the definition of GMK_EXPORT in two, and putting each part in a different file. The way they were together before is how programmers are used to see this stuff on Windows; splitting them will not something people will expect. I'm sure you can see this in quite a few packages out there. Here's a random example from GMP's gmp.h, slightly edited for clarity: #define __GMP_DECLSPEC_EXPORT __declspec(__dllexport__) #define __GMP_DECLSPEC_IMPORT __declspec(__dllimport__) #if __GMP_LIBGMP_DLL #if __GMP_WITHIN_GMP /* compiling to go into a DLL libgmp */ #define __GMP_DECLSPEC __GMP_DECLSPEC_EXPORT #else /* compiling to go into an application which will link to a DLL libgmp */ #define __GMP_DECLSPEC __GMP_DECLSPEC_IMPORT #endif #else /* all other cases */ #define __GMP_DECLSPEC #endif If you didn't like the symbol MAIN, then we can use any other symbol, like BUILDING_GMAKE or whatever. But having this split in two is not something I'd recommend. I think it's bad for maintenance, if nothing else. There will be more disruption I think. Looking forward to it ;-) One other thing: I changed the pump function to read from a FD but write to a FILE*, because all our other uses of stdout/stderr use FILE* and it's not wise to mix them. It works fine. While I was in there I noticed the handling of the text/binary mode. I wonder if this is not quite right. It seems to me that since we're going to be writing to stdout/stderr anyway, if we're going to set the mode at all we should be setting the mode on the temporary file to match the mode on stdout/stderr, before we write to it, rather than setting the mode on stdout/stderr to match the temporary file. What do you think? Make never changes the I/O mode of its standard streams. And it shouldn't, since everything Make itself writes or reads is always plain text. Since the standard streams always start in text mode, your suggestion boils down to make input from the temporary file be always in text mode. That wouldn't be right. The issue here is that Make has no idea what mode will be used by its children. We redirect the children's standard streams to a file, but we cannot force them to use this or that mode (nor should we, because that is up to the child program). Since we have no idea, and we must copy to our stdout/stderr everything the child wrote, we must assume the worst. And the worst is that the child did use binary mode, and as result wrote there some bytes that cannot be read in text mode without corrupting child's output. Examples include the ^Z byte, which stops text-mode reads and writes, and lone CR characters that get stripped by text-mode reads to disappear without a trace. So we use binary mode to read from the temporary file, because that's the only way to guarantee that everything the child wrote will be read verbatim. We then need to write that stuff to our output in the same mode, to produce identical output, as if the intermediate temporary file never happened. Does this make sense? If you want a practical example to illustrate this conundrum, just ask. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: possible solution for -Otarget recurse (was: Re: Some serious issues with the new -O option)
On Sun, 2013-05-05 at 19:36 +0300, Eli Zaretskii wrote: However, I wonder what was the reason for splitting the definition of GMK_EXPORT in two, and putting each part in a different file. Well, because the gnumake.h file is intended to be installed into the user's /usr/include or similar, and included in a binary package build of make such as RPM or DEB or whatever, and be included by the user's code, and when it's included there it should NEVER (IIUC) be using the in-make variant. I wanted to separate that in-make variant out so that users would never see it or have the possibility of accidentally using it, so I moved it into our internal headers which are never installed anywhere outside our source tree and would not be included in a GNU make binary package. This is slightly more potential maintenance on our part internally but is much safer for the user which is a tradeoff I'll almost always choose. However, if you really want it back the way it was please do choose a more unique name than MAIN. Something prefixed with GMK_ at least. One other thing: I changed the pump function to read from a FD but write to a FILE*, because all our other uses of stdout/stderr use FILE* and it's not wise to mix them. It works fine. While I was in there I noticed the handling of the text/binary mode. I wonder if this is not quite right. It seems to me that since we're going to be writing to stdout/stderr anyway, if we're going to set the mode at all we should be setting the mode on the temporary file to match the mode on stdout/stderr, before we write to it, rather than setting the mode on stdout/stderr to match the temporary file. What do you think? Make never changes the I/O mode of its standard streams. And it shouldn't, since everything Make itself writes or reads is always plain text. Since the standard streams always start in text mode, your suggestion boils down to make input from the temporary file be always in text mode. Well, I assumed that something that invoked make could set the mode and then make could inherit that mode. I don't know if that's how it works or not in Windows. And of course I doubt anyone does that. I understand your point. I just wonder if this difference might end up being visible to the user in some way. But, we'll leave it as-is. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: possible solution for -Otarget recurse (was: Re: Some serious issues with the new -O option)
From: Paul Smith psm...@gnu.org Cc: bug-make@gnu.org Date: Sun, 05 May 2013 12:56:48 -0400 On Sun, 2013-05-05 at 19:36 +0300, Eli Zaretskii wrote: However, I wonder what was the reason for splitting the definition of GMK_EXPORT in two, and putting each part in a different file. Well, because the gnumake.h file is intended to be installed into the user's /usr/include or similar, and included in a binary package build of make such as RPM or DEB or whatever, and be included by the user's code, and when it's included there it should NEVER (IIUC) be using the in-make variant. That's true, but it's common to have the same header be included when building the application or library which provides the exported services. However, if you really want it back the way it was please do choose a more unique name than MAIN. Something prefixed with GMK_ at least. OK, will do. Make never changes the I/O mode of its standard streams. And it shouldn't, since everything Make itself writes or reads is always plain text. Since the standard streams always start in text mode, your suggestion boils down to make input from the temporary file be always in text mode. Well, I assumed that something that invoked make could set the mode and then make could inherit that mode. I don't know if that's how it works or not in Windows. And of course I doubt anyone does that. The text/binary mode cannot be inherited the way file descriptors are. That mode is entirely private to each application, and the startup code provided by MS (to which MinGW is compatible) unconditionally initializes it to text mode. Each application decides on its own how it should treat its standard streams; most leave them at the default text mode, but some switch to binary. Examples of the latter are ports of 'cat' (cat foo bar should create a file that is identical to 'foo'), carefully done ports of 'tr' and Sed (otherwise you cannot edit in or out CR characters), etc. I understand your point. I just wonder if this difference might end up being visible to the user in some way. If the users will see any difference, it means (barring bugs in Make) the child program used binary I/O where it would use text I/O had its stdout/stderr not been redirected. But this belongs to the broader class of problems with -O, not unlike the colored screen output that was discussed here a few days ago. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: possible solution for -Otarget recurse (was: Re: Some serious issues with the new -O option)
From: Paul Smith psm...@gnu.org Cc: stefano.lattar...@gmail.com, bug-make@gnu.org Date: Fri, 03 May 2013 17:17:44 -0400 -O in no way changes that behavior, all it does is ensure that output from any individual line or target of the recipe will not interfere with any other individual line or target. At the Make's internals level, -O indeed doesn't change that behavior. But, whether we want it or not, Make users perceive the order in which they see the output as the order in which things got executed. So any significant changes in the order we show the results of remaking a target will, or at least might, be interpreted as changes in behavior. While it is OK for a non-default switch to change behavior, the modified behavior must make sense, or users will complain. For the default behavior, it must make a lot of sense. Without -O, output might be messy at times, but there's never any doubt in user's mind what happened: she understands that the jobs whose output appears on the screen at the same time run in parallel. Add -O (in its current implementation), and the user now stares at a puzzle: output appears serialized, but the order of that serialization is hard to make sense of. I'm not even sure I understand completely what confuses me when I look at such output. Perhaps my mind tries to interpret that as a non-parallel run, where a prerequisite target is always remade before the target that's dependent on that prerequisite. Or maybe some other mental mechanism is at work here. All I can say is that the order in which -O shows output confuses the heck out of me, and I need to look at it for a long time, reading and re-reading its portions, before I can convince myself that everything that needed to be remade was indeed remade, and in correct order. I guess that means I will seldom if ever use -O. And if that doesn't bother anyone, then let's just leave it at that. Either I'm an odd one out, or you will hear from others who might explain this better than I could. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: possible solution for -Otarget recurse (was: Re: Some serious issues with the new -O option)
On Fri, 2013-05-03 at 12:55 -0400, Paul Smith wrote: Suppose we do this: if we're about to invoke a line marked recursive and we're in -Otarget mode, then before we run it we'll show the current contents of the temp file (using the normal synchronized output function). I've implemented this feature and it seems to work as expected. I also implemented the fix to the duplicate output being shown in some cases. I have two open issues I want to address before calling this feature done: first, fix make's writing of the command it's going to run (for rules that don't start with @) as that's not working right. Second, fix the enter/leave issue. It turns out that these are currently somewhat bound together so I may have to solve the second one first. Oh, and a renaming as well :-) Eli, I did some cleanup in job.c to try to make things simpler and reduce duplication. I tried to be careful but it's quite possible I did something to disrupt the Windows version again. It's up to you if you want to fix any problems now or wait until I solve the above two issues and look at it all at the same time. There will be more disruption I think. One other thing: I changed the pump function to read from a FD but write to a FILE*, because all our other uses of stdout/stderr use FILE* and it's not wise to mix them. It works fine. While I was in there I noticed the handling of the text/binary mode. I wonder if this is not quite right. It seems to me that since we're going to be writing to stdout/stderr anyway, if we're going to set the mode at all we should be setting the mode on the temporary file to match the mode on stdout/stderr, before we write to it, rather than setting the mode on stdout/stderr to match the temporary file. What do you think? ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
possible solution for -Otarget recurse (was: Re: Some serious issues with the new -O option)
I have a solution for this problem that I think will work well, and will be simple to implement. Suppose we do this: if we're about to invoke a line marked recursive and we're in -Otarget mode, then before we run it we'll show the current contents of the temp file (using the normal synchronized output function). This will ensure that output from lines before the recursive make will be shown before the targets in the recursive make. It's not 100% identical but I can't see any way to do better. Thoughts? On Fri, 2013-05-03 at 16:39 +0300, Eli Zaretskii wrote: then how about if this exemption will only be applied if the recipe has a single command? In this case, if a recipe consisted of only one line then every target in the submake will be output immediately when it's finished, but as soon as you add another line to the recipe, like @echo Build is done!, now all of a sudden you get no output from the entire sub-make until the end. That would be too confusing. If the single-command-in-recursive-invocation is _not_ the use case which -Otarget is optimized for, then what use case is? -Otarget is not really about recursive invocations at all. It's there for the non-recursive targets. It would be nice if it worked well for the recursive targets, too, obviously. Consider every target in the entirety of build, including all submakes and all their targets as well, as one long list of targets (for example the results of a serial build). The fraction of those targets that are invoking sub-makes will be, at most, very small. -Otarget wants to collect the output of each individual target, then show the output for each target as soon as that target finishes. That's what users (should) expect when using this option. In the case of recursive make targets, this presents an unsolveable contradiction: how can you both show the output of EVERY target (including the targets run by the submake) as soon as it completes, and still not show the output of any target (including the targets that invoke a submake) before it's complete? You can't do both! The -Omake option chooses the latter as more important: it will delay the output until the submake is complete. The -Otarget option chooses the former as more important: it wants to behave properly for the large majority of targets which are not invoking a recursive make. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
