Re: Default output-sync setting (was: Re: [bug #33138] .PARLLELSYNC enhancement with patch)
Let me add my voice as a user. If you are one of the lucky people whose builds consist mostly of 1 line of output per rule then you will rarely have any trouble in a good build but try interpreting error messages from compiler/tool X when they're 10 lines from the file that they refer to and don't include the filename in the error message. Try working it out when it's not your code that's being built. This suggests an optimisation where you buffer 5 lines in memory and open a file if there's more but I wouldn't be surprised if the gain is quite small as I suspect that the average build step does so much IO as to make a buffered temp file for the output seem a relatively tiny overhead. I have certainly spent a lot of time wondering why xyz.cpp doesn't have a line 370. I faced this 5 years ago and we had to come up with our own solution rather than wait to convince GNU mission control that we had a problem - which was the only choice given the great difficulty at that job of trying to get permission to make open source contributions. It has already been a very long time coming. You've got it now, thank goodness, in whatever shape or form. Please turn it on. As for reasons: a) I would be surprised if there weren't other changes in the next release that give some people pause for thought before upgrading - why worry about 1 more? e.g. globbing differences between various recent releases caused more pain to me than this is likely to. b) Demand is usually only going to come from the few anyhow. I think this is one of those features that few people will know to turn on but many will benefit anyhow. The reason is that you will start to be able to write generic make output analysis tools. I don't know if you guys have seen Electric Insight, for example. I don't intend this as a plug but I can't help it - it's just the only non-inhouse example that I can think of but I have seen a number of others at various companies. This is a trivial example where you can see that parsing is taking most of the time in this parallel build: https://docs.google.com/file/d/0BwJNUxZZ7qItQlRrUjBmM0tTTDg/edit?usp=sharing It lets you see the state of your parallel build - gives you an intuitive idea of where the time is going and what the errors are and so on. You can't implement tools like this across a general a lot of builds unless they have contiguous recipe output and the surest way to get that is for it to be on by default. You also need recipe separators and timing data but I think you'll realise that eventually now that this step is taken. c) Lets benchmark it on something like an android build and you can make a decision after that? Heck, lets try it with various things and be happy it works. Regards, Tim On 29 April 2013 03:53, Eli Zaretskii e...@gnu.org wrote: From: Paul Smith psm...@gnu.org Cc: bug-make@gnu.org Date: Sun, 28 Apr 2013 22:03:39 -0400 Now that we seem to have a workable solution for output synchronization for both POSIX and Windows systems, I wonder if we shouldn't consider enabling it as the default mode when parallel builds are running. I don't think we should do this unless users demand it, which means we should wait for the next release and some time beyond before making the decision. FWIW, I'm running 8-way parallel Make jobs for many months now, and never had any trouble interpreting their output, neither when they succeed, nor when they don't. The synchronized output operation annoys me slightly in that there are distinct time intervals when Make says nothing at all, and then the output appears in large chunks that get dumped en-masse to the screen. IOW, the user experience is different and takes time to get used to. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make -- You could help some brave and decent people to have access to uncensored news by making a donation at: http://www.thezimbabwean.co.uk/friends/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Default output-sync setting (was: Re: [bug #33138] .PARLLELSYNC enhancement with patch)
Date: Mon, 29 Apr 2013 09:58:50 +0100 From: Tim Murphy tnmur...@gmail.com Cc: bug-make@gnu.org bug-make@gnu.org try interpreting error messages from compiler/tool X when they're 10 lines from the file that they refer to and don't include the filename in the error message. That's unrelated: interpreting such output will be hard no matter what order it comes in. b) Demand is usually only going to come from the few anyhow. I think this is one of those features that few people will know to turn on but many will benefit anyhow. The reason is that you will start to be able to write generic make output analysis tools. People who write programs that parse Make output will find and use this switch anyway; we could announce that prominently in NEWS if you think the feature could go unnoticed. But that is irrelevant to interactive human users and their use cases. c) Lets benchmark it on something like an android build and you can make a decision after that? Heck, lets try it with various things and be happy it works. I already tried it, and I'm not happy, FWIW. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Default output-sync setting (was: Re: [bug #33138] .PARLLELSYNC enhancement with patch)
On 29 April 2013 16:19, Eli Zaretskii e...@gnu.org wrote: Date: Mon, 29 Apr 2013 09:58:50 +0100 From: Tim Murphy tnmur...@gmail.com Cc: bug-make@gnu.org bug-make@gnu.org try interpreting error messages from compiler/tool X when they're 10 lines from the file that they refer to and don't include the filename in the error message. That's unrelated: interpreting such output will be hard no matter what order it comes in. cc fred.c -c -o fred.o cc bob.c -c -o bob.o error on line 20 -X Which one? It gets worse when you scale up. Regards, Tim http://www.thezimbabwean.co.uk/friends/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Default output-sync setting (was: Re: [bug #33138] .PARLLELSYNC enhancement with patch)
Date: Mon, 29 Apr 2013 16:40:03 +0100 From: Tim Murphy tnmur...@gmail.com Cc: bug-make@gnu.org bug-make@gnu.org cc fred.c -c -o fred.o cc bob.c -c -o bob.o error on line 20 -X Which one? Make will actually tell you which one, something like: makefile:342: recipe for target 'oo/i386/acl-errno-valid.o' failed make: *** [oo/i386/acl-errno-valid.o] Error 1 (Copied from a failed make -j8 run I just did for Emacs.) ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Default output-sync setting (was: Re: [bug #33138] .PARLLELSYNC enhancement with patch)
On Mon, Apr 29, 2013 at 10:30 AM, Tim Murphy tnmur...@gmail.com wrote: cc fred.c -c -o fred.o cc bob.c -c -o bob.o error on line 20 -X error on line 30 - error on line 330 - makefile:342: recipe for target 'fred.o' failed makefile:350: recipe for target 'bob.o' failed ? Doctor, my hammer has a head so large that I always hit my thumb Throw out that hammer and get a non-broken one The GNU tool working around the brokenness of some non-free software? Some would call that collaboration in retaining your chains. :-/ I haven't had a chance to play with the new functionality in the repository, so I'll merely caution that having major new functionality that will affect the user experience be on by default runs the risk of there being a problem and having the new version be tarred-and-feathered, and the projects that would most benefit from the new functionality instead turning it off. If there are other fixes and/or features in this release that people are waiting for, burning them on this would be unfriendly. Philip Guenther ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Default output-sync setting (was: Re: [bug #33138] .PARLLELSYNC enhancement with patch)
Come now - the broken excuse is an excuse. There's plenty of crap free software out there and some poor bastard trying to build it who can't change the source because the people who own it think it should be make's problem. :-) Cheers, Tim On 29 April 2013 19:00, Philip Guenther guent...@gmail.com wrote: On Mon, Apr 29, 2013 at 10:30 AM, Tim Murphy tnmur...@gmail.com wrote: cc fred.c -c -o fred.o cc bob.c -c -o bob.o error on line 20 -X error on line 30 - error on line 330 - makefile:342: recipe for target 'fred.o' failed makefile:350: recipe for target 'bob.o' failed ? Doctor, my hammer has a head so large that I always hit my thumb Throw out that hammer and get a non-broken one The GNU tool working around the brokenness of some non-free software? Some would call that collaboration in retaining your chains. :-/ I haven't had a chance to play with the new functionality in the repository, so I'll merely caution that having major new functionality that will affect the user experience be on by default runs the risk of there being a problem and having the new version be tarred-and-feathered, and the projects that would most benefit from the new functionality instead turning it off. If there are other fixes and/or features in this release that people are waiting for, burning them on this would be unfriendly. Philip Guenther -- You could help some brave and decent people to have access to uncensored news by making a donation at: http://www.thezimbabwean.co.uk/friends/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Default output-sync setting (was: Re: [bug #33138] .PARLLELSYNC enhancement with patch)
Date: Mon, 29 Apr 2013 18:30:37 +0100 From: Tim Murphy tnmur...@gmail.com Cc: bug-make@gnu.org bug-make@gnu.org cc fred.c -c -o fred.o cc bob.c -c -o bob.o error on line 20 -X error on line 30 - error on line 330 - makefile:342: recipe for target 'fred.o' failed makefile:350: recipe for target 'bob.o' failed ? You need to look in both anyway. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Default output-sync setting (was: Re: [bug #33138] .PARLLELSYNC enhancement with patch)
Date: Mon, 29 Apr 2013 19:33:10 +0100 From: Tim Murphy tnmur...@gmail.com Cc: Eli Zaretskii e...@gnu.org, bug-make@gnu.org bug-make@gnu.org Come now - the broken excuse is an excuse. There's plenty of crap free software out there and some poor bastard trying to build it who can't change the source because the people who own it think it should be make's problem. Look, no one is claiming that this feature is useless. I just spent two days researching and implementing it on Windows, something I wouldn't do if I thought it wasn't important to have. What we are talking here is whether to have this turned on by default. I hope you are not going to claim that adding -O to a Make command line to get the new behavior is too hard. By contrast, having to type -Onone to have the _old_ behavior back might very well annoy a few. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Default output-sync setting (was: Re: [bug #33138] .PARLLELSYNC enhancement with patch)
One doesn't have to suffer the problems and learn the option exists afterwards. In the end I can understand why a new feature might not be default to start with - until a lot of people have used it and are sure that it works everywhere. Cheers, Tim On 29 April 2013 20:21, Eli Zaretskii e...@gnu.org wrote: Date: Mon, 29 Apr 2013 19:33:10 +0100 From: Tim Murphy tnmur...@gmail.com Cc: Eli Zaretskii e...@gnu.org, bug-make@gnu.org bug-make@gnu.org Come now - the broken excuse is an excuse. There's plenty of crap free software out there and some poor bastard trying to build it who can't change the source because the people who own it think it should be make's problem. Look, no one is claiming that this feature is useless. I just spent two days researching and implementing it on Windows, something I wouldn't do if I thought it wasn't important to have. What we are talking here is whether to have this turned on by default. I hope you are not going to claim that adding -O to a Make command line to get the new behavior is too hard. By contrast, having to type -Onone to have the _old_ behavior back might very well annoy a few. -- You could help some brave and decent people to have access to uncensored news by making a donation at: http://www.thezimbabwean.co.uk/friends/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Default output-sync setting (was: Re: [bug #33138] .PARLLELSYNC enhancement with patch)
Eli: cc fred.c -c -o fred.o cc bob.c -c -o bob.o error on line 20 -X error on line 30 - error on line 330 - makefile:342: recipe for target 'fred.o' failed makefile:350: recipe for target 'bob.o' failed You need to look in both anyway. That is true of the very specific example Tim had given, in response to a particular objection which his example did genuinely address. Think for a moment longer and you shall see that there are other cases where your answer won't apply. The interleaved output from make -j n is often much harder to read. (When the output isn't even line-interleaved, but has some command's output starting part way through a line of the output of another command, life gets even harder.) Even in the specific example: on which file should I be looking at line 20, at line 30, or line 330 ? The answer look in every failing file for every failing line number scales quadratically, which is not good. Even when I know which line a message relates to, it can be hard to work out how that message relates to that line (especially if there's a preprocessor involved and the undefined symbol xxyyz relates to a line in which xxyyz never appears, but only arises after expansion of a mess of macros). It gets nasty fast to have to look in several files, at line 30 of each, and work out which of them is the one - relates to; and it gets easy to end up mistaking which one it relates to and fixing the wrong one. In a make with -j 12ish compiling many files, the starts and ends of compiles interleave with one another and with the output of running commands. Some of that output is errors, which (if the commands being run are well-behaved - not always true) shall mean the affected files are named by make in the output when they finish. Some output is warnings, that may actually be important, but the command succeeds: there's no marker indicating the end of a file that succeeds, so there are many files that warning might have come from; most likely one of those started recently, but not always easy to tell which. When hundreds of files are compiling - why else use -j 12ish ? - it can get hard to keep track of which are currently active at any given line of output, even when every file has both start and end markers for its output, e.g. if some files take much longer than others. The output from make -j several is hard to make sense of. Your shiny new feature really does sound (I haven't had occasion to try it) like a vast improvement. That *is* a strong case for turning it on by default. Naturally, before doing so, it's important to do ample testing: and one way to do that is to initially release make with it off by default but available, so that lots of folk can try it and give you feed-back. All the same, once it's been well tested, if no serious problems arise (and aren't fixed by your collective ingenuity) this is a good thing to turn on by default. Philip Doctor, my hammer has a head so large that I always hit my thumb Throw out that hammer and get a non-broken one The GNU tool working around the brokenness of some non-free software? Some would call that collaboration in retaining your chains. This problem doesn't arise exclusively with non-free software, so this is an emotive argument, an appeal to bad example. Programmers, generally, are slap-dash about the things they throw together to build the software they're so proud of having (in their own opinions) written so nicely and well. They throw together shell scripts to wrap the compiler with some other commands that they also want to run on the same files, or that they need run before handing off their output to the compiler (e.g. they invented their own way of doing localisation, because it's so much more fun to reinvent the wheel than learn to use gettext): and they're profoundly negligent of error-reporting because it's always worked for me so anyone else inheriting their code is stuck with the results. They write contorted shell-scripts inline in the make-file as the command for a rule, using the [ condition ] action short-hand for if [ condition ]; then action; fi and tack on || true to avoid having make report the command as failed when [ condition ] was false; so the rule never gets reported as failing, even when it does, but that's OK, because the error messaeges tell you what failed. The hideous monstrosities of make files that get generated by helper scripts that work round the deficiencies of make (almost invariably, in fact, they work round the author's poor knowledge of how to use make well - but those who inherit the code are stuck with the results) end up doing perverse things that most on this list would never dream of. I'd love to answer that with so replace the bad scripts and hideously wrapped (and warped) build system with something well written, but the poor bug-fixer who's inherited the code from the oh-so-clever author is stuck with the need to get things fixed today, which doesn't
Default output-sync setting (was: Re: [bug #33138] .PARLLELSYNC enhancement with patch)
Now that we seem to have a workable solution for output synchronization for both POSIX and Windows systems, I wonder if we shouldn't consider enabling it as the default mode when parallel builds are running. I understand that this will be a change that could be visible (beyond the collection of output) due to using a temp file instead of a terminal. Of course people can still use -Onone if they want old behavior. However assuming the new mode works and is reliable, and is not a performance bottleneck, I'm hard-pressed to see why a well-ordered output would not be preferable to just about everyone, and hence shouldn't be the default. Of course it's possible that writing to a file, rather than spewing to stdout, WOULD be noticeably performance impacting at least in some situations/systems. Comments? ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Default output-sync setting (was: Re: [bug #33138] .PARLLELSYNC enhancement with patch)
From: Paul Smith psm...@gnu.org Cc: bug-make@gnu.org Date: Sun, 28 Apr 2013 22:03:39 -0400 Now that we seem to have a workable solution for output synchronization for both POSIX and Windows systems, I wonder if we shouldn't consider enabling it as the default mode when parallel builds are running. I don't think we should do this unless users demand it, which means we should wait for the next release and some time beyond before making the decision. FWIW, I'm running 8-way parallel Make jobs for many months now, and never had any trouble interpreting their output, neither when they succeed, nor when they don't. The synchronized output operation annoys me slightly in that there are distinct time intervals when Make says nothing at all, and then the output appears in large chunks that get dumped en-masse to the screen. IOW, the user experience is different and takes time to get used to. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
