Re: Make, MAKE_TERMOUT, color escape sequences, TTYs, and PTYs

[Brian Vandenberg]

On the one hand I can see some value in this effort, but it seems simpler
for build maintainers to overtly request colorized output if that's what
they want.  If the tools start generating escape sequences for other than
colorizing text it may not be sufficient, but in general it seems like a
better strategy.

However, if you're going to go down this path ... it would be extremely
helpful if there was a built-in mechanism to automatically send output to
individual logs; adding that functionality as part of this effort seems
like it wouldn't be a terribly difficult thing to do.

I've recently made changes to our build to get colorized output not just in
the terminal but in logs as well, and added an option to get rid of
colorized output if they don't want it.  At the heart of what I'm doing:

* Using .ONESHELL with bash
* At the beginning of every recipe is stuff for logging that dup()s
stdout/stderr so I can print log paths to the terminal for recipes that
fail (using a trap in bash), then either redirects output to logs or tees
to both the logs & terminal.
* Added a check to see if a variable (eg, "NO_COLOR") is set or not to
decide whether to pass flags to clang/gcc to colorize output

For developers like me who are more likely to "cat" the logs this works
wonderfully, with the caveat that greping is a pain because of the escape
sequences in the logs, but that's a hindrance I'm willing to put up with in
exchange for the other conveniences.

-brian


On Wed, Jun 1, 2016 at 7:56 AM, Paul Smith  wrote:

> On Tue, 2016-05-31 at 15:06 -0700, Josh Triplett wrote:
> > However, if programs start observing those variables, that seems
> > highly likely to lead to potential breakage in makefiles, for one key
> > reason: those variables remain in the environment even for programs
> > run with stdout/stderr redirected or piped.
>
> I agree that this is a complication.  My thought here was that these
> variables would only be monitored by a few specific tools where it makes
> sense, not by every tool which might possibly want to know whether
> output was a TTY or not.
>
> For example, I would think that GCC/binutils/clang/cmake/etc. would
> definitely WANT to consult these variables, so that colorized warnings,
> etc. could be emitted.
>
> On the other hand, I would expect that simpler tools like ls, grep, etc.
> that might generate colorized output would NOT want to consult these
> variables, because the likelihood that they are being used in some other
> context where the variables are not accurate (and where the risk of
> doing the wrong thing) is very high.
>
> In general I anticipated that use of these variables would be restricted
> to development toolchain programs where it's highly likely that the
> variable values are correct and not vestigial, and in the rare
> situations where they are not people can unset them in the recipe or use
> explicit tool options that disable colorized/etc. output (which is
> probably safer in those situations where the output is postprocessed
> anyway).
>
> > In order to collect output via a pty, make will need to continuously
> > read data from the pty and write it out to the temporary file.  I was
> > thinking of doing so either by forking an extra process to do the
> > copying when launching a command, or by having the parent process do
> > this for all commands it runs as part of its pselect loop.
>
> This is the main reason why I didn't want to mess with ptys (in addition
> to general portability): not being able to use simple redirection is a
> big complication.
>
> I am not excited about the idea of adding a read/write loop into the
> pselect loop.  I think this will really slow down the build if programs
> make is invoking are forced to wait because their output is hung writing
> to the pipe, until make gets around to servicing input on all the pipes.
>  People are using GNU make on systems with 32, 64, even more CPU cores
> with high -j values.
>
> On the other hand, it's not going to work to have make fork a copy of
> itself just to manage those pipes.  Make processes can get very large
> for today's large, non-recursive builds.  Even with COW fork
> capabilities, each forked instance of make itself will be a significant
> overhead.  I don't like that idea.
>
> On the gripping hand, make has always been a trivial installation
> operation because it's just one executable which is completely
> relocatable.  Having it build a little extra program to do data
> shoveling from the pipe then installing that, having make find it, etc.
> is a big extra complication.
>
> Of course, a simpler solution would just be for make to invoke cat(1) or
> something to handle this instead of providing its own program.  Possibly
> we'd need to allow the makefile author to reset this variable to some
> other tool so the name should be in a make variable.  I don't know if
> the overhead of this will be an issue: will having 64 instances of
> cat(1) running with your 64 instances 

Re: Make, MAKE_TERMOUT, color escape sequences, TTYs, and PTYs

[Tristan Wibberley]

Hi,

I can imagine this will be difficult to make complete, useful and bug free,
there are many issues that will appear.

I'd suggest saving a lot of time in development while still allowing the
feature in its earliest forms and leaving room for issues to be resolved at
will by providing hooks for a .so to replace the start, reap, and output
pipelines.

If you implement this by refactoring then the standard mechanism can be
available for extenders to use in the implementation.

The existing system with variables can be provided as a hello-world example
.so

Regards,
Tristan
On 29 May 2016 16:05, "Paul Smith"  wrote:

> On Sat, 2016-05-28 at 16:09 -0700, Josh Triplett wrote:
> > If make's own stdout/stderr refers to a PTY, make could create PTYs in
> > place of pipes, collect output that way, and synchronize it to its own
> > stdout/stderr as it does now.
>
> Just for clarity: GNU make doesn't use pipes to collect output, it uses
> IO redirection to temporary files.
>
> I don't see any serious issue with this proposal, as long as it's
> appropriately managed for portability.
>
> ___
> Bug-make mailing list
> Bug-make@gnu.org
> https://lists.gnu.org/mailman/listinfo/bug-make
>
___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Re: Make, MAKE_TERMOUT, color escape sequences, TTYs, and PTYs

[Paul Smith]

On Sat, 2016-05-28 at 16:09 -0700, Josh Triplett wrote:
> If make's own stdout/stderr refers to a PTY, make could create PTYs in
> place of pipes, collect output that way, and synchronize it to its own
> stdout/stderr as it does now.

Just for clarity: GNU make doesn't use pipes to collect output, it uses
IO redirection to temporary files.

I don't see any serious issue with this proposal, as long as it's
appropriately managed for portability.

___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

Make, MAKE_TERMOUT, color escape sequences, TTYs, and PTYs

[Josh Triplett]

Thank you very much for the new "Integrating make" chapter of the GNU
Make manual; I really appreciate the documentation for the jobserver.

The new "Terminal Output" section
(https://www.gnu.org/software/make/manual/make.html#Terminal-Output)
documents variables "MAKE_TERMOUT" and "MAKE_TERMERR", which indicate
that programs should assume output to the corresponding file descriptor
goes to a terminal even though isatty returns false on that fd.

However, if programs start observing those variables, that seems highly
likely to lead to potential breakage in makefiles, for one key reason:
those variables remain in the environment even for programs run with
stdout/stderr redirected or piped.

Suppose a program that defaulted to --color=auto started checking
MAKE_TERMOUT, and when set, the autodetection would assume it can safely
print color even if output doesn't go to a TTY.  Then, deep inside a
makefile (or inside a script invoked from that makefile), some command
includes a pipeline based on that program, passing its output through a
filter like sed or awk, or writing it to a file used as part of the
build proces.  The program would then print color escape sequences to
the pipeline or built file, almost certainly not what the caller
expected.

I don't see any way a program checking MAKE_TERMOUT could avoid this
scenario.  If its stdout refers to a pipe, the program has no way of
knowing if that pipe matches the one make provided, or a shell pipeline.

I do have a proposal for an alternative that would address this problem,
and avoid the need for programs to override isatty when autodetecting if
they can use terminal escape sequences.

What if make had an option to collect output via PTYs, rather than
pipes?  This option would only work on systems that support PTYs, but
almost every modern system provides such support (as terminal emulators
typically require it).  (And on a system that doesn't have such support,
users would just lose out on color output and similar.)

If make's own stdout/stderr refers to a PTY, make could create PTYs in
place of pipes, collect output that way, and synchronize it to its own
stdout/stderr as it does now.  The programs run this way could call
isatty as normal, and would find out that their output does indeed refer
to a TTY.  And running a command with its output redirected or piped
would then make its output *not* a TTY, so that case would work
automatically as well.

Does this sound like a plausible solution?

- Josh Triplett

___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make