Re: [bug #61226] A regression prevents generation of missing included dependency files.

[Paul Smith]

On Mon, 2021-10-25 at 17:51 -0400, Dmitry Goncharov via Bug reports and
discussion for GNU make wrote:
> On Monday, October 25, 2021, Alejandro Colomar (man-pages) <
> alx.manpa...@gmail.com> wrote:
> > Why do I do this?  Because, if you remove a file from your tree, an
> > old .d file will require that file and make your build fail,
> > requiring you to clean before making again.
> 
> This has been solved long ago. Check the awesome advanced dependency
> generation article by Paul. 

FYI:

  http://make.mad-scientist.net/papers/advanced-auto-dependency-generation/

The issue you raise (as well as others) is discussed, hopefully
clearly, in this article [*].


[*] 
http://make.mad-scientist.net/papers/advanced-auto-dependency-generation/#depdelete

Re: [bug #61226] A regression prevents generation of missing included dependency files.

[Dmitry Goncharov via Bug reports and discussion for GNU make]

On Monday, October 25, 2021, Alejandro Colomar (man-pages) <
alx.manpa...@gmail.com> wrote:

> However, there's a downside to this:  The payment for this robustness is a
> non-negligible time cost.  The single-process time for a no-op make is
> around 10 s


>
>  10s is about 100 to 1000 times longer than what make is capable of.

Regards, Dmitry

Re: [bug #61226] A regression prevents generation of missing included dependency files.

[Dmitry Goncharov via Bug reports and discussion for GNU make]

On Monday, October 25, 2021, Alejandro Colomar (man-pages) <
alx.manpa...@gmail.com> wrote:
>
>
> Why do I do this?  Because, if you remove a file from your tree, an old .d
> file will require that file and make your build fail, requiring you to
> clean before making again.
>
>

This has been solved long ago. Check the awesome advanced dependency
generation article by Paul.

Regards, Dmitry

Re: [bug #61226] A regression prevents generation of missing included dependency files.

[Alejandro Colomar (man-pages)]

Hi all,

Regarding the generation of .d files, I may be a bit paranoid, but I 
like to regenerate them unconditionally at the beginning of every make 
invocation.  I wouldn't trust on .d from my last build, since they may 
be outdated.


Why do I do this?  Because, if you remove a file from your tree, an old 
.d file will require that file and make your build fail, requiring you 
to clean before making again.


To be more precise, consider the following case:

old tree:

//foo.h
int a;

//foobar.h
#include "foo.h"

//unchanged.h
#include "foobar.h"


new tree:

//bar.h
int a;

//foobar.h
#include "bar.h"

//unchanged.h
#include "foobar.h"


foobar.d (or foobar.h.d, depending on your naming conventions) will 
state that foobar.h depends on foo.h, which doesn't exist anymore, and 
thus the build will fail.


My build system works no matter what, since it always has the latest 
dependencies list.  I don't even remember the last time that I had to do 
a 'make clean'.  I've only done recently at some points, to test the 
Makefile itself, but not because I needed it.


See part of my Makefile implementation below, if you're interested in it.

However, there's a downside to this:  The payment for this robustness is 
a non-negligible time cost.  The single-process time for a no-op make is 
around 10 s (i.e., if I run 2 consecutive makes, the second will take 10 
seconds no matter what).  I can reduce that to 1.5 s with 'make -j', but 
it's still a considerable overhead.  But, considering that a full 
project build is around 10 minutes, I can assume that.


To have an idea of the overhead compared to the project size, here are 
some numbers of the library to which those correspond:


$ sloccount include/ src/
[...]
SLOCDirectory   SLOC-by-Language (Sorted)
14322   src ansic=11812,cpp=2510
9237include ansic=9237
[...]
 generated using David A. Wheeler's 'SLOCCount'.

$ find include/ src/ -type f | wc -l
720

Cheers,

Alex


---

[...]

UNITS_c   := $(sort $(shell find $(SRCDIR_m) -type f | grep '\.c$$'))
UNITS_cxx := $(sort $(shell find $(SRCDIR_m) -type f | grep '\.cxx$$'))
UNITS_h   := $(sort $(shell find $(INCLUDEDIR__m)/ -type f | grep '\.h$$'))
UNITS_hxx := $(sort $(shell find $(INCLUDEDIR__m)/ -type f | grep 
'\.hxx$$'))
UNITS_src := $(patsubst $(SRCDIR_m)/%, $(builddir__m)/%, $(UNITS_c) 
$(UNITS_cxx))


[...]

UNITS_c_d   := $(addsuffix .d,$(filter %.c,$(UNITS_src)))
UNITS_cxx_d := $(addsuffix .d,$(filter %.cxx,$(UNITS_src)))
UNITS_h_d   := $(patsubst $(INCLUDEDIR)/%, $(builddir)/%.d, $(UNITS_h))
UNITS_hxx_d := $(patsubst $(INCLUDEDIR)/%, $(builddir)/%.d, $(UNITS_hxx))
UNITS_d := $(UNITS_c_d) $(UNITS_cxx_d) $(UNITS_h_d) $(UNITS_hxx_d)

[...]

.PHONY: $(UNITS_d)
$(UNITS_h_d):   $(builddir)/%.d: $(INCLUDEDIR)/% Makefile | $$(@D)/.
$(CC)  $(CFLAGS)   -I $(INCLUDEDIR) -M -MT $<.gch -MF $@ $<
$(UNITS_hxx_d): $(builddir)/%.d: $(INCLUDEDIR)/% Makefile | $$(@D)/.
$(CXX) $(CXXFLAGS) -I $(INCLUDEDIR) -M -MT $<.gch -MF $@ $<

$(UNITS_c_d):   $(builddir_)/%.d: $(SRCDIR)/% Makefile | $$(@D)/.
$(CC)  $(CFLAGS)   -I $(INCLUDEDIR) -M -MT $(builddir_)/$*.i -MF $@ $<
$(UNITS_cxx_d): $(builddir_)/%.d: $(SRCDIR)/% Makefile | $$(@D)/.
$(CXX) $(CXXFLAGS) -I $(INCLUDEDIR) -M -MT $(builddir_)/$*.i -MF $@ $<

[...]

include $(UNITS_d)


--
Alejandro Colomar
Linux man-pages comaintainer; https://www.kernel.org/doc/man-pages/
http://www.alejandro-colomar.es/

Re: -V, --verbose, as opposite of -s, --silent, --quiet

[Alejandro Colomar (man-pages)]

Hi Paul!

On 10/25/21 10:32 PM, Paul Smith wrote:

On Mon, 2021-10-25 at 22:28 +0200, Alejandro Colomar (man-pages) wrote:

Since I use '--warn-undefined-variables', and also to help
readability to someone who may not know about this usage and may
wonder what does that $(V) mean, I used the following:

V :=
$(V).SILENT:


Looks good.

Something you might consider is using:

   V ?=

instead of ":=" which would allow someone to run:

   V=1 make

successfully (or put it into the environment in some other way).



Hmmm.  I didn't consider that case.  Thanks!

I recently replaced all of my ?= where I expected that a variable may be 
set in the command line by :=, just because I noticed that I could set 
it anyway in the arguments and it would work equally fine, so := would 
probably simplify make's (and the human parser's) checks, but this case 
seems a valid argument for ?=.


Cheers,

Alex

--
Alejandro Colomar
Linux man-pages comaintainer; https://www.kernel.org/doc/man-pages/
http://www.alejandro-colomar.es/

Re: -V, --verbose, as opposite of -s, --silent, --quiet

[Paul Smith]

On Mon, 2021-10-25 at 22:28 +0200, Alejandro Colomar (man-pages) wrote:
> Since I use '--warn-undefined-variables', and also to help
> readability to someone who may not know about this usage and may
> wonder what does that $(V) mean, I used the following:
> 
> V :=
> $(V).SILENT:

Looks good.

Something you might consider is using:

  V ?=

instead of ":=" which would allow someone to run:

  V=1 make

successfully (or put it into the environment in some other way).

Re: -V, --verbose, as opposite of -s, --silent, --quiet

[Alejandro Colomar (man-pages)]

Hello Paul, Dmitry, and David!

> On Sat, Oct 23, 2021 at 6:06 AM Paul Smith  > wrote:
>
> On Sat, 2021-10-23 at 03:01 +0200, Alejandro Colomar (man-pages) 
wrote:

>  > I'd like a project to use '--silent' by default, to have readable
>  > output, and hide most of the full commands, which would be too
>  > noisy.
>  >
>  > So, ideally, I'd like to have 'MAKEFLAGS += --silent' in the
>  > Makefile.
>
> Actually what you really want is the .SILENT special target.
>
> 
https://www.gnu.org/software/make/manual/html_node/Special-Targets.html#index-_002eSILENT
> 


>
> That leads to this best-practice way to handle verbose output:
>
> http://make.mad-scientist.net/managing-recipe-echoing/
> 
>
>

Hmm!  Yes, I knew about .SILENT, but didn't know about this usage of it. 
 It's interesting and simple enough, so I've started using it.


Since I use '--warn-undefined-variables', and also to help readability 
to someone who may not know about this usage and may wonder what does 
that $(V) mean, I used the following:


V :=
$(V).SILENT:

Thanks!


On 10/23/21 10:28 PM, David Boyce wrote:
BTW we use a slightly different and, I think, slightly improved version 
of what's shown there (see below). Improvements are:


1. Either the verbose or the concise output is shown, vs both in verbose 
mode as the original does.


I thought about it, and I prefer printing both for the following reasons:

The verbose output tends to use many lines for each command, since it 
may easily use 500 characters per command.  Having a short line that 
uses uppercase at the beginning makes it much easier to know where each 
command starts.


Also, if you put that into a log file, then you can easily grep for the 
short line that you know you want to search, and next to it will be the 
long line (something like a dictionary).


And anyway, one short line per command (typically around 5-8 lines) 
won't hurt readability too much.


2. It uses $(info ...) instead of echo which saves a fork/exec per 
recipe (note: this is safe only when used in the first recipe line).


Yup!  I also use $(info ...) since mad scientist showed it to me.  It 
made a huge impact in performance.



3. It provides a default "MAKING $@" message as demonstrated in the example.


Nah, I prefer explicit $(info ...) lines always.  Makes it easier to 
read, and also I prefer showing the command that runs every target.


Cheers,

Alex


4. A special case is used to ensure the clean target is always verbose.

David

$ cat Makefile
.PHONY: all
all: hola

ifeq ($(V),)
   vb = $(info $(or $1,MAKING $(@F)))
   .SILENT:
endif

hola: hola.o
         $(call vb)
         $(CC) -o $@ $(LDFLAGS) $^ $(LDLIBS)

%.o: %.c
         $(call vb,CC $( $(@F))
         $(CC) -c -o $@ $(CPPFLAGS) $(CFLAGS) $<

.PHONY: clean
clean: clean_recipe = $(RM) *.o hola
clean:
         @$(info $(clean_recipe))
         @$(clean_recipe)


$ make clean; make
rm -f *.o hola
CC hola.c -> hola.o
MAKING hola

$ make clean; make V=1
rm -f *.o hola
cc -c -o hola.o   hola.c
cc -o hola  hola.o


--
Alejandro Colomar
Linux man-pages comaintainer; https://www.kernel.org/doc/man-pages/
http://www.alejandro-colomar.es/

Re: [bug #61226] A regression prevents generation of missing included dependency files.

[Dmitry Goncharov via Bug reports and discussion for GNU make]

On Sun, Oct 17, 2021 at 11:17 PM Britton Kerin  wrote:
> I got that but it sounds like the proposal is for it to do one thing
> if the file is a .d and otherwise something else?

The proposal is to see if the recipe is empty.

There is this statement in the manual
"If a rule has no prerequisites or recipe, and the target of the rule
is a nonexistent file, then make imagines this target to have been
updated whenever its rule is run."
It is not clear how this behavior could be implemented without
checking the recipe.

regards, Dmitry

Re: [bug #61226] A regression prevents generation of missing included dependency files.

[Dmitry Goncharov via Bug reports and discussion for GNU make]

On Mon, Oct 25, 2021 at 6:12 AM Edward Welbourne  wrote:

> Surely that would solve your problem, without invasive surgery in GNU
> make, or special-case handling of .d files anywhere but your make file ?

Actually, my makefiles do not suffer from this. i use the technique
described here 
https://github.com/dgoncharov/fun/blob/master/autodeps/include_as_part_of_the_dag,
which avoids including .d files.
makefiles which include .d files suffer.


regards, Dmitry

Re: [bug #61226] A regression prevents generation of missing included dependency files.

[Paul Smith]

On Mon, 2021-10-25 at 10:11 +, Edward Welbourne wrote:
> Dmitry Goncharov (17 October 2021 18:33) wrote:
> > i think, make should not print a warning when a .d file is missing.
> > make should proceed and create the missing file. However, when .d
> > is present, but make cannot include it, then make should print an
> > error and stop.
> ...
> 
> Surely that would solve your problem, without invasive surgery in GNU
> make, or special-case handling of .d files anywhere but your make
> file ?

Two points:

First for Eddy's response: the issue we are thinking about isn't how
one person could rewrite makefiles to work differently: the issue is
that a somewhat serious backward compatibility break has been
introduced that could break thousands of makefiles, and we have to
resolve that before the next release.

Second for Britton's response: as far as I can tell no one is
suggesting that make would check the name of an included file and if it
ended in ".d" make would behave differently in some way than if it were
named ".od" or whatever.  Certainly I hope we all agree that kind of
check is not appropriate.  If you check his patch you'll see the change
Dmitry proposed was different: it didn't check the name of the target,
it checked to see if the recipe of the target was empty.  That is, it
would distinguish between this:

   include foo.xx
   foo.xx:

and this:

   include foo.xx
   foo.xx: ; : nothing

In both cases "foo.xx" is not created but (with Dmitry's change) the
first case would not give an error while the second one would give an
error, because it has a recipe.

The _name_ of the target is irrelevant.

Re: [PATCH] Fix type errors in win32.

[Eli Zaretskii]

> From: U2FsdGVkX1 
> Date: Mon, 25 Oct 2021 10:55:03 +0800
> Cc: U2FsdGVkX1 
> 
> * src/commands.c (fatal_error_signal): DWORD type should be unsigned
> * src/dir.c (print_dir_data_base): Fix format type mismatch
> * src/function.c (windows32_openpipe): Use the WINAPI GetStdHandle function 
> instead
> * src/getopt.c (_getopt_internal): Improve code readability

Thanks.  A couple of comments to some of the hunks you propose:

> --- a/src/dir.c
> +++ b/src/dir.c
> @@ -1119,7 +1119,7 @@ print_dir_data_base (void)
>else if (dir->contents->dirfiles.ht_vec == 0)
>  {
>  #ifdef WINDOWS32
> -  printf (_("# %s (key %s, mtime %I64u): could not be 
> opened.\n"),
> +  printf (_("# %s (key %s, mtime %llu): could not be opened.\n"),

This change is problematic, because the default C runtime used by Make
doesn't support %llu, at least not on Windows versions older than
Windows 10.  MinGW64 links programs by default with an extension
library of its own, which provides ANSI-compatible replacements for
printf and stuff, and those do support %llu, but we don't want to make
the sources compile only with those tools.

Are there any problems with using %I64u there? if so, what problems
did you see?

> -  tmpErr = (HANDLE)_get_osfhandle (errfd);
> +  tmpErr = GetStdHandle (errfd);

This one I don't understand.  _get_osfhandle is the documented way of
obtaining the OS handle for any CRT file descriptor, so what's wrong
with using it here?  By contrast, GetStdHandle isn't even explicitly
documented to serve this purpose, i.e. to obtain a handle from a CRT
file descriptor (since errfd comes from 'fileno').  maybe it can do
that, but why use undocumented functionality when a documented one is
available?

Re: [bug #61226] A regression prevents generation of missing included dependency files.

[Edward Welbourne]

Dmitry Goncharov (17 October 2021 18:33) wrote:
> i think, make should not print a warning when a .d file is missing. make
> should proceed and create the missing file. However, when .d is present, but
> make cannot include it, then make should print an error and stop.

Given that I always generated .d files as side-effects of generating .o
files, a missing .d file always meant the .o was missing, so my make
files didn't need to know what the .o file depended on, aside from its
primary source file, so I just used

include $(wildcard $(OBJECTS:%.o=%.d))

Surely that would solve your problem, without invasive surgery in GNU
make, or special-case handling of .d files anywhere but your make file ?

Eddy.