Re: [bug #61226] A regression prevents generation of missing included dependency files.

[Alejandro Colomar (man-pages)]

On 10/26/21 3:20 PM, Alejandro Colomar (man-pages) wrote:

Hi Paul and Dmitry,

On 10/26/21 1:16 AM, Paul Smith wrote:

On Mon, 2021-10-25 at 17:51 -0400, Dmitry Goncharov via Bug reports and
discussion for GNU make wrote:

On Monday, October 25, 2021, Alejandro Colomar (man-pages) <
alx.manpa...@gmail.com> wrote:

Why do I do this?  Because, if you remove a file from your tree, an
old .d file will require that file and make your build fail,
requiring you to clean before making again.


This has been solved long ago. Check the awesome advanced dependency
generation article by Paul.


FYI:

   
http://make.mad-scientist.net/papers/advanced-auto-dependency-generation/


The issue you raise (as well as others) is discussed, hopefully
clearly, in this article [*].


[*] 
http://make.mad-scientist.net/papers/advanced-auto-dependency-generation/#depdelete 





Thanks for sharing that great article!

I'll definitely have to have a look at many or all of the articles in 
that website!


I've already applied some of the improvements there, reducing no-op make 
times to 0.7 s in single-process and 0.35 with 'make -j'.  I'll keep 
adding the other improvements too.


In the end, I couldn't use all of the features that the article shows.

Since I do a lot of static analysis to the files, I also have 
*.checkpatch.touch (and similar) targets created in the dependencies 
files, and since those tools can't generate dependencies files as side 
effects, I need an explicit rule for creating them.  Otherwise, if I 
create all of them at the compilation stage, it is very likely that at 
some point I only run static analysis and not compilation, and I'll have 
outdated dependencies files the next time I run static analysis, and I 
won't notice.


Maybe I could overcome this with some complicated workaround, but I 
don't think it's worth it: 0.7 s is already a very acceptable time :-) 
And it would probably be a bit obscure and maybe slower for full builds.


So I'm forced to create dependencies files not as side effects, but as 
real targets:



DEPTARGETS = -MT $<.checkpatch.touch -MT $<.clang-tidy.touch \
 -MT $<.cppcheck.touch -MT $<.cpplint.touch
DEPHTARGETS = $(DEPTARGETS) -MT $<.gch
DEPCTARGETS = $(DEPTARGETS) -MT $(builddir_)/$*.i
$(UNITS_h_d):   $(builddir)/%.d: $(INCLUDEDIR)/% $(mk) | $$(@D)/.
$(CC)  $(CFLAGS)   -I $(INCLUDEDIR) -M -MP $(DEPHTARGETS) -MF $@ $<
$(UNITS_hxx_d): $(builddir)/%.d: $(INCLUDEDIR)/% $(mk) | $$(@D)/.
$(CXX) $(CXXFLAGS) -I $(INCLUDEDIR) -M -MP $(DEPHTARGETS) -MF $@ $<

$(UNITS_c_d):   $(builddir_)/%.d: $(SRCDIR)/% $(mk) | $$(@D)/.
$(CC)  $(CFLAGS)   -I $(INCLUDEDIR) -M -MP $(DEPCTARGETS) -MF $@ $<
$(UNITS_cxx_d): $(builddir_)/%.d: $(SRCDIR)/% $(mk) | $$(@D)/.
$(CXX) $(CXXFLAGS) -I $(INCLUDEDIR) -M -MP $(DEPCTARGETS) -MF $@ $<





Thanks!

Alex



Cheers,

Alex





--
Alejandro Colomar
Linux man-pages comaintainer; https://www.kernel.org/doc/man-pages/
http://www.alejandro-colomar.es/

Re: [bug #61226] A regression prevents generation of missing included dependency files.

[Alejandro Colomar (man-pages)]

Hi Paul and Dmitry,

On 10/26/21 1:16 AM, Paul Smith wrote:

On Mon, 2021-10-25 at 17:51 -0400, Dmitry Goncharov via Bug reports and
discussion for GNU make wrote:

On Monday, October 25, 2021, Alejandro Colomar (man-pages) <
alx.manpa...@gmail.com> wrote:

Why do I do this?  Because, if you remove a file from your tree, an
old .d file will require that file and make your build fail,
requiring you to clean before making again.


This has been solved long ago. Check the awesome advanced dependency
generation article by Paul.


FYI:

   http://make.mad-scientist.net/papers/advanced-auto-dependency-generation/

The issue you raise (as well as others) is discussed, hopefully
clearly, in this article [*].


[*] 
http://make.mad-scientist.net/papers/advanced-auto-dependency-generation/#depdelete



Thanks for sharing that great article!

I'll definitely have to have a look at many or all of the articles in 
that website!


I've already applied some of the improvements there, reducing no-op make 
times to 0.7 s in single-process and 0.35 with 'make -j'.  I'll keep 
adding the other improvements too.


Cheers,

Alex


--
Alejandro Colomar
Linux man-pages comaintainer; https://www.kernel.org/doc/man-pages/
http://www.alejandro-colomar.es/

Re: [bug #61226] A regression prevents generation of missing included dependency files.

[Paul Smith]

On Mon, 2021-10-25 at 17:51 -0400, Dmitry Goncharov via Bug reports and
discussion for GNU make wrote:
> On Monday, October 25, 2021, Alejandro Colomar (man-pages) <
> alx.manpa...@gmail.com> wrote:
> > Why do I do this?  Because, if you remove a file from your tree, an
> > old .d file will require that file and make your build fail,
> > requiring you to clean before making again.
> 
> This has been solved long ago. Check the awesome advanced dependency
> generation article by Paul. 

FYI:

  http://make.mad-scientist.net/papers/advanced-auto-dependency-generation/

The issue you raise (as well as others) is discussed, hopefully
clearly, in this article [*].


[*] 
http://make.mad-scientist.net/papers/advanced-auto-dependency-generation/#depdelete

Re: [bug #61226] A regression prevents generation of missing included dependency files.

[Dmitry Goncharov via Bug reports and discussion for GNU make]

On Monday, October 25, 2021, Alejandro Colomar (man-pages) <
alx.manpa...@gmail.com> wrote:

> However, there's a downside to this:  The payment for this robustness is a
> non-negligible time cost.  The single-process time for a no-op make is
> around 10 s


>
>  10s is about 100 to 1000 times longer than what make is capable of.

Regards, Dmitry

Re: [bug #61226] A regression prevents generation of missing included dependency files.

[Dmitry Goncharov via Bug reports and discussion for GNU make]

On Monday, October 25, 2021, Alejandro Colomar (man-pages) <
alx.manpa...@gmail.com> wrote:
>
>
> Why do I do this?  Because, if you remove a file from your tree, an old .d
> file will require that file and make your build fail, requiring you to
> clean before making again.
>
>

This has been solved long ago. Check the awesome advanced dependency
generation article by Paul.

Regards, Dmitry

Re: [bug #61226] A regression prevents generation of missing included dependency files.

[Alejandro Colomar (man-pages)]

Hi all,

Regarding the generation of .d files, I may be a bit paranoid, but I 
like to regenerate them unconditionally at the beginning of every make 
invocation.  I wouldn't trust on .d from my last build, since they may 
be outdated.


Why do I do this?  Because, if you remove a file from your tree, an old 
.d file will require that file and make your build fail, requiring you 
to clean before making again.


To be more precise, consider the following case:

old tree:

//foo.h
int a;

//foobar.h
#include "foo.h"

//unchanged.h
#include "foobar.h"


new tree:

//bar.h
int a;

//foobar.h
#include "bar.h"

//unchanged.h
#include "foobar.h"


foobar.d (or foobar.h.d, depending on your naming conventions) will 
state that foobar.h depends on foo.h, which doesn't exist anymore, and 
thus the build will fail.


My build system works no matter what, since it always has the latest 
dependencies list.  I don't even remember the last time that I had to do 
a 'make clean'.  I've only done recently at some points, to test the 
Makefile itself, but not because I needed it.


See part of my Makefile implementation below, if you're interested in it.

However, there's a downside to this:  The payment for this robustness is 
a non-negligible time cost.  The single-process time for a no-op make is 
around 10 s (i.e., if I run 2 consecutive makes, the second will take 10 
seconds no matter what).  I can reduce that to 1.5 s with 'make -j', but 
it's still a considerable overhead.  But, considering that a full 
project build is around 10 minutes, I can assume that.


To have an idea of the overhead compared to the project size, here are 
some numbers of the library to which those correspond:


$ sloccount include/ src/
[...]
SLOCDirectory   SLOC-by-Language (Sorted)
14322   src ansic=11812,cpp=2510
9237include ansic=9237
[...]
 generated using David A. Wheeler's 'SLOCCount'.

$ find include/ src/ -type f | wc -l
720

Cheers,

Alex


---

[...]

UNITS_c   := $(sort $(shell find $(SRCDIR_m) -type f | grep '\.c$$'))
UNITS_cxx := $(sort $(shell find $(SRCDIR_m) -type f | grep '\.cxx$$'))
UNITS_h   := $(sort $(shell find $(INCLUDEDIR__m)/ -type f | grep '\.h$$'))
UNITS_hxx := $(sort $(shell find $(INCLUDEDIR__m)/ -type f | grep 
'\.hxx$$'))
UNITS_src := $(patsubst $(SRCDIR_m)/%, $(builddir__m)/%, $(UNITS_c) 
$(UNITS_cxx))


[...]

UNITS_c_d   := $(addsuffix .d,$(filter %.c,$(UNITS_src)))
UNITS_cxx_d := $(addsuffix .d,$(filter %.cxx,$(UNITS_src)))
UNITS_h_d   := $(patsubst $(INCLUDEDIR)/%, $(builddir)/%.d, $(UNITS_h))
UNITS_hxx_d := $(patsubst $(INCLUDEDIR)/%, $(builddir)/%.d, $(UNITS_hxx))
UNITS_d := $(UNITS_c_d) $(UNITS_cxx_d) $(UNITS_h_d) $(UNITS_hxx_d)

[...]

.PHONY: $(UNITS_d)
$(UNITS_h_d):   $(builddir)/%.d: $(INCLUDEDIR)/% Makefile | $$(@D)/.
$(CC)  $(CFLAGS)   -I $(INCLUDEDIR) -M -MT $<.gch -MF $@ $<
$(UNITS_hxx_d): $(builddir)/%.d: $(INCLUDEDIR)/% Makefile | $$(@D)/.
$(CXX) $(CXXFLAGS) -I $(INCLUDEDIR) -M -MT $<.gch -MF $@ $<

$(UNITS_c_d):   $(builddir_)/%.d: $(SRCDIR)/% Makefile | $$(@D)/.
$(CC)  $(CFLAGS)   -I $(INCLUDEDIR) -M -MT $(builddir_)/$*.i -MF $@ $<
$(UNITS_cxx_d): $(builddir_)/%.d: $(SRCDIR)/% Makefile | $$(@D)/.
$(CXX) $(CXXFLAGS) -I $(INCLUDEDIR) -M -MT $(builddir_)/$*.i -MF $@ $<

[...]

include $(UNITS_d)


--
Alejandro Colomar
Linux man-pages comaintainer; https://www.kernel.org/doc/man-pages/
http://www.alejandro-colomar.es/

Re: [bug #61226] A regression prevents generation of missing included dependency files.

[Dmitry Goncharov via Bug reports and discussion for GNU make]

On Sun, Oct 17, 2021 at 11:17 PM Britton Kerin  wrote:
> I got that but it sounds like the proposal is for it to do one thing
> if the file is a .d and otherwise something else?

The proposal is to see if the recipe is empty.

There is this statement in the manual
"If a rule has no prerequisites or recipe, and the target of the rule
is a nonexistent file, then make imagines this target to have been
updated whenever its rule is run."
It is not clear how this behavior could be implemented without
checking the recipe.

regards, Dmitry

Re: [bug #61226] A regression prevents generation of missing included dependency files.

[Dmitry Goncharov via Bug reports and discussion for GNU make]

On Mon, Oct 25, 2021 at 6:12 AM Edward Welbourne  wrote:

> Surely that would solve your problem, without invasive surgery in GNU
> make, or special-case handling of .d files anywhere but your make file ?

Actually, my makefiles do not suffer from this. i use the technique
described here 
https://github.com/dgoncharov/fun/blob/master/autodeps/include_as_part_of_the_dag,
which avoids including .d files.
makefiles which include .d files suffer.


regards, Dmitry

Re: [bug #61226] A regression prevents generation of missing included dependency files.

[Paul Smith]

On Mon, 2021-10-25 at 10:11 +, Edward Welbourne wrote:
> Dmitry Goncharov (17 October 2021 18:33) wrote:
> > i think, make should not print a warning when a .d file is missing.
> > make should proceed and create the missing file. However, when .d
> > is present, but make cannot include it, then make should print an
> > error and stop.
> ...
> 
> Surely that would solve your problem, without invasive surgery in GNU
> make, or special-case handling of .d files anywhere but your make
> file ?

Two points:

First for Eddy's response: the issue we are thinking about isn't how
one person could rewrite makefiles to work differently: the issue is
that a somewhat serious backward compatibility break has been
introduced that could break thousands of makefiles, and we have to
resolve that before the next release.

Second for Britton's response: as far as I can tell no one is
suggesting that make would check the name of an included file and if it
ended in ".d" make would behave differently in some way than if it were
named ".od" or whatever.  Certainly I hope we all agree that kind of
check is not appropriate.  If you check his patch you'll see the change
Dmitry proposed was different: it didn't check the name of the target,
it checked to see if the recipe of the target was empty.  That is, it
would distinguish between this:

   include foo.xx
   foo.xx:

and this:

   include foo.xx
   foo.xx: ; : nothing

In both cases "foo.xx" is not created but (with Dmitry's change) the
first case would not give an error while the second one would give an
error, because it has a recipe.

The _name_ of the target is irrelevant.

Re: [bug #61226] A regression prevents generation of missing included dependency files.

[Edward Welbourne]

Dmitry Goncharov (17 October 2021 18:33) wrote:
> i think, make should not print a warning when a .d file is missing. make
> should proceed and create the missing file. However, when .d is present, but
> make cannot include it, then make should print an error and stop.

Given that I always generated .d files as side-effects of generating .o
files, a missing .d file always meant the .o was missing, so my make
files didn't need to know what the .o file depended on, aside from its
primary source file, so I just used

include $(wildcard $(OBJECTS:%.o=%.d))

Surely that would solve your problem, without invasive surgery in GNU
make, or special-case handling of .d files anywhere but your make file ?

Eddy.

Re: [bug #61226] A regression prevents generation of missing included dependency files.

[Britton Kerin]

On Sun, Oct 17, 2021 at 4:44 PM Dmitry Goncharov
 wrote:
>
> On Sun, Oct 17, 2021 at 3:05 PM Britton Kerin  wrote:
>
> > If I understand right that the idea is a special case for .d files
>
> The question being discussed is what to do when make cannot include a 
> makefile.

I got that but it sounds like the proposal is for it to do one thing
if the file is a .d and otherwise something else?

Britton

Re: [bug #61226] A regression prevents generation of missing included dependency files.

[Dmitry Goncharov via Bug reports and discussion for GNU make]

On Sun, Oct 17, 2021 at 3:05 PM Britton Kerin  wrote:

> If I understand right that the idea is a special case for .d files

The question being discussed is what to do when make cannot include a makefile.

regards, Dmitry

Re: [bug #61226] A regression prevents generation of missing included dependency files.

[Britton Kerin]

On Sun, Oct 17, 2021 at 8:33 AM Dmitry Goncharov
 wrote:
>
> Follow-up Comment #8, bug #61226 (project make):
>
> > > Switching to -include robs the user of a useful message, should there be a
> real issue.
>
> > I'm not sure what this means: in what situation do we lose a useful
> message?
>
> -include robs the user of a not readable or corrupted .d file, even though
> user's intervention is required.
>
> i think, hand written included makefiles require different error handling than
> generated .d files.
>
> A missing hand written makefile is an error. The user has the choice of
> 'include' vs '-include' as you described above. I agree with that logic.
>
> On the other hand, a missing .d file (or any other included makefile generated
> by a rule) is not an error. It is simply a clean build.
>
> i think, make should not print a warning when a .d file is missing. make
> should proceed and create the missing file. However, when .d is present, but
> make cannot include it, then make should print an error and stop.
>
>
> The traditional make behavior, which is still in 4.3 is exactly this.
> i think, we should retain this behavior.
>
>
> in regards to the change in main.c see the following possibilities
>
> 1. Revert the change in main.c.
> 2. Modify the change in main.c to allow for missing .d files, but print an
> error and stop in the other cases.

If I understand right that the idea is a special case for .d files I
dislike this sort of thing in make.  All the various C-centric rules
etc. I tend to just disable even when building C, and this sounds
worse because it's not just a disableable library.

Admittedly I haven't used the generated dependency files in a while
but if some additional semantics are needed to support them well I'd
prefer it be exposed as a general feature rather than hidden.  This
might add a bit of interface  complexity but that's still better
hidden special case behavior IMO.

Britton