[bug #58734] gmake does not check for the existence of a file before complaining it is missing

[Paul D. Smith]

Follow-up Comment #11, bug #58734 (project make):

I understand the lack of time.

As I mentioned I've implemented and pushed a fix for the directory caching
issue.  When I follow the directions here as best as I could, the build
succeeded and didn't throw any warnings or errors with this change installed.

So, I believe this issue to be resolved.

___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

[bug #58734] gmake does not check for the existence of a file before complaining it is missing

[Jörg Schilling]

Follow-up Comment #10, bug #58734 (project make):

I am sorry, but there are problems in gmake that only show
up with large or complex makefiles. I cannot give simpler
examples as I cannot spend an infinite amount of time in trying
to find a smaller example.

I however already explained to you how you could make gmake
compatible to make in this area. Since even the classic make
did cache file time stamps, it is obvious that things can 
be done in a compatible way.

- You may cache whatever you like as long as you do not
  break things that are part of the basic functionality of
  make since 1976. So do just not make the wrong conclusions
  from cached time stamps.

- If a cached time stamp results in the assumption that
  the related file is up to date, this is OK. This may be
  wrong if the file was removed meanwhile, but I believe
  this is not a problem.

- If a file was the target of a rule command that did exit()
  with code zero, assume "younger than any file" without
  checking the file time stamp again.

- If the cached time results in the assumption that the
  file is out of date or missing, check the time stamp
  again. This is what seems to be missing in gmake.

Just a note: assuming a file is out of date and rerunning the
rule command is more expensive than calling stat() for the file.
The cases where you need to rerun stat() are rare and thus do
not affect the overall performance.

Claiming that the file does not exist (this is what happens
with gmake) is incompatible to the behavior of make. If you
are familiar with the source code of gmake, it should be
obvious how to fix it. Just call stat() if you believe that
a file does not exist.

___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

[bug #58734] gmake does not check for the existence of a file before complaining it is missing

[Paul D. Smith]

Update of bug #58734 (project make):

  Status:None => Duplicate  
 Assigned to:None => psmith 
 Open/Closed:Open => Closed 

___

Follow-up Comment #9:

The reproduction case here is too complicated for me to have the time to
really understand.  If you wanted to provide a simpler repro case I'd be happy
to look at it.  Also, it's helpful if you use LC_ALL=C when running make so
that error messages can be looked up in the source code.

However I ran this repro case using the latest version which has a fix for bug
#41273 and it appeared to work correctly, so I'm closing this as a duplicate.

Thanks for reporting!

___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

[bug #58734] gmake does not check for the existence of a file before complaining it is missing

[Jörg Schilling]

Follow-up Comment #8, bug #58734 (project make):

Sorry for the typo, the command in step 8 of course should be:

  find . -type l -exec rm {} +

___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

[bug #58734] gmake does not check for the existence of a file before complaining it is missing

[Jörg Schilling]

Follow-up Comment #7, bug #58734 (project make):

You are mistaken. My makefile describes enough of the files to allow any
non-buggy make implementation to work correctly.

If gmake does not check the state of a file before it claims that this file is
missing, then gmake is broken.

Steps to reproduce:

1) download the current state of schilytools from:
https://sourceforge.net/projects/schilytools/files/

 The currently up to date file is:
https://sourceforge.net/projects/schilytools/files/schily-2020-07-18.tar.bz2

2) unpack the archive and chdir to "schily-2020-07-18"

3) call "make" and wait for the compilation to finish

4) call "make install" to get smake and dmake installed in /opt/schily/bin

5) add /opt/schily/bin to your PATH

6) chdir to libshedit

7) install the attachec gmake-fail.mk

8) remove symlinks and OBJ by calling:

   find . -type d-exec rm {} +

9) call "smake -f gmake\-fail.mk"

and see:

smake -f gmake-fail.mk 
==> MAKING SYMLINKS in .
...

10) repeat step 8)

11) call "dmake -f gmake-fail.mk"

and see:

dmake -f gmake\-fail.mk   
==> MAKING SYMLINKS in .
...

12) repeat setp 8)

13) call gmake -f gmake-fail.mk GMAKE_NOWARN=true

==> MAKING SYMLINKS in .
gmake: *** Keine Regel vorhanden, um das Ziel »inputc.c«, 
  benötigt von »OBJ/x86_64-linux-gcc/inputc.o«, zu erstellen.  Schluss.

So gmake is the only make implementation that complains here.

(file #49528)
___

Additional Item Attachment:

File name: gmake-fail.mk  Size:1 KB




___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

[bug #58734] gmake does not check for the existence of a file before complaining it is missing

[Paul D. Smith]

Follow-up Comment #6, bug #58734 (project make):

> I forgot to mention, that this happens in parallel and in serial mode. So
your assumption is wrong.

I said, running the command twice can only happen with parallel builds
enabled.  I stand behind that statement as correct.  I was careful to make a
distinction between that and the problem of files not existing.

> I am also not going to add unneeded hidden files when there is already a
file that is created as a result of the command that has been run.

I explained this but just realized that message went to the mailing list, not
to the bug:

If you want parallel builds to work correctly you _must_ "introduce unneeded
files".  This is a deficiency in the POSIX standard (it does not provide any
way to explain to make that a single invocation of a recipe generates multiple
output files) and if you want to write portable makefiles that will also work
properly with parallelism enabled then there's no other way to solve that
problem.

_As a result of_ introducing the hidden files to fix the parallel builds
issue, that _will also_ fix the caching problem.

In short, if you ensure that your makefile works properly with parallel builds
then you won't have a caching problem.  This is why the caching problem is not
a huge issue for every user: virtually every GNU make user these days enables
parallel mode, and they want their makefile to work properly with it.

> It would be so easy just to re-check the time stamp of every cached file
that is believed to be out of date or missing.

The only purpose of the cache is reducing the number of stat() calls make has
to do.  Obviously once make determines that a target is already up to date, it
doesn't have to check it again.  If you say that for all other files make
should always re-check, then there's no reason to cache.

If you're saying something different, such as as long as make has not run a
command that might possibly change some timestamp that it can cache the mod
time ... well ... that's pretty much exactly what I said in my list of
possible enhancements (again this went to the mailing list unfortunately).

> Time stamp checking is also faster than re-running a command for a related
rule.

Of course!  But it's not faster than not checking the timestamp in the first
place, which is what the directory cache is for.  If your makefile describes
the target and prerequisite relationships completely, without omitting any
relevant information, then the directory cache provides performance
improvements.  If you don't, then it can do the wrong thing.

As I've written already, it's clear from experience that some makefiles don't
provide sufficient information and the cache should be modified to address
that.

___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

[bug #58734] gmake does not check for the existence of a file before complaining it is missing

[Jörg Schilling]

Follow-up Comment #5, bug #58734 (project make):

I forgot to mention, that this happens in parallel and in serial mode. So your
assumption is wrong.

Caching is not a bad thing if it helps to speed up things. I did not make
related tests and I am not going to doubt here because this would put effort
on the wrong target. 

I am also not going to add unneeded hidden files when there is already a file
that is created as a result of the command that has been run.

The main problem for this bug is that gmake does not implement the cache the
right way.

If you like to use caching the right way, you may do so whenever this caching
contains information that leads to the assumption that a target is up to
date.

The gmake command failed not because of caching in general but because of the
fact that gmake does not correct invalid cache content.

Please keep in mind that the current implementation of gmake causes gmake to
stop working with a wrong error message. If you run gmake again, it works
further on, up to the next similar problem while other make implementations
just continue to work with the first call.

This behavior of gmake is annoying and slows down the compilation.

It would be so easy just to re-check the time stamp of every cached file that
is believed to be out of date or missing. Time stamp checking is also faster
than re-running a command for a related rule.

A time stamp in "make" may have three values (the last one is a range):

- File dos not exist. This may become invalid if cached.

- Younger than any file. This will never become invalid for the lifetime of
make.

- A specific time stamp. This may become invalid and rechecking is faster than
running a command again. In special as only the time stamp of a single file
needs to be checked in such a case.

Just a note: I've seen commands that have been run twice as a result of not
verifying the third case in gmake.

BTW: some of the problems I have with gmake would go away if there was not an
unlucky combination of contradictory behavior.

___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

Re: [bug #58734] gmake does not check for the existence of a file before complaining it is missing

[Paul Smith]

On Thu, 2020-07-09 at 16:00 +, Edward Welbourne wrote:
> By the sounds of it, you have four files that are all created by a
> single rule, which isn't a scenario make is good at.  So work-arounds
> are tricky:

If you want to use parallel builds you must find a workaround for this
issue, or you'll run the same recipe multiple times.  That has nothing
to do with the directory cache: that's because in POSIX there is no way
to explain to make that a single invocation of a recipe will generate
multiple files as output.

That lack doesn't bother non-parallel make, and since POSIX doesn't
define parallelism if you're using pure POSIX (no parallelism) it works
fine to just say:

targ1 targ2 targ3: prereq1 prereq2 prereq3
recipe

But if you use -j then you can't use this simple idiom and you'll have
to do something more sophisticated.

If you want your makefile to be POSIX-compatible AND work with -j AND
you have recipes that generate multiple outputs, you have only one real
solution I'm aware of: use a sentinel file like this:

   targ1 targ2 targ3: .sentinel ;

   .sentinel: prereq1 prereq2 prereq3
   recipe
   @touch .sentinel

As a happy side-effect, if you implement this fix for multiple commands
being run then you will ALSO have solved the directory cache problem.

The directory cache problem happens when some recipe does some work
behind make's back, with results that make needs to know about.  If you
explicitly state what your recipes are _really_ doing in your makefile,
you won't have directory cache problems.

> persuade the maintainer of gmake to introduce a significant
> pessimisation.

Well, in fact the directory cache IS a problem, and I've long
recognized that.  The many emails to this list and elsewhere make it
clear.

However, it is ALSO a big performance gain, and furthermore it only
impacts makefiles which don't provide full dependency information and
so are, at least, sub-optimal (still POSIX compliant, of course).

So I would like to fix things so it still provides benefits to well-
written makefiles but also doesn't have these downsides for other
makefiles.  I have considered many options but so far none have gotten
past concept stage:

* Add an option to simply disable it (maybe adding .POSIX: does that).
  Obviously a very blunt hammer.

* Add a facility to dirty the cache; say, every time make runs a
  process (either a recipe or $(shell ...)) it removes cached info for
  all directories.  This would work but also would potentially lose a
  lot of performance, depending on the build (basically the more
  updating your build does the less the directory cache helps).

* Figure out a way to detect whether a directory has been updated, and
  dirty its cache if so.  This is probably the best option, and for
  POSIX systems this is simple: you just look at the modification time
  of the directory.  But for Windows systems (and VMS) it's not nearly 
  so simple.  So we'd need to figure out how to manage that.

For best effect we can combine #2 and #3, and only check for updated
directories if we've invoked some process since the last time.

Re: [bug #58734] gmake does not check for the existence of a file before complaining it is missing

[Edward Welbourne]

Jörg Schilling (9 July 2020 16:18)
> What I can say to help fixing the problem is that truss(1) shows that
> gmake calls stat(2) as a block on a larger list of files at startup
> and at that time correctly finds that the file in question does not
> exist.

gmake caches stat results and makes the reasonable assumption that they
only get invalidated for files that are targets of rules it executes.

> Then it executes some rules that have the side effect to create the
> file in question (a symlink to a source file).
>
> Gmake however never again calls stat(2) on that file but incorrectly
> claims that the file does not exist, eventhough it exists at the time
> it is really needed.

That sounds reasonable; if your make files don't say that their rule
creates a file, make doesn't know it's been created (or updated).
Rechecking for whether missing files have mysteriously appeared without
apparent cause would be a horrible pessimisation for most users of make.

> Please note that the fact that there is no explicit rule for that
> file, is caused by a workaround for another gmake bug that would cause
> a specific command to be called 4x concurrently, resulting in
> overwritten results.

Then try looking for a different work-around.
Your present work-around is a bug in your make files.

By the sounds of it, you have four files that are all created by a
single rule, which isn't a scenario make is good at.  So work-arounds
are tricky: but you're more likely to get help finding a better
work-around than to persuade the maintainer of gmake to introduce a
significant pessimisation.

Eddy.

[bug #58734] gmake does not check for the existence of a file before complaining it is missing

[Jörg Schilling]

Follow-up Comment #4, bug #58734 (project make):

This is one of the gmake bugs that does not seem to exist with trivial
makefiles. This is why I am apparently unable to give a small example to
repeat that problem. If you really need a makefile to repeat the problem, I
can give you a modified makefile that would cause the problem in a larger
project that yu would need to fetch for the test (this is the schilytools
project).

What I can say to help fixing the problem is that truss(1) shows that gmake
calls stat(2) as a block on a larger list of files at startup and at that time
correctly finds that the file in question does not exist.

Then it executes some rules that have the side effect to create the file in
question (a symlink to a source file).

Gmake however never again calls stat(2) on that file but incorrectly claims
that the file does not exist, eventhough it exists at the time it is really
needed.

Please note that the fact that there is no explicit rule for that  file, is
caused by a workaround for another gmake bug that would cause a specific
command to be called 4x concurrently, resulting in overwritten results.

BTW: I write portable software, so I cannot use a non portable feature (e.g.
your &: proposal) within a leaf makefile.

___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

[bug #58734] gmake does not check for the existence of a file before complaining it is missing

[anonymous]

Follow-up Comment #3, bug #58734 (project make):

> directory caching

... which I was convinced was flawed in:

https://lists.gnu.org/archive/html/bug-make/2016-10/msg00019.html

I bet that's not the problem here, but, until we see the example Dmitry
requested...

___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

[bug #58734] gmake does not check for the existence of a file before complaining it is missing

[Paul D. Smith]

Follow-up Comment #2, bug #58734 (project make):

> either the program is run twice

This can only happen if parallel mode is enabled, and is the correct behavior
assuming you've written your rule like this:


one two: three ; touch one two


This means the same thing as:


one: three ; touch one two
two: three ; touch one two


(this interpretation is required by the POSIX standard).  If parallelism is
enabled make may see that two is out of date before one finishes updating it,
and so it will run the rule twice.

In serial mode this should never happen (I've never seen it happen): if you
see it please give an example.

> or a depending rule that needs the created file incorrectly complains that
the file does not exist.

I've never seen this happen either: please provide an example.  The only
possible reason this could happen would be due to directory caching but I
don't think so.

If you want to declare a rule that builds multiple outputs, and you have GNU
make 4.3 or above, you can use the new "&:" separator to tell make that one
invocation will build both targets:


one two &: three ; touch one two


___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

[bug #58734] gmake does not check for the existence of a file before complaining it is missing

[Dmitry Goncharov]

Follow-up Comment #1, bug #58734 (project make):

Can you please attach a makefile that demonstrates the issue?

___

Reply to this item at:

  

___
  Сообщение отправлено по Savannah
  https://savannah.gnu.org/

[bug #58734] gmake does not check for the existence of a file before complaining it is missing

[Jörg Schilling]

URL:
  

 Summary: gmake does not check for the existence of a file
before complaining it is missing
 Project: make
Submitted by: schily
Submitted on: Wed 08 Jul 2020 02:59:25 PM UTC
Severity: 3 - Normal
  Item Group: None
  Status: None
 Privacy: Public
 Assigned to: None
 Open/Closed: Open
 Discussion Lock: Any
   Component Version: None
Operating System: None
   Fixed Release: None
   Triage Status: None

___

Details:

A file that has been created by a program that has more than one output file
is not handled correctly by gmake.

Depending on how the rules are written, either the program is run twice or a
depending rule that needs the created file incorrectly complains that the file
does not exist.

gmake should check for the time stamp of a file whenever it actually needs the
file and believes it does not exist or is outdated.




___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/