Re: Goodbye to GNU make's "build.sh" ... ?

2022-06-27 Thread Paul Smith
On Sun, 2022-06-26 at 22:00 -0400, Dmitry Goncharov wrote:
> What was the original driving force to introduce build.sh?

As mentioned, the goal of "build.sh" is to allow systems without an
already-existing make program to bootstrap themselves, by providing a
way to build GNU make without using "make".

> i think, people use shell globbing much more often than $(wildcard)
> or glibc's glob impl.  bash's glob impl is derived from the same old
> rms impl and has the bug fixed. And this allows the situation to
> continue.

Sorry I think I lost the thread here.

Just to remind, two things: first, the glob issue is not the primary
one; even if we leave glob the same way it is right now we'll still
have the build.sh problem.

And second, today GNU make uses its built-in glob ONLY on systems which
don't provide a GNU version of glob in their libc.  So for any system
which uses GNU libc, we are using the libc version of glob and whatever
bugs it contains.



Re: Goodbye to GNU make's "build.sh" ... ?

2022-06-26 Thread Paul Smith
On Sun, 2022-06-26 at 16:57 -0400, Dmitry Goncharov wrote:
> Have you considered to to avoid glob from gnulib? Make has its one
> impl of glob which it uses on certain systems.
> We could use this impl and maintain it on all systems.

While the original issue came up WRT glob, in actuality the problem is
far deeper.  The later versions of gnulib will pull significant content
for the findprog-in module, for example, which we do rely on quite
significantly in the GNU make fastpath.

However my plan was to preserve this internal version of glob and use
it for systems where we couldn't use the gnulib version, such as on
Windows.

> Aside from dependencies, one problem with the gnulib impl of glob is
> that it has a bug on some filesystems.

I would prefer to have that fixed in gnulib, than continue to maintain
a local copy just to avoid this issue.

Seems like it's not so much that the patch was rejected; indeed they
seem to agree there's a bug.  But they wanted more work on it and it
sort of fell apart.  I have no idea how much effort it would be to push
hard enough to get a change merged.



Re: Goodbye to GNU make's "build.sh" ... ?

2022-06-26 Thread Paul Smith
On Sun, 2022-06-26 at 17:39 +0300, Eli Zaretskii wrote:
> > The problem is that the process for "make-less" systems in the past
> > has been: run configure, then run build.sh.  But we won't be able
> > to use the environment created by configure in this "new" build.sh,
> > because it works in tandem with the makefiles.  The generated
> > config.h etc. will assume lots of things that simply aren't true
> > anymore, since we don't have make.
> 
> I thought of making build.sh compile and link make without any
> configure step, similar to what build_w32.bat does.
> 
> > So to me this is equivalent to my option #1, don't use gnulib at
> > all (or at least, use it incredibly sparingly).
> 
> No, I'm talking about not using Gnulib only for stage-1 build, the
> one that produces a Make binary capable of building the full
> version.  I was not talking about avoiding to use Gnulib for the
> final build whose results are installed.

This is sort of what Sven was suggesting.  It could work, but configure
does a lot of things: not only does it generate a config.h but it also
finds compilers, figures out what options they accept, etc.  Obviously
we don't need to do all that for a minimal build, but still the Windows
ecosystem is much simpler because it's so homogeneous; even then
build_w32.bat doesn't support building with Clang for Windows, for
example.  Generating a config.h on any random system is much harder
than having one pre-defined for the Windows32 API.

The goal of this "mini-make" must be to be able to compile automake-
generated makefiles (since that's what's needed in order to work with a
gnulib-enabled system).  Or at least, the Makefile.am that GNU make and
its gnulib modules use.

Since POSIX make does support wildcard syntax (e.g., "foo: *.c" is
legal) we would need glob/fnmatch of some sort, unless we can guarantee
that automake won't use it, for example.

We can hopefully disable other advanced features such as ar parsing,
job control, etc. which will certainly help simplify things.

There's some amount of disingenuity here since make is really just the
top of the iceberg when it comes to bootstrapping: you can't run
configure without a full libc, compiler, coreutils etc. and how can you
build those without a make?  So either we're talking about cross-
compilation, or else this "minimake" would need to be sufficient to
build more than just make itself.



Re: Goodbye to GNU make's "build.sh" ... ?

2022-06-26 Thread Paul Smith
On Sun, 2022-06-26 at 07:34 +0100, Sven C. Dack wrote:
> 3. Allow to build a minimal Make executable, which provides basic and
> traditional Make functionality and does not rely on gnulib, and then
> use it as a bootstrap.

It's certainly a possibility.  But there is already so much ifdef etc.
in GNU make, I'm loathe to increase it dramatically by trying to carve
out some sort of minimalist, but still sufficiently powerful, version
of make.  And, the things we get from gnulib are not really "extra
things" that we can just jettison, so much as replacements for basic
POSIX functions that not all platforms provide, or where the provided
versions are broken in some way on some platform.

If the GNU make source code was well-organized and cleanly modular this
would probably be a far more reasonable proposition but anyone who's
looked at the code can attest that this is far from the case.



Re: Goodbye to GNU make's "build.sh" ... ?

2022-06-26 Thread Paul Smith
On Sun, 2022-06-26 at 08:41 +0300, Eli Zaretskii wrote:
> It is sad that Gnulib maintainers aren't prepared to cater to a GNU
> project, and an important one such as Make.  These are special
> requirements that only a handful of GNU project could ever have, and
> for good reasons, so supporting their needs should be a no-brainer.

I suspect their position is that it IS a no-brainer, but with exactly
the opposite result: this requirement is needed by exactly one project
(GNU make, since once you have it you're all set) and so undertaking
all that work in gnulib is not resource-effective.

> > This leaves me with two options:
> > 
> >    1. Stop using gnulib, or at least sharply limit the modules we
> > will include to those with trivial-enough configurations.
> >    2. Abandon the build.sh script and require an existing make
> > program in order to build a new version of GNU make.
> 
> #2 would require that we promise to have Make 4.3 available from here
> to eternity.  Is that feasible?

Well, it doesn't have to be GNU make.  It just needs to be some make
that can use automake generated makefiles.  However, it does require
that such a make exists.

There are two different situations: first, bringing up GNU make on a
non-GNU system where you have some make, but not GNU make.  Maybe you
have BSD make or something.  Then the question is, is your existing
make capable of using automake-generated makefiles?

Second, you might be bootstrapping an entirely new system without ANY
make at all.  Now you need to get one.  I don't know whether the other
versions of make available have facilities to build without an existing
make.  Here you might need an older version of GNU make which has a
build.sh.  Of course here you likely already have a cross-build
environment so you could just cross-compile make as well.

> Can you list the Gnulib functions we currently use?

You can find them in the bootstrap.conf file:

https://git.savannah.gnu.org/cgit/make.git/tree/bootstrap.conf

See the list at the end.

The newly-added strtoll() is causing the proximate issue because it
relies on unistd.h which generates an enormous sed script in the
makefile.

This really came up because I was trying to find a way to include the
latest gnulib globbing library.  Adding "glob" to this pulls in a HUGE
number of prerequisites.  But even after undoing this I discovered
strtoull() is bad enough.

My idea was to keep the current glob/fnmatch to use on systems which
couldn't run "configure" (for example with the build_w32.bat script)
and use the gnulib glob/fnmatch for all systems that could run
configure.

But, configure is not enough: you also need make.  The current build.sh
script contains some minimal amount of magic needed to transform the
simple files we used in GNU make 4.3; see
https://git.savannah.gnu.org/cgit/make.git/tree/build.sh#n69
but making this work for the complex targets is frankly more than I
want to take on.

So then I wondered, why even both with gnulib glob?  Maybe we should
just take ownership of the ancient glob we already have and say, if you
have a system with GNU libc you get the latest, else you get this
basically functional version.

> Perhaps we could then prepare a fake-gnulib module with trivial
> implementations of those functions, which could be used by build.sh
> instead of the real Gnulib functions.

This seems odd to me: I mean, isn't that just replacing gnulib with
what it already is?  The point of gnulib is to provide replacements on
systems that don't provide these.  I don't want to create a new "micro-
gnulib" project.

> (Some Gnulib functions are replacements for those found in every C
> library, and those could be simply ignored in the build.sh build,
> relying on libc to do a reasonably good job.)

Yes, we could say that build.sh can only be used on systems which are
essentially POSIX-compliant and don't need a lot of fixup.

The problem is that the process for "make-less" systems in the past has
been: run configure, then run build.sh.  But we won't be able to use
the environment created by configure in this "new" build.sh, because it
works in tandem with the makefiles.  The generated config.h etc. will
assume lots of things that simply aren't true anymore, since we don't
have make.

So to me this is equivalent to my option #1, don't use gnulib at all
(or at least, use it incredibly sparingly).



Re: Goodbye to GNU make's "build.sh" ... ?

2022-06-26 Thread Paul Smith
On Sun, 2022-06-26 at 02:14 +0200, Henrik Carlqvist wrote:
> Would  this "old version of make" have to be GNU make?

It would need to be some version of make that is supported by automake.

As far as I'm aware, automake-generated makefiles are currently
intended to run with any POSIX-compliant version of make; I believe
that BSD make qualifies.  Certainly SunPro make from the late Jörg
Shilling would qualify, but I believe it requires a pre-existing make
to build (I haven't tried in quite a while).  And now that we've lost
him I'm not sure who will take up maintenance of that project.

Also, I've heard calls for automake maintainers to simplify the
generated makefiles by assuming GNU make and taking advantage of GNU
make features: then we really WOULD be in a quandary.  But I have no
idea how likely such a thing is to happen.

> The scary part of relying on an old version of some software is that
> old versions sometimes suffer from bit rot caused by changes in
> language standards or API changes in libraries.

Indeed.

> Maybe we would have to rely on cross compiling GNU make for new
> systems?

Another solution, to be sure.



Goodbye to GNU make's "build.sh" ... ?

2022-06-25 Thread Paul Smith
I'm trying to decide what the future is for GNU make's "build.sh"
bootstrapping script. As you may recall, this script is provided to
allow GNU make to build on systems which don't already have an instance
of make installed. Its goal is to build the first make binary, without
of course all the fancy parts of avoiding rebuilds, generating
dependency files, etc.

Unfortunately, this really cannot work if GNU make is going to rely on
gnulib, because a MANY gnulib modules require not only autoconf to
work, but also automake. That is, it's not enough to just run configure
and then you get a set of source files and header files that you can
compile by hand by just invoking the compiler. You must ALSO run make,
because the modules provide makefile recipes that invoke sed, etc. to
convert files into their final form.

I had a discussion about this with the Gnulib maintainers a while ago:

https://lists.gnu.org/archive/html/bug-gnulib/2019-09/msg00041.html

However the gnulib maintainers were disinclined to modify the practices
of the gnulib modules.

This leaves me with two options:

   1. Stop using gnulib, or at least sharply limit the modules we will
  include to those with trivial-enough configurations.
   2. Abandon the build.sh script and require an existing make program
  in order to build a new version of GNU make.

#1 is what I followed for GNU make 4.3, which has a limited subset of
carefully-chosen modules. However this becomes harder over time. For
example any module that needs unistd.h requires a very complex automake
rule.

If #2 is chosen, then a bootstrap process would involve first obtaining
an older version of make, such as GNU make 4.3 or lower, and building
that with its build.sh, then using the resulting make to build the
newer version.

I'm interested in thoughts about these options.



Re: Crash in 'find_and_set_default_shell()'

2022-06-19 Thread Paul Smith
On Sun, 2022-06-19 at 08:47 +0300, Eli Zaretskii wrote:
> > I do have a memory that some versions of Visual Studio, up until
> > relatively recently, have non-standard implementations of
> > snprintf()
> > but hopefully it's standard enough to manage this.
> 
> If we now rely on ANSI-standard behavior of snprintf in the Windows
> port, we need the MinGW build to use -D__USE_MINGW_ANSI_STDIO=1, to
> force replacement of the MS version of snprintf with MinGW's own
> reimplementation, which is ANSI-standard.  That's because linking
> Make against MSVCRT.DLL will use a non-compliant snprintf in that DLL
> (MinGW cannot link against proprietary C runtime libraries that come
> with later versions of Visual Studio).

As best as I recall, the non-standard part of the old snprintf() was
that it returned -1 if the buffer wasn't large enough, rather than the
number of chars that would be needed.

The change made here doesn't rely on that behaviour.

However I realize now that I need to forcibly add a nul terminator
because the old snprintf() on Windows didn't nul-terminate the string
if the buffer wasn't large enough.

Maybe I'll just punt on that and simply allocate a large-enough buffer.

Were there other differences in old snprintf()?



Re: Crash in 'find_and_set_default_shell()'

2022-06-18 Thread Paul Smith
On Wed, 2022-05-11 at 08:00 +0200, Gisle Vanem wrote:
> The crash and wild call-stack seems to be caused
> by an overflow to 'sprintf(sh_path, ..)'. But replacing
> with 'snprintf()' works w/o any crash:

I applied changes based on this: some didn't need sprintf() at all; the
rest I used snprintf().

I do have a memory that some versions of Visual Studio, up until
relatively recently, have non-standard implementations of snprintf()
but hopefully it's standard enough to manage this.



Re: Potential Bug: `.PHONY` targets and order-only prerequisites

2022-05-21 Thread Paul Smith
On Sat, 2022-05-21 at 14:34 -0400, Dmitry Goncharov wrote:
> On Sat, May 21, 2022 at 12:25 PM Paul Smith  wrote:
> > Maybe what you're saying is that make should throw an error or
> > warning if you try to add an order-only prerequisite to a phony
> > target, telling you that it will have no effect on your makefile?
> 
> Having a phony target depend (usually indirectly, but there are
> examples of immediate dependency) on an order-only prerequisite is
> quite common. This happens when the top target is marked phony and
> there is an order-only prerequisite down the graph.

Agreed we would never mention an indirect dependency.  That is not an
invalid use of order-only prerequisites anyway; the order-only
prerequisite will be relevant only for its immediate target not
"parents" of that target and if the immediate target is not phony then
the o-o prerequisite is entirely valid there.

I can't think of a reason to add an order-only prerequisite directly to
a phony target on purpose, but I suppose it could happen as the result
of some variable expansion or something which is annoying to avoid.

But I do not plan to add any sort of warning like this at the moment.



Re: floating point exception 8 when running make

2022-05-21 Thread Paul Smith
On Sat, 2022-05-21 at 12:17 -0700, Valery Tolkov wrote:
> Exact target file doesn't matter, all files give the same error. If I
> do a clean build without any existing targets, it works. But second
> time it fails again.
> 
> > ...
> > Must remake target `bin/dbg/clr/clr.o'.
> > clang++ --config ./warnings.cfg -g -O0 -DDEBUG -std=c++2a -
> > fmodules  -I . -I vt -D HAVE_STDDEF_H -D HAVE_STDINT_H -D
> > HAVE_SYS_TYPES_H -D OSX -D UNIT_TESTS -c clr/clr.cc -o
> > bin/dbg/clr/clr.o
> > Putting child 0x61b94230 (bin/dbg/clr/clr.o) PID 35065 on the
> > chain.
> > Live child 0x61b94230 (bin/dbg/clr/clr.o) PID 35065 
> > Reaping losing child 0x61b94230 PID 35065 
> > make: *** [bin/dbg/clr/clr.o] Floating point exception: 8
> > Removing child 0x61b94230 PID 35065 from chain.

It is not make that's failing, it's your compiler (clang++).  Make is
just reporting to you that the command it invoked (the compiler) exited
because it (the compiler) got a floating point exception error.

You'll have to figure out what's wrong with your compiler, or why your
code is causing your compiler to exit this way.



Re: Potential Bug: `.PHONY` targets and order-only prerequisites

2022-05-21 Thread Paul Smith
On Sat, 2022-05-21 at 19:06 +0200, Alejandro Colomar wrote:
> By "once all children are complete" you are implying the "existence"
> of the children (which make(1) doesn't really check, but one can
> think of it as if it did).

Perhaps that's the confusion.  Make doesn't care about files at all per
se: it treats the filesystem as a key/value store where the key is the
pathname and the value is the mod time.  It will look up each target in
that store (filesystem) and if it doesn't exist it gets a special
"oldest" timestamp so that the recipe is run.  After the recipe is run
make gives the node a special "newest" timestamp so targets that list
it as a prerequisite are out of date.  It's not an error (to make) if
the key (pathname) still doesn't exist after the recipe runs.

Make really only cares about its internal graph.  The filesystem is
just a place to keep target state in between invocations of make (and
of course, a way for other tools to let make know that something
changed), since make doesn't have its own database.

Having make complain for any non-phony target that didn't actually
create its target would violate POSIX, and probably cause hundreds of
thousands of makefiles out there to throw complaints where people
forgot or didn't want to bother with using .PHONY (not all makefiles
were written for GNU make, and .PHONY is a GNU make feature).



Re: Potential Bug: `.PHONY` targets and order-only prerequisites

2022-05-21 Thread Paul Smith
On Wed, 2022-05-18 at 14:36 -0700, Jacob Kopczynski wrote:
> The thing that the docs refer to as "impose order" is not a single
> thing, but two. I would characterize a normal prerequisite as doing
> three things rather than two:
> - update-marking: cause a target to be marked out of date if the
> prereq is marked out of date
> - require-existence: require the prereq to be built successfully at
> least once before the target is built
> - imposed-order: require the prereq to be built before the target, if
> both are being built

I guess I've just been using make for too long because I don't
understand the distinction you're trying to make between the last two.
There's no difference here, and the extra comment at the end "if both
are being built" is not meaningful (or anyway I don't understand what
it means).

Let's review how make works: it reads the makefile(s) and builds a
directed acyclic graph where every target is a node and every
prerequisite relationship is an edge between two nodes.

Then starting with each node representing a goal target (either the
first target in the makefile, or the target(s) given on the command
line) make performs a depth-first walk of the graph starting at that
node.  "Walking the graph" consists of processing each child node
recursively, in order, and once all children are complete make compares
the timestamp of each (not order-only) child node to the timestamp of
the current node.  If the current node is out of date, then it is
updated (by running its recipe), else we do nothing.  Then this node is
complete and we return to the processing of the parent node, or if
we're the goal target then we're done.

I don't see any way that you can talk about "require-existence"
separately from "imposed-order", and the attempt to do so is (it seems
to me) just creating confusion.

> For .PHONY targets, require-existence always mandates update-marking,
> which is why the overlap between those two cases is particularly
> confusing - a phony target which drops only update-marking is a
> contradiction in terms, so the natural assumption to make is that the
> effect of "order-only" prereqs is not that, and is instead something
> non-paradoxical.

Order-only prerequisites weren't created because people wanted to use
them with .PHONY targets.  They were created for use with non-phony
targets.

You can add them into phony targets, but they make no difference there
as we described.

Maybe what you're saying is that make should throw an error or warning
if you try to add an order-only prerequisite to a phony target, telling
you that it will have no effect on your makefile?

Perhaps.  I'm generally not inclined to do that sort of thing but maybe
it would be helpful.  Many people have asked for an equivalent to a GCC
-Wall flag for GNU make that gives warnings about things that are
probably not doing what you expected.



Re: Potential Bug: `.PHONY` targets and order-only prerequisites

2022-05-18 Thread Paul Smith
On Wed, 2022-05-18 at 10:22 -0700, Jacob Kopczynski wrote:
> I believe I understand. The name "order-only" is highly misleading
> and should be changed - it does considerably more than "only"
> "order"; the only thing it does not do is check the timestamp.

As described in the docs there are only two things a prerequisite can
do: impose order and effect the out-of-date decision of the target. 
Order-only prerequisites do one of them (impose order) and don't do the
other (effect out-of-date decisions).

What are the "considerably more" things that order-only prerequisites
do, besides impose order?

I do agree that the docs need clarification though.



Re: Potential Bug: `.PHONY` targets and order-only prerequisites

2022-05-17 Thread Paul Smith
On Tue, 2022-05-17 at 22:32 +, Martin Dorey wrote:
> >  all your targets are .PHONY, and thus are always rebuilt anyway
> 
> If you "make down", the rule for "down-clean" doesn't run.  They're
> only rebuilt if something causes them to be considered.
> 
> >  order-only prerequisites are totally irrelevant and have no impact
> > on
> > your makefile.
> 
> If you comment-out the order-only prerequisite that says that the up
> target depends on down-clean, then the recipe for down-clean doesn't
> get run when you "make up".

Sure, of course.

What I was trying to say was, any rule that would normally be run will
be run regardless of order-only or not, because all the targets in the
makefile are phony.  So adding or removing the order-only operator in a
prerequisites list makes no difference to how the targets will be
processed, in this specific makefile.



Re: Potential Bug: `.PHONY` targets and order-only prerequisites

2022-05-17 Thread Paul Smith
On Tue, 2022-05-17 at 17:20 -0400, Paul Smith wrote:
> this says two things: first, that b and c will both be rebuilt (if
> necessary) before a's recipe is started,

I guess I should be more clear about the "(if necessary)".  What I mean
is the same as if you had run "make b" or "make c" and make had decided
it needed to rebuild those targets.

In other words, the "if necessary" has nothing to do with "a" or
whether "a" needs to be rebuilt.  It only relates to "b" and "c" and
whether they need to be rebuilt, on their own.  If those targets don't
exist, or are phony, or they have prerequisites which cause them to be
out of date, they will be rebuilt.  If they exist, are not phony, and
have no prerequisites that cause them to be out of date, they will not
be rebuilt.

Once both "b" and "c" are completed, then make will decide whether "a"
needs to be rebuilt.  For that decision make will ignore any order-only
prerequisites and only consider the normal prerequisites.



Re: Potential Bug: `.PHONY` targets and order-only prerequisites

2022-05-17 Thread Paul Smith
On Tue, 2022-05-17 at 14:00 -0700, Jacob Kopczynski wrote:
> I'm unsure whether this is a bug or just undocumented, but I found a
> confusing interaction in a simple Makefile:

You are misreading the documentation.  I will quote:

> A normal prerequisite makes two statements: first, it imposes an
> order in which recipes will be invoked: the recipes for all
> prerequisites of a target will be completed before the recipe for the
> target is run. Second, it imposes a dependency relationship: if any
> prerequisite is newer than the target, then the target is considered
> out-of-date and must be rebuilt.

Hopefully that is clear.  To give an example, if you have:

  a: b c

this says two things: first, that b and c will both be rebuilt (if
necessary) before a's recipe is started, and second, that if either b
or c are newer than a (if a is out of date with respect to b or c),
then a will be considered out of date and rebuilt.

The docs go on to say:

> Occasionally, however, you have a situation where you want to impose
> a specific ordering on the rules to be invoked without forcing the
> target to be updated if one of those rules is executed. In that case,
> you want to define order-only prerequisites.

What this means is that if you have:

  a: b | c

then the ordering statement (that both b and c will be rebuilt (if
necessary) before a's recipe is started) is still in place for both
prerequisites b and c.

However, the second statement, that a is considered out of date if it's
older than the prerequisite, only is in effect for b, and not for c.

"c" is an "order-only" prerequisite in that it is ordered before "a",
but if it's updated then that doesn't mean "a" will be updated.

Since all your targets are .PHONY, and thus are always rebuilt anyway,
order-only prerequisites are totally irrelevant and have no impact on
your makefile.



Re: Archive Members as Targets

2022-05-11 Thread Paul Smith
On Tue, 2022-05-10 at 22:45 +0200, Michael Lehn wrote:
> But not on the Linux boxes there make always rebuild everything. On
> all machines I am using GNU Make.

This is because whatever GNU/Linux distribution is being used, has
configured their binutils so that ar is in "deterministic mode" by
default, which is bogus and breaks make.

On such a system you'll have to pass the "U" option to ar, to disable
deterministic mode.



Re: Crash in 'find_and_set_default_shell()'

2022-05-10 Thread Paul Smith
On Tue, 2022-05-10 at 15:03 +0200, Gisle Vanem wrote:
> SHELL := env "PATH=$(PATH)" /bin/bash

Well, I dunno.  The problem is that at some point you have to choose
which command to use to invoke something.  The SHELL variable is
intended to contain a shell program that make will exec().  Here you're
saying that either (a) make has to invoke a shell which will invoke the
SHELL command (and how does it choose which shell?  It clearly can't
use the SHELL variable...), or else (b) make has to parse this string
and break it up into words that it can use to call exec() without going
through a shell: normally make leaves this sort of command parsing up
to the shell.

I will say that this does work as expected and doesn't throw an error
with the latest GNU make Git version, on GNU/Linux.



Re: [PATCH v1 2/2] misc: Replace strcmp with memcmp when it obviously works

2022-04-24 Thread Paul Smith
I applied both of these patches, thanks!



Re: Wooden toy

2022-04-20 Thread Paul Smith
On Wed, 2022-04-20 at 21:36 +0800, Eliza Tan via Bug reports and
discussion for GNU make wrote:
> How are you ?

Sorry, this got approved in moderation when it shouldn't have been.



Re: A translate mistake in Gnu make 4.3

2022-04-01 Thread Paul Smith
On Fri, 2022-04-01 at 16:06 +0800, calvin wrote:
> "Flags" should be translated to "标志" not "旗标" in Chinese

Thanks for your interest!  All GNU make translations are handled
through The Translation Project:

  https://translationproject.org/html/welcome.html

Please report any issues to the GNU make translation team(s) there. 
Please provide info on exactly what message(s) you're referring to
rather than just showing a single word, since translations are done on
complete sentences or statements.

Cheers!



Re: GNU Make bug report: broken dry-run functionality with sub-make invocations

2022-03-21 Thread Paul Smith
(We generally prefer to use inline replies on the GNU lists, rather
than top-posted replies, thanks)

On Mon, 2022-03-21 at 13:22 +, Ambrus Sumegi wrote:
> If the invocation is a function, i.e., `$(make,"external_target") |
> tee logs/external_task.log` then Make knows exactly where the call to
> the sub-make ends without having to parse a shell command. So, when
> running with the -n switch, it can simply print "make external_target
> | tee logs/external_task.log" and proceed to show the output of `make
> external_target -n`

So if someone runs make -n it would be OK if only the make was invoked,
without actually sending any of the output to the tee (because clearly
the tee would not be invoked)?  That idea seems just as fraught, of not
moreso, as the original behavior.

What if your recipe looked like this:

test -f afile || $(MAKE) -C foo

If you change your makefile to use the putative function like this:

test -f afile || $(make -C foo)

Are you saying that if "make -n" is invoked this recipe will ALWAYS run
the sub-make, regardless of whether the "afile" file exists or not?

If you check makefiles out in the world you'll see that many times the
recipe lines that are used to invoke sub-makes are complex with lots of
additional shell operations in the same recipe line.  IMO a facility
that reached into the middle of all that scripting and plucked out the
sub-make by itself and ran it without any of that surrounding context,
when make -n was given, would be at least as dangerous as what we have
now.



Re: GNU Make bug report: broken dry-run functionality with sub-make invocations

2022-03-21 Thread Paul Smith
On Mon, 2022-03-21 at 09:34 +, Ambrus Sumegi wrote:
> For the record, I’ve thought of a sort-of-solution to the “would you
> have Make parse the shell command” question over the weekend. If the
> sub-make was called through a function rather than a variable, the
> whole issue could be a lot more contained. Having $(make, “ passed to sub-make>”) as the canonical form of invoking other Make
> instances could guarantee that no other program gets called when the
> -n switch is used. This way any elaborate multi-line command
> containing $(MAKE) could more or less be fixed with a sed command to
> completely eliminate this problem class.

I don't see how this would work.

The command you wanted to run was:

$(MAKE) external_target | tee logs/external_task.log

You wanted to pipe the output of the make command to another command. 
How does putting make in a function allow this to happen?
 
> Of course, one could still do something like $(make, $(shell
> (“some_reckless_command”)), but it would be more obvious, and Make
> could output an explicit warning about a shell being called together
> with the sub-make invocation, or even refuse to execute said shell
> call. Since on Fri 18/03/2022 20:14, psm...@gnu.org pointed out that
> this is also a potential security problem, a future release could
> deprecate the $(MAKE) variable and switch to the function-based
> invocation, with shell calls disabled within it by default for the
> sake of that extra bit of security.

It's simply not possible to make this change.  People want to do all
kinds of complicated things before or after or instead of invoking the
sub-make, so the full power of the shell needs to be available.

If you wanted to simply run a make command with no surrounding shell
operations, then running:

$(MAKE) ...

all by itself _already_ does exactly what you want.  It's only when you
need to add extra operations into the same recipe line that you run
into problems with "-n", and that's exactly the place where using a
make function instead won't work.



Re: [PATCH] Author: Nenghe Wang <545934...@qq.com>

2022-03-20 Thread Paul Smith
Thank you for the patch.

It's most helpful if you can provide an example of the warning you're
trying to address, when you send the patch.

Cheers!


On Sat, 2022-03-19 at 19:42 +0800, boxues...@126.com wrote:
> From: wnh <545934...@qq.com>
> 
> Adjust header file order in src/remote-cstms.c
> 
> Put #include "job.h" in front of #include "commands.h" to
> avoid compile warning
> ---
>  src/remote-cstms.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/src/remote-cstms.c b/src/remote-cstms.c
> index e336928..1c82ded 100644
> --- a/src/remote-cstms.c
> +++ b/src/remote-cstms.c
> @@ -20,8 +20,8 @@ this program.  If not, see
> .  */
>  
>  #include "makeint.h"
>  #include "filedef.h"
> -#include "commands.h"
>  #include "job.h"
> +#include "commands.h"
>  #include "debug.h"
>  
>  #include 




Re: Bug report made anonymously should be assigned to my username on Savannah for receiving updates on it

2022-03-18 Thread Paul Smith
On Fri, 2022-03-18 at 21:12 +, Lahfa Samy wrote:
> I've reported this bug anonymously : 
> https://savannah.gnu.org/bugs/index.php?62200 and would like to
> receive updates/comments on it by mail on my Savannah account, I
> don't know if the bug report could be assigned to me or the "posted
> by" field could be updated to my username on Savannah (AkechiShiro
> ).

Unfortunately I'm not aware of an way to reset the submitter of the
bug.

However, you can add yourself to the CC list to receive any updates
made to it: look at the bottom of the bug to find the Mail Notification
Carbon-Copy List and add yourself.



Re: GNU Make bug report: broken dry-run functionality with sub-make invocations

2022-03-18 Thread Paul Smith
On Fri, 2022-03-18 at 17:48 +, Martin Dorey wrote:
> Maybe putting it in the form of a patch on the latest git source will
> help it over the finish line:

I'm OK with adding some short text about this into the man page.  As
David mentions it may be important enough to do that since command
being run by make even when the user gives "-n" could give unexpected
or even unpleasant consequences.

However, in general I want to be clear that the man page is not, and is
not intended to be, user documentation.  It's a reference page.  The
man page is intended for people who ALREADY KNOW how to run make and
write makefiles, but need to quickly remind themselves of some syntax
that might have slipped their mind, and they don't want to have to
search the manual for it.

make and makefiles are entirely too complex a topic to be addressed in
a man page.

If you want to learn about how something works, not just remind
yourself how to use it, you need to be reading the user manual not the
man page.  Everything should be documented in full and understandable
detail in the user manual.

https://www.gnu.org/software/make/manual/html_node/index.html



Re: GNU Make bug report: broken dry-run functionality with sub-make invocations

2022-03-17 Thread Paul Smith
On Thu, 2022-03-17 at 18:27 +, Martin Dorey wrote:
> That coped with -nj --no-print-directory on the one version of Make
> that I tested it with, but I don't know how portable that would
> prove.

Modern versions of make guarantee a canonical format of MAKEFLAGS such
that you can parse them relatively easily, and with confidence.

The details are in the manual I believe but the short is:

 * All options that have "short" variants with no arguments (for
   example --dry-run -> -n) are converted to the short variants and put
   into the first word, with no leading "-"
 * All other options are added after that.
 * Short options have a single dash prefix and if they have an argument
   it's attached to the option
 * Long options have a double-dash prefix (obviously) and if they have
   an argument it's attached to the option with an "="
 * Options that have both short and long forms, prefer the short form.

So, for -n, you can use:

$(findstring n,$(word 1,$(MAKEFLAGS)))

and it will expand to "n" if dry run is enabled (regardless of which
form the option was given in), or empty if not.

So, the OP could use something like this:

DRYRUN = $(findstring n,$(word 1,$(MAKEFLAGS)))

 main_target: create_logdirs
 $(MAKE) external_target $(if $(DRYRUN),,| tee logs/external_task.log)

There are other alternatives. For example, you could add "+" as a
prefix to the recipe in the create_logdirs so that the directories are
created even when "-n" is given.


But ultimately Martin's comment is correct: this is not a bug in make
and there's no possible way that make could do anything "better" than
what it does. At some level, especially if you're writing recursive
makefile environments, your recipes have to be written to be resilient
to the possibility that make was invoked with "-n".



Re: [PATCH] Fix src/function.c build failure on gcc-12.

2022-02-21 Thread Paul Smith
On Mon, 2022-02-21 at 08:59 +, Edward Welbourne wrote:
> and that's assuming a 32-bit int; the signed range is from -
> 2,147,483,647 to 2,147,483,648.  However, may I suggest the following
> (which I know included in the GPL'd cfengine sources at some point):

This computation is already included in GNU make (in Git) as the
constant INTSTR_LENGTH (which uses sizeof(uintmax_t) as the basis),
from a previous, similar suggestion you made a few months ago :).

I modified this and a few other buffers that used static lengths, to
use the INTSTR_LENGTH macro.



Re: Make losing jobserver tokens on Windows

2022-02-21 Thread Paul Smith
On Mon, 2022-02-21 at 15:56 +0100, Magnus Ihse Bursie wrote:
> I'm trying to built it from source right now (and it does not seem 
> complicated), but even so, an official binary makes sure I'm not 
> introducing any issues from my local build environment.

There is no such thing as an "official binary" for any GNU/FSF
software.  The GNU project publishes source code, only.  Any binary you
find is just someone's unofficial build.

"gnuwin32.sourceforge.net" is in no way affiliated with GNU or the FSF
and doesn't look like it's been maintained in 11+ years, so I would
discourage anyone from using it anymore.

> > I believe that the goal of the Cygwin port is to implement a POSIX
> > layer on top of Windows, so I'm assuming that when you compile GNU
> > make with Cygwin it uses the POSIX implementation of the
> > jobserver.   This implementation relies on certain POSIX behaviors
> > of pipes and signals and it's possible that the Cygwin wrappers
> > here have an issue that is leading to this problem.

> Is there some way I can verify this hypothesis (e.g. debug output 
> flags), other than looking at the Cygwin source on how they 
> configure/build make?

I have no idea, since we don't support or maintain the Cygwin port of
GNU make.  You could go ask the Cygwin folks.  However, they definitely
provide the source code for all the software they make available so you
could also go find that and see if there are any patches applied.



Re: [PATCH] Fix src/function.c build failure on gcc-12.

2022-02-21 Thread Paul Smith
On Mon, 2022-02-21 at 08:59 +, Edward Welbourne wrote:
> and that's assuming a 32-bit int; the signed range is from -
> 2,147,483,647 to 2,147,483,648.  However, may I suggest the following
> (which I know included in the GPL'd cfengine sources at some point):

This computation is already included in GNU make (in Git) as the
constant INTSTR_LENGTH (which uses sizeof(uintmax_t) as the basis),
from a previous, similar suggestion you made a few months ago :).



Re: GNU Make app and right text output with echo

2022-02-19 Thread Paul Smith
On Sat, 2022-02-19 at 17:40 +0200, Eli Zaretskii wrote:
> > From: 
> > Date: Sat, 19 Feb 2022 14:25:47 +0100 (CET)
> > 
> > Problem is wisible on follow example of file Makefile.win:
> > 
> > all : 
> >   echo $(pkg-config --libs glib-2.0)
> 
> This is wrong, you want
> 
> all : 
> echo $(shell pkg-config --libs glib-2.0)
> 
> The 'shell' function is missing.

For questions like this you probably want the help-m...@gnu.org mailing
list FYI.

I think the best solution is to escape the "$" so it's passed to the
shell rather than interpreted by make; using the shell function in
recipes is to be discouraged IMO.

So use:

  all : 
 echo $$(pkg-config --libs glib-2.0)

This of course assumes that all other aspects of this system are set up
correctly: running POSIX operations like this on a Windows system
typically requires a lot of other configuration.



Re: Make losing jobserver tokens on Windows

2022-02-18 Thread Paul Smith
On Fri, 2022-02-18 at 10:18 +0100, Magnus Ihse Bursie wrote:
> make[2]: INTERNAL: Exiting with 1 jobserver tokens available; should
> be 24!
> 
> This effectively turns the highly parallelized builds into 
> single-threaded builds, and is absolutely detrimental for
> performance. :-( On the flip side, this also makes for the perfect
> testing environment to really get to the bottom of this issue.
> 
> So, my first question is: Is this a known problem? Are there any 
> workarounds/fixes to get around it?

Please be sure to tell us which version of GNU make you're using.

I am not aware of any issues like this.  However, I recommend that you
try using the Visual Studio build of GNU make and see if that works
better.  I'm not very familiar with the Cygwin version of GNU make;
it's not maintained by us but my understanding is that they've needed
to apply some patches (maybe this understanding is outdated though).

I believe that the goal of the Cygwin port is to implement a POSIX
layer on top of Windows, so I'm assuming that when you compile GNU make
with Cygwin it uses the POSIX implementation of the jobserver.  This
implementation relies on certain POSIX behaviors of pipes and signals
and it's possible that the Cygwin wrappers here have an issue that is
leading to this problem.

The Windows jobserver uses Windows semaphores: it's a simpler
implementation than the POSIX one and may be more reliable since it's
using a native W32 implementation.

Note, all the above is just my personal noodling based on no actual
knowledge, as I'm not really a Windows person.

Maybe Eli will have some other thoughts.

> Otherwise: Any suggestions on how to go on and debug this? I am
> willing to build and test an instrumented debug build of make, but I
> will need assistance to find my way around the source and spot likely
> candidates for the source of the problem.

Well, the implementation is in the src/w32/s32os.c file in the make
source code.

But if the problem occurs reliably with a cygwin build and doesn't
happen with a Visual Studio build, then my suspicion is that it's an
issue with Cygwin rather than GNU make (since we don't see this kind of
problem on other POSIX-based systems).

Of course it could be a bug in GNU make where we are implicitly
assuming some kind of behavior which isn't actually guaranteed by
POSIX.



Re: oddball messages on FreeBSD that cause make to fail the testsuite

2022-02-14 Thread Paul Smith
On Mon, 2022-02-14 at 18:54 -0500, Dennis Clarke wrote:
> variable 'plugin_is_GPL_compatible' [-Wmissing-variable-declarations]

This is happening because you're adding extra compile-time options to
the standard GNU make build, and those options are getting passed
through to the test suite.

Most likely we should expend a little effort to sanitize the options
provided to the compiler in the test suite to remove unneeded things
like extra warnings.

However, in general when compiling external tools it's probably not a
good idea to force extra warning flags etc. into the build, unless
you're actually trying to develop or debug the code.  If you're just
trying to build it to use it, then adding extended warnings usually
just generates spurious issues.



Re: Errors in man pages of make

2022-02-07 Thread Paul Smith
On Mon, 2022-02-07 at 18:18 +0100, Helge Kreutzmann wrote:
> > This channel is fine, but please always do include the release of
> > GNU make that you're looking at when you submit changes (run "make
> > --version").
> 
> This I don't know. I receive the man pages from Fedora, SUSE,
> Archlinux, Debian without any further information, so the string
> appears in any, some or all of those version.

I see.  It's unfortunate that our man page doesn't include the version
it documents in the page itself.  I will look into adding that.

> > Note that there are no translations of the man page provided with
> > GNU make.
> 
> Yes, that's why it is included in manpages-l10n. If you are
> interested in establishing man page translations within GNU make
> (using po4a or similar), please let me know.

Currently all our translations are handled through the TP
(https://translationproject.org/html/welcome.html ).  I'm not sure if
they support translations of man pages etc. or what would be involved
with distributing other translations in the GNU make package.

At first glance I lean towards documentation translations being
provided separately from the GNU make package: it seems more useful to
allow these to be updated and distributed asynchronously.



Re: Errors in man pages of make

2022-02-06 Thread Paul Smith
On Sat, 2022-02-05 at 11:24 +0100, Helge Kreutzmann wrote:
> I'm now reporting the errors for your project. If future reports
> should use another channel, please let me know.

This channel is fine, but please always do include the release of GNU
make that you're looking at when you submit changes (run "make
--version").

Note that there are no translations of the man page provided with GNU
make.

> Man page: make.1
> Issue: I doesn't appear in SYNOPSIS, why not format »target«
> instead?

I addressed this, thanks!

> --
> Man page: make.1
> Issue: B -> B;
> 
> "Internal option B uses to pass the jobserver pipe read and write file "
> "descriptor numbers to B see the section B "JOBSERVER> for details"
> --
> Man page: make.1
> Issue: I → I,
> 
> "If the job to be run is not a B then B will close the "
> "jobserver pipe file descriptors before invoking the commands, so that the "
> "command can not interfere with the I and the command does not "
> "find any unusual file descriptors."

I can't find either of the above two texts in the GNU make man page,
neither the current latest nor any other previous version going back to
the first committed version, in 1990.  There has never been any
description of the jobserver in any version of the man page.

All I can assume is that your distribution has provided you with a
version of the man page that is modified from the standard one.



Re: Bug in $(shell ...) I can't understand

2022-02-06 Thread Paul Smith
On Sun, 2022-02-06 at 17:12 -0500, Dmitry Goncharov wrote:
> This behavior is correct, is not it?
> 
> $ cat makefile
> SHELL:=/badpath/bash
> value:=$(shell echo hello world)
> all: ; $(info $(value))
> $ make >/dev/null
> make: /badpath/bash: No such file or directory
> $
> 
> stdout is redirected to /dev/null, but the user still wants to see
> errors. Is not this scenario what stderr was invented for?

Yes, but that's sent directly to stderr so make's shell function is not
even involved in that.

The weird behavior we're considering is that make was printing the
_stdout_ of the invoked shell to make's stderr, if the shell exited
with code 127 (see the source I mentioned previously).

> On Sun, Feb 6, 2022 at 4:48 PM Paul Smith  wrote:
> > I decided this was a bug and changed the behavior for the next
> > release.
> 
> What is the new behavior?

The above makefile behaves the same, and this makefile:

  $ cat Makefile
  out := $(shell bad-command 2>&1)
  all: ; @echo 'out = $(out)'

gives:

  $ make
  out = /bin/sh: 1: bad-command: not found

instead of what we get with 4.3:

  $ make
  /bin/sh: 1: bad-command: not found
  out =



Re: Bug in $(shell ...) I can't understand

2022-02-06 Thread Paul Smith
On Sun, 2022-02-06 at 15:21 -0500, Paul Smith wrote:
> I'm not sure this is correct.  I will need to think about it.

I decided this was a bug and changed the behavior for the next release.



Re: Bug in $(shell ...) I can't understand

2022-02-06 Thread Paul Smith
On Sun, 2022-02-06 at 20:18 +0300, Dmitry V. Levin wrote:
> 4175643 write(2, "/bin/sh: bad-program: command no"..., 
> 40) = 40
> 4175640 <... read resumed>"/bin/sh: bad-program: command no"..., 200) = 40
> 4175640 read(5,  
> 4175643 +++ exited with 127 +++
> 4175640 <... read resumed>"", 160)  = 0
> 4175642 +++ exited with 127 +++
> 4175640 write(2, "/bin/sh: bad-program: command no"..., 40) = 40
> 4175640 write(1, "out = ", 6) = 6
> 4175640 write(1, "\n", 1)    = 1
> 4175640 read(4, "", 4096) = 0
> 4175640 write(2, "make: *** No targets.  Stop.\n", 29) = 29
> 4175640 +++ exited with 2 +++

Yes, I saw this as well.  But this doesn't help me understand WHY it's
happening.  There's something make is doing here that's causing this
but I don't understand what it is.  Make is reading the output of the
pipe, like we want, but somehow it's then rewriting the output to its
own stderr... why is that happening?

Oh.  I see the problem.  If the shell exits with an exit code of 127
then make's shell function assumes that it failed because the sub-
process was not found, and it actually writes its output to stderr!!

function.c:func_shell_base() contains:

/* shell_completed() will set shell_function_completed to 1 when the
   child dies normally, or to -1 if it dies with status 127, which is
   most likely an exec fail.  */

if (shell_function_completed == -1)
  {
/* This likely means that the execvp failed, so we should just
   write the error message in the pipe from the child.  */
fputs (buffer, stderr);
fflush (stderr);
  }
else
  ...

I'm not sure this is correct.  I will need to think about it.

> Try the following instead:
> 
> out := $(shell bad-program 2>&1 ||:)
> $(info out = $(out))
> 
> $ make
> out = /bin/sh: bad-program: command not found

Sure, there are definitely workarounds: I suggested a different one in
my message.  I was trying to understand why the behavior happened, not
how to work around it.



Bug in $(shell ...) I can't understand

2022-02-06 Thread Paul Smith
OK, someone posted a question to SO and that led me to an hour or more
of banging my head against a wall trying to understand what's
happening... and I can't.

The problem is that the user would like to invoke $(shell ...) and
capture errors, even errors that the program being run doesn't exist. 
The shell function only captures stdout, not stderr.  This simple idea
won't work of course:

  out := $(shell bad-program 2>&1)
  $(info out = $(out))

  $ make
  /bin/sh: bad-program: not found
  out =

because this redirects the output of the bad-program, but it's the
shell printing the error not the program.  So I suggested this:

  out := $(shell $(SHELL) -c 'bad-program' 2>&1)
  $(info out = $(out))

This SHOULD work: the outer shell is invoking a sub-shell with the sub-
shell's stderr redirected to its stdout, then the subshell will print
the not found error to its stderr (redirected to stdout).

But it DOES NOT WORK!

  $ make
  /bin/sh: bad-program: not found
  out =

What is happening here?!?!  I wrote a little C program to verify my
thinking:

/* - */
#include 
#include 
#include 

int main(int argc, char** argv)
{
char *args[4] = {"/bin/sh", "-c",
 "/bin/sh -c bad-program 2>&1", NULL};

int pid = fork();
if (pid == 0) {
execv(args[0], args);
} else {
int st;
waitpid(pid, , 0);
}

return 0;
}
/* - */

But, this works as expected.  Something make is doing is causing
problems here.  I tried building both with and without posix_spawn and
both behave the same so it's nothing like that.

If you force the sub-shell to create its own subshell, then it works! 
This works:

  out := $(shell $(SHELL) -c 'bad-program | cat' 2>&1)
  $(info out = $(out))

  $ make
  out = /bin/sh: bad-program: not found




Re: [PATCH] RFC: add --shuffle[=seed] argument support

2022-02-05 Thread Paul Smith
Nice work!

On Sat, 2022-02-05 at 22:04 +, Sergei Trofimovich wrote:
> Some high level TODOs:

For this amount of change it's likely that you'll need to provide
copyright assignment paperwork.  Let me know if you'd like more details
about this.

> - No documentation for the optin yet.

This feature would also need a set of regression tests.

> - No(?) environment passing for recursive make.  I would prefer to
> share the same see across all calls to get easier reproducers.

You explicitly disabled this, though... was there a reason?

> - The dependency traversal is naive and uses potentially unbounded
> stack.
> - srand() / rand() is used from system libc.  This might make it
> harder to reproduce a failure on another machine.  But maybe it's
> fine.

There are a few issues here:

I recommend you try your version of GNU make on a bunch of different
real codebases and make sure it still works (and of course, create the
above-mentioned regression tests).

First, I think it's not correct to shuffle the goaldeps.  The goals
that are specified on the command line should always be invoked in the
order that the user requested.  It would be bad to convert:

  make clean all install

to:

  make install clean all

Second, you cannot rearrange the first prerequisite.  The first
prerequisite always must be placed in $<.  Consider the simple pattern
rule:

  %.o : %.c
  $(CC) $(CFLAGS) -c -o $@ $<

  foo.o: foo.c foo.h bar.h baz.h

If you rearrange the prerequisites to "bar.h foo.h foo.c baz.h" it will
not work well :).

Lastly, I'm not entirely sure that this is the best way to do the
shuffle.  Similar to my concern above about the first prerequisite,
this change will break makefiles that rely on the order of
prerequisites in perfectly legitimate ways; for example:

  foo%: arg%1 arg%2 arg%3 arg%4
  bld $< $(word 3,$^) $(word 2,$^) $(word 4,$^)

The concept you want to implement is not the shuffling of the actual
prerequisites, it's the shuffling of the BUILDING of the prerequisites.
The list of deps should not be modified but instead when make goes to
build the deps it should build them in a different order than they
appear in the prerequisites list.

Put another way, you don't want to change the structure of make's
dependency graph; you just want to change the order in which it's
walked.

If you do it this way you don't have to worry about any of the
reordering issues I raise above, because the values of $<, $^, etc.
won't change, and also you won't have to worry about deep recursions
etc. because you'll just be rearranging the targets that are being
built.

But, I think making it work this way will be trickier to code.



Re: Idea of triggering bugs in users' Makefiles

2022-02-04 Thread Paul Smith
On Fri, 2022-02-04 at 09:13 +, Sergei Trofimovich wrote:
> 1. Enable parallel builds by GNU make by default
> 
> 2. Do not run dependencies in deterministic order by default:

GNU make (unlike Ninja) is bound by the POSIX standard in terms of its
behaviors.  The POSIX standard makes very clear the way in which make
operates on prerequisites and neither of these changes are conforming
to the standard.

So, it's not possible to modify the default behavior of GNU make in
these  ways.  Not to mention that GNU make is also a foundational
component of hundreds of thousands of build systems and changes that
would cause a large number of them to break isn't acceptable, even in
pursuit of a good cause.

The idea of having a non-deterministic order of prerequisite builds is
something others have suggested but no one has implemented.  Of course
it would not be possible to move the first prerequisite as that one is
special but randomly rearranging the rest of them could be done.  This
would require an option to enable of course; it could not be the
default behavior.




Re: [BUG] --warn-undefined-variable is not triggered in prerequisites

2022-01-28 Thread Paul Smith
On Fri, 2022-01-28 at 01:09 +0100, Alejandro Colomar (man-pages) wrote:
> I'd like make to warn about this.  It took me a while to debug 
> a Makefile bug, which I thought was not happening, since make should
> have warned me.  Isn't this supposed to trigger the warning?

In previous versions of GNU make, the value of MAKEFLAGS set inside a
makefile is only re-parsed once before all makefiles are read (from the
environment) and again after all makefiles are read.

So, changes to this variable within the makefile don't take effect
until after all makefiles are parsed.  In your situation the variable
in question is part of a prerequisite list and that's expanded while
parsing the makefile, so the option is not set yet and you don't get
warnings.  Undefined variables used in a recipe WOULD be warned about
since recipes are expanded after all makefiles are parsed.

To make this option do what you want you must pass it in either via the
command line:

   $ make --warn-undefined-variables

or the environment:

   $ MAKEFLAGS=--warn-undefined-variables make

or:

   $ GNUMAKEFLAG=--warn-undefined-variables make

In the next release of GNU make, this has been changed; from the NEWS
file:

* If the MAKEFLAGS variable is modified in a makefile, it will be re-parsed
  immediately rather than after all makefiles have been read.  Note that
  although all options are parsed immediately, some special effects won't
  appear until after all makefiles are read.




Re: Paths above current/working directory don't work with suffix rules

2022-01-24 Thread Paul Smith
On Mon, 2022-01-24 at 21:27 +0100, Adam Tuja wrote:
> SRCS = a.c b.c ../c-1up.c
> OBJS = $(SRCS:.c=.o)
> 
> .c.o:
> $(CC) $(CFLAGS) -c -o $@ $<
> 
> $(PROG): $(OBJS)
> 
> Now, when I understand it, it works. It's not the same thing as
> objects are placed beside source files but step forward nevertheless.

Did you mean to write something different in your last sentence?  It 
IS the same thing as objects placed beside source files, unless I
misunderstand what you mean by "beside".

This makefile will always write the object files into the same
directory as the source files.  As long as you want to do that, then
you can use suffix rules or pattern rules very easily.

It's when you want object files to go into a different directory from
source files that suffix rules won't work well (you can use VPATH to
help somewhat).




Re: Paths above current/working directory don't work with suffix rules

2022-01-24 Thread Paul Smith
On Mon, 2022-01-24 at 19:09 +0100, Adam Tuja wrote:
> It doesn't matter how with many levels above current directory the
> path has (../ ../../ ...), it's still the same.

Suffix rules are very limited: this is why pattern rules were invented.

As long as you're happy with creating the object file in the current
directory, AND you're happy with ensuring that no two directories ever
have the same source file name, then it's possible to use suffix rules
to build them.

However you can see from the rule:

  .c.o:

that there's nothing here to tell make where the source file will be,
so clearly that cannot work by itself.  A suffix rule is identical to
the pattern rule:

  %.o : %.c

so, the prefix of the target and prerequisite MUST be identical
strings.  The string "c-1up" is not identical to the string "../c-1up", 
and so this rule cannot match.

You can use VPATH to give make a list of directories to use to search
for source files.  So, in your makefile if you add this:

  VPATH = .. ../..

when make wants to build c-1up.c it will look for ./c-1up.c, then
../c-1up.c, then ../../c-1up.c, and build the first one it finds.

https://www.gnu.org/software/make/manual/html_node/Directory-Search.html

> PS.
> Not to mention that if I don't specify target (all) then it only
> processes one file

Make has to choose SOME target to build by default, if you don't
specify one.  Make always chooses the first target in the makefile by
default.  So, move your "all" target to the top so it's the first
target and it will work as you expect.

https://www.gnu.org/software/make/manual/html_node/Goals.html




Re: Expansion of $(eval..)

2022-01-22 Thread Paul Smith
On Sat, 2022-01-22 at 13:59 +0100, Gisle Vanem wrote:
>define add_c_src> 
>  VPATH += $(1)
>  C_SRC += $(addprefix $(1)/, $(2))
>  $(info Number of 'C_SRC': $(words $(C_SRC)))
>endef
> 
>$(eval $(call add_c_src, frmts/ceos, ceosopen.c))
>$(eval $(call add_c_src, frmts/ceos2, ceos.c ceosrecipe.c ceossar.c))
> 
># ... plus a lot more
> 
> But I'm curious about how this gets expanded

It gets expanded identically to every other variable or function: left
to right, but from the inside out.

In this case you have two functions: $(info ...) and inside that you
have $(call ...).  Since $(call ...) is the inner function it's
expanded first.

That means the "first level" of expansion happens before eval is
invoked, and the variable is simply expanded as a text string with no
knowledge of makefile constructs.  Replacing $(1), running the
$(addprefix ..), AND RUNNING THE $(info ...) all happens here because
these are all simple references.

So the info is invoked by call before any of the variables are assigned
(because the assign is done by eval), then the result of that expansion
is handed to eval to evaluate.

You can figure out what eval will see by replacing it with info, like
this:

  $(info $(call add_c_src, frmts/ceos, ceosopen.c))

Now this will print the text that is passed to eval to be evaluated.

You probably want to escape the non-argument references, so that call
will not run them and instead they will be passed to eval:

   define add_c_src
 VPATH += $(1)
 C_SRC += $$(addprefix $(1)/, $(2))
 $$(info Number of 'C_SRC': $$(words $$(C_SRC)))
   endef





Re: $(info xxx) output interleaved with other $(info) output

2022-01-19 Thread Paul Smith
On Wed, 2022-01-19 at 15:49 -0800, Paul Eggert wrote:
> On 1/19/22 15:28, Paul Eggert wrote:
> > Proposed patch attached.
> 
> I see that patch's commit message has the wrong URL for the bug
> report.  Here's a fixed patch, attached. Only the commit message is
> changed.

I examined this method and in fact, it's not possible for it to be
called with anything other than exactly one argument so all the mess
around recombining multiple arguments is useless.

I rewrote this function and applied this change:

Author: Paul Smith 
Date:   2022-01-19 15:49:19 -0800

Avoid interleaved $(info ) output

Since $(info ) function output is written using two system calls for the
message and the newline, it's possible for output from other parallel
make job to sneak in between them.

Reported by Paul Eggert , who saw the report from
Lars Ingebrigtsen <https://bugs.gnu.org/53358>;.

* src/function.c (func_error): Paul provided a fix but instead I rewrote
the entire function: it's not possible for it to be invoked with
anything other than exactly one argument so don't bother concatenating
the arguments.

diff --git a/src/function.c b/src/function.c
index c27f573b..d7c13923 100644
--- a/src/function.c
+++ b/src/function.c
@@ -1171,41 +1171,25 @@ func_strip (char *o, char **argv, const char *funcname 
UNUSED)
 static char *
 func_error (char *o, char **argv, const char *funcname)
 {
-  char **argvp;
-  char *msg, *p;
-  size_t len;
-
-  /* The arguments will be broken on commas.  Rather than create yet
- another special case where function arguments aren't broken up,
- just create a format string that puts them back together.  */
-  for (len=0, argvp=argv; *argvp != 0; ++argvp)
-len += strlen (*argvp) + 2;
-
-  p = msg = alloca (len + 1);
-  msg[0] = '\0';
-
-  for (argvp=argv; argvp[1] != 0; ++argvp)
-{
-  strcpy (p, *argvp);
-  p += strlen (*argvp);
-  *(p++) = ',';
-  *(p++) = ' ';
-}
-  strcpy (p, *argvp);
-
   switch (*funcname)
 {
 case 'e':
-  OS (fatal, reading_file, "%s", msg);
+  OS (fatal, reading_file, "%s", argv[0]);

 case 'w':
-  OS (error, reading_file, "%s", msg);
+  OS (error, reading_file, "%s", argv[0]);
   break;

 case 'i':
-  outputs (0, msg);
-  outputs (0, "\n");
-  break;
+  {
+size_t len = strlen (argv[0]);
+char *msg = alloca (len + 2);
+strcpy (msg, argv[0]);
+msg[len] = '\n';
+msg[len + 1] = '\0';
+outputs (0, msg);
+break;
+  }

 default:
   OS (fatal, *expanding_var, "Internal error: func_error: '%s'", funcname);




Re: Invalid use of const pointer?

2022-01-19 Thread Paul Smith
On Tue, 2022-01-18 at 10:56 -0500, Joe Filion wrote:
> I could recommend changing line 557 to something like: 
> 
>   const char * cp = strchr (nptr, ‘%’);

Good idea, thanks.  I cleaned up this section of code.




Re: Linux /proc/loadavg

2022-01-19 Thread Paul Smith
On Sat, 2022-01-15 at 10:29 +, Sven C. Dack wrote:
> I have recently been looking into this again, because of a new
> feature I am currently testing, and so far can only repeat what I
> have already said in the past. The /proc/loadavg file under Linux
> with a vanilla kernel is behaving as expected, meaning, the number of
> active processes can very well exceed the number of cores.

I spent some time yesterday building a large codebase with different
options and configurations.

I agree with your assessment: the value here is the number of runnable
jobs, not the number of running jobs.  I'm not exactly sure where the
original issue came from, so I'll fix this for the next release.




Re: [PATCH] Fix nonzero detection in integer parsing

2022-01-17 Thread Paul Smith
On Thu, 2022-01-13 at 23:31 +0100, Jouke Witteveen wrote:
> I would like to draw attention to this patch again, since without
> it the intcmp function is misbehaving.

Thanks for reminding me: somehow the original got deleted from my inbox
without being applied.  I've now applied it to my Git repo and will
push shortly.




Re: Invalid use of const pointer?

2022-01-17 Thread Paul Smith
On Sun, 2022-01-09 at 20:02 -0500, Joe Filion wrote:
> If interested, I found another similar construct in another area of
> the code. Don’t worry, this appears to be the last one. 
> 
> On line 557 of implicit.c:
>   p = strchr (nptr, '%');
> nptr is a const pointer, but p is used as a non-const pointer later
> on and modifies the string.

I see this line, but I don't see where p is used to modify the string.

Maybe I am missing something?  Can you show where the modification
happens, not just the strchr()?




Re: .SILENT: clobbered by .SILENT: with_target

2022-01-17 Thread Paul Smith
On Wed, 2022-01-12 at 14:22 -0900, Britton Kerin wrote:
> > You can see that this example mimics your .silent example. 
> > Your makefile provided a prerequisite to .SILENT. Make then knows
> > that .SILENT has a prerequisite.
> 
> I agree that it's consistent syntax, but semantically it's bad.  The
> expectation and the manual both imply that .SILENT: will have a
> global effect, and a  real union-of-effects would have .SILENT:
> meaning everything and .SILENT: some_target redundant.  The root of
> the problem is that the rule syntax has been recycled for an
> unrelated purpose, especially so for directives without
> prerequisites.

Another option you can consider is to add:

  MAKEFLAGS += --silent

to enforce silent mode.

This doesn't address the issue you raise but maybe it will help with
the problem you're trying to solve.




Re: Invalid use of const pointer?

2022-01-11 Thread Paul Smith
On Tue, 2022-01-11 at 10:57 +, Edward Welbourne wrote:
> Indeed.  The compiler is allowed to place a string literal in read-
> only memory, where modifying it (even if you do "put it back the way
> it was" later) is an access violation.  Passing such a const char *
> to your function would thus crash.

Of course, I'm very well aware of all of this.

But, in fact, it's not possible for make to pass a string literal to
this function.  So your scenario will never happen.

As I've said in other messages in this thread, the right way to fix
this, if we wanted to fix it, would be to allocate separate memory and
copy the substring into that so that the original string would not be
modified.  It's the WRONG behavior to start de-constifying swaths of
the program to allow a low-level method to modify strings.




Re: Invalid use of const pointer?

2022-01-09 Thread Paul Smith
On Sun, 2022-01-09 at 11:06 +0100, Henrik Carlqvist wrote:
> On Sat, 08 Jan 2022 17:29:33 -0500 Paul Smith  wrote:
> > It turns out to be innocuous because none of the callers care that
> > the value of the input string is modified if we return a different
> > string, but it's still wrong and should be fixed.
> 
> If so, the easy and more correct fix might be to to remove const from
> the function variable declarations rather than to restore the value.

It's not that easy.  Removing the const from the argument means that we
have to change all the caller's types to remove const, and that means
all the callers of those methods have to remove const, etc.

In any event, the bug still exists whether you say the argument is
const or not: the expectation when this function is called is that
after it returns the string passed to it has the same content as before
it was called.




Re: Invalid use of const pointer?

2022-01-08 Thread Paul Smith
On Sat, 2022-01-08 at 22:34 +0100, Henrik Carlqvist wrote:
> But what about the case marked with <--X above? To me it seems as if
> the function returns after having modified const char *name bu using
> userend.

You're right, that's wrong.

It turns out to be innocuous because none of the callers care that the
value of the input string is modified if we return a different string,
but it's still wrong and should be fixed.

Thanks for noticing!




Re: Segafult while running make(1) from /lib/init/rc with -j

2022-01-08 Thread Paul Smith
On Sat, 2022-01-08 at 21:37 +0100, Alejandro Colomar (man-pages) wrote:
> Hi Dmitry,
> On 1/7/22 17:48, Dmitry Goncharov wrote:
> > On Thu, Jan 6, 2022 at 2:13 PM Alejandro Colomar (man-pages)
> >  wrote:
> > >I could try to write a simpler Makefile
> > That would be good. We need to be able to reproduce the crash.
> 
> I couldn't reproduce it with a simple Makefile with a few includes
> and a few sleeps.
> 
> Would you mind if I send you the script with which I generated the 
> Makefiles, and you run it in a virtual machine?

If it's not possible to reproduce the crash outside of your environment
then better would be for you to build GNU make with debugging enabled:

  make CFLAGS=-g

then run it so it crashes, then investigate the generated core file
with a debugger (gdb) and generate a stack trace to see where things
are crashing.




Re: Invalid use of const pointer?

2022-01-08 Thread Paul Smith
On Sat, 2022-01-08 at 19:47 +0100, Henrik Carlqvist wrote:
> On Sat, 08 Jan 2022 10:37:17 -0500
> Paul Smith  wrote:
> > The const-correct way to write this code would be to allocate a new
> > string and modify that
> 
> Another correct way to do this would be to not declare the input
> variable *name as const, but that would need to spread up to calling
> functions.

Indeed.  We explicitly don't want to do that.

> There are cases in tilde_expand when *userend is restored restored to
> '/' after having being altered to '\0'. In those cases at least no
> permanent changes has been made to the const string seen from the
> calling functions point of view.

Correct.

>  But now, with both userend and pwent set it seems as if the calling
> function will have its const string modified. If this final case were
> fixed at least no calling function would suffer from a modified const
> string.

I'm not sure what you mean here.  It is never the case that the
incoming string (name) is ever modified under any circumstances, as far
as the calling function is concerned.

If the incoming string needs to expanded then a new string is allocated
and returned from this function containing the expansion.  If the
incoming string is not expanded, then no new string is allocated and 0
(null pointer) is returned.




Re: Invalid use of const pointer?

2022-01-08 Thread Paul Smith
On Fri, 2022-01-07 at 18:28 -0500, Joe Filion wrote:
> Line 3094 of read.c is: 
> 
> char *userend = strchr (name + 1, '/'); 
> 
> The name parameter is a const pointer, so the overloaded version of
> strchr that takes a const pointer as the first parameter should also
> return a const pointer.

You might be thinking of C++.  GNU make is written in C, not C++, and
there's no such thing as an "overloaded" function in C.  There's only
one strchr() and it has this signature (according to the C standard):

  char *strchr(const char *s, int c);

>  But userend is not a const pointer and is used to modify the string
> later in the code. I’m a bit surprised the compiler allows this and I
> realize this could just be a misunderstanding of something on my
> part. Does anyone else find this construct suspicious?

It is kind of gross, yes.  It relies on some internal knowledge that,
in fact, the string passed in is never truly constant (that we never
pass a string constant to this function, that might have been stored in
read-only memory).

The const-correct way to write this code would be to allocate a new
string and modify that (or, rework the entire program to use the
equivalent of C++'s std::string_view), but the author of this code
(might be me, might be someone else: I didn't investigate) decided that
the quick and dirty way had enough benefits to outweigh the gross-ness.




Re: Oddness with intermediate files / .INTERMEDIATE

2021-12-30 Thread Paul Smith
On Wed, 2021-12-29 at 18:04 -0500, Paul Smith wrote:
> This is just not a good idea anyway, and now that we have the
> is_explicit value in the file structure we don't need it so I removed
> that code.  Unfortunately is_explicit is not correctly preserved in
> pattern_search() so I fixed this (plus the above tests) and now
> everything seems to pass.

I did push these changes yesterday.  It seems that the Git hook that
sends email was not working so no email went out about it but FYI.




Oddness with intermediate files / .INTERMEDIATE

2021-12-29 Thread Paul Smith
While looking through some old bugs in Savannah and trying to reproduce
them I ran across this change in behavior between GNU make 4.3 and
current Git master HEAD:

  $ cat Makefile
  1.all: 1.q ; touch $@
  %.q: 1.r ; touch $@
  %.r: ; touch $@

Note here, 1.r is mentioned explicitly in the makefile, but only as a
prerequisite of a pattern rule.  GNU make 4.3 treats this as an
intermediate file:

  $ rm -f 1.* ; make -r
  touch 1.r
  touch 1.q
  touch 1.all
  rm 1.r

In current master HEAD, it is not considered intermediate:

  $ rm -f 1.* ; ~/src/make/make -r
  touch 1.r
  touch 1.q
  touch 1.all

This seems OK, because the docs are pretty clear that if a file is
mentioned explicitly as a prerequisite it's not intermediate with no
special-casing of prerequisites of pattern targets.  In any event this
change needs to be noted in NEWS.

However, it seems that even forcing the file to be considered
intermediate does not work in master:

  $ cat Makefile
  1.all: 1.q ; touch $@
  %.q: 1.r ; touch $@
  %.r: ; touch $@
  .INTERMEDIATE: 1.r

  $ rm -f 1.*; make -r
  touch 1.r
  touch 1.q
  touch 1.all

Investigating it appears that the reason is that the 1.r target is
marked as secondary:

  1.r:
  #  Implicit rule search has been done.
  #  Implicit/static pattern stem: '1'
  #  File is an intermediate prerequisite.
> #  File is secondary (prerequisite of .SECONDARY).
  #  Last modified 2021-12-29 13:23:54.184652114
  #  File has been updated.
  #  Successfully updated.

This prevents it from being removed even though it's been marked
intermediate.

This happens here in pattern_search():

  /* We don't want to delete an intermediate file that happened
 to be a prerequisite of some (other) target. Mark it as
 secondary.  We don't want it to be precious as that disables
 DELETE_ON_ERROR etc.  */
  if (f != 0)
f->secondary = 1;
  else
f = enter_file (imf->name);

In previous versions of make, the then-branch is not taken here and
secondary is not set.  In master, it is.

This is just not a good idea anyway, and now that we have the
is_explicit value in the file structure we don't need it so I removed
that code.  Unfortunately is_explicit is not correctly preserved in
pattern_search() so I fixed this (plus the above tests) and now
everything seems to pass.

I suspect that we have too many options controlling explicit /
intermediate / secondary / notintermediate.  We should probably take a
comprehensive look at these values and minimize them if possible.




Re: escaped newline in macro expansion in command line

2021-12-29 Thread Paul Smith
On Fri, 2021-12-24 at 23:30 +, Humm wrote:
> (woops, sorry for replying off-list first; mutt doesn’t like me)
> 
> Quoth Paul Smith:
> > In your example the backslash is part of a variable expansion: it's
> > INSIDE the variable expansion so it will be handled by make as part
> > of expanding the variable and won't ever be passed to the shell.
> 
> POSIX.1-2017 says about this (98624-98631):

Yep, I'm quite familiar with the text.  I'm not entirely sure that this
particular usage was envisioned, where the backslash appears inside a
variable reference.





Re: escaped newline in macro expansion in command line

2021-12-24 Thread Paul Smith
On Fri, 2021-12-24 at 10:45 +, Humm wrote:
> Consider the Makefile:
> 
> .POSIX:
> M = word
> all:
> echo ${M:word=a\
> b}
> 
> I believe, as the escaped newline is in a command line, the expected
> behavior is to let the macro expand to it, and thus to run the
> command
> 
> echo a\
> b
> 
> with output `ab`.  The actual output is `a b`.

I understand what you're saying, but it's not true that the expected
behavior is to run the command:

  echo a\
  b

In your example the backslash is part of a variable expansion: it's
INSIDE the variable expansion so it will be handled by make as part of
expanding the variable and won't ever be passed to the shell.

It's handled as if you wrote:

  M = word
  _X = ${M:word=a\
  b

  all: ; echo ${_X}

Where you would expect to see "a b".

There is an argument to be made that the backslash should be managed as
if it were part of a command line and not part of a variable assignment
because it appears in a recipe, even though it's part of a make
variable expansion.  If the backslash were handled that way then make
would see this:

  echo ${M:word=ab}

after the backslash/newline was removed and it would run the command:

  echo ab

which is still not

  echo a\
  b

but would give equivalent output in this case.  I don't know offhand
what the right behavior is.  No version of make I've been able to find
actually works this latter way, removing the backslash first.





Re: make -j does not work on RedHat7.7 VM

2021-12-23 Thread Paul Smith
On Thu, 2021-12-23 at 12:49 +, Zhu, Mason wrote:
> In GNU Make 3.82, it seems that -j option will be finally added if
> Make determines my VM has the parallel build capability.

It has never been the case that any version of GNU make has
automatically enabled parallel builds by itself.  It's always been
required that the user request it via the -j option.

If you are seeing parallelism in 3.82 then someone provided -j on the
command line or in MAKEFLAGS before starting make.  I don't believe
that GNU make 3.82 allows -j to be added to MAKEFLAGS inside the
makefile and take effect in the current invocation of make.

> However in GNU Make 4.2.1, there is no parallel build if I does not
> explicitly set -jN option.

Does this mean that you discovered a way to make parallel builds work
as expected using GNU make 4.2.1?  Just by adding this argument?
 Exactly how did you do this?

If so we'll need to fully understand how you are enabling parallel
builds in GNU make 3.82 and what change you made to make it work in GNU
make 4.2.1.




Re: [PATCH] doc: note that $(wildcard) is implemented in terms of glob(3)

2021-12-20 Thread Paul Smith
Ævar Arnfjörð Bjarmason  writes:
> Aside: I found it difficult to figure out how to submit patches to
> this project. The Savannah page suggests the bug tracker, but as af
> writing (and I'm logged in, as "avar") that link appears greyed out
> in the web interface, and has a .

I added a note to the README file on submitting patches and a note to
the docs and README.git pointing there.

As for the bug tracker I can't explain why it appears greyed out for
you: it's never happened to me and when I visited just now without
being logged in I was still able to see the bug tracker.




Re: Compilation error with GCC

2021-12-19 Thread Paul Smith
On Sun, 2021-12-19 at 20:04 +0100, Jouke Witteveen wrote:
> The patches "file #52422" and "file #52428" should resolve your
> issues.

I pushed these changes to Git.




Re: Confusing message when make starts in current directory

2021-12-15 Thread Paul Smith
On Wed, 2021-12-15 at 17:59 +0100, Andrea Monaco wrote:
> In my opinion, those messages should always come with a chdir.

Well, make can't know whether the previous make started in the same
directory or not, at least not reliably.

If you do something like the very common, and POSIX-compatible:

   subdir:
   cd subdir && $(MAKE)

rather than the GNU make-specific "$(MAKE) -C subdir", then it's not
make doing the chdir, the shell does it before make is ever invoked.

I guess make could provide its current directory to its child make in
some kind of environment variable, then the child make could compare it
and omit the message if they were the same.

But that seems like a lot of work for a relatively cosmetic issue.

Also, it would mean that the recursive counter would have holes; that
is it might go from "make[1]" to "make[4]" if the 2 and 3 recursions
were in the same directory.  I don't know why that would be an issue
but could be...

Cheers!




Re: .SECONDEXPANSION problems

2021-12-13 Thread Paul Smith
On Sun, 2021-12-12 at 20:35 -0500, Dmitry Goncharov wrote:
> On Sun, Dec 12, 2021 at 2:15 PM Paul Smith  wrote:
> > Did something happen when it stopped working, like you updated to a
> > different version of GNU make?
> 
> i bet this make is built from the current git.

Thanks Dmitry.  I was fooled by the "since some time" text which made
it sound like this started happening quite a while ago.




Re: .SECONDEXPANSION problems

2021-12-12 Thread Paul Smith
On Thu, 2021-12-09 at 12:25 +0100, Gisle Vanem wrote:
> Since some time the cool '.SECONDEXPANSION' feature has stopped
> working for me.

Did something happen when it stopped working, like you updated to a
different version of GNU make?

>  In a Makefile, I have many rules to link module .DLLs:
> 
> My problem is that 'EXTRA_CFLAGS' is NOT picked up by the above
> pattern compile rule when the .SECONDEXPANSION rule is used. This
> used to work for sure.

I can't understand the piece of the makefile you provided.  I don't see
anything immediately wrong with it, but I haven't been able to
reproduce this problem in my own experiments.  Can you create a small,
self-contained example that shows the problem you're seeing?

> BTW. Perhaps my problem is related to:
>https://savannah.gnu.org/bugs/?60799

As far as I can see from your example you don't have either ";" or "#"
anywhere in any of your prerequisites so I don't see how that issue
would be related.




Re: [PATCH] doc: note that $(wildcard) is implemented in terms of glob(3)

2021-12-05 Thread Paul Smith
On Fri, 2021-12-03 at 22:10 +0100, Ævar Arnfjörð Bjarmason wrote:
> Would a patch that's updated to note that, and discusses the behavior
> in older versions be acceptable for inclusion?

I added a note about this to the docs.  Thanks for pointing it out!




Re: [PATCH] doc: note that $(wildcard) is implemented in terms of glob(3)

2021-12-03 Thread Paul Smith
On Fri, 2021-12-03 at 11:08 +0100, Ævar Arnfjörð Bjarmason wrote:
> The motivation for this patch is that I've seen code like this in the
> wild:
> 
> FILES = $(sort $(wildcard t*.sh))
> 
> I thought that it would be documented that that sort of thing was
> redundant, but I didn't find any mention of it in the manual. After
> some source spelunking I came up with this addition, which would have
> helped me when I consulted the manual.

Originally it was not _guaranteed_ that wildcard yields sorted output.

However, the output did happen to be sorted in older versions of GNU
make.  In GNU make 3.82, that was changed to yield unsorted output.  As
of GNU make 4.3, it was changed back and it is now guaranteed that
wildcard results are sorted:

https://savannah.gnu.org/bugs/index.php?52076

I thought that I had updated the documentation to make this guarantee
clear but perhaps I forgot?




Re: GPL Interpretation on load [Was: [bug #61594] suggest new $(hash ...) function]

2021-12-01 Thread Paul Smith
On Wed, 2021-12-01 at 09:33 -0500, rsbec...@nexbridge.com wrote:
> That is understood. Is this an official GNU Make policy because it is
> not specified that way in GPL. Has the GNU Make team modified their
> copy of the GPL license because it is not indicated as a modified
> version?

I'm not sure what you're asking here.  As Eli says, most GNU projects
that provide the ability to load modules like this, have a similar type
of restriction.

I don't see how this goes against the GPL license or would require an
extension or exception to the GPL license.  You, as the user of the
program, have received all your rights under the GPL.  There's nothing
in the GPL that says or implies that the implementation of a program
licensed under the GPL, must allow anything at all to be loaded
regardless of license.

Note that the GPL only applies to distribution, it doesn't apply to
use.  Just creating and using a plugin yourself doesn't mean it has to
also be provided to everyone under a GPL-compatible license. It's only
if you distribute the plugin that the GPL comes into play.  Whether
this is actually legally enforceable is not something I am qualified to
discuss... this is the way the GNU project wants plugins to be handled.

If you want to understand more about the process and philosophy here,
you can read about it in the GCC plugins section (GNU make uses the
same thinking):

https://gcc.gnu.org/onlinedocs/gccint/Plugin-API.html
https://gcc.gnu.org/wiki/GCC_Plugins




Re: [PATCH] Fix type errors in win32.

2021-11-28 Thread Paul Smith
On Mon, 2021-10-25 at 15:32 +0300, Eli Zaretskii wrote:
> the default C runtime used by Make doesn't support %llu, at least not
> on Windows versions older than Windows 10.

Hrm, I just added a new use of sprintf() with %lld into GNU make, to
format a long long variable.  It worked for me in my trivial Windows
tests but I am using Windows 10 with MSVC 2019 so pretty new.  I don't
have the facilities to test older systems.

Should I be using %I64d instead on Windows, to allow support for older
systems?




Re: [PATCH 3/3] Introduce $(compare ...) for numerical comparison

2021-11-28 Thread Paul Smith
On Sun, 2021-11-28 at 15:49 +0100, Jouke Witteveen wrote:
> I fully agree, but was not aware of the robustness of INTSTR_LENGTH.
> It felt a bit fragile when I spotted its definition in makeint.h.

The C standard defines the largest unsigned long long value
as 18446744073709551615, the largest signed long long value
as 9223372036854775807, and the smallest signed long long value as -
9223372036854775808.  So, the definition cannot be wrong in any
standards-conforming implementation of C.

> File sizes are an interesting application indeed. If you want me to
> change the patches to use strtoll, I would need some help since I am
> not sure how to set things up so that we get a fallback
> implementation on platforms where it is missing.

I already did it locally, thanks for the offer.  I enhanced the tests
to check that the above (signed) values are parsed correctly.

I just changed the gnulib import of "strtol" to "strtoll".  strtoll()
is required by the C99 standard (which we were forced to require in GNU
make as well, since gnulib requires it) so it's pretty much everywhere
these days.  There are a few older platforms where it has incorrect
implementations that gnulib will take care of for us.

Cheers!




Re: [PATCH 3/3] Introduce $(compare ...) for numerical comparison

2021-11-28 Thread Paul Smith
On Sun, 2021-11-28 at 08:24 +0100, Jouke Witteveen wrote:
> On the user side, strcmp could now probably be defined something like
> $(and $(intcmp $(words $1),$(words $2)),$(findstring x$1x,x$2x))

I don't think this is equivalent since a putative strcmp would also do
greater / less than comparison (like intcmp does).  Of course that
leads into all sorts of i18n / locale issues, but likely we'd just punt
(like C's strcmp does) and use ASCII ordering; we've already taken that
route elsewhere.

You can probably gin up something that works using $(sort ...) but it
seems tricky.  Anyway I'm not suggesting this be added right now but
just keeping options open for the future.




Re: [PATCH 3/3] Introduce $(compare ...) for numerical comparison

2021-11-28 Thread Paul Smith
On Sun, 2021-11-28 at 14:57 +0100, Jouke Witteveen wrote:
> > Since the two arguments are equal, it doesn't matter which of LHS
> > or RHS we return.
> 
> They could differ for instance when one of them contains a '+'-sign.
> My reason for using LHS is that we already have a string for it.

I don't think that it's necessary return the exact string.  If the user
wanted a string match they can do that other ways.  Returning the
"absolute value" (stripping leading +/-, leading 0's, etc.) seems more
useful to me.

>  By using sprintf, we need to make the buffer big enough, which in
> turn requires INTSTR_LENGTH to be fitting for whatever width a long
> has on the current platform.

That's already the case; GNU make defines this constant to be large
enough to hold the largest possible (8-byte) integer value, regardless
of platform.  This is used in various places when showing output etc.

> > However, now that I think about it I need to change the code more: we
> > need to be using "long long" here not just "long".  While on Linux etc.
> > a "long" is 8 bytes, on Windows "long" is only 4 bytes.
> 
> I was hoping this would not be necessary, and I cannot think of a
> typical use case where make is a good fit for dealing with large
> integers. The benefit of "long" is that strtol is more widely
> available than strtoll.

I see what you mean, but I _really_ don't like the idea of GNU make
working differently on different platforms, even if such use cases are
rare.  I can imagine a situation where, for example, someone wants to
compare the sizes of files and if one of the files is >4G then it will
work on Linux and fail on Windows.




Re: [PATCH 3/3] Introduce $(compare ...) for numerical comparison

2021-11-28 Thread Paul Smith
On Sun, 2021-11-28 at 08:24 +0100, Jouke Witteveen wrote:
> Thanks for sending this message, I would have otherwise prepared and
> sent an updated patch series today. My plan was to expand to RHS in
> the two-argument case if both values are equal. I assume you also
> updated the documentation where needed? If so, there's nothing I have
> to add and I'm looking forward to the new functionality.

Yes, I updated the docs.

Since the two arguments are equal, it doesn't matter which of LHS or
RHS we return.

The change I made was:

  argv += 2;

  if (*argv == NULL)
{
  if (lhs == rhs)
{
  char buf[INTSTR_LENGTH+1];
  sprintf (buf, "%ld", lhs);
  o = variable_buffer_output(o, buf, strlen (buf));
}
  return o;
}

However, now that I think about it I need to change the code more: we
need to be using "long long" here not just "long".  While on Linux etc.
a "long" is 8 bytes, on Windows "long" is only 4 bytes.




Re: [PATCH 3/3] Introduce $(compare ...) for numerical comparison

2021-11-27 Thread Paul Smith
On Mon, 2021-11-15 at 20:49 +0100, Jouke Witteveen wrote:
> It may be obscure, but how about we implement this as well? Sure, the
> two-argument form of $(compare) will be a little inconsistent, but it
> may be useful.

I applied this three-patch set.

I left the argument order as you originally specified it, and I added
support for a 2-argument form that provides a trivial equality as
above.  Maybe it's weird but I think it might be useful.

The one change I made is changing the name of the function from
"compare" to "intcmp": I felt "compare" was too generic.  Sometime we
may want to introduce other types of comparison such as "strcmp" etc.

Let me know what you think about this: I've only applied it to my local
Git repository but not pushed it yet.




Re: [PATCH 3/3] Introduce $(compare ...) for numerical comparison

2021-11-14 Thread Paul Smith
On Wed, 2021-11-10 at 18:19 +0100, Jouke Witteveen wrote:
> On Mon, Nov 8, 2021 at 4:08 PM Paul Smith  wrote:
> 
> > On Fri, 2021-07-16 at 14:04 +0200, Jouke Witteveen wrote:
> > > $(compare 
> > > @var{lhs},@var{rhs},@var{lt-part}[,@var{eq-part}[,@var{gt-part}]])
> > Let me ask this: would it be better to use a format of:
> >$(compare , , [, [, ]])
> 
>   $(if $(compare ,,,they_are_equal,),,)
> 
> In fact, in the original design you could get away with just the
> three-argument version of $(compare) in combination with $(if),
> $(and) and $(or). This is not the case for the alternative design.

Can you explain this, perhaps with an example or two?  I don't quite
follow.  It seems that:

  $(if $(compare ,,equal),,)

works fine with the alternative syntax.

It's even possible to allow $(compare ,) with no other
arguments and say that if they are equal then it expands to the value,
else it expands to the empty string, to give a very short-circuited
equality statement.

I do agree, though, that we already have a way to test equality (via
filter-out) and although it will treat "1" and "01" as different this
is probably a rare problem.

> I also took a look at other languages. Nearly everywhere I could find
> three-way comparisons, the possible outcomes are listed as LT, EQ,
> GT, in that order.

I'm certainly willing to be convinced that the original idea is better,
but this argument doesn't hold much water for me.  I don't think that
changing the order of the extra arguments will cause much cognitive
dissonance for people.




Re: [PATCH 3/3] Introduce $(compare ...) for numerical comparison

2021-11-08 Thread Paul Smith
On Fri, 2021-07-16 at 14:04 +0200, Jouke Witteveen wrote:
> +@item $(compare 
> @var{lhs},@var{rhs},@var{lt-part}[,@var{eq-part}[,@var{gt-part}]])

Let me ask this: would it be better to use a format of:

  $(compare , , [, [, ]])

Then the rule is, if the values are equal we get the  part, if lhs
is less than rhs we get , and if lhs is greater than rhs we get
.

If  is not present then the invocation devolves to:

  $(compare , , , )

that is, the fourth arg is used for not equal.

If  is also not present then  is used if the value is equal
else it expands to the empty string.




Re: [bug #61226] A regression prevents generation of missing included dependency files.

2021-10-25 Thread Paul Smith
On Mon, 2021-10-25 at 17:51 -0400, Dmitry Goncharov via Bug reports and
discussion for GNU make wrote:
> On Monday, October 25, 2021, Alejandro Colomar (man-pages) <
> alx.manpa...@gmail.com> wrote:
> > Why do I do this?  Because, if you remove a file from your tree, an
> > old .d file will require that file and make your build fail,
> > requiring you to clean before making again.
> 
> This has been solved long ago. Check the awesome advanced dependency
> generation article by Paul. 

FYI:

  http://make.mad-scientist.net/papers/advanced-auto-dependency-generation/

The issue you raise (as well as others) is discussed, hopefully
clearly, in this article [*].


[*] 
http://make.mad-scientist.net/papers/advanced-auto-dependency-generation/#depdelete




Re: -V, --verbose, as opposite of -s, --silent, --quiet

2021-10-25 Thread Paul Smith
On Mon, 2021-10-25 at 22:28 +0200, Alejandro Colomar (man-pages) wrote:
> Since I use '--warn-undefined-variables', and also to help
> readability to someone who may not know about this usage and may
> wonder what does that $(V) mean, I used the following:
> 
> V :=
> $(V).SILENT:

Looks good.

Something you might consider is using:

  V ?=

instead of ":=" which would allow someone to run:

  V=1 make

successfully (or put it into the environment in some other way).




Re: [bug #61226] A regression prevents generation of missing included dependency files.

2021-10-25 Thread Paul Smith
On Mon, 2021-10-25 at 10:11 +, Edward Welbourne wrote:
> Dmitry Goncharov (17 October 2021 18:33) wrote:
> > i think, make should not print a warning when a .d file is missing.
> > make should proceed and create the missing file. However, when .d
> > is present, but make cannot include it, then make should print an
> > error and stop.
> ...
> 
> Surely that would solve your problem, without invasive surgery in GNU
> make, or special-case handling of .d files anywhere but your make
> file ?

Two points:

First for Eddy's response: the issue we are thinking about isn't how
one person could rewrite makefiles to work differently: the issue is
that a somewhat serious backward compatibility break has been
introduced that could break thousands of makefiles, and we have to
resolve that before the next release.

Second for Britton's response: as far as I can tell no one is
suggesting that make would check the name of an included file and if it
ended in ".d" make would behave differently in some way than if it were
named ".od" or whatever.  Certainly I hope we all agree that kind of
check is not appropriate.  If you check his patch you'll see the change
Dmitry proposed was different: it didn't check the name of the target,
it checked to see if the recipe of the target was empty.  That is, it
would distinguish between this:

   include foo.xx
   foo.xx:

and this:

   include foo.xx
   foo.xx: ; : nothing

In both cases "foo.xx" is not created but (with Dmitry's change) the
first case would not give an error while the second one would give an
error, because it has a recipe.

The _name_ of the target is irrelevant.




Re: -V, --verbose, as opposite of -s, --silent, --quiet

2021-10-23 Thread Paul Smith
On Sat, 2021-10-23 at 03:01 +0200, Alejandro Colomar (man-pages) wrote:
> I'd like a project to use '--silent' by default, to have readable
> output, and hide most of the full commands, which would be too
> noisy. 
> 
> So, ideally, I'd like to have 'MAKEFLAGS += --silent' in the
> Makefile.

Actually what you really want is the .SILENT special target.

https://www.gnu.org/software/make/manual/html_node/Special-Targets.html#index-_002eSILENT

That leads to this best-practice way to handle verbose output:

http://make.mad-scientist.net/managing-recipe-echoing/




Re: Make language

2021-10-12 Thread Paul Smith
On Tue, 2021-10-12 at 14:12 +, Bartol Hrg wrote:
> I'm really displeased and agitated.

I hope that letting us know what you saw helps you feel better!




Re: Compilation error on master branch (Commit c5d4b7b)

2021-10-03 Thread Paul Smith
On Sun, 2021-10-03 at 23:23 +0200, Mohammad Akhlaghi wrote:
> I then ran the standard './configure' and 'make' commands to build
> it, but the build failed with the attached error messages when
> building 'src/guile.c'.

These errors appear to be coming from compiling src/read.c, not
src/guile.c.

I strongly suspect these are false-positive warnings but I'll take a
look.




Re: Parallel job instance identifiers?

2021-09-06 Thread Paul Smith
On Fri, 2021-08-13 at 14:14 +0100, Howard Chu via Bug reports and
discussion for GNU make wrote:
> I'm looking for a way to expose a job ID number to the individual
> jobs, in a consecutive range from 1-[number of jobs]. Not just
> unique numbers, for that we could just use $$ already. The purpose
> is to e.g. assign non-overlapping network port numbers when firing
> off a number of client/server test scripts in parallel.
> 
> Is there anything that could do this now, exposed as a Make variable?

I don't see any way make can provide something like this, at least not
using the model it does today.  The current requirement is that each
read from the pipe is of exactly 1 byte and there's no way to guarantee
a unique value in a single byte of data if there are >256 possible
values of course.

One possibility would be for make to write out bytes with values
0,1,2,3... then when it gets to 255 start over again: this would give
unique values for any -j <= 256 and you'd only have to worry about
wrapping for very large values of -j.

One reason I'm not excited about this idea is that I've been seriously
contemplating switching from the current pipe-based jobserver control
method, to using POSIX named semaphores instead (at least as the
default).  There is support for these in most systems these days, and
it's already how Windows works (with the Windows equivalent).  The
advantage here is it's just easier in lots of ways to not have to pass
open file descriptors between make invocations.  I could use a named
pipe but semaphores are more obviously built for this.  But of course
there's no possibility of uniquely identifying a semaphore lock.

The other downside of using a semaphore is I was thinking of
implementing a "fast fail" feature where as soon as any make in the
jobserver domain died all the other makes would stop quickly as well.
 This is something people seem to really want.  One way to implement
this is to write "error bytes" into the jobserver pipe when we've hit
"failure mode"; whenever a make instance reads the "error byte" instead
of the "normal byte" it would finish any outstanding jobs then stop.

This is pretty straightforward although it's not ideal: make can
potentially do a lot of work with its single token without trying to
read from the jobserver for example.

Of course a single semaphore can't help here.  If we go with "named X"
we'll need a directory to hold it; one idea would be to write a
"failed" file into that directory that all make's could check for.




Re: Report 3 UBSan integer overflow bugs found by an automatic fuzzer

2021-09-05 Thread Paul Smith
On Wed, 2021-06-30 at 17:34 +, He  Jingxuan wrote:
> We tested GNU make with an automatic tool (based on the fuzzer AFL).
> A number of test cases triggering UBSan integer overflow errors were
> generated. We manually checked those test cases and filtered out
> benign cases. Finally, we identified and report 3 cases that could
> trigger bugs. Below is the information for reproducing the bugs.

I made changes to resolve these issues.  Thanks!




Re: GNU make man page typo

2021-09-05 Thread Paul Smith
On Thu, 2021-09-02 at 20:37 +, John Marshall wrote:
> On 2 Sep 2021, at 06:40, Zach Petch wrote:
> > The second paragraph under DESCRIPTION reads,
> > 
> > > To prepare to use make, you must write a file called the makefile
> > that describes the relationships among files in your program, and
> > the states the commands for updating each file.
> 
> The same text was previously in make.texi but was updated (rather
> astonishingly) 30 years ago in 45fa41f:
> 
>  To prepare to use @code{make}, you must write a file called
>  the @dfn{makefile} that describes the relationships among files
> -in your program, and the states the commands for updating each file.
> +in your program and provides commands for updating each file.

I've applied this change to the man page, thanks!




Re: [bug #60774] make hangs on fcntl lock when using -O and stdout is /dev/null

2021-07-26 Thread Paul Smith
On Sun, 2021-07-25 at 17:29 -0700, David Boyce wrote:
> It's simple to detect this on a Unix-like host (see sample script
> below) and my contention is that this is "plenty portable enough"

Ah.  Yes, determining if stdout is /dev/null can be done.  I was
definitely having an off day yesterday and forgot about fstat()!

Thanks for that reminder.




Re: [bug #60774] make hangs on fcntl lock when using -O and stdout is /dev/null

2021-07-25 Thread Paul Smith
On Sun, 2021-07-25 at 16:04 -0400, Dmitry Goncharov wrote:
> > It's clear that /dev/null is a special case, however, because the
> > order in which data is written to it cannot be detected so what's
> > the point in preserving any order?
> 
> Maybe Mike can tell us.

Based on the bug report I don't think Mike knows.  It didn't appear to
me that he had purposefully locked /dev/null, or was even aware that it
was locked.  Is there something else on his system that's doing it?  Is
it a change to the kernel that cause locks here to hang?  We don't
know.

> > What I'm saying is that IF make can detect that its stdout is going
> > to /dev/null then it shouldn't lock at all, because it's not
> > necessary to do so in order to meet the goals of the -O option, and
> > doing so introduces unnecessary latency in the build.
> 
> Even if we come up with a portable mechanism to detect /dev/null,
> what about other files? What about /dev/stdout or /dev/pts/5? i don't
> think, it is possible or even reasonable to attempt to special case
> all these files.

What I'm trying to say is that even if we do special-case /dev/null we
definitely shouldn't special case other files, because /dev/null is
truly a unique situation which is fundamentally different from
/dev/pts/5 etc., and is deserving of special handling.

However, it's moot because I don't think we can detect it.

> > The  question is what to do about supporting -O on these systems.
> A private temp file will work.

Yes.  It's good to have that capability.

Another option would be to use a POSIX named semaphore.  The advantage
of this is that you don't have to worry about temporary file creation,
permissions, and cleaning up.





Re: [bug #60774] make hangs on fcntl lock when using -O and stdout is /dev/null

2021-07-25 Thread Paul Smith
On Sun, 2021-07-25 at 15:25 -0400, Paul Smith wrote:
> There's no reason that those two disjoint sets of output need to be
> synchronized WITH EACH OTHER to meet the goal of the -O option. Given
> the choice between allowing output to go to these two different
> locations in parallel vs. serializing them with each other, I don't
> think anyone would prefer the latter.

In any event, what I'm really trying to say is this: locking stdout
rather than some other way of lock sharing was done quite deliberately
and intentionally and does provide tangible benefits that can't be
easily duplicated using other locking methods.  It wasn't just a
careless implementation choice.

Maybe those benefits don't outweigh the problems they cause, or maybe
they do.  But either way it needs to be considered.




Re: [bug #60774] make hangs on fcntl lock when using -O and stdout is /dev/null

2021-07-25 Thread Paul Smith
On Sun, 2021-07-25 at 13:11 -0400, Dmitry Goncharov wrote:
> On Sun, Jul 25, 2021 at 10:03 AM Paul Smith  wrote:
> > Writing to things like /dev/lp0 SHOULD be locked, IMO: you don't
> > want multiple concurrent writers to your printers or your terminal!
> 
> The user intentionally redirects make to a printer along with another
> tool. Should make be smarter than the user and refuse?

I suppose you mean in the same way Mike reported, the "other tool"
locks the device (e.g. printer) and now if you use -O then make's lock
prevents make from writing to it (or vice versa).

I have no problem with make refusing to write to a device that's locked
in general.  If users don't want that to happen, they can just not use
the -O option.  Or if they do want to use the -O option and they really
do want to write to the device even if it's locked, it's trivial enough
to hide it from make by introducing a intermediate step, like this:

  $ make | cat > /dev/lp0

It's clear that /dev/null is a special case, however, because the order
in which data is written to it cannot be detected so what's the point
in preserving any order?

> > To me the most compelling reason to change the current locking
> > behavior is not this: I agree with David that simply special-casing
> > /dev/null could be a good solution; if you're redirecting to
> > /dev/null why even HAVE a lock?
> 
> Mike wanted to specify -O and redirect to /dev/null. Again, why
> should the tool be smarter the user?

What I'm saying is that IF make can detect that its stdout is going to
/dev/null then it shouldn't lock at all, because it's not necessary to
do so in order to meet the goals of the -O option, and doing so
introduces unnecessary latency in the build.

The problem is, I don't know of any portable way of determining whether
stdout is going to /dev/null, or not.  The only way I know of to do it
is to check /proc//fd/1 and that's pretty much just available in
Linux (and requires you to have procfs mounted which not all Linux
systems do).

> > The most compelling reason to change the current behavior is that
> > unfortunately BSD-based systems don't support locking file
> > descriptors that aren't real files: so if you pipe make's output to
> > something on a BSD-based system like MacOS or FreeBSD, for example,
> > you get a ton of error messages.
> 
> One mechanism to avoid these error messages would be to stat stdout
> and check S_IFREG.

Certainly I didn't mean to suggest that detecting this situation is
hard (actually the simplest way is to just check the error code when we
try to take the lock and "do something else" if it fails).  The
question is what to do about supporting -O on these systems.

> >  submake: ; $(MAKE) -C $@ >sub.out
> > The way the implementation is today, the outer make and the submake
> > magically have different "output lock domains",
> 
> This is an interesting situation.
> 
> i considered this behavior to be a violation of the contract. The
> make manual says that when -O is specified the output of the parent
> and children is synchronized.

I don't agree; the -O option tells make that the user wants to see the
output of each rule not mixed with the output of other rules.  That's
all it means: any other deeper or more technical meaning is an accident
of the current implementation and can't be relied on.

Here the output to stdout is synchronized, and the output to the
redirected file is synchronized.

There's no reason that those two disjoint sets of output need to be
synchronized WITH EACH OTHER to meet the goal of the -O option.  Given
the choice between allowing output to go to these two different
locations in parallel vs. serializing them with each other, I don't
think anyone would prefer the latter.

> make is a portable tool and this magic does not happen on windows.

That's true but there are, unfortunately, some edge cases where Windows
and POSIX systems behave differently.  As long as the builds succeed in
largely similar ways and the edge cases don't cause incorrect builds,
we live with them.

My goal is not to provide an implementation of make which supports only
the least common denominator of all platforms.  The goal is to provide
the best implementation of make for the GNU system, and provide best-
effort support for other platforms that people care about as well.





Re: [bug #60774] make hangs on fcntl lock when using -O and stdout is /dev/null

2021-07-25 Thread Paul Smith
On Sat, 2021-07-24 at 20:35 -0400, Dmitry Goncharov wrote:
> On Sat, Jul 24, 2021 at 3:02 PM David Boyce 
> wrote:
> > I can’t think of a file other than /dev/null which would
> > appropriately be shared with unrelated processes in a “w” (write)
> > condition.
> 
> /dev/zero, /dev/lp0, /dev/lp1, /dev/pts, etc.

/dev/zero should not be written TO, it's only read from.  Writing to
things like /dev/lp0 SHOULD be locked, IMO: you don't want multiple
concurrent writers to your printers or your terminal!

To me the most compelling reason to change the current locking behavior
is not this: I agree with David that simply special-casing /dev/null
could be a good solution; if you're redirecting to /dev/null why even
HAVE a lock?  Assuming you can determine this it would be better to
simply turn it off and avoid the overhead.

The most compelling reason to change the current behavior is that
unfortunately BSD-based systems don't support locking file descriptors
that aren't real files: so if you pipe make's output to something on a
BSD-based system like MacOS or FreeBSD, for example, you get a ton of
error messages.

> i recognize the simplicity of using stdout. However, i also dislike
> adding pieces of code for a set of special files. The user will
> always find a way to screw it.

Using stdout was chosen intentionally because it gives one big
advantage, which is why I've struggled with the idea of replacing it:

It gives a simple and elegant solution to the situation where a
submake's output is redirected to a different file.  Suppose you have
this:

submake: ; $(MAKE) -C $@ >sub.out

The way the implementation is today, the outer make and the submake
magically have different "output lock domains", because their stdout
goes to different files: the submake doesn't have to wait for the lock
of the outer make to write content to its output, and vice versa.  This
is a huge benefit, and I'm loathe to give it up.




Re: [bug #60798] Make does not compile with GCC 11.1.0

2021-06-19 Thread Paul Smith
On Sat, 2021-06-19 at 16:23 +0100, Dmitrii Pasechnik wrote:
> > I don't think I understand.
> 
> p[] occupies a continuous memory area, starting from p[0], but p[-1]
> is not guaranteed by the standard to be accessible for you (well,
> unless you control the memory layout by placing p in a struct, not as
> the 1st member, but then why would you need p[-1] in the 1st place?)
> 
> What's unclear about it?

The problem is that your initial premise is not correct.  p here is not
an array, it's a pointer.  If we had declared it like this:

char p[10]

then p[-1] would be problematic.  But, we don't declare that.

We declare it like this:

char *p

so p is a pointer.  It points to some memory, and it can (and does)
move both forward AND backward through that memory, via p+X and p-X.
That's perfectly legitimate and not illegal or a hack in any way.

I guess the compiler believes that it's possible that p might point to
the very start of some legitimate memory, and if that were true then
p[-1] (or, equivalently, *(p-1)) would be undefined behavior.

But, as human programmers we can examine this code and understand that
actually, it's never possible for p to point to the first character of
the array: we know that eval_strings->idx is never 0, so we know that p
will always be incremented past the beginning of the memory buffer:

  p = value = alloca (len);
  for (i = 0; i < eval_strings->idx; ++i)
{
...
  *(p++) = ' ';
}
  p[-1] = '\0';




Re: [bug #60798] Make does not compile with GCC 11.1.0

2021-06-19 Thread Paul Smith
On Sat, 2021-06-19 at 15:31 +0100, Dmitrii Pasechnik wrote:
> On Fri, Jun 18, 2021 at 12:53:32PM -0400, Paul D. Smith wrote:
> 
> > Follow-up Comment #1, bug #60798 (project make):
> > FWIW, this warning is not valid in this situation.  The code is
> > correct; p will always be pointing into a valid buffer and never
> > pointing at the first character in that buffer, so p[-1] always
> > points to valid memory.
> 
> It's undefined behaviour in C to point to such a location, isn't it?

To point to which such location?  As I said, we know for a fact that p
does not point at the first character of the array.  It's clearly legal
in C to walk backwards through an array!!  And referencing the previous
element in an array is certainly not a hack.

I don't think I understand.




Re: shell assignment operator documentation missing 'not'

2021-05-30 Thread Paul Smith
On Tue, 2021-05-25 at 10:33 +, Ronald Hoogenboom wrote:
> In the info file documenting the shell assignment operator in section
> 3.7 "reading makefiles", there is the phrase:
> 
> "...that variable becomes a simple variable (and will thus be re-
> evaluated on each reference)." 
> 
> A simple variable is NOT re-evaluated on each reference, so the word
> "not" is missing in the parenthesized part.

Actually, it's correct to not have the "not" there: the problem is that
the variable is a recursive variable not a simple variable.

If it were a simple variable then the two assignments ":=" and "!="
would be identical in behavior.

Thanks for pointing out this error in the documentation!




Re: basename function in 4.3 cygwin

2021-05-21 Thread Paul Smith
On Fri, 2021-05-21 at 08:31 +, Ronald Hoogenboom wrote:
> The difference happens when a suffix contains one or more
> backslashes. This is sometimes needed to escape special behavior of
> meta characters in the shell. The basename function in 3.81 would
> return everything up to the last period like documented in the info
> file, but the basename function in 4.3 apparently considers the
> backslash in the suffix as a path separator (I guess...).

Can you provide a repro case?

I don't don't know much about Windows, but I don't understand how you
can tell the difference between a "suffix containing backslashes"
versus a directory that contains a suffix.

Is it not allowed for a directory to contain a suffix in Windows or
something?

E.g., is "C:\foo\bar.biz\baz" not a valid file named "biz" in a
directory named "C:\foo\bar.biz" ?




Re: basename function in 4.3 cygwin

2021-05-21 Thread Paul Smith
On Fri, 2021-05-21 at 08:31 +, Ronald Hoogenboom wrote:
> I have noticed a nasty difference in behavior between 4.3 and 3.81
> versions of gnu-make in cygwin.
> 
> I’m not sure if this qualifies as a bug per-se, but it is a nasty
> difference in behavior anyway. At least this should be documented in
> the info file.

We don't develop or support (or document) the cygwin port of GNU make
on this list.  If you can reproduce this behavior on the native Windows
port of GNU make (see the README.W32 for how to build it) then we will
definitely want to fix it.

If it is only reproducible in the cygwin port but not the native
Windows port, please file an issue with the Cygwin development team.

Thanks!




Re: [bug #60538] Environment variable overrides explicit assignment

2021-05-06 Thread Paul Smith
On Thu, 2021-05-06 at 17:11 +, Martin Dorey wrote:
> So MAKEOVERRIDES is exported, though perhaps it doesn't need to be,
> but isn't expanded before being exported.

Oh duh!!  I didn't provide any overrides.

You're right, it is exported.  OK I guess I'll have to go back and look
at the code but it will not be today.




Re: [bug #60538] Environment variable overrides explicit assignment

2021-05-06 Thread Paul Smith
On Thu, 2021-05-06 at 17:11 +, Martin Dorey wrote:
> > The manual doesn't actually say that MAKEOVERRIDES is exported like
> MAKEFLAGS but I assume it is.
> 
> I was going to just say that MAKEFLAGS would be expanded before being
> exported, and indeed it is, but, having tried it, the results aren't
> quite what I would have expected, even having just skimmed the
> section on MAKEOVERRIDES that Paul just cited.
> 
> $ make TESTNAME=badger 2>&1 | grep MAKE
> MAKEFLAGS= -- TESTNAME=badger
> ... eliding mentions of MAKE in my environment already...
> MAKEOVERRIDES=${-*-command-variables-*-}
> MAKELEVEL=1
> $ 
> 
> So MAKEOVERRIDES is exported, though perhaps it doesn't need to be,
> but isn't expanded before being exported.

Without seeing your makefile I can't say what you're seeing but I don't
see it exported:

  $ echo 'all:;@echo "mo=$$MAKEOVERRIDES"' | make -f-
  mo=





  1   2   3   4   5   6   7   8   9   10   >