Re: [PATCH v2 1/6] * expand.c (variable_name_extract): extract variable name and strip prefix.
Hi, Paul and other developers,
2014-08-18 22:35 GMT+08:00 Paul Smith psm...@gnu.org:
On Mon, 2014-08-18 at 21:27 +0800, Macpaul Lin wrote:
Variables used in conditional lines usually has '$',
'(', and ')' prefix, and etc.
We can use vairable_name_extract() to extract pure variable
name without these prefix.
Hello. Thanks for your work on GNU make!
Can you provide some sort of summary of the feature you've created, what
its goals are, examples, etc.?
It can be helpful to contact me or someone on the development team
before (or while) doing feature work for GNU make. As a GNU project,
for example, we'll need to get copyright assignment paperwork for any
contribution of significant size and this takes some time.
Also, we may have suggestions or alternative implementations or concerns
that should be considered. Of course these can always be addressed
later, but it can save some effort to think about them up-front.
And finally, note that fully-formed changes need to have at least a stab
at changes to the manual (I usually rework these but it's helpful to
have a starting point) and some additions to the regression test suite,
if possible.
Cheers!
It's very happy to receive your response!
Any kind of paperwork of copyright assigment is okay!
Sure, alternative implementations or any other concenrs need to be discussed.
GNU tools are widely delopyed and used in every single second.
I hope the patches could help people instead of bring them disasters.
About the paperwork, because I'm living oudside of U.S. If it can be done by
email or fax is good. If postmail is necessary will be okay, too. Just take
longer round trip time of these documentations.
The following is the description of the purpose of these patches.
* expand.c (variable_name_extract): extract variable name and strip prefix.
* variable.h: Add support of struct value_records and parent_records
* variable.c: Add support for constructing value_records and parent_records
* read.c: Construct the dependency chain between parent and target variable
* load.c: add support of parent_records
* main.c: add --dep parameter for print variable dependency
This set of patches were created to help developers to understand
the dependencies between compile options or any other variables defined
in makefiles.
The dependencies especially indicate the relationship between a variable
parent (assigner) and a variable itself as a child (assignee).
The relationship between variable parent and variable child usually defined
by conditional lines (ifeq/ifneq/ifdef/ifndef).
By recording the operation history of these conditional lines, we can keep
the operation history of a variable child and its variable parents.
These operation histories are stored in value_record and parent_records.
A value_record is a linking list to record each value was assigned to a
variable caused by a conditional line. And the parent_record will keep the
variables which made the value is assigned to this child value.
Developers can simply dump all these informations by simply use command line
`make --dep` to dump the dependency chain stored in makefile for a software
project.
Software projects is growing bigger and bigger nowadays. By adding this
--dep features may also help developers not only to understand the relationship
between definitions in makefiles and also test if their makefile has problem
(loop) in conditional checking for variables.
Here comes the examples:
Makefile:
A = AA
B = BB
ifeq ($(A),AA)
C = CC
endif
ifeq ($(A),AA)
D = DD
ifeq ($(B),BB)
C += CCC
endif
endif
ifeq ($(D),DD)
ifeq ($(B),BB)
E = EE
endif
endif
Output of `make --dep`
A = AA
B = BB
C = CC CCC
value: CC, parent: A, value: AA;
value: CCC, parent: A, value: AA; parent: B, value: BB;
D = DD
value: DD, parent: A, value: AA;
E = EE
value: EE, parent: D ,value DD; parent: B, value: BB;
parent name: D, value: DD, parent: A, value AA;
The following are some thought and futhur extension of current implementations.
1. Since the make variable evaluation were stateless and no records stored.
This implementation will use much more memory and linear search for linking list
to value_records and parent records.
I think we can add something conditional checking if the --dep option has been
enabled. There is no need to maintain these records if --dep option was not
enabled.
2. Loop detection and blacklists support.
Actually, I've found some makefile will wrote like this.
A = AA
ifeq ($(A),)
A =
endif
This kind of codes will lead a loop for a variable link a parents to itself.
When dump all parents will cause a infinite loop.
Some project may really need this kind of code for build.
I think add a blacklist to avoid make to assign this kind of variable
as parent could help.
3. Support print depth when dump dependency informations?
If the dependency chain of a variable is very deep.
Should we support the depth parameters to limits the depth of
printing
Re: [PATCH v2 1/6] * expand.c (variable_name_extract): extract variable name and strip prefix.
This sounds like an extremely useful debugging feature. I have often
had the problem of getting the wrong build parameters but not being
sure exactly why because of the great complexity of makefiles that are
trying to build many different sorts of object files all with slight
variations that are based on lots of configuration options.
In the end one has to pepper the makefile with $(info) statements to
find out and this can be impossible when dealing with a customer.
I think the commandline option might not be appropriate and the
feature should absolutely not be active or use memory unless the
commandline option is on.
As for whether the implementation is correct or optimal or fits make
standards or whether it should really be part of the core gnumake
(instead of being some sort of frequently updated patch/fork) I don't
know.
Regards,
Tim
On 21 August 2014 08:25, Macpaul Lin macp...@gmail.com wrote:
Hi, Paul and other developers,
2014-08-18 22:35 GMT+08:00 Paul Smith psm...@gnu.org:
On Mon, 2014-08-18 at 21:27 +0800, Macpaul Lin wrote:
Variables used in conditional lines usually has '$',
'(', and ')' prefix, and etc.
We can use vairable_name_extract() to extract pure variable
name without these prefix.
Hello. Thanks for your work on GNU make!
Can you provide some sort of summary of the feature you've created, what
its goals are, examples, etc.?
It can be helpful to contact me or someone on the development team
before (or while) doing feature work for GNU make. As a GNU project,
for example, we'll need to get copyright assignment paperwork for any
contribution of significant size and this takes some time.
Also, we may have suggestions or alternative implementations or concerns
that should be considered. Of course these can always be addressed
later, but it can save some effort to think about them up-front.
And finally, note that fully-formed changes need to have at least a stab
at changes to the manual (I usually rework these but it's helpful to
have a starting point) and some additions to the regression test suite,
if possible.
Cheers!
It's very happy to receive your response!
Any kind of paperwork of copyright assigment is okay!
Sure, alternative implementations or any other concenrs need to be discussed.
GNU tools are widely delopyed and used in every single second.
I hope the patches could help people instead of bring them disasters.
About the paperwork, because I'm living oudside of U.S. If it can be done by
email or fax is good. If postmail is necessary will be okay, too. Just take
longer round trip time of these documentations.
The following is the description of the purpose of these patches.
* expand.c (variable_name_extract): extract variable name and strip prefix.
* variable.h: Add support of struct value_records and parent_records
* variable.c: Add support for constructing value_records and parent_records
* read.c: Construct the dependency chain between parent and target variable
* load.c: add support of parent_records
* main.c: add --dep parameter for print variable dependency
This set of patches were created to help developers to understand
the dependencies between compile options or any other variables defined
in makefiles.
The dependencies especially indicate the relationship between a variable
parent (assigner) and a variable itself as a child (assignee).
The relationship between variable parent and variable child usually defined
by conditional lines (ifeq/ifneq/ifdef/ifndef).
By recording the operation history of these conditional lines, we can keep
the operation history of a variable child and its variable parents.
These operation histories are stored in value_record and parent_records.
A value_record is a linking list to record each value was assigned to a
variable caused by a conditional line. And the parent_record will keep the
variables which made the value is assigned to this child value.
Developers can simply dump all these informations by simply use command line
`make --dep` to dump the dependency chain stored in makefile for a software
project.
Software projects is growing bigger and bigger nowadays. By adding this
--dep features may also help developers not only to understand the
relationship
between definitions in makefiles and also test if their makefile has problem
(loop) in conditional checking for variables.
Here comes the examples:
Makefile:
A = AA
B = BB
ifeq ($(A),AA)
C = CC
endif
ifeq ($(A),AA)
D = DD
ifeq ($(B),BB)
C += CCC
endif
endif
ifeq ($(D),DD)
ifeq ($(B),BB)
E = EE
endif
endif
Output of `make --dep`
A = AA
B = BB
C = CC CCC
value: CC, parent: A, value: AA;
value: CCC, parent: A, value: AA; parent: B, value: BB;
D = DD
value: DD, parent: A, value: AA;
E = EE
value: EE, parent: D ,value DD; parent: B, value: BB;
parent name: D, value: DD, parent: A, value AA;
The following are some thought and futhur extension
Re: POSIX ruling on up-to-date vs. identical timestamps
On Thu, Aug 21, 2014 at 4:32 PM, Eric Blake ebl...@redhat.com wrote: Make folks: You may want to check out http://austingroupbugs.net/view.php?id=857 and add comments and/or change GNU make behavior accordingly. There, the argument is made that HP-UX make behavior is nicer than GNU's current behavior when two files have identical timestamps: HP-UX considers the file as out-of-date, while GNU make considers it up-to-date. A strict reading of POSIX can argue that GNU's behavior was required, but this reading has been called into question. GNU's behavior is an optimization that avoids needless churn on file systems with course timestamps (well, FAT still exists, but these days, MOST file systems have sub-second resolution, so it is harder to get files with identical timestamps without actually touch'ing them that way) - but it risks leaving a tree in an incomplete state. HP-UX behavior guarantees the rules are run, even if they were not strictly necessary, but has the nice property that the tree is never left in an incomplete state due to unfortunate timing on a file system with course timestamps. The POSIX recommendation was therefore that GNU should change its behavior to act like HP-UX, and consider identical timestamps as out-of-date, because the standard will be fixed to allow HP-UX behavior. Autoconf folks: The section of the autoconf manual that discusses this should probably be modernized, particularly if changes to POSIX and/or GNU make result from this discussion. https://www.gnu.org/software/autoconf/manual/autoconf.html#Timestamps-and-Make Some projects that want to have deterministic builds by using the gitian builder [1] use libfaketime to set the timestamps to a known constant, and gnu make trips up on this, occasionally causing broken builds and othertimes autoconf declares suck a build system insane. There was a bug in libfaketime so that the nanosecond field wasn't cleared and this allowed you to avoid both these bugs if the planets were aligned correctly (which has since been fixed). I agree that these issues should be fixed in make and autoconf. [1] https://gitian.org/ -- Eric Blake eblake redhat com+1-919-301-3266 Libvirt virtualization library http://libvirt.org ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: POSIX ruling on up-to-date vs. identical timestamps
On Thu, Aug 21, 2014 at 4:57 PM, Ray Donnelly mingw.andr...@gmail.com wrote: On Thu, Aug 21, 2014 at 4:32 PM, Eric Blake ebl...@redhat.com wrote: Make folks: You may want to check out http://austingroupbugs.net/view.php?id=857 and add comments and/or change GNU make behavior accordingly. There, the argument is made that HP-UX make behavior is nicer than GNU's current behavior when two files have identical timestamps: HP-UX considers the file as out-of-date, while GNU make considers it up-to-date. A strict reading of POSIX can argue that GNU's behavior was required, but this reading has been called into question. GNU's behavior is an optimization that avoids needless churn on file systems with course timestamps (well, FAT still exists, but these days, MOST file systems have sub-second resolution, so it is harder to get files with identical timestamps without actually touch'ing them that way) - but it risks leaving a tree in an incomplete state. HP-UX behavior guarantees the rules are run, even if they were not strictly necessary, but has the nice property that the tree is never left in an incomplete state due to unfortunate timing on a file system with course timestamps. The POSIX recommendation was therefore that GNU should change its behavior to act like HP-UX, and consider identical timestamps as out-of-date, because the standard will be fixed to allow HP-UX behavior. Autoconf folks: The section of the autoconf manual that discusses this should probably be modernized, particularly if changes to POSIX and/or GNU make result from this discussion. https://www.gnu.org/software/autoconf/manual/autoconf.html#Timestamps-and-Make Some projects that want to have deterministic builds by using the gitian builder [1] use libfaketime to set the timestamps to a known constant, and gnu make trips up on this, occasionally causing broken builds and othertimes autoconf declares suck a build system insane. Freudian slip? I meant such, not suck. There was a bug in libfaketime so that the nanosecond field wasn't cleared and this allowed you to avoid both these bugs if the planets were aligned correctly (which has since been fixed). I agree that these issues should be fixed in make and autoconf. [1] https://gitian.org/ -- Eric Blake eblake redhat com+1-919-301-3266 Libvirt virtualization library http://libvirt.org ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: POSIX ruling on up-to-date vs. identical timestamps
Eric Blake wrote: You may want to check out http://austingroupbugs.net/view.php?id=857 and add comments and/or change GNU make behavior accordingly. Let's leave GNU 'make' alone. Its behavior is better for rules like this: copy: original cp -p original copy I've added a comment to the Austin Group page. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: POSIX ruling on up-to-date vs. identical timestamps
On Thu, 21 Aug 2014, Eric Blake wrote: The POSIX recommendation was therefore that GNU should change its behavior to act like HP-UX, and consider identical timestamps as out-of-date, because the standard will be fixed to allow HP-UX behavior. A change like this may result in some builds which never complete (or take much longer) because the timestamp rules require that something be rebuilt (tail chasing). This is particularly a problem for recursive builds which may enter the same directories multiple times. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: POSIX ruling on up-to-date vs. identical timestamps
Ray Donnelly wrote: There was a bug in libfaketime so that the nanosecond field wasn't cleared That sounds like it's a different issue. If a program botches the nanosecond component of timestamps, it shouldn't matter whether 'make' uses the traditional/GNU or the HP-UX approach; either way, you can get into trouble. If I'm misunderstanding the libfaketime issue, could you please explain it with an example? ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: POSIX ruling on up-to-date vs. identical timestamps
The obvious compromise would be to change the behavior only in the presence of the .POSIX: special target. On Thu, Aug 21, 2014 at 9:13 AM, Paul Eggert egg...@cs.ucla.edu wrote: Eric Blake wrote: You may want to check out http://austingroupbugs.net/view.php?id=857 and add comments and/or change GNU make behavior accordingly. Let's leave GNU 'make' alone. Its behavior is better for rules like this: copy: original cp -p original copy I've added a comment to the Austin Group page. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: POSIX ruling on up-to-date vs. identical timestamps
On Thu, Aug 21, 2014 at 12:27 PM, Ray Donnelly mingw.andr...@gmail.com wrote: On Thu, Aug 21, 2014 at 8:03 PM, David Boyce david.s.bo...@gmail.com wrote: The obvious compromise would be to change the behavior only in the presence of the .POSIX: special target. Sounds pragmatic; the repeatable builds people would probably like a solution that doesn't require changing Makefiles though, either an env. var or a command line option? The command line is always possible for a special target given Paul's clever --eval option: make --eval .POSIX: ... And for an EV approach it should be possible to add that to an exported MAKEFLAGS. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: POSIX ruling on up-to-date vs. identical timestamps
David Boyce wrote: The obvious compromise would be to change the behavior only in the presence of the .POSIX: special target. We should limit .POSIX to what POSIX requires. Even if the ruling stands POSIX won't require the HP-UX behavior, so .POSIX should be independent of this issue. It'd be OK to introduce a new special target to enable the HP-UX behavior. .EQUAL_TIMES_ARE_OUT-OF-DATE, say. We could document the new target next to .LOW_RESOLUTION_TIME, since they're related issues. The new target could act like .LOW_RESOLUTION_TIME, except that it imposes HP-UX rather than low-res behavior. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
