Hi all. I've set Reply-To: to the bug-make@gnu.org list; I'm hoping we can keep the discussion there since I don't subscribe to kbuild-devel.
I'm working on getting the next release of GNU make, 3.81, out the door (amazing!) The weekend before last I released 3.81rc1 for people to test. A day or two ago, Art Haas <[EMAIL PROTECTED]> emailed me that he was having problems using kbuild with this version. The previous version, 3.81beta4, works fine. The symptoms are that much of the kernel was rebuilding over gain every time he ran make, even after he'd just done a top-down build. I pulled the 2.6.15 kernel and sure enough, I see the same behavior. I delved into the kbuild infrastructure and I found the problem. The kbuild system uses a trick to force rebuilds if the command line changes for a given target (normally make only rebuilds if the target is out of date--some versions of make, like Solaris make, have a .KEEP_STATE feature but GNU make does not support this). Here's a stripped-down example of what kbuild does: .PHONY: FORCE %.o : %.c FORCE $(if_changed_rule ...) if_changed_rule = $(if $(strip $? $(call arg-check, $(cmd_$(1)), $(cmd_$@)) ),\ @set -e; \ $(rule_$(1))) The FORCE prerequisite causes ALL .o targets that match this rule to be considered by make to be out of date (because FORCE is always out-of-date and it's a prerequisite of the .o). The trick is in the if_changed_rule macro: it tests whether any prerequisites have changed ($? is the changed prerequisites; it'll be empty if none have changed) and whether the command lines have changed (the call to arg-check). If those values are true (non-empty) it expands to the rule. Otherwise it expands to nothing. GNU make takes several shortcuts to provide efficiency in places where it doesn't matter, and so if it sees an empty command it doesn't try to run a shell. In this way, kbuild gets the benefit of checking the command line for every target without paying a price for useless shells being invoked during the build. However, this trick as implemented is accidentally relying on what may be a misbehavior on GNU make's part: one that was changed in the latest rc release. This is causing rebuilds to happen. In previous versions of GNU make, prerequisites that didn't exist were not included in the $? variable. In the new version that's been changed (fixed?) so that all out-of-date prerequisites are included in the $? variable, even if they don't exist. The old behavior allowed this rule to work, because even though FORCE was out-of-date and would normally always appear in $?, it didn't exist as a file and this exception caused it to be left out. So, the value of $? was empty in the old version if the only prerequisite that was considered out-of-date was the non-existent file FORCE. In the new version of GNU make, the value of $? is FORCE in that situation, so the test in if_changed_rule is always true and it always evaluates to the compile line, and rebuilds. Neither the GNU make manual nor the POSIX definition of make gives us clear direction as to the correct behavior in this particular situation. SuS says: $? The $? macro shall evaluate to the list of prerequisites that are newer than the current target. It shall be evaluated for both target and inference rules. The GNU make manual says: $? The names of all the prerequisites that are newer than the target, with spaces between them. So... is a non-existent file "newer than the target"? This specific situation is not addressed. However, other versions of make (SysV make for example) interpret a non-existent file as out-of-date and DO include it in $?. Given the meaning of "newer than the current target" to make (that it causes the target to be rebuilt) and the implied meaning of $? (a list of the prerequisites that cause the target to be rebuilt), I feel that the new behavior is correct and the old behavior is incorrect. So. If the change is correct, how should we proceed? Obviously it's not hard to change kbuild to fix the majority of the problem; replace the above macro with something like: if_changed_rule = $(if $(strip $(filter-out FORCE,$?) $(call arg-check, $(cmd_$(1)), $(cmd_$@)) ),\ @set -e; \ $(rule_$(1))) and all will be well. There are other, similar macros that need this change as well. And, there are other places where this $(filter-out FORCE,...) is already done, so it's apparently come up before. I attach a patch here which makes this fix in kbuild. The above solution is backward-compatible (will work with older versions of GNU make), so that's nice. But of course, current releases will still be broken unless patched. Is it OK to say, if you upgrade GNU make you have to patch kbuild, unless you want to rebuild everything every time? Ouch. There is another slight problem: some targets have extra PHONY prerequisites as well. For example, here: .tmp_kallsyms1.o .tmp_kallsyms2.o .tmp_kallsyms3.o: %.o: %.S scripts FORCE $(call if_changed_dep,as_o_S) from the root Makefile. Now, "scripts" here is PHONY, so it will always appear in $? in the new version of make as well. Unless we want to add all those PHONY target to the $(filter-out ...) test in the if_* macros we're going to get some rebuild: this causes the final link to be done again even if nothing has changed. That's probably not a big deal since most people WILL change something before they build. Nevertheless, it's unpleasant. I don't know of any way, given the current features of GNU make, to solve this problem generically (like, "remove all PHONY targets from this list of targets"). Well, I'll stop typing now :-). Please send me your thoughts.
--- scripts/Kbuild.include-dist 2006-01-02 22:21:10.000000000 -0500 +++ scripts/Kbuild.include 2006-03-01 10:42:30.398316278 -0500 @@ -73,8 +73,8 @@ # function to only execute the passed command if necessary # >'< substitution is for echo to work, >$< substitution to preserve $ when reloading .cmd file # note: when using inline perl scripts [perl -e '...$$t=1;...'] in $(cmd_xxx) double $$ your perl vars -# -if_changed = $(if $(strip $? $(call arg-check, $(cmd_$(1)), $(cmd_$@)) ), \ +# +if_changed = $(if $(strip $(filter-out FORCE,$?) $(call arg-check, $(cmd_$(1)), $(cmd_$@)) ), \ @set -e; \ $(echo-cmd) \ $(cmd_$(1)); \ @@ -82,7 +82,7 @@ # execute the command and also postprocess generated .d dependencies # file -if_changed_dep = $(if $(strip $? $(filter-out FORCE $(wildcard $^),$^)\ +if_changed_dep = $(if $(strip $(filter-out FORCE,$?) $(filter-out FORCE $(wildcard $^),$^)\ $(call arg-check, $(cmd_$(1)), $(cmd_$@)) ), \ @set -e; \ $(echo-cmd) \ @@ -94,6 +94,6 @@ # Usage: $(call if_changed_rule,foo) # will check if $(cmd_foo) changed, or any of the prequisites changed, # and if so will execute $(rule_foo) -if_changed_rule = $(if $(strip $? $(call arg-check, $(cmd_$(1)), $(cmd_$@)) ),\ +if_changed_rule = $(if $(strip $(filter-out FORCE,$?) $(call arg-check, $(cmd_$(1)), $(cmd_$@)) ),\ @set -e; \ $(rule_$(1)))
-- ------------------------------------------------------------------------------- Paul D. Smith <[EMAIL PROTECTED]> Find some GNU make tips at: http://www.gnu.org http://make.paulandlesley.org "Please remain calm...I may be mad, but I am a professional." --Mad Scientist