Re: make modules kicks the first module directory twice

2000-11-15 Thread Bruce Evans

On Tue, 14 Nov 2000, Warner Losh wrote:

 make depend is already *REQUIRED* when you are updating a kernel from
 an older version of the kernel.  For config -r FOO kernels it isn't.

It is neither necessary nor sufficient for updating.  config -r removes
conflicting garbage from previous builds.  Removing the conflicting
garbage is sufficient.  `make clean' normally suffices to remove it,
but there are cases where .depend itself is garbage and must be removed
to unbreak `make depend'.

 Even a make clean after a make depend will require that make depend be
 run again.

This was broken in rev.1.348 of sys/conf/files, etc.  `make clean'
removes the generated bus interface headers, and there are no longer
enough dependencies for make(1) to know how to rebuild them.  There
are enough dependencies to rebuild them for `make depend', and for
plain `make' when .depend doesn't exist (minor magic involving
${BEFORE_DEPEND}).  .depend still doesn't contain all the dependencies
on headers, because at least the generated bus interface .c files are
handled too specially.

Bruce



To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message



Re: make modules kicks the first module directory twice

2000-11-14 Thread David O'Brien

On Mon, Nov 13, 2000 at 09:17:54PM -0700, Warner Losh wrote:
 The implications are that make obj isn't done unless you've run make
 depend first.  If a new directory is added and a make depend isn't
 run, then the modules won't get built into the obj tree, but instead
 will be built into $S/modules.

Having modules wind up in two trees is not acceptable IMHO.
I'd rather take a major compile time hit and be deterministic than not.


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message



Re: RQ review: [was: Re: make modules kicks the first module directory twice]

2000-11-14 Thread David O'Brien

On Mon, Nov 13, 2000 at 08:02:47PM -0800, Marcel Moolenaar wrote:
 Any objections?

Yes.
 
 (patches follow for your convenience)

[its easier to read patches when they aren't quoted in their entirety ;-)]

modules-depend:
  @mkdir -p ${.OBJDIR}/modules
  !   cd $S/modules; env ${MKMODULESENV} ${MAKE} obj
  !   env ${MKMODULESENV} ${MAKE} depend

This is broken for non -j case.

-- 
-- David  ([EMAIL PROTECTED])
  GNU is Not Unix / Linux Is Not UniX


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message



Re: make modules kicks the first module directory twice

2000-11-14 Thread Warner Losh

In message [EMAIL PROTECTED] "David O'Brien" writes:
: On Mon, Nov 13, 2000 at 09:17:54PM -0700, Warner Losh wrote:
:  The implications are that make obj isn't done unless you've run make
:  depend first.  If a new directory is added and a make depend isn't
:  run, then the modules won't get built into the obj tree, but instead
:  will be built into $S/modules.
: 
: Having modules wind up in two trees is not acceptable IMHO.

But they are both in the $S tree. :-)

: I'd rather take a major compile time hit and be deterministic than not.

I'd rather not.  We don't do an implicit make obj in the rest of the
tree.  If I go build the world, and then someone adds a new program to
the tree, you are in the same boat.  If you cd to that program and
type make it will wind up in . rather than /usr/obj.  Completely
deterministic, the same thing will happen every time you do the
scenario.

make depend is already *REQUIRED* when you are updating a kernel from
an older version of the kernel.  For config -r FOO kernels it isn't.
Even a make clean after a make depend will require that make depend be
run again.

But before making major changes to this, let's see Peter Wemm's new
all singing all dancing config work does for us.  I'd rather see what
he's come up with than argue further on this.

Warner


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message



Re: make modules kicks the first module directory twice

2000-11-14 Thread David O'Brien

[stable dropped, this should have only been in a single list to start with!!]

On Tue, Nov 14, 2000 at 01:27:32PM -0700, Warner Losh wrote:
 : I'd rather take a major compile time hit and be deterministic than not.
 
 I'd rather not.  We don't do an implicit make obj in the rest of the
 tree.  

It is in `make world' if /usr/obj exists.  Otherwise how does all the
.o's get in /usr/obj/ ?

 If I go build the world, and then someone adds a new program to
 the tree, you are in the same boat.

Nope, `make world' will DTRT.

 If you cd to that program and type make it will wind up in . rather
 than /usr/obj.

Yes, and how many times do we have to tell people to run
``make cleandir  make cleandir'' or 
``rm -rf /usr/obj/* ; cd /usr/src ; make cleandir ''

 Completely deterministic, the same thing will happen every time you do
 the scenario.

Ok, completely deterministic given enough detail -- detail which 99% of
the time will not be provided in email to the lists saying this and that
is broken.

 But before making major changes to this, let's see Peter Wemm's new
 all singing all dancing config work does for us.  I'd rather see what
 he's come up with than argue further on this.

Earlier today, I too I was wondering if his plan makes all this OBE.
 
-- 
-- David  ([EMAIL PROTECTED])
  GNU is Not Unix / Linux Is Not UniX


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message



Re: make modules kicks the first module directory twice

2000-11-14 Thread Warner Losh

In message [EMAIL PROTECTED] "David O'Brien" writes:
: [stable dropped, this should have only been in a single list to start with!!]

Agreed.  My summary:
o We disagree about the support impact
o Peter's stuff may OBE this whole thread
o My change shouldn't be committed until we know it will have no
  support impact, or it can be fixed to be more robust.

: On Tue, Nov 14, 2000 at 01:27:32PM -0700, Warner Losh wrote:
:  : I'd rather take a major compile time hit and be deterministic than not.
:  
:  I'd rather not.  We don't do an implicit make obj in the rest of the
:  tree.  
: 
: It is in `make world' if /usr/obj exists.  Otherwise how does all the
: .o's get in /usr/obj/ ?

I should have stated this more clearly.  We don't have an implicit
make obj anywhere else in the tree for the default "all" target.  We
do have it for other, special targets.

:  If I go build the world, and then someone adds a new program to
:  the tree, you are in the same boat.
: 
: Nope, `make world' will DTRT.

No.  Not if you compile that one file.  Also, make all won't do the
right thing.  That's my point.  We have extra special targets that are
all singing all dancing that do the right thing, but the plain vanilla
ones act in a plain vanilla way.

Again, make world isn't the default target.  It is an extra special
target that does special things.

I've been burned by the example that I sighted.

:  If you cd to that program and type make it will wind up in . rather
:  than /usr/obj.
: 
: Yes, and how many times do we have to tell people to run
: ``make cleandir  make cleandir'' or 
: ``rm -rf /usr/obj/* ; cd /usr/src ; make cleandir ''

A few, but a lot less than I'd expect :-).

:  Completely deterministic, the same thing will happen every time you do
:  the scenario.
: 
: Ok, completely deterministic given enough detail -- detail which 99% of
: the time will not be provided in email to the lists saying this and that
: is broken.

True.  But it would be rare enough that people would have two
different kernels, with different revs of the sources in the same
source tree and that those differences would cause problems.  It would
sure be hard to find it, I'll grant that.

I think it would be rare enough that we won't get mail on it, but I
could be wrong about that.

:  But before making major changes to this, let's see Peter Wemm's new
:  all singing all dancing config work does for us.  I'd rather see what
:  he's come up with than argue further on this.
: 
: Earlier today, I too I was wondering if his plan makes all this OBE.

He sure has been silent on all of this. :-)

I do think that we're all in agreement that make obj all should be
changed to make obj\n make all to make the parallel case work.  The
rest shouldn't be committed until we have it more robust.

I'll take alook to see if there's a robust way to know if we need to
run make obj or not.


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message



Re: RQ review: [was: Re: make modules kicks the first module directory twice]

2000-11-14 Thread Marcel Moolenaar

David O'Brien wrote:
 
 modules-depend:
   @mkdir -p ${.OBJDIR}/modules
   !   cd $S/modules; env ${MKMODULESENV} ${MAKE} obj
   !   env ${MKMODULESENV} ${MAKE} depend
 
 This is broken for non -j case.

Yes, this was known. The right diff was given at the beginning of the
message including the comment that the original patch had this breakage
:-)

-- 
Marcel Moolenaar
  mail: [EMAIL PROTECTED] / [EMAIL PROTECTED]
  tel:  (408) 447-4222


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message



Re: make modules kicks the first module directory twice

2000-11-13 Thread Bruce Evans

On Sun, 12 Nov 2000, Marcel Moolenaar wrote:

 Makoto MATSUSHITA wrote:
  
  % make -j 2 modules
  cd ../../modules  env MAKEOBJDIRPREFIX=/usr/src/sys/compile/GENERIC/modules 
KMODDIR=/boot/kernel make obj all
  === 3dfx
  === 3dfx
  Warning: Object directory not changed from original /usr/src/sys/modules/3dfx
  (... ok, break it ...)
 
 The problem is in the fact that the Makefile (the one in /sys/conf)
 contains something like:
 
   ${MAKE} obj all
 
 and
 
   ${MAKE} obj depend
 
 The net effect is that these targets are built in parallel, which
 obviously isn't right. The following solves the problem (i386 only):

That's just one of the problems :-).  "make obj all" is usually an
error, but in Makefile.${MACHINE} it should be just a bad example,
since the `obj' and `all' targets should be built sequentially and then
the object directories will exist by the time make(1) recurses into
them for the `all' target.  This doesn't work right for the -j case.
(In the above example, the targets are built concurrently and race
each other.  This is bad when the `all' target wins the race.  The
`obj' target runs faster, so it usually wins the race except in the
first directory (3dfx)).  More .ORDER statements in *.mk are required.

 Index: Makefile.i386
 ===
 RCS file: /home/ncvs/src/sys/conf/Makefile.i386,v
 retrieving revision 1.212
 diff -u -r1.212 Makefile.i386
 --- Makefile.i386   2000/10/29 09:47:50 1.212
 +++ Makefile.i386   2000/11/13 07:49:00
 @@ -271,11 +271,13 @@
  
  modules:
 @mkdir -p ${.OBJDIR}/modules
 -   cd $S/modules  env ${MKMODULESENV} ${MAKE} obj all
 +   cd $S/modules  env ${MKMODULESENV} ${MAKE} obj  \
 +   env ${MKMODULESENV} ${MAKE} all
  
  modules-depend:
 @mkdir -p ${.OBJDIR}/modules
 -   cd $S/modules  env ${MKMODULESENV} ${MAKE} obj depend
 +   cd $S/modules  env ${MKMODULESENV} ${MAKE} obj $$ \
 +   env ${MKMODULESENV} ${MAKE} depend
  
  modules-clean:
 cd $S/modules  env ${MKMODULESENV} ${MAKE} clean
 

`' should never be used in shell commands in makefiles, although it
may be only a bad example.  This is because multiple commands are
executed in the same shell in the -j case, and `' gives non-simple
commands which may defeat the shell's -e setting.  E.g., the command:

cd /; set -e; cd /nonesuch  false; rm -rf *

removes everything under "/", not everything under /nonesuch, despite
checking that the cd to /nonesuch worked.

Bruce



To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message



Re: make modules kicks the first module directory twice

2000-11-13 Thread Makoto MATSUSHITA


bde (In the above example, the targets are built concurrently and race
bde each other.  This is bad when the `all' target wins the race.  The
bde `obj' target runs faster, so it usually wins the race except in the
bde first directory (3dfx)).  More .ORDER statements in *.mk are required.

Thank you for giving us a details of this problem. But can we fix this
problem with .ORDER statements? Putting ".ORDER: obj all" or alike to
Makefile doesn't fix to me (maybe I misunderstand the usage of .ORDER
statement)...

bde `' should never be used in shell commands in makefiles, although it
bde may be only a bad example.  This is because multiple commands are
bde executed in the same shell in the -j case, and `' gives non-simple
bde commands which may defeat the shell's -e setting.

What should we do if we want to check the existence of a directory and
kick one (not two or more) command after chdir to that directory? Much
Makefiles in our FreeBSD repository employ "cd ${dir}  command" to
do this... Should we say

if [ -d ${dir} ]; then \
(cd ${dir}; command) \
else
false
fi  

or exist() directive of make(1) ?

***

Anyway, attached below is current sample patch to fix this problem.
"make obj" and "make all" are separated to two individual commands
(redundant 'cd' clause, but it should work).

-- -
Makoto `MAR' MATSUSHITA

Index: Makefile
===
RCS file: /lab/FreeBSD/FreeBSD.cvs/src/release/Makefile,v
retrieving revision 1.585
diff -c -r1.585 Makefile
*** Makefile2000/11/12 11:04:11 1.585
--- Makefile2000/11/13 05:37:12
***
*** 831,837 
@rm -f ${RD}/kernels/*.ko
@cd ${.CURDIR}/../sys/${MACHINE}/conf  config ${KERNEL}
@cd ${.CURDIR}/../sys/compile/${KERNEL}  \
!   make kernel-depend  \
make ${KERNEL_FLAGS} modules  \
make modules-reinstall DESTDIR=${RD}/kernels  \
  
--- 831,837 
@rm -f ${RD}/kernels/*.ko
@cd ${.CURDIR}/../sys/${MACHINE}/conf  config ${KERNEL}
@cd ${.CURDIR}/../sys/compile/${KERNEL}  \
!   make modules-depend  \
make ${KERNEL_FLAGS} modules  \
make modules-reinstall DESTDIR=${RD}/kernels  \
  
Index: Makefile.alpha
===
RCS file: /lab/FreeBSD/FreeBSD.cvs/src/sys/conf/Makefile.alpha,v
retrieving revision 1.79
diff -c -r1.79 Makefile.alpha
*** Makefile.alpha  2000/10/29 09:47:50 1.79
--- Makefile.alpha  2000/11/13 10:09:34
***
*** 310,320 
  
  modules:
@mkdir -p ${.OBJDIR}/modules
!   cd $S/modules  env ${MKMODULESENV} ${MAKE} obj all
  
  modules-depend:
@mkdir -p ${.OBJDIR}/modules
!   cd $S/modules  env ${MKMODULESENV} ${MAKE} obj depend
  
  modules-clean:
cd $S/modules  env ${MKMODULESENV} ${MAKE} clean
--- 310,322 
  
  modules:
@mkdir -p ${.OBJDIR}/modules
!   cd $S/modules  env ${MKMODULESENV} ${MAKE} obj
!   cd $S/modules  env ${MKMODULESENV} ${MAKE} all
  
  modules-depend:
@mkdir -p ${.OBJDIR}/modules
!   cd $S/modules  env ${MKMODULESENV} ${MAKE} obj
!   cd $S/modules  env ${MKMODULESENV} ${MAKE} depend
  
  modules-clean:
cd $S/modules  env ${MKMODULESENV} ${MAKE} clean
Index: Makefile.i386
===
RCS file: /lab/FreeBSD/FreeBSD.cvs/src/sys/conf/Makefile.i386,v
retrieving revision 1.212
diff -c -r1.212 Makefile.i386
*** Makefile.i386   2000/10/29 09:47:50 1.212
--- Makefile.i386   2000/11/13 10:10:03
***
*** 271,281 
  
  modules:
@mkdir -p ${.OBJDIR}/modules
!   cd $S/modules  env ${MKMODULESENV} ${MAKE} obj all
  
  modules-depend:
@mkdir -p ${.OBJDIR}/modules
!   cd $S/modules  env ${MKMODULESENV} ${MAKE} obj depend
  
  modules-clean:
cd $S/modules  env ${MKMODULESENV} ${MAKE} clean
--- 271,283 
  
  modules:
@mkdir -p ${.OBJDIR}/modules
!   cd $S/modules  env ${MKMODULESENV} ${MAKE} obj
!   cd $S/modules  env ${MKMODULESENV} ${MAKE} all
  
  modules-depend:
@mkdir -p ${.OBJDIR}/modules
!   cd $S/modules  env ${MKMODULESENV} ${MAKE} obj
!   cd $S/modules  env ${MKMODULESENV} ${MAKE} depend
  
  modules-clean:
cd $S/modules  env ${MKMODULESENV} ${MAKE} clean
Index: Makefile.ia64
===
RCS file: /lab/FreeBSD/FreeBSD.cvs/src/sys/conf/Makefile.ia64,v
retrieving revision 1.4
diff -c -r1.4 Makefile.ia64
*** Makefile.ia64   2000/10/29 09:47:50 1.4
--- Makefile.ia64   2000/11/13 10:10:41
***
*** 275,285 
  
  modules:
@mkdir -p ${.OBJDIR}/modules
!   cd $S/modules  env ${MKMODULESENV} ${MAKE} obj all
  
  modules-depend:

Re: make modules kicks the first module directory twice

2000-11-13 Thread Bruce Evans

On Mon, 13 Nov 2000, Makoto MATSUSHITA wrote:

 bde (In the above example, the targets are built concurrently and race
 bde each other.  This is bad when the `all' target wins the race.  The
 bde `obj' target runs faster, so it usually wins the race except in the
 bde first directory (3dfx)).  More .ORDER statements in *.mk are required.
 
 Thank you for giving us a details of this problem. But can we fix this
 problem with .ORDER statements? Putting ".ORDER: obj all" or alike to
 Makefile doesn't fix to me (maybe I misunderstand the usage of .ORDER
 statement)...

It didn't work for me either :-).  I put it in bsd.obj.mk near the `obj'
target, with the idea that this would cover all uses of the obj target.
The problem seems to be that we both put it in the wrong place.  I think
it needs to be in bsd.subdir.mk for this case and in bsd.obj.mk for most
cases.

 bde `' should never be used in shell commands in makefiles, although it
 bde may be only a bad example.  This is because multiple commands are
 bde executed in the same shell in the -j case, and `' gives non-simple
 bde commands which may defeat the shell's -e setting.
 
 What should we do if we want to check the existence of a directory and
 kick one (not two or more) command after chdir to that directory? Much
 Makefiles in our FreeBSD repository employ "cd ${dir}  command" to
 do this... Should we say
 
   if [ -d ${dir} ]; then \
   (cd ${dir}; command) \
   else
   false
   fi  
 
 or exist() directive of make(1) ?

Just use a semicolon instead of "" ("cd foo; command").  This gives
multiple single commands, and make(1) execs sh(1) with -e, so the shell
exits if any of the simple commands fails.  ("simple" here is a technical
term.  See sh.1.)

Bruce



To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message



Re: make modules kicks the first module directory twice

2000-11-13 Thread David O'Brien

On Sun, Nov 12, 2000 at 11:58:39PM -0700, Warner Losh wrote:
 In message [EMAIL PROTECTED] Makoto MATSUSHITA writes:
 : It does not fix this problem. However, if we separate the execution of
 : "make obj" and "make all", we can avoid (again, not *fix*) the problem.
 : Maybe this change is reasonable; there is few meaning doing "make
 : kernel-depend" in "doMODULES" target.
 
 I think that make has no business doing an implicit make obj for the
 all target.

Someone has to run `make obj' for the modules tree.  How are you doing it
locally?

 I have been running with patches in my tree that doesn't do the obj
 target if .depend exists.  

can you post them for review.

 Of course, you have to be more careful about running make depend in
 that case

Now *that* sounds scarry to me.  We'll have people all over the lists who
forget to and would now get rather bitten.  From the sounds of it, your
patch doesn't create as robust a world, but maybe it does.

-- 
-- David  ([EMAIL PROTECTED])
  GNU is Not Unix / Linux Is Not UniX


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message



Re: make modules kicks the first module directory twice

2000-11-13 Thread David O'Brien

On Mon, Nov 13, 2000 at 07:51:11PM +0900, Makoto MATSUSHITA wrote:
 ! cd $S/modules  env ${MKMODULESENV} ${MAKE} obj all
 --- 310,322 
 ! cd $S/modules  env ${MKMODULESENV} ${MAKE} obj
 ! cd $S/modules  env ${MKMODULESENV} ${MAKE} all


I can certainly commit this type of fix.  The combined make invocation
was an optimization someone recommended.

-- 
-- David  ([EMAIL PROTECTED])
  GNU is Not Unix / Linux Is Not UniX


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message



Re: make modules kicks the first module directory twice

2000-11-13 Thread Marcel Moolenaar

Bruce Evans wrote:
 
  Index: Makefile.i386
  ===
  RCS file: /home/ncvs/src/sys/conf/Makefile.i386,v
  retrieving revision 1.212
  diff -u -r1.212 Makefile.i386
  --- Makefile.i386   2000/10/29 09:47:50 1.212
  +++ Makefile.i386   2000/11/13 07:49:00
  @@ -271,11 +271,13 @@
 
   modules:
  @mkdir -p ${.OBJDIR}/modules
  -   cd $S/modules  env ${MKMODULESENV} ${MAKE} obj all
  +   cd $S/modules  env ${MKMODULESENV} ${MAKE} obj  \
  +   env ${MKMODULESENV} ${MAKE} all
 
   modules-depend:
  @mkdir -p ${.OBJDIR}/modules
  -   cd $S/modules  env ${MKMODULESENV} ${MAKE} obj depend
  +   cd $S/modules  env ${MKMODULESENV} ${MAKE} obj $$ \
  +   env ${MKMODULESENV} ${MAKE} depend
 
   modules-clean:
  cd $S/modules  env ${MKMODULESENV} ${MAKE} clean
 
 
 `' should never be used in shell commands in makefiles, although it
 may be only a bad example.

Mostly that, yes. The first try for a "fix" was:

cd $S/modules  env ${MKMODULESENV} ${MAKE} obj
cd $S/modules  env ${MKMODULESENV} ${MAKE} all

But since $S expands to "../.." and -j runs a single shell, the first cd
will succeed, but not the second.

I simply copied the "construction" in the second and posted fix.

I'll commit a fix with just semi-colons today for all architectures if
someone hasn't done that already by that time.

-- 
Marcel Moolenaar
  mail: [EMAIL PROTECTED] / [EMAIL PROTECTED]
  tel:  (408) 447-4222


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message



Re: make modules kicks the first module directory twice

2000-11-13 Thread Marcel Moolenaar

David O'Brien wrote:
 
 On Mon, Nov 13, 2000 at 07:51:11PM +0900, Makoto MATSUSHITA wrote:
  ! cd $S/modules  env ${MKMODULESENV} ${MAKE} obj all
  --- 310,322 
  ! cd $S/modules  env ${MKMODULESENV} ${MAKE} obj
  ! cd $S/modules  env ${MKMODULESENV} ${MAKE} all
 
 I can certainly commit this type of fix.  The combined make invocation
 was an optimization someone recommended.

No you can't. $S expands to "../.." which only works for the first cd in
the -jX case. The second cd will fail.

-- 
Marcel Moolenaar
  mail: [EMAIL PROTECTED] / [EMAIL PROTECTED]
  tel:  (408) 447-4222


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message



Re: make modules kicks the first module directory twice

2000-11-13 Thread Marcel Moolenaar

Marcel Moolenaar wrote:
 
 David O'Brien wrote:
 
  On Mon, Nov 13, 2000 at 07:51:11PM +0900, Makoto MATSUSHITA wrote:
   ! cd $S/modules  env ${MKMODULESENV} ${MAKE} obj all
   --- 310,322 
   ! cd $S/modules  env ${MKMODULESENV} ${MAKE} obj
   ! cd $S/modules  env ${MKMODULESENV} ${MAKE} all
 
  I can certainly commit this type of fix.  The combined make invocation
  was an optimization someone recommended.
 
 No you can't.

Oops. A smiley was intended here. I'm not the authoritive kind of guy
:-)

-- 
Marcel Moolenaar
  mail: [EMAIL PROTECTED] / [EMAIL PROTECTED]
  tel:  (408) 447-4222


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message



Re: make modules kicks the first module directory twice

2000-11-13 Thread Warner Losh

In message [EMAIL PROTECTED] "David O'Brien" writes:
: On Sun, Nov 12, 2000 at 11:58:39PM -0700, Warner Losh wrote:
:  In message [EMAIL PROTECTED] Makoto MATSUSHITA writes:
:  : It does not fix this problem. However, if we separate the execution of
:  : "make obj" and "make all", we can avoid (again, not *fix*) the problem.
:  : Maybe this change is reasonable; there is few meaning doing "make
:  : kernel-depend" in "doMODULES" target.
:  
:  I think that make has no business doing an implicit make obj for the
:  all target.
: 
: Someone has to run `make obj' for the modules tree.  How are you doing it
: locally?

Right now we do it twice.  Once in make dpeend and again in make
all.  My patch removes it from make all.

:  I have been running with patches in my tree that doesn't do the obj
:  target if .depend exists.  
: 
: can you post them for review.

Sure.  It is i386 only right now.  should be easy to port.

:  Of course, you have to be more careful about running make depend in
:  that case
: 
: Now *that* sounds scarry to me.  We'll have people all over the lists who
: forget to and would now get rather bitten.  From the sounds of it, your
: patch doesn't create as robust a world, but maybe it does.

I'm not sure what you're criteria for a robust world is here.  If the
directory doesn't exist, it will warn the user.  If new directories
appear, then there is a problem in that it will build in
$S/modules/foo rather than in the object tree.

I really don't think that we should be doing the make obj twice.  Once
for make depend and then again for make all.  Maybe I'm just too
impatient to sit still for it :-).  doing it during make all has the
effect of creating a race.  since the obj and all targets are done in
parallel for -j N, we have similar problems if the obj leg runs more
slowely than the all leg.  At least this restores deterministic
behavior to the build.

Clearly something better needs to happen than either what we're doing
now, or my small band-aid.

Warner

Index: Makefile.i386
===
RCS file: /home/imp/FreeBSD/CVS/src/sys/conf/Makefile.i386,v
retrieving revision 1.212
diff -u -r1.212 Makefile.i386
--- Makefile.i386   2000/10/29 09:47:50 1.212
+++ Makefile.i386   2000/11/13 18:12:18
@@ -270,8 +270,11 @@
 MKMODULESENV=  MAKEOBJDIRPREFIX=${.OBJDIR}/modules KMODDIR=${KODIR}
 
 modules:
-   @mkdir -p ${.OBJDIR}/modules
-   cd $S/modules  env ${MKMODULESENV} ${MAKE} obj all
+   @if [ ! -d ${.OBJDIR}/modules ]; then \
+   echo You must run make depend before building modules; \
+   exit 1; \
+   fi
+   cd $S/modules  env ${MKMODULESENV} ${MAKE} all
 
 modules-depend:
@mkdir -p ${.OBJDIR}/modules


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message



Re: make modules kicks the first module directory twice

2000-11-13 Thread David O'Brien

On Mon, Nov 13, 2000 at 01:19:38PM -0500, Marcel Moolenaar wrote:
 I'll commit a fix with just semi-colons today for all architectures if
 someone hasn't done that already by that time.

Can you post a patch first.  There seems to be some subtleties here that
might make a review useful.
 
-- 
-- David  ([EMAIL PROTECTED])
  GNU is Not Unix / Linux Is Not UniX


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message



Re: make modules kicks the first module directory twice

2000-11-13 Thread David O'Brien

On Mon, Nov 13, 2000 at 12:23:08PM -0700, Warner Losh wrote:
 :  I think that make has no business doing an implicit make obj for the
 :  all target.
 : Someone has to run `make obj' for the modules tree.  How are you doing it
 : locally?
 
 Right now we do it twice.  Once in make dpeend and again in make
 all.  My patch removes it from make all.
..snip.. 
 I'm not sure what you're criteria for a robust world is here.  If the
 directory doesn't exist, it will warn the user.

It doesn't warn the user, it errors out (possibly a suttle distinction
I'm making).  Also in the past a `make depend' for the kernel was not
required.  Just highly suggested.  Are we really prepared to make it a
requirement now?

To tell the truth, IMHO the modules should just build in the current
directory (and thus could share some .o's with the kernel build).

-- 
-- David  ([EMAIL PROTECTED])
  GNU is Not Unix / Linux Is Not UniX


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message



Re: make modules kicks the first module directory twice

2000-11-13 Thread Warner Losh

In message [EMAIL PROTECTED] "David O'Brien" writes:
: It doesn't warn the user, it errors out (possibly a suttle distinction
: I'm making).  Also in the past a `make depend' for the kernel was not
: required.  Just highly suggested.  Are we really prepared to make it a
: requirement now?

I am, but others might not be.

: To tell the truth, IMHO the modules should just build in the current
: directory (and thus could share some .o's with the kernel build).

Me too.  I'm tired of building things twice :-)

Warner


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message



Re: make modules kicks the first module directory twice

2000-11-13 Thread Marcel Moolenaar

David O'Brien wrote:
 
 On Mon, Nov 13, 2000 at 01:19:38PM -0500, Marcel Moolenaar wrote:
  I'll commit a fix with just semi-colons today for all architectures if
  someone hasn't done that already by that time.
 
 Can you post a patch first.  There seems to be some subtleties here that
 might make a review useful.

Sure.

BTW: I'm also looking at Warner's patch. Maybe that's the better fix for
it, but I have to dig into the Makefiles a bit more to get a better
picture...

-- 
Marcel Moolenaar
  mail: [EMAIL PROTECTED] / [EMAIL PROTECTED]
  tel:  (408) 447-4222


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message



Re: make modules kicks the first module directory twice

2000-11-13 Thread Makoto MATSUSHITA


marcel No you can't. $S expands to "../.." which only works for the
marcel first cd in the -jX case. The second cd will fail.

Ouch... give me one more chance to submit a patch. Here's summary:

* src/release/Makefile should use 'module-depend' while
  checking dependancy of modules (not kernel-depend).
* For parallel build: "command  command" - "command; command"
* For parallel build: "make obj depend" - "make obj; make depend"
* Avoid to do run "obj" target again in "modules" target
  (comes from an email of [EMAIL PROTECTED])

-- -
Makoto `MAR' MATSUSHITA

Index: Makefile
===
RCS file: /lab/FreeBSD/FreeBSD.cvs/src/release/Makefile,v
retrieving revision 1.585
diff -c -r1.585 Makefile
*** Makefile2000/11/12 11:04:11 1.585
--- Makefile2000/11/13 05:37:12
***
*** 831,837 
@rm -f ${RD}/kernels/*.ko
@cd ${.CURDIR}/../sys/${MACHINE}/conf  config ${KERNEL}
@cd ${.CURDIR}/../sys/compile/${KERNEL}  \
!   make kernel-depend  \
make ${KERNEL_FLAGS} modules  \
make modules-reinstall DESTDIR=${RD}/kernels  \
  
--- 831,837 
@rm -f ${RD}/kernels/*.ko
@cd ${.CURDIR}/../sys/${MACHINE}/conf  config ${KERNEL}
@cd ${.CURDIR}/../sys/compile/${KERNEL}  \
!   make modules-depend  \
make ${KERNEL_FLAGS} modules  \
make modules-reinstall DESTDIR=${RD}/kernels  \
  
Index: Makefile.alpha
===
RCS file: /lab/FreeBSD/FreeBSD.cvs/src/sys/conf/Makefile.alpha,v
retrieving revision 1.79
diff -c -r1.79 Makefile.alpha
*** Makefile.alpha  2000/10/29 09:47:50 1.79
--- Makefile.alpha  2000/11/14 00:42:39
***
*** 252,258 
echo ${CFILES} | tr -s ' ' '\12' | sed 's/\.c/.o/' | \
  sort -u | comm -23 - dontlink | \
  sed 's,../.*/\(.*.o\),rm -f \1;ln -s ../GENERIC/\1 \1,'  makelinks
!   sh makelinks  rm -f dontlink
  
  kernel-tags:
@[ -f .depend ] || { echo "you must make depend first"; exit 1; }
--- 252,258 
echo ${CFILES} | tr -s ' ' '\12' | sed 's/\.c/.o/' | \
  sort -u | comm -23 - dontlink | \
  sed 's,../.*/\(.*.o\),rm -f \1;ln -s ../GENERIC/\1 \1,'  makelinks
!   sh makelinks; rm -f dontlink
  
  kernel-tags:
@[ -f .depend ] || { echo "you must make depend first"; exit 1; }
***
*** 309,338 
  MKMODULESENV= MAKEOBJDIRPREFIX=${.OBJDIR}/modules KMODDIR=${KODIR}
  
  modules:
!   @mkdir -p ${.OBJDIR}/modules
!   cd $S/modules  env ${MKMODULESENV} ${MAKE} obj all
  
  modules-depend:
@mkdir -p ${.OBJDIR}/modules
!   cd $S/modules  env ${MKMODULESENV} ${MAKE} obj depend
  
  modules-clean:
!   cd $S/modules  env ${MKMODULESENV} ${MAKE} clean
  
  modules-cleandepend:
!   cd $S/modules  env ${MKMODULESENV} ${MAKE} cleandepend
  
  modules-cleandir:
!   cd $S/modules  env ${MKMODULESENV} ${MAKE} cleandir
  
  modules-tags:
!   cd $S/modules  env ${MKMODULESENV} ${MAKE} tags
  
  modules-install modules-install.debug:
!   cd $S/modules  env ${MKMODULESENV} ${MAKE} install
  
  modules-reinstall modules-reinstall.debug:
!   cd $S/modules  env ${MKMODULESENV} ${MAKE} install
  
  config.o:
${NORMAL_C}
--- 309,342 
  MKMODULESENV= MAKEOBJDIRPREFIX=${.OBJDIR}/modules KMODDIR=${KODIR}
  
  modules:
!   @if [ ! -d ${.OBJDIR}/modules ]; then \
!   echo You must run make depend before building modules; \
!   exit 1; \
!   fi
!   cd $S/modules; env ${MKMODULESENV} ${MAKE} all
  
  modules-depend:
@mkdir -p ${.OBJDIR}/modules
!   cd $S/modules; env ${MKMODULESENV} ${MAKE} obj
!   env ${MKMODULESENV} ${MAKE} depend
  
  modules-clean:
!   cd $S/modules; env ${MKMODULESENV} ${MAKE} clean
  
  modules-cleandepend:
!   cd $S/modules; env ${MKMODULESENV} ${MAKE} cleandepend
  
  modules-cleandir:
!   cd $S/modules; env ${MKMODULESENV} ${MAKE} cleandir
  
  modules-tags:
!   cd $S/modules; env ${MKMODULESENV} ${MAKE} tags
  
  modules-install modules-install.debug:
!   cd $S/modules; env ${MKMODULESENV} ${MAKE} install
  
  modules-reinstall modules-reinstall.debug:
!   cd $S/modules; env ${MKMODULESENV} ${MAKE} install
  
  config.o:
${NORMAL_C}
Index: Makefile.i386
===
RCS file: /lab/FreeBSD/FreeBSD.cvs/src/sys/conf/Makefile.i386,v
retrieving revision 1.212
diff -c -r1.212 Makefile.i386
*** Makefile.i386   2000/10/29 09:47:50 1.212
--- Makefile.i386   2000/11/14 00:43:16
***
*** 212,218 
echo ${CFILES} | tr -s ' ' '\12' | sed 's/\.c/.o/' | \
  sort -u | comm -23 - dontlink | \
  sed 's,../.*/\(.*.o\),rm 

Re: make modules kicks the first module directory twice

2000-11-13 Thread Marcel Moolenaar

[-stable removed from the cc list]

Makoto MATSUSHITA wrote:
 
 marcel No you can't. $S expands to "../.." which only works for the
 marcel first cd in the -jX case. The second cd will fail.
 
 Ouch...

Sorry :-)

 give me one more chance to submit a patch. Here's summary:

You're not going to like this:

   modules-depend:
 @mkdir -p ${.OBJDIR}/modules
 !   cd $S/modules; env ${MKMODULESENV} ${MAKE} obj
 !   env ${MKMODULESENV} ${MAKE} depend

For the non parallel case this doesn't work, because the "${MAKE}
depend" is not run from the right directory (ie "$S/modules") because
the second make doesn't get started from the same shell. Example:

gauss% more ~/mf
all:
cd ../marcel; pwd
pwd

*** Non-parallel case

gauss% cd /xtra/ncvs
gauss% make -f ~/mf
cd ../marcel; pwd
/xtra/marcel
pwd
/xtra/ncvs

*** Parallel case

gauss% make -f ~/mf -j2
cd ../marcel; pwd
/xtra/marcel
pwd
/xtra/marcel

...other than that... :-)

-- 
Marcel Moolenaar
  mail: [EMAIL PROTECTED] / [EMAIL PROTECTED]
  tel:  (408) 447-4222


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message



RQ review: [was: Re: make modules kicks the first module directory twice]

2000-11-13 Thread Marcel Moolenaar

Makoto MATSUSHITA wrote:
 
 Ouch... give me one more chance to submit a patch. Here's summary:

I see no reason to not commit Makoto-san's patches with the fix he sent
me for the modules-depend target. The fix is (modulo indentation):

   modules-depend:
 !   cd $S/modules; env ${MKMODULESENV} ${MAKE} obj; \
 !   env ${MKMODULESENV} ${MAKE} depend

It contains everything mentioned so far, which is:

 * src/release/Makefile should use 'module-depend' while
   checking dependancy of modules (not kernel-depend).
 * For parallel build: "command  command" - "command; command"
 * For parallel build: "make obj depend" - "make obj; make depend"
 * Avoid to do run "obj" target again in "modules" target
   (comes from an email of [EMAIL PROTECTED])

Any objections?

(patches follow for your convenience)

 Index: Makefile
 ===
 RCS file: /lab/FreeBSD/FreeBSD.cvs/src/release/Makefile,v
 retrieving revision 1.585
 diff -c -r1.585 Makefile
 *** Makefile2000/11/12 11:04:11 1.585
 --- Makefile2000/11/13 05:37:12
 ***
 *** 831,837 
 @rm -f ${RD}/kernels/*.ko
 @cd ${.CURDIR}/../sys/${MACHINE}/conf  config ${KERNEL}
 @cd ${.CURDIR}/../sys/compile/${KERNEL}  \
 !   make kernel-depend  \
 make ${KERNEL_FLAGS} modules  \
 make modules-reinstall DESTDIR=${RD}/kernels  \
 
 --- 831,837 
 @rm -f ${RD}/kernels/*.ko
 @cd ${.CURDIR}/../sys/${MACHINE}/conf  config ${KERNEL}
 @cd ${.CURDIR}/../sys/compile/${KERNEL}  \
 !   make modules-depend  \
 make ${KERNEL_FLAGS} modules  \
 make modules-reinstall DESTDIR=${RD}/kernels  \
 
 Index: Makefile.alpha
 ===
 RCS file: /lab/FreeBSD/FreeBSD.cvs/src/sys/conf/Makefile.alpha,v
 retrieving revision 1.79
 diff -c -r1.79 Makefile.alpha
 *** Makefile.alpha  2000/10/29 09:47:50 1.79
 --- Makefile.alpha  2000/11/14 00:42:39
 ***
 *** 252,258 
 echo ${CFILES} | tr -s ' ' '\12' | sed 's/\.c/.o/' | \
   sort -u | comm -23 - dontlink | \
   sed 's,../.*/\(.*.o\),rm -f \1;ln -s ../GENERIC/\1 \1,'  makelinks
 !   sh makelinks  rm -f dontlink
 
   kernel-tags:
 @[ -f .depend ] || { echo "you must make depend first"; exit 1; }
 --- 252,258 
 echo ${CFILES} | tr -s ' ' '\12' | sed 's/\.c/.o/' | \
   sort -u | comm -23 - dontlink | \
   sed 's,../.*/\(.*.o\),rm -f \1;ln -s ../GENERIC/\1 \1,'  makelinks
 !   sh makelinks; rm -f dontlink
 
   kernel-tags:
 @[ -f .depend ] || { echo "you must make depend first"; exit 1; }
 ***
 *** 309,338 
   MKMODULESENV= MAKEOBJDIRPREFIX=${.OBJDIR}/modules KMODDIR=${KODIR}
 
   modules:
 !   @mkdir -p ${.OBJDIR}/modules
 !   cd $S/modules  env ${MKMODULESENV} ${MAKE} obj all
 
   modules-depend:
 @mkdir -p ${.OBJDIR}/modules
 !   cd $S/modules  env ${MKMODULESENV} ${MAKE} obj depend
 
   modules-clean:
 !   cd $S/modules  env ${MKMODULESENV} ${MAKE} clean
 
   modules-cleandepend:
 !   cd $S/modules  env ${MKMODULESENV} ${MAKE} cleandepend
 
   modules-cleandir:
 !   cd $S/modules  env ${MKMODULESENV} ${MAKE} cleandir
 
   modules-tags:
 !   cd $S/modules  env ${MKMODULESENV} ${MAKE} tags
 
   modules-install modules-install.debug:
 !   cd $S/modules  env ${MKMODULESENV} ${MAKE} install
 
   modules-reinstall modules-reinstall.debug:
 !   cd $S/modules  env ${MKMODULESENV} ${MAKE} install
 
   config.o:
 ${NORMAL_C}
 --- 309,342 
   MKMODULESENV= MAKEOBJDIRPREFIX=${.OBJDIR}/modules KMODDIR=${KODIR}
 
   modules:
 !   @if [ ! -d ${.OBJDIR}/modules ]; then \
 !   echo You must run make depend before building modules; \
 !   exit 1; \
 !   fi
 !   cd $S/modules; env ${MKMODULESENV} ${MAKE} all
 
   modules-depend:
 @mkdir -p ${.OBJDIR}/modules
 !   cd $S/modules; env ${MKMODULESENV} ${MAKE} obj
 !   env ${MKMODULESENV} ${MAKE} depend
 
   modules-clean:
 !   cd $S/modules; env ${MKMODULESENV} ${MAKE} clean
 
   modules-cleandepend:
 !   cd $S/modules; env ${MKMODULESENV} ${MAKE} cleandepend
 
   modules-cleandir:
 !   cd $S/modules; env ${MKMODULESENV} ${MAKE} cleandir
 
   modules-tags:
 !   cd $S/modules; env ${MKMODULESENV} ${MAKE} tags
 
   modules-install modules-install.debug:
 !   cd $S/modules; env ${MKMODULESENV} ${MAKE} install
 
   modules-reinstall modules-reinstall.debug:
 !   cd $S/modules; env ${MKMODULESENV} ${MAKE} install
 
   config.o:
 ${NORMAL_C}
 Index: Makefile.i386
 ===
 RCS file: /lab/FreeBSD/FreeBSD.cvs/src/sys/conf/Makefile.i386,v
 

Re: make modules kicks the first module directory twice

2000-11-13 Thread Warner Losh

In message [EMAIL PROTECTED] Marcel Moolenaar writes:
: BTW: I'm also looking at Warner's patch. Maybe that's the better fix for
: it, but I have to dig into the Makefiles a bit more to get a better
: picture...

The implications are that make obj isn't done unless you've run make
depend first.  If a new directory is added and a make depend isn't
run, then the modules won't get built into the obj tree, but instead
will be built into $S/modules.  This isn't too bad because unless you
have multiple kernels that are skewed in version that you'll be doing
make installs from from the same tree (which isn't supported,
officially, now, but does accidentally usually work).

Warner



To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message



Re: make modules kicks the first module directory twice

2000-11-12 Thread Warner Losh

In message [EMAIL PROTECTED] Makoto MATSUSHITA writes:
: It does not fix this problem. However, if we separate the execution of
: "make obj" and "make all", we can avoid (again, not *fix*) the problem.
: Maybe this change is reasonable; there is few meaning doing "make
: kernel-depend" in "doMODULES" target.

I think that make has no business doing an implicit make obj for the
all target.  I have been running with patches in my tree that doesn't
do the obj target if .depend exists.  Of course, you have to be more
careful about running make depend in that case, but right now the make
obj in the all target is hiding this problem.

So I would agree with your patch and your proposal.

Warner


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message



Re: make modules kicks the first module directory twice

2000-11-12 Thread Marcel Moolenaar

Makoto MATSUSHITA wrote:
 
 % make -j 2 modules
 cd ../../modules  env MAKEOBJDIRPREFIX=/usr/src/sys/compile/GENERIC/modules 
KMODDIR=/boot/kernel make obj all
 === 3dfx
 === 3dfx
 Warning: Object directory not changed from original /usr/src/sys/modules/3dfx
 (... ok, break it ...)

The problem is in the fact that the Makefile (the one in /sys/conf)
contains something like:

${MAKE} obj all

and

${MAKE} obj depend

The net effect is that these targets are built in parallel, which
obviously isn't right. The following solves the problem (i386 only):

Index: Makefile.i386
===
RCS file: /home/ncvs/src/sys/conf/Makefile.i386,v
retrieving revision 1.212
diff -u -r1.212 Makefile.i386
--- Makefile.i386   2000/10/29 09:47:50 1.212
+++ Makefile.i386   2000/11/13 07:49:00
@@ -271,11 +271,13 @@
 
 modules:
@mkdir -p ${.OBJDIR}/modules
-   cd $S/modules  env ${MKMODULESENV} ${MAKE} obj all
+   cd $S/modules  env ${MKMODULESENV} ${MAKE} obj  \
+   env ${MKMODULESENV} ${MAKE} all
 
 modules-depend:
@mkdir -p ${.OBJDIR}/modules
-   cd $S/modules  env ${MKMODULESENV} ${MAKE} obj depend
+   cd $S/modules  env ${MKMODULESENV} ${MAKE} obj $$ \
+   env ${MKMODULESENV} ${MAKE} depend
 
 modules-clean:
cd $S/modules  env ${MKMODULESENV} ${MAKE} clean

-- 
Marcel Moolenaar
  mail: [EMAIL PROTECTED] / [EMAIL PROTECTED]
  tel:  (408) 447-4222


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message