Re: [kbuild-devel] Some feedback on using kbuild
G'day, I'll just briefly unlurk on the subject of kbuild before getting back to the kconfig bughunt. Sam Ravnborg wrote: On Mon, Jun 24, 2002 at 08:49:34AM +1000, Keith Owens wrote: [...] There is absolutely no requirement that all kernel developers uses the same SCM system. Agreed. But this means that when dealing with the kernel build system we need to plan for the case where someone is working in a workarea built from a combination of two or more trees, partial trees, or patches from different sources kept in different SCMs. So we need to plan for the lowest common denominator in SCM systems, which is none at all i.e. getting a bunch of tarballs from ftp.kernel.org and elsewhere. Obviously the kernel build system should work for everyone irrespective of the SCM system in use. This put at least the following demands: 1) Separate OBJ and SRC tree 2) That kbuild does not touch any files in the SRC tree Yes, this is the absolute minimum useful behaviour for a build system. For example autoconf/automake work this way in user space. Note that kbuild 2.4 doesn't work this way (or at least it didn't before Kai got started). Without shadow trees, I had to maintain 8 separate source trees to support those combinations. [...] Manually tracking and replicating changes across multiple source trees takes far too much time and is error prone. All the above look like ordinary problems for me, and we can only agree that some kind of tool support is needed during daily work with this amount of different trees. So the point for discussion is solely: a) Where is this functionality best located b) How to implement this I have seen NO discussion of the above items so far. Therefore I bring it up now together with my opinions. Fair enough. Shadow tree functionality = The functionality to support shadow trees are best located in an SCM tool. If the developer or group in question does not use a SCM tool - or maybe a non-coherent setup with two different SCM tools then there is a need to support this at another place. The best place is in between the kbuild and the trees managed. Sure, this is basically an SCM problem, and in an ideal world would be handled by the SCM tool and kbuild itself would just see a unified view of files. However in the real world we need to plan for the case where the SCM is too dumb to do this and too (ahem) precious to be fixed to do this, or even nonexistant, so we end up having to do it in kbuild. This is the same reason why we have the extraordinary complexity of autoconf and automake for usermode programs: it's regrettably necessary to deal with imperfect and unfixable platforms. How to implement shadow trees = within the SCM tool. It no SCM tool is used then a simple script that takes all updates from one or more separate SRC trees and copy them to a common place could do it. And then afterwards we've got to untangle this mess again, completely reliably and automatically. The beauty of having proper shadow trees is that the trees are kept completely separate at all times except in the compiler. There is *no* danger of ending up sending the wrong patch to the wrong place or checking the wrong file into the wrong SCM. [...] In some (most?) SCM systems, the checkout process on a file resets the timestamp to when it was checked in. I have worked with the following SCM systems: RCS, Clearcase, BitKeeper None of them had this BUG! CVS does this. Maybe there's a way to turn it off, I don't know. Another set of problems with an SCM is separating the change sets when you [...] do the check in. [...] In Cleacase I just establish two different views. No problem. I BitKeeper I just execute bk clone twice. No problem. So I do not recognize the problems that you describe above. But it could be me that is unfamilar with CVS, PRCS, arch, ??? It is a problem with every SCM too dumb to know about change sets, in particular CVS. Remember, CVS is used by everyone too poor for ClearCase and too cautious or oldfashioned for BK. When shadow trees were first suggested I argued against them, but experience has shown that they are better than an SCM system. I agree with Keith, shadow trees rock. I think they are probably the single most useful feature of kbuild 2.5. I fervently hope we end up with shadow trees or something like them by the end of Linux 2.5. Greg. -- the price of civilisation today is a courageous willingness to prevail, with force, if necessary, against whatever vicious and uncomprehending enemies try to strike it down. - Roger Sandall, The Age, 28Sep2001. --- Sponsored by: ThinkGeek at http://www.ThinkGeek.com/ ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
Re: [kbuild-devel] Some feedback on using kbuild
On Tue, 25 Jun 2002 23:06:39 +1000, Greg Banks [EMAIL PROTECTED] wrote: I agree with Keith, shadow trees rock. I think they are probably the single most useful feature of kbuild 2.5. I fervently hope we end up with shadow trees or something like them by the end of Linux 2.5. Greg covered almost all the points I was planning to make. One other point: kbuild must not assume that it is running on Linux. Users must be able to build the Linux kernel from _any_ evironment that supports Posix and has a _small_ set of GNU tools. This includes Solaris, Cygwin and other host systems. kbuild must not assume that non-standard tools such as SCM are available. We do not even assume that yacc, lex, loadkeys or Perl are available. These tools may be needed for doing development on certain drivers but kbuild must not require anything beyond gcc and the standard small set of utilities. Bottom line: to support add on code and patch sets kbuild has to do the work itself. --- Sponsored by: ThinkGeek at http://www.ThinkGeek.com/ ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
Re: [kbuild-devel] Some feedback on using kbuild
[CCs trimmed] [Sam Ravnborg] Obviously the kernel build system should work for everyone irrespective of the SCM system in use. This put at least the following demands: 1) Separate OBJ and SRC tree 2) That kbuild does not touch any files in the SRC tree Agreed. It looks like Kai is close to that point. Actually VPATH does all the work for you if you let it, though it looks as though Kai (along with kaos) is taking a more manual approach to get more control. (Come to think of it, VPATH can almost do shadow trees - though it would fail to comprehend different layers of Makefiles.) The functionality to support shadow trees are best located in an SCM tool. If the developer or group in question does not use a SCM tool - or maybe a non-coherent setup with two different SCM tools then there is a need to support this at another place. The best place is in between the kbuild and the trees managed. [Greg Banks] However in the real world we need to plan for the case where the SCM is too dumb to do this and too (ahem) precious to be fixed to do this, or even nonexistant, so we end up having to do it in kbuild. This is the same reason why we have the extraordinary complexity of autoconf and automake for usermode programs: it's regrettably necessary to deal with imperfect and unfixable platforms. Ah, but note that the Linux kernel build does *not* use autoconf or automake. Why not? Because the platform *is* seen as fixable after all. If your old, weird platform has strings.h instead of string.h - tough, you'll have to fix your platform before you compile mkdep.c or split-include.c. If you don't happen to have a perror() - tough again. We assume not only a decent ANSI C compiler - we assume a specific range of gcc versions. Etc. It no SCM tool is used then a simple script that takes all updates from one or more separate SRC trees and copy them to a common place could do it. And then afterwards we've got to untangle this mess again, completely reliably and automatically. The beauty of having proper shadow trees is that the trees are kept completely separate at all times except in the compiler. There is *no* danger of ending up sending the wrong patch to the wrong place or checking the wrong file into the wrong SCM. There is that. But if you regard cobble together a build-only source tree as the first stage of your build process, this shouldn't be a problem. [...] In some (most?) SCM systems, the checkout process on a file resets the timestamp to when it was checked in. CVS does this. Maybe there's a way to turn it off, I don't know. Huh? Not for me. It sets current time on anything *I* check out. Maybe there's a way to turn that screwy feature *on* I agree with Keith, shadow trees rock. I think they are probably the single most useful feature of kbuild 2.5. I'll have to take your word for it - I guess I'm still at Keith's early stages where he was arguing against them before seeing how useful they apparently really are. Peter --- Sponsored by: ThinkGeek at http://www.ThinkGeek.com/ ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
Re: [kbuild-devel] Some feedback on using kbuild
[Greg Banks] I think the problem of Makefile bits in shadow trees is really quite difficult. Keith's solution of pre-processing Makefiles and Makefile.appends from all the shadow trees into a combined Makefile doesn't handle all the cases but is the best attempt I've seen so far. Agreed.. So now we assume BK? What's next, Python 2.1? Touché. No, my point was not that we can assume BK, but that we can assume the developer is willing to install whatever tools he needs to get the job done. I think the assumption is valid, assuming the developer has some choice in the matter - i.e., do not dictate a specific SCM or even a specific version of gcc (though there is a fairly limited range of acceptible gccs). This would be the case if the build process were simple and linear and consisted of just cobbling together a combined source tree and then building a kernel image. But in my experience it comprises a number of loops where I go back and fix simple compile errors (either my own or the latest IDE breakage from the mainline kernel) and do a partial rebuild. A solution where I have to cobble together 174 MB of kernel source every time I fix a one-line compile error is not useful. Another good point. You seem to be full of those. Anyway, the cobble together script will most likely build a symlink tree, not a whole separate copy, so you probably wouldn't have to rebuild it *every* compile. At least, if I were writing an ad hoc shadow tree preprocessor, that's how I'd do it. Then when you are just fixing one-liners, you aren't adding or removing files from the tree so you don't rebuild the link forest. Here's part of a brand new checkout: Ah, I wasn't thinking 'cvs co / export', I was thinking 'cvs update', which is sane. If you have a brand new checkout, you probably don't have a preexisting set of object files yet, so I'm not convinced that you can actually break your compile this way. (I do agree it's a design bug in cvs.) Peter --- Sponsored by: ThinkGeek at http://www.ThinkGeek.com/ ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
Re: [kbuild-devel] Some feedback on using kbuild
Sam Ravnborg wrote: On Wed, Jun 26, 2002 at 12:36:11AM +1000, Greg Banks wrote: I think the problem of Makefile bits in shadow trees is really quite difficult. Keith's solution of pre-processing Makefiles and Makefile.appends from all the shadow trees into a combined Makefile doesn't handle all the cases but is the best attempt I've seen so far. Keith's approach is the only working attempt to handle shadow trees outside a modern SCM tool. So obviously it is the best so far. Actually the SuperH port uses a sort of poor man's shadow tree, where we get a shadow tree from CVS and then use a script to symlink the shadow tree's files into the main tree. I understand this approach was copied from another port, although I can't remember which. So now we assume BK? What's next, Python 2.1? People are not forced to use BK whatsoever. Last time I looked there were still regulary patches available at kernel.org - where does this come into the picture. Sorry, this was a rhetorical question, I'll try to be clearer next time. But on the other hand, if people want decent support for parrallel development then a modern (not CVS) SCM tools is perferable. Be that BitKeeper, PRCS, arch or whatever. Some people, I believe a significant number, are still stuck in the CVS dark ages. CVS still has some advantages, e.g. you can get a free, public, remote CVS repository at sourceforge.net for basically zero effort. I won't attempt to argue that CVS is better than BK (although I might argue its more practical than PRCS for some tasks). But the reality is that it's still out there, it's mature and it's quirks are well known by those who use it. The reality is also that some people have no SCM at all, our beloved leader being one these until recently. If people refuses to use existing tools solving a specific problem then thay are on their own. I see no point in integrating this deep into kbuild because some people do not want to, or do not understand how to do parrallel development using a proper SCM tool. The problem is that when you're dealing with a combined tree built from source published by multiple people with varying SCM setups (including none at all) you soon end up working with the lowest common denominator of all those SCM setups, i.e. none at all. Hence, to get all the benefits of (say) BK, we *all* need to be running BK. Until we reach this state of Nirvana, tree management will be very very painful for some people. This does not stop any attemp to make a simple wrapper that creates and maintain a BUILD_TREE. To check timestamps and link accordinly should not take too much time, at least not at the second run. Ok, why don't you and Peter Samuelson get together, create such a thing and we can compare it against kbuild2.5? If it's simple and a win, great! Greg. -- the price of civilisation today is a courageous willingness to prevail, with force, if necessary, against whatever vicious and uncomprehending enemies try to strike it down. - Roger Sandall, The Age, 28Sep2001. --- This sf.net email is sponsored by: Jabber Inc. Don't miss the IM event of the season | Special offer for OSDN members! JabConf 2002, Aug. 20-22, Keystone, CO http://www.jabberconf.com/osdn ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Problem with configurators and customized help texts perarchitecture.
Greetings all, Since kernel 2.5.3, the monolithic Configure.help file has been broken up into a hundred or so Config.help files. This division potentially allows for customized help texts for different architectures. For example, the CONFIG_SMP help text is different in arch/i386/Config.help and arch/ppc/Config.help. However, a limitation of the current configurators can make this a moot point. Robert Love recently posted a Configurable NR_CPUS for 2.5.24 patch which adds a help text for CONFIG_NR_CPUS. The help text for his patch is for the i386 arch only. Other architectures could display a help text thusly: For 32-bit architectures: CONFIG_NR_CPUS This allows you to specify the maximum number of CPUs which this kernel will support. The maximum supported value is 32 and the minimum value which makes sense is 2. This is purely to save memory - each supported CPU adds approximately eight kilobytes to the kernel image. For 64-bit architectures: CONFIG_NR_CPUS This allows you to specify the maximum number of CPUs which this kernel will support. The maximum supported value is 64 and the minimum value which makes sense is 2. This is purely to save memory - each supported CPU adds approximately eight kilobytes to the kernel image. I put together a trivial patch which adds the appropriate help text to the 12 other architectures effected by Robert's patch. The problem is that the current configurators generate a list of Config.help files which can have other architectures listed first. Here is a snippet from scripts/header.tk: # Now pick out right help text: set message [exec find . -name Config.help | xargs sed -n I don't know what determines the order in which find finds files, but on my machine I get this subset of files: ./drivers/telephony/Config.help ./arch/sparc64/Config.help ./arch/i386/Config.help ./arch/ppc/8260_io/Config.help The arch/sparc64/Config.help file is found before the arch/i386/Config.help file, so if a CONFIG_NR_CPUS help text is in sparc64, that is what I get when configuring for i386. Now, I realize that this particular problem could be easily solved by simply rewording the help text for CONFIG_NR_CPUS so that one text would be appropriate and accurate for all archs. For those of you wondering why we even have multiple copies of help texts scattered around different arch/xxx/Config.help files, one answer is that some distributions cut out the non-relevant arch source code from their shipped source trees. If customized help texts have any value, the configurators will have to be upgraded so that either non-relevant arches are ignored, or the relevant arch is searched first. Any volunteers? Comments? Steven --- This sf.net email is sponsored by: Jabber Inc. Don't miss the IM event of the season | Special offer for OSDN members! JabConf 2002, Aug. 20-22, Keystone, CO http://www.jabberconf.com/osdn ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
Re: [kbuild-devel] Some feedback on using kbuild
[Sam Ravnborg]
This does not stop any attemp to make a simple wrapper that
creates and maintain a BUILD_TREE.
To check timestamps and link accordinly should not take too much
time, at least not at the second run.
[Greg Banks]
Ok, why don't you and Peter Samuelson get together, create such a thing and
we can compare it against kbuild2.5? If it's simple and a win, great!
Not sure about a win - but the following is certainly simple. It took
me 5 or 10 minutes. Perhaps there are cases it doesn't handle, but
I'm thinking it should work for most things.
TOTALLY UNTESTED, since it's intended solely as a proof of concept.
Also, I don't have any shadow trees already set up.
Peter
#!/bin/sh
# mkshadow: a hack to make a build tree from shadow source trees
#
# If something doesn't like the symlinks, replace 'ln -s' with 'ln'.
# Then you get hard links which should work anywhere, but which easily
# get out of sync with reality when you patch / edit the original
# files, depending on your editor.
if [ -z $KBUILD_SRCTREE_000 ]; then
echo no KBUILD_SRCTREE_000, nothing to do
exit 0
fi
obj=$KBUILD_OBJTREE
test -n $obj || obj=.
src=$(set | sed -ne '/^KBUILD_SRCTREE_[0-9][0-9][0-9]=/s//p')
test -n $src || src=.
cd $obj
for s in $src; do
if [ ${s#/} = $s ]; then
echo source tree '$s' is not absolute
exit 1
elif [ $(cd $s; pwd -P) = $(pwd -P) ]; then
echo source tree $(pwd -P) is also the object tree
echo this is not allowed
exit 1
fi
(cd $s; find * -type d) | xargs mkdir -p;
(cd $s;
exec find * \( -type d -exec mkdir \{} \; \) -o \
\( -type f ! -name \*.prepend ! -name \*.append -print \) ) |
while read f; do
rm -f $f;
ln -s $s/$f $f
done
done
for s in $src; do
(cd $s;
exec find * -type f -name \*.prepend ) |
while read f; do
base=${f%.prepend}
if [ -f $base ]; then mv $base $base.; else touch $base.; fi
cat $s/$f $base. $base
rm $base.
done
(cd $s;
exec find * -type f -name \*.append ) |
while read f; do
base=${f%.append}
if [ -f $base ]; then mv $base $base.; else touch $base.; fi
cat $s/$f $base. $base
rm $base.
done
done
---
This sf.net email is sponsored by: Jabber Inc.
Don't miss the IM event of the season | Special offer for OSDN members!
JabConf 2002, Aug. 20-22, Keystone, CO http://www.jabberconf.com/osdn
___
kbuild-devel mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/kbuild-devel
Re: [kbuild-devel] Some feedback on using kbuild
[I wrote]
(cd $s; find * -type d) | xargs mkdir -p;
(cd $s;
exec find * \( -type d -exec mkdir \{} \; \) -o \
\( -type f ! -name \*.prepend ! -name \*.append -print \) ) |
while read f; do
This is redundant - the second mkdir is not needed.
Should be:
(cd $s; exec find * -type d) | xargs mkdir -p;
(cd $s;
exec find * -type f ! -name \*.prepend ! -name \*.append) |
while read f; do
Still untested,
Peter
PS - I still (unlike Sam) like Keith's kbuild2.5. I think it is a
cleaner and more flexible design than the existing makefiles. I just
don't see the need for explicit shadow tree support. (Separate src
and obj dirs are *definitely* needed, though.)
As Kai continues to hack on the existing Rules.make system, I've
changed my mind about what an unfixable mess it was. Obviously some
of its flaws were fixable after all (or some features of kbuild2.5
were mergeable after all, depending on how you look at it). I still
prefer kbuild2.5, but by a significantly smaller margin than I did
before Kai started.
---
This sf.net email is sponsored by: Jabber Inc.
Don't miss the IM event of the season | Special offer for OSDN members!
JabConf 2002, Aug. 20-22, Keystone, CO http://www.jabberconf.com/osdn
___
kbuild-devel mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/kbuild-devel
