Re: [kbuild-devel] Re: State of the new config build system
Hi Keith, The main reason is to convert absolute dependency names to $(xxx) followed by a relative name, where xxx is one of the KBUILD_OBJTREE or KBUILD_SRCTREE_nnn variables. This conversion allows users to rename their source and object trees and to compile on one machine and install on another over NFS without being bitten by absolute dependencies. I really need to do that conversion using the current values of the kbuild variables, the variables might have changed on the next make. Yes, I understand, but this could be done as well at the start of the make run, couldn't it? My new design for module symbol versions requires that the version data be stored immediately after the compile. That also requires processing after each compile using the current environment. This sounds worse ... damned modversions, I still think it was one of the biggest mistakes in Linux history and an one which will be probably very hard to get rid of. Anyway, why do you need to process it immediately? Have a nice fortnight -- Martin `MJ' Mares [EMAIL PROTECTED] http://atrey.karlin.mff.cuni.cz/~mj/ Faculty of Math and Physics, Charles University, Prague, Czech Rep., Earth Minimalist definition of maximalism: `more!'. ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
Re: [kbuild-devel] Re: State of the new config build system
Hi Keith,
That is exactly what kbuild 2.5 does. The slowdown occurs when
massaging the -MD dependencies from absolute names to relative path
names. To support separate source and object trees, renaming of trees,
different names in local and NFS mode etc., the massage code needs a
list of where all the files are before it can convert the absolute
dependencies produced by gcc. Reading and indexing that file for every
compile is _slow_.
I didn't follow the thread from the very beginning nor did I study
your makefiles carefully, because I don't have much time for kernel
hacking these days, but maybe I won't miss the pond :)
Is there any reason for processing all the files for each compile
instead of merging them to a single file once at the start of the make?
I use it in one of my projects, there is the relevant part of the
Makefile:
# Black magic with dependencies. It would be more correct to make depend.new
# a prerequisite for depend, but depend.new often has the same timestamp
# as depend which would confuse make a lot and either force remaking anyway
# or (as in current versions of GNU make) erroneously skipping the remaking.
-include depend
depend: force
if [ -s depend.new ] ; then build/mergedeps depend depend.new ; depend.new ;
fi
force:
# Implicit rules
obj/%.o: %.c
DEPENDENCIES_OUTPUT=depend.new $@ $(CC) $(CFLAGS) -c -o $@ $
The DEPENDENCIES_OUTPUT mode is much more convenient than gcc -Mx as it
avoids scattering the relevant information over many files. The mergedeps
script is a simple Perl script which takes care of merging the dependencies
gathered during the previous run of make to the depend file for the next
run. It can do a lot of fixups and translations, here is a trivial
example:
#!/usr/bin/perl
@ARGV == 2 or die Usage: mergedeps base update;
foreach $a (@ARGV) {
open F, $a or next;
$t = ;
while (F) {
$t .= $_;
if (! /\\$/) {
($t =~ /^(.*):/) || die Parse error at $t;
$rules{$1} = $t;
$t = ;
}
}
close F;
}
open(F, . $ARGV[0]) || die Unable to write output file;
foreach $a (sort keys %rules) {
print F $rules{$a};
}
close F;
Have a nice fortnight
--
Martin `MJ' Mares [EMAIL PROTECTED] http://atrey.karlin.mff.cuni.cz/~mj/
Faculty of Math and Physics, Charles University, Prague, Czech Rep., Earth
Light-year? One-third less calories than a regular year.
___
kbuild-devel mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/kbuild-devel
Re: [kbuild-devel] Re: State of the new config build system
On Sun, 6 Jan 2002 09:55:49 +0100, Martin Mares [EMAIL PROTECTED] wrote: Is there any reason for processing all the files for each compile instead of merging them to a single file once at the start of the make? The main reason is to convert absolute dependency names to $(xxx) followed by a relative name, where xxx is one of the KBUILD_OBJTREE or KBUILD_SRCTREE_nnn variables. This conversion allows users to rename their source and object trees and to compile on one machine and install on another over NFS without being bitten by absolute dependencies. I really need to do that conversion using the current values of the kbuild variables, the variables might have changed on the next make. My new design for module symbol versions requires that the version data be stored immediately after the compile. That also requires processing after each compile using the current environment. ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
On Mon, 31 Dec 2001 20:03:59 -0800, Mike Touloumtzis [EMAIL PROTECTED] wrote: On Fri, Dec 28, 2001 at 12:57:58PM +1100, Keith Owens wrote: Unlike the broken make dep, kbuild 2.5 extracts accurate dependencies by using the -MD option of cpp and post processing the cpp list. The post processing code is slow because the current design requires every compile to read a complete list of all the files, giving O(n^2) effects. Mark 2 of the core code will use a shared database with concurrent update so post processing is limited to looking up just the required files, instead of reading the complete list every time. Why not use '$(GCC) -c -Wp,-MD,foo.d foo.c' to generate the dependencies as a side effect of the regular compile step? This enables you to skip the initial dependency preprocessing step entirely, and could lead to a speedup over even the current fastdep system. You still have to massage the dependencies but you can do it based on the side-effect dependency output of the _previous_ build, to whatever degree that output exists. That is exactly what kbuild 2.5 does. The slowdown occurs when massaging the -MD dependencies from absolute names to relative path names. To support separate source and object trees, renaming of trees, different names in local and NFS mode etc., the massage code needs a list of where all the files are before it can convert the absolute dependencies produced by gcc. Reading and indexing that file for every compile is _slow_. Larry McVoy has sent me the source code to an mmapped database (from bitkeeper). Using a shared mmapped database should speed the process up considerably. ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
Re: [kbuild-devel] Re: State of the new config build system
[EMAIL PROTECTED] (Peter Samuelson) wrote on 31.12.01 in
[EMAIL PROTECTED]:
[Alan Cox]
find $TOPDIR -name *.cf -exec cat {} \; Configure.help
[Horst von Brand]
cat `find $TOPDIR -name *.cf` Configure.help #;-)
[Arnaldo Carvalho de Melo]
whatever is faster, do you have trustable benchmark numbers? ;)
Fewer forks vs. increased parallelism ... depends on the nature of your
What parallelism? Neither version seems to have any.
bottlenecks, I guess, and cold vs. hot cache. Or you could have it
both ways:
find $TOPDIR -name \*.cf | xargs -n10 cat Configure.help
...where 10 is tuned by benchmarking. (:
Not *there* you actually do have parallelism.
Yes, its a joke, have a nice 2002 all!
Yeah, same from me..
Yeah, yeah, AOL that stuff ...
MfG Kai
___
kbuild-devel mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
Alan Cox [EMAIL PROTECTED] said:
Something like:
find $TOPDIR -name *.cf -exec cat {} \; Configure.help
Make that:
cat `find $TOPDIR -name *.cf` Configure.help #;-)
--
Horst von Brand [EMAIL PROTECTED]
Casilla 9G, Vin~a del Mar, Chile +56 32 672616
___
kbuild-devel mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
[Alan Cox]
find $TOPDIR -name *.cf -exec cat {} \; Configure.help
[Horst von Brand]
cat `find $TOPDIR -name *.cf` Configure.help #;-)
[Arnaldo Carvalho de Melo]
whatever is faster, do you have trustable benchmark numbers? ;)
Fewer forks vs. increased parallelism ... depends on the nature of your
bottlenecks, I guess, and cold vs. hot cache. Or you could have it
both ways:
find $TOPDIR -name \*.cf | xargs -n10 cat Configure.help
...where 10 is tuned by benchmarking. (:
Yes, its a joke, have a nice 2002 all!
Yeah, same from me..
Peter
___
kbuild-devel mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/kbuild-devel
Re: [kbuild-devel] Re: State of the new config build system
On Saturday 29 December 2001 05:43 pm, Eric S. Raymond wrote: Tom Rini [EMAIL PROTECTED]: unless (ISA or PCI) suppress dependent IDE Just a minor point, but what about non-PCI/ISA ide? The CML1 rules seem to imply that this set is empty. There are, apparently, paralell port IDE devices. I've never seen one, but we've got drivers for them. See PARIDE and paride_devices. Rob ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
Re: [kbuild-devel] Re: State of the new config build system
Just a minor point, but what about non-PCI/ISA ide? The CML1 rules seem to imply that this set is empty. There are, apparently, paralell port IDE devices. I've never seen one, but we've got drivers for them. See PARIDE and paride_devices. There are IDE drives on just about every conceivable bus or interface. ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
Re: [kbuild-devel] Re: State of the new config build system
On Fri, Dec 28, 2001 at 03:39:02PM -0500, Eric S. Raymond wrote: It may be that the reason our experiences have been different is because we focus on different target languages. But I think my experience is an existence proof that there *is* demand for localization and that meeting it can have useful results. Is your native language something different thæn english or Al's? Localization for technical messages sucks. badly. Just take a look at a european computer magazine, you will find lots of english words in the text because there is no german/frensh/whatever one. Trying to use different grammar doesn't help the understanding. Christoph -- Of course it doesn't work. We've performed a software upgrade. ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
Re: [kbuild-devel] Re: State of the new config build system
[EMAIL PROTECTED] said: unless (ISA or PCI) suppress dependent IDE Just a minor point, but what about non-PCI/ISA ide? Eric is merely representing the _existing_ rules. Changing the behaviour can come later - that shouldn't be done at the same time as introducing CML2. -- dwmw2 ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
Re: [kbuild-devel] Re: State of the new config build system
On Sun, Dec 30, 2001 at 05:14:22PM +, David Woodhouse wrote: [EMAIL PROTECTED] said: unless (ISA or PCI) suppress dependent IDE Just a minor point, but what about non-PCI/ISA ide? Eric is merely representing the _existing_ rules. Changing the behaviour can come later - that shouldn't be done at the same time as introducing CML2. Yes, but what I was getting at was that these constraints will change (either because they were incorrect or no longer aplicable). Either way, why not fix bugs now? (since there are non-PCI/ISA ide, which is why I kept that example to start with). -- Tom Rini (TR1265) http://gate.crashing.org/~trini/ ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
Re: [kbuild-devel] Re: State of the new config build system
On Sun, Dec 30, 2001 at 05:14:22PM +, David Woodhouse wrote: [EMAIL PROTECTED] said: unless (ISA or PCI) suppress dependent IDE Just a minor point, but what about non-PCI/ISA ide? Eric is merely representing the _existing_ rules. Changing the behaviour can come later - that shouldn't be done at the same time as introducing CML2. Existing rules allow non-PCI/ISA IDE. Its a bug, not a change of behaviour. -- Russell King ([EMAIL PROTECTED])The developer of ARM Linux http://www.arm.linux.org.uk/personal/aboutme.html ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
Re: [kbuild-devel] Re: State of the new config build system
On Sun, 30 Dec 2001, Christoph Hellwig wrote: On Fri, Dec 28, 2001 at 03:39:02PM -0500, Eric S. Raymond wrote: It may be that the reason our experiences have been different is because we focus on different target languages. But I think my experience is an existence proof that there *is* demand for localization and that meeting it can have useful results. Is your native language something different thæn english or Al's? Localization for technical messages sucks. badly. Just take a look at a european computer magazine, you will find lots of english words in the text because there is no german/frensh/whatever one. Trying to use different grammar doesn't help the understanding. For some people it helps when the text is in e.g. German although the technical words are still English. The most important point I see is: If the tanslation works similar to gettext, IOW there's a seperate directory that contains the complete translations I can't see problem for the normal kernel hacker: You don't have to care about the translations but if someone wants to provide a translation to e.g. Esperanto he can always do so by adding a file with the translated texts. People like you and me who prefer the English version can always use it but other people who prefer the translated messages can use them instead. Christoph cu Adrian ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
At Fri, 28 Dec 2001 23:20:01 + (GMT), Alan Cox wrote: Frankly, I find it very amusing that advocates of i18n efforts tend to be either British or USAnians. Folks, get real - your languages are too close to show where the problems are. I can see how doing that gives you a warm fuzzy feeling, but could you please listen to those of us who have to deal with the resulting mess for real? The biggest advocates I see are from the Middle-East and Japan. We already have people providing translations for Configure.help in several languages. Yes. We JF Project (Japan) is still keeping translating Configure.help into Japanese for the stable kernel version 2.0, 2.2, and 2.4. We have some interest in distributing our translated-Configure.help, but, such distribution needs so-high-precious technical translation. I think to leave quality control of Configure.help from developer is not good, we have to be so careful, and it's a big problem. In addition, I think we need a framework for keeping up to date with the latest Configure.help against translated Configure.help. Consistency between original Configure.help and translated-Configure.help must be kept. IMHO, for example, if CONFIG_FOO is changed between 2.4.16 and 2.4.17, then (translated-)CONFIG_FOO must show in original English, even if we have only 2.4.16-translated-CONFIG_FOO, and so on. -- gotom ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
Re: [kbuild-devel] Re: State of the new config build system
Eric S. Raymond wrote: Linus Torvalds [EMAIL PROTECTED]: Eric, this is the _wrong_approach_. I want /local/ files, not global ones. First: where should the prompt-string definitions for capability symbols that occur in multiple port trees live? Proposal: the main cml script in linux root dir should be as little as possible. it will include all the arch/*/ cml files (for really specifific options) and you move other item into the subdir drivers/ (the natural place, all file who should not be in driver/ are by definition arch specific). Second: Forward references, and references across the tree, mean that there is a class of symbols that have theoretically natural home directories but which would have to be declared elsewhere in order to be defined at the point of first reference. (A potential solution to this would be to improve the CML2 compiler's handling of forward references.) No. CML2 could be improved to handle che forward references, but not user that will use line config. Third: I could hack my installer to break Configure.help up into a bunch of little component CML files distributed through the tree... but Configure.help doesn't currently contain any markup that says where to direct each entry to. The Makefile should help you. Fourth: There's still the localization issue. If it's your ukase that this is not an important problem, then I'll accept that -- but I haven't heard you say that yet, so I'm not sure you've considered it enough. PROPOSAL: You add a tool to build a big file from the sparse symbols.cml. Translator will use this file as references, adn your CML2 will use translated big files or the default sparse little files. This should not be a problem, because a translator will read documentation (unlike the most user), so you can explain how to do this work. (And the 'diff' could be a friend to the translators). kbuild-2.5 have already support for 'clean' driver (clean: driver that don't touch existing files). I like it. If CML2 could handle natively also these change it would great. The problem is the use of multiple sources dir. I think you and Keith should coordinate this work. And I find clean if also configuration files go into makefiles. giacomo PS: Keith: How you handle the obsolete files? (foo.c in the main source. the patch in source src1 will remove this file). Actually I have create a shell script kpatch: a new implementation of scripts/patch-kernel, that handles normal, testing and testing/incr patches, dont-use patches, multiple sources and new (and clean) destination dir). ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
Hi, Taking nothing away from Tridge, I like Tridge, I'd like to see numbers. I'm sure that Tridge's stuff is great, but we were very motivated to go well beyond the normal effort when we wrote this code. How large is your core db stuff? The thing I like about tdb is that it is very simple, only recently growing over 1024 lines. Anton ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
On Fri, 28 Dec 2001, Linus Torvalds wrote: Having per-function comment blocks, in contrast, makes sense to have inline: - you read the comment when you read the function - you might even update the comment when you update the function - you have a reasonable 1:1 relationship. Personally I'd like to see each C file have a header like this too, describing in a few lines what the functions in this file are supposed to do. This should make it easier for people to figure out not just what each C file is about, but also if they should spend their time wading through this particular C file when in search of some piece of code. regards, Rik -- Shortwave goes a long way: irc.starchat.net #swl http://www.surriel.com/ http://distro.conectiva.com/ ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
On Fri, 28 Dec 2001, Legacy Fishtank wrote: A per-driver metadata file is IMHO clearly the preferred solution. Note that it doesn't need to be per-driver: there are good reasons to have combined files too. For example, things like architecture config could all be in one file, along with similar drivers (ie 3com network devices, whatever). Linus ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
On Sat, 29 Dec 2001, Keith Owens wrote: Yes, some of the problems with mkdep can be fixed in the current design but there is one problem that is inherently unfixable. make dep is a manual process so it relies on users knowing when they have to rerun make dep AND THEY DON'T DO IT! Don't be silly. Make the dependency file itself depend on all the files it describes, and add a makefile rule to re-generate it. Poof, problem gone. Dependencies _do_ change when your .config changes, Only if you do them wrong. Look at mkdep.c - it statically determines the complete list of header files that _can_ be included, and does not care at all about what config options there are. that are included varies. gcc -MD gets this exactly right, gcc knows which files it read. Bzzt, it knows the subset of files to read, nothing more. Linus ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
Re: [kbuild-devel] Re: State of the new config build system
On Fri, Dec 28, 2001 at 05:31:51PM -0500, Eric S. Raymond wrote: When I talk about rules that use architecture symbols to suppress things like bus types I have in mind things like this: [snip] unless (ISA or PCI) suppress dependent IDE Just a minor point, but what about non-PCI/ISA ide? unless PCI suppress dependent USB HOTPLUG_PCI And there's hope this will die soon too (USB) ... unless (X86 or ALPHA or MIPS32 or PPC) suppress usb or SPARC or SPARC64 (iirc) or ARM (once !pci usb is allowed)... unless (X86 and PCI and EXPERIMENTAL) or PPC or ARM or SPARC suppress dependent IEEE1394 Wouldn't the experimental be global? And maybe the PCI too? It seems to me *extremely* unlikely that a typical patch from a PPC maintainer would mess with any of these! They're rules that are likely to be written once at the time a new port is added to the tree and seldom or ever changed afterwards. But they will be modified for new arch X, or when constraint X (like PCI) is removed. -- Tom Rini (TR1265) http://gate.crashing.org/~trini/ ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
Re: [kbuild-devel] Re: State of the new config build system
On Sat, Dec 29, 2001 at 05:43:54PM -0500, Eric S. Raymond wrote: Tom Rini [EMAIL PROTECTED]: unless (ISA or PCI) suppress dependent IDE Just a minor point, but what about non-PCI/ISA ide? The CML1 rules seem to imply that this set is empty. RiscPC: CONFIG_PCI=n CONFIG_ISA=n CONFIG_ARCH_ACORN=y Yet, we have in drivers/ide: if [ $CONFIG_ARCH_ACORN = y ]; then dep_bool 'ICS IDE interface support' CONFIG_BLK_DEV_IDE_ICSIDE $CONFIG_ARCH_ACORN dep_bool ' ICS DMA support' CONFIG_BLK_DEV_IDEDMA_ICS $CONFIG_BLK_DEV_IDE_ICSIDE dep_bool 'Use ICS DMA by default' CONFIG_IDEDMA_ICS_AUTO $CONFIG_BLK_DEV_IDEDMA_ICS define_bool CONFIG_BLK_DEV_IDEDMA $CONFIG_BLK_DEV_IDEDMA_ICS dep_bool 'RapIDE interface support' CONFIG_BLK_DEV_IDE_RAPIDE $CONFIG_ARCH_ACORN fi So I guess I've found a bug. -- Russell King ([EMAIL PROTECTED])The developer of ARM Linux http://www.arm.linux.org.uk/personal/aboutme.html ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
Re: [kbuild-devel] Re: State of the new config build system
Russell King [EMAIL PROTECTED]: On Sat, Dec 29, 2001 at 05:43:54PM -0500, Eric S. Raymond wrote: Tom Rini [EMAIL PROTECTED]: unless (ISA or PCI) suppress dependent IDE Just a minor point, but what about non-PCI/ISA ide? The CML1 rules seem to imply that this set is empty. RiscPC: CONFIG_PCI=n CONFIG_ISA=n CONFIG_ARCH_ACORN=y Yet, we have in drivers/ide: if [ $CONFIG_ARCH_ACORN = y ]; then dep_bool 'ICS IDE interface support' CONFIG_BLK_DEV_IDE_ICSIDE $CONFIG_ARCH_ACORN dep_bool ' ICS DMA support' CONFIG_BLK_DEV_IDEDMA_ICS $CONFIG_BLK_DEV_IDE_ICSIDE dep_bool 'Use ICS DMA by default' CONFIG_IDEDMA_ICS_AUTO $CONFIG_BLK_DEV_IDEDMA_ICS define_bool CONFIG_BLK_DEV_IDEDMA $CONFIG_BLK_DEV_IDEDMA_ICS dep_bool 'RapIDE interface support' CONFIG_BLK_DEV_IDE_RAPIDE $CONFIG_ARCH_ACORN fi So I guess I've found a bug. I have removed the constraint in question. -- a href=http://www.tuxedo.org/~esr/;Eric S. Raymond/a Let us hope our weapons are never needed --but do not forget what the common people knew when they demanded the Bill of Rights: An armed citizenry is the first defense, the best defense, and the final defense against tyranny. If guns are outlawed, only the government will have guns. Only the police, the secret police, the military, the hired servants of our rulers. Only the government -- and a few outlaws. I intend to be among the outlaws. -- Edward Abbey, Abbey's Road, 1979 ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
On Fri, Dec 28, 2001 at 01:54:42AM +0100, Dave Jones wrote: How far down the list was make it not take twice as long to build the kernel as kbuild 2.4 ? Keith mentioned O(n^2) effects due to each compile operation needing to reload the dependancies etc. Each compile needs to reload deps??? Ug. IMHO if you are doing to shake up the entire build system, you should Do It Right(tm) and build a -complete- dependency graph -once-. Jeff ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
On Fri, 28 Dec 2001 04:26:48 -0500, Legacy Fishtank [EMAIL PROTECTED] wrote: On Fri, Dec 28, 2001 at 01:54:42AM +0100, Dave Jones wrote: How far down the list was make it not take twice as long to build the kernel as kbuild 2.4 ? Keith mentioned O(n^2) effects due to each compile operation needing to reload the dependancies etc. Each compile needs to reload deps??? Ug. IMHO if you are doing to shake up the entire build system, you should Do It Right(tm) and build a -complete- dependency graph -once-. We have one complete dependency graph for the explicit dependencies. What is slow is extracting the implicit dependencies after an object has been compiled, i.e. the files that it includes. Actually extracting the implicit dependencies is fast, converting them to standard names is fast, what is slow is _reading_ the big list that maps from absolute names to standardized names. I need the big list in order to remove absolute names in the dependency trees. kbuild 2.4 forces a complete recompile if you rename a tree, including if you build on one system then try to install via NFS on a second system. kbuild 2.5 can cope with trees being renamed and trees having different names on local and NFS mounted systems. That flexibility comes at a cost. All I need to do is have one server process that reads the big list once and the other client processes talk to the server. Much less data involved means faster conversion from absolute to standardized names. ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
Ah, OK, I get it. Hey, would it help to have a dbm interface compat library which uses mmap instead of building the db in brk() space? mmap for db file seems to be slower. For basic db hash usage and raw speed nothing seems to touch tdb (Tridge's db hack). Its also portable code which is important since the tool has to be built on the compiling host. Personally I've always considered make dep good enough. Its trying to solve the extra .5% that probably can be solved by careful use of make clean when CML realises a critical rule changed (SMP etc) Alan ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
On Fri, 28 Dec 2001 14:14:37 + (GMT), Alan Cox [EMAIL PROTECTED] wrote: Ah, OK, I get it. Hey, would it help to have a dbm interface compat library which uses mmap instead of building the db in brk() space? mmap for db file seems to be slower. For basic db hash usage and raw speed nothing seems to touch tdb (Tridge's db hack). Its also portable code which is important since the tool has to be built on the compiling host. lm sent me the bk mdbm code but I will look at tdb as well. Four acronyms in one sentance, I must be a phb :). Personally I've always considered make dep good enough. Its trying to solve the extra .5% that probably can be solved by careful use of make clean when CML realises a critical rule changed (SMP etc) http://prdownloads.sourceforge.net/kbuild/kbuild-2.5-history.tar.bz2 Especially makefile-2.5_make_dep.html, 9 reasons why make dep is broken as designed. Some are fixable in the current system, others are inherently unfixable. I skipped that page when I did my presentation at the 2.5 developer's conference. ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
On Fri, Dec 28, 2001 at 02:14:37PM +, Alan Cox wrote: Ah, OK, I get it. Hey, would it help to have a dbm interface compat library which uses mmap instead of building the db in brk() space? mmap for db file seems to be slower. I'll need to see some numbers to back up that statement, please. If you look at the graphs produced by LMbench, they tell you exactly what you need to know. It's true that for very small files, 8K and under, using read() to access them is faster than using mmap, due to the extra work of setting up and tearing down the mapping. To quantify this, a 4KB open/read/close is 500MB/sec, but an open/mmap/access/unmap/close is 425MB/sec. By the time we hit 16K, mmap wins by 15% and just gets better from there. And that all assumes you are doing large reads, which in db code you are not. So mmap will look better even on the small files if you are doing little DB style accesses. For basic db hash usage and raw speed nothing seems to touch tdb (Tridge's db hack). Taking nothing away from Tridge, I like Tridge, I'd like to see numbers. I'm sure that Tridge's stuff is great, but we were very motivated to go well beyond the normal effort when we wrote this code. A multithreaded version of the code that I sent to Keith was doing 455,000 lookups/second on an 8way 200Mhz R4400 SGI box in 1996. Each lookup was locked. If you assume perfect scaling (it was) and you assume the locks took 0 time (they didn't), that's 1.75 usecs for each lookup. On a machine with horrible memory latency and a large dataset. We designed the MDBM code to be scalable (its 64bit clean), portable (runs on 20+ platforms today), multiplatform (metadata is stored in network byte order on disk), and fast (we knew exactly what the instruction and data cache footprint was for hot cache, and we made sure that you did at most 2 disk accesses, 1 was typical, to get at any item in a cold cache). SGI uses this code for their name server, every process mmaps the name server cache. We use this code all over BitKeeper. Its also portable code which is important since the tool has to be built on the compiling host. The code works on Windows, MacOS X, and basically all Unix platforms. Yeah, yeah, I pounding my chest and I'm sorry, but I get beat up all the time that the BK license doesn't let you reuse code and this code is part of BK that we broke out and licensed under the GPL. The point being that if there is reusable code in BK, we're willing to let people use it under whatever license they want. It would be nice if that actually happened after all the yelling and screaming. -- --- Larry McVoy lm at bitmover.com http://www.bitmover.com/lm ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
[ Btw, Jeff, any reason why you changed your name to Legacy Fishtank? It took a few mails before I noticed that it also said garzik in the fine print;] One thing that this big flame-war has brought up is that different people like different things. There may be a simpler solution to this: have the core dependency files generated from some other file format. My pet peeve is centralized knowledge. I absolutely detested the first versions of cml2 for having a single config file, and quite frankly I don't think Eric has even _yet_ separated things out enough - why does the main rules.cml file have architecture-specific info, for example? That's a big step backwards as far as I'm concerned - we didn't use to have those stupid global files, and each architecture could do it's own config rules. Eric never got the point that to me, modularity is _the_ most important thing for maintenance. Something I also asked for the config system at least a year ago was to have Configure.help split up. Never happened. It's still one large ugly file. Driver or architecture maintainers still can't just change _their_ small fragment, they have to touch a global file that they don't own. So if somebody really wants to help this, make scripts that generate config files AND Configure.help files from a distributed set. And once you do that, you could even imagine creating the old-style config files (without the automatic checking and losing some information) from the information. Linus ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
So if somebody really wants to help this, make scripts that generate
config files AND Configure.help files from a distributed set. And once you
do that, you could even imagine creating the old-style config files
Something like:
find $TOPDIR -name *.cf -exec cat {} \; Configure.help
or changing the tools to look for
Documentation/Configure/CONFIG_SMALL_BANANA
??
Alan
___
kbuild-devel mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
Linus Torvalds [EMAIL PROTECTED]: My pet peeve is centralized knowledge. I absolutely detested the first versions of cml2 for having a single config file, and quite frankly I don't think Eric has even _yet_ separated things out enough - why does the main rules.cml file have architecture-specific info, for example? I'm not certain what you're objecting to, and I want to understand it. There are rules that use architecture symbols to suppress things like bus types. I presume that's not a problem for you, but tell me if it is. My best guess is that you're objecting to the archihacks and kernelhacking menus, or the architecture-dependent derivations down around line 330. In general what's going on here is actually the beginnings of an attempt to replace architecture-dependent questions with architecture-*independent* questions. It looks kind of ugly right now because it's too early in the game to mess with the config-symbol namespace -- but, for example, I want to merge the MATH_EMULATION and MATHEMU symbols eventually. And there ought to be a generic set of toggles for kernel-debugging that present to the user as cross-platform capabilities rather than platform- specific switches. In those two menus I've gathered together architecture-specific symbols that I think ought to merge into cross-platform capabilities. But I know there is other cruft in there for historical reasons. Since you've brought up the point, I'll do a cleanup pass on these and see how much I can exile to the arch/*/rules.cml files. There isn't really any help for the ceoss-platform derivations. There are exactly four of these. I've worked hard at holding them to a minimum: derive HAVE_DEC_LOCK from (SMP and (ALPHA or X86_CMPXCHG)) or SPARC or PPC derive HIGHMEM from HIGHMEM4G or HIGHMEM64G or SPARC derive MAC_HID from (ALL_PPC and INPUT!=n) or (MAC and INPUT_ADBHID) derive PC_KEYB from ARM_PC_KEYB or MIPS_PC_KEYB If you notice that each right-hand part includes port symbols from at least two different architectures, I think it will be clearer why these are necessary. CML1's way of doing this had the problem that it was hard to know by inspection of the rulebase under what circumstances a given symbol was actually turned on. This is why CML2 has a rule that each symbol is derived (or occurs in a menu) exactly once. With some work I could relax this restriction, but I don't want to -- it's a major factor in keeping the rulebase's complexity down in the range that a human brain can mentally model. That's a big step backwards as far as I'm concerned - we didn't use to have those stupid global files, and each architecture could do it's own config rules. Eric never got the point that to me, modularity is _the_ most important thing for maintenance. Oh, no, I got that all right. What I have been trying to do is trade off correctly between modularity (which helps maintenance) and the advantages to the configurator *users* of having a global capability namespace, single-apex menu structure, and the symbols-to-prompts mapping in one file. These choices weren't made at random. You don't readily see their advantages because you have a nose-to-the-code, maintainer perspective (quite properly so, in most cases). But in designing the configuration system, simplifying life for *users* is just as important, if not more so. Sometimes this implies not going as far in the direction you favor direction as you might like (monolithic Configure.help is an example). Something I also asked for the config system at least a year ago was to have Configure.help split up. Never happened. It's still one large ugly file. Driver or architecture maintainers still can't just change _their_ small fragment, they have to touch a global file that they don't own. Yes, there are two reasons for this. The contingent, historical reason is that I wanted to get Configure.help in good shape before thinking about dispersing it. That work is now done (though you haven't kept up to date with it). The design reason is that having a single file with all the symbol-to-prompt mappings in it is really helpful when you want to localize the rulebase for another language. I'm still leaning towards keeping symbols.cml together just to make it easier for people to do and distribute translations of it. I think this is an issue that is rising in importance. I have no problem with assuming that kernel hackers are English-literate, but it's no longer an assumption we should make about people *building* kernels. I want to encourage CML2 and question-string localizations for French. And German. And Thai. And Ethiopian. -- a href=http://www.tuxedo.org/~esr/;Eric S. Raymond/a If I were to select a jack-booted group of fascists who are perhaps as large a danger to American society as I could pick today, I would pick BATF [the Bureau of Alcohol, Tobacco, and Firearms]. -- U.S. Representative John Dingell, 1980
[kbuild-devel] Re: State of the new config build system
On Fri, Dec 28, 2001 at 08:42:44PM +1100, Keith Owens wrote: All I need to do is have one server process that reads the big list once and the other client processes talk to the server. Much less data involved means faster conversion from absolute to standardized names. Actually, if you use the mdbm code, you can have a server process which reads the data, stashes it in the db, touchs ./i_am_done, and exits. client processes do a while (!exists(i_am_done)) usleep(10); m = mdbm_open(db, O_RDONLY, 0, 0); val = mdbm_fetch_str(m, key); etc. No sockets, no back and forth, runs at mmap speed. If you tell me what the data looks like that needs to be in the DB, i.e., how to get it, I'll code you up the server side. -- --- Larry McVoy lm at bitmover.com http://www.bitmover.com/lm ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
On Fri, 28 Dec 2001, Eric S. Raymond wrote: The design reason is that having a single file with all the symbol-to-prompt mappings in it is really helpful when you want to localize the rulebase for another language. I'm still leaning towards keeping symbols.cml together just to make it easier for people to do and distribute translations of it. I think this is an issue that is rising in importance. I have no problem with assuming that kernel hackers are English-literate, but it's no longer an assumption we should make about people *building* kernels. I want to encourage CML2 and question-string localizations for French. And German. And Thai. And Ethiopian. You are nuts. OK, you've got these translations. Now what? $FOO adds a new option. Should he, by any chance, supply all relevant translations in the same patch? No? Good, so we are going to have them permanently out of sync. Better yet, option changes its meaning. Now we have English variant with new semantics and Thai one with the old. Happy, happy, joy, joy... And that's aside of the real problem with internationalization - quality of translations _sucks_. Always. Yes, USAnian to English is easy. But that's it. I've tried to use LANG=ru_RU.koi8-r. It had lasted a couple of days. You end up reconstructing the English original and translating it to Russian - and boy, does that process annoy... Frankly, I find it very amusing that advocates of i18n efforts tend to be either British or USAnians. Folks, get real - your languages are too close to show where the problems are. I can see how doing that gives you a warm fuzzy feeling, but could you please listen to those of us who have to deal with the resulting mess for real? ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
More numbers. I coded up a little program (included below) which
reads paths from stdin, lstats() them, and builds an MDBM of inode -
pathname entries. I ran that 10 times on the 2.4 kernel, which had 8679
files matching *.[chSs]. I did a little tuning of page size and inital
DB size (reduces page split costs) and got it down to 105 millisecs from
200, so we're at 12 usecs per item. Then I removed the mdbm_store()
call so I was doing everything except that. That took 7 usecs/item.
Write path summary: the mdbm_store() cost is about 5 usecs/item, which
is about right. To build a DB of the same number of items as source
files in the kernel should cost less than 50 milliseconds for the DB
part of the work. In other words, it's basically free.
OK, on to the read path. I generated the list of inodes as an ascii file
and wrote another program to open the mdbm and fetch each one. Ran that
10 times, it cost 40 milliseconds to look up all the items, so that's
about 4 usecs/item including the read of the data from stdin. That's
slower than I think it should be and I may go look to see what is going
on, but it's plenty fast for the config/build system.
Here's the code. Sorry about the perlisms, wait, no I'm not, I like those,
but it will make you look at it twice before it makes sense.
--
/*
* inode.c - create an MDBM of inode - path mappings
*/
#include sys/types.h
#include sys/stat.h
#include sys/fcntl.h
#include stdio.h
#include unistd.h
#include mdbm.h
#define unless(x) if (!(x))
#define fnext(buf, f) fgets(buf, sizeof(buf), f)
#define u32 unsigned int
void
chomp(char *s)
{
unless (s *s) return;
while (*s (*s != '\n')) s++;
*s = 0;
}
u32
inode(char *path)
{
struct stat sb;
if (lstat(path, sb)) return (0);
return ((u32)sb.st_ino);
}
int
main()
{
charbuf[1024];
MDBM*m;
datum k, v;
u32 ino;
unlink(ino.mdbm);
unless (m = mdbm_open(ino.mdbm, O_RDWR|O_CREAT, 0644, 410)) {
perror(ino.mdbm);
exit(1);
}
mdbm_pre_split(m, 128);
while (fnext(buf, stdin)) {
chomp(buf);
unless (ino = inode(buf)) {
perror(buf);
continue;
}
printf(%u\n, ino);
k.dptr = (void*)ino;
k.dsize = sizeof(u32);
v.dptr = buf;
v.dsize = strlen(buf) + 1;
if (mdbm_store(m, k, v, MDBM_INSERT)) {
perror(buf);
exit(1);
}
}
mdbm_close(m);
exit(0);
}
--
/*
* read.c - read items from the mdbm
*/
#include sys/types.h
#include sys/stat.h
#include sys/fcntl.h
#include stdio.h
#include unistd.h
#include mdbm.h
#define unless(x) if (!(x))
#define fnext(buf, f) fgets(buf, sizeof(buf), f)
#define u32 unsigned int
int
main()
{
charbuf[1024];
MDBM*m;
datum k, v;
u32 ino;
unless (m = mdbm_open(ino.mdbm, O_RDONLY, 0644, 0)) {
perror(ino.mdbm);
exit(1);
}
while (fnext(buf, stdin)) {
ino = atoi(buf);
continue;
k.dptr = (void*)ino;
k.dsize = sizeof(u32);
v = mdbm_fetch(m, k);
unless (v.dsize) {
perror(buf);
exit(1);
}
}
mdbm_close(m);
exit(0);
}
___
kbuild-devel mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
Alexander Viro [EMAIL PROTECTED]: I think this is an issue that is rising in importance. I have no problem with assuming that kernel hackers are English-literate, but it's no longer an assumption we should make about people *building* kernels. I want to encourage CML2 and question-string localizations for French. And German. And Thai. And Ethiopian. You are nuts. OK, you've got these translations. Now what? $FOO adds a new option. Should he, by any chance, supply all relevant translations in the same patch? No? No. The usual way to handle this, of course, is to fall back on the English where you don't have translations. Imperfect, but liveable. Good, so we are going to have them permanently out of sync. Better yet, option changes its meaning. Now we have English variant with new semantics and Thai one with the old. Happy, happy, joy, joy... Which is why there are organized translation groups that do periodic translation updates for software that has registered with them. This doesn't eliminate the problem, but it can keep it within manageable bounds that make having localizations better than not. I deal with this regularly with respect to fetchmail. Anyway, options change semantics only very rarely in the kernel rulebase. Trust me on this as I've been maintaining the CML2 rulebase for 18 months; I have a better handle on the frequency of these events than *anyone* else. You are worrying about a non-problem in this case. And that's aside of the real problem with internationalization - quality of translations _sucks_. Always. No, not always. I read French, Italian, and Spanish; I can puzzle out technical prose in a couple of other languages. I can read fetchmail's .po files and *see* that they don't suck. Frankly, I find it very amusing that advocates of i18n efforts tend to be either British or USAnians. That's not my experience. I've had technical problems with GNU gettext (unrelated to quality of translation) severe enough that I've come very close to dropping localization support twice. The people who plead with me not to drop it have been non-Anglophones. It may be that the reason our experiences have been different is because we focus on different target languages. But I think my experience is an existence proof that there *is* demand for localization and that meeting it can have useful results. -- a href=http://www.tuxedo.org/~esr/;Eric S. Raymond/a I do not find in orthodox Christianity one redeeming feature. -- Thomas Jefferson ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
Legacy Fishtank [EMAIL PROTECTED]: For single-file drivers, I like Becker's (correct credit?) system... about 10 lines of metadata is embedded in a C comment, and it includes the Config.in and Configure.help info. I proposed implementing something like this about a year ago (to replace the nasty centralized knowledge in the MAINTAINERS and CREDITS files) and was shot down. I'd be happy to take another swing at this problem once the kbuild-2.5/CML2 transition is done. But I don't think we should let it block us from having the good results we can get from that change. -- a href=http://www.tuxedo.org/~esr/;Eric S. Raymond/a Those who make peaceful revolution impossible will make violent revolution inevitable. -- John F. Kennedy ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
On Fri, 28 Dec 2001, Alan Cox wrote:
So if somebody really wants to help this, make scripts that generate
config files AND Configure.help files from a distributed set. And once you
do that, you could even imagine creating the old-style config files
Something like:
find $TOPDIR -name *.cf -exec cat {} \; Configure.help
For old tools..
or changing the tools to look for
Documentation/Configure/CONFIG_SMALL_BANANA
small banana? Freud would go wild.
But no. I don't want it under the Documentation directory: I'd much rather
have them _together_ with the config file.
So the config file format could be something that includes the docs, and
you could do something like
find . -name '*.cf' -exec grep '^+' {} \; Configure.help
for old tools, and nw tools would just automatically get the docs from the
same place they get the config info.
And there would _never_ be more than a few entries per config file: you
can imagine having a separate config file for PCI 100Mbps ethernet drivers
and one for ISA drivers.
The current Configure.help is 25k _lines_, and over a megabyte in size. I
would never consider that good taste in programming, why should I consider
it good in documentation?
Linus
___
kbuild-devel mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
On Fri, 28 Dec 2001, Eric S. Raymond wrote: I'm not certain what you're objecting to, and I want to understand it. There are rules that use architecture symbols to suppress things like bus types. I presume that's not a problem for you, but tell me if it is. It _is_ a problem for me, because I want to do diffstat on a patch from a PPC maintainer, and if I see anything non-PPC, loud ringing noises go off in my head. I want that diffstat to say _only_ arch/ppc/... include/asm-ppc/... and nothing else. That way I know that I don't have to worry. In contrast, if it starts talking about Documentation/Configure.help and the main config file, I start worrying. For example, that MATHEMU thing is just ugly. It was perfectly fine in the per-architecture version, now it suddenly has magic dependencies just because different architectures call it different things, and different architectures have different rules on when it's needed. Linus ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
On Fri, 28 Dec 2001, Legacy Fishtank wrote: I think one thing to note is that dependencies is that if you are smart about it, dependencies -really- do not even change when your .config changes. Absolutely. I detest gcc -MD, exactly because it doesn't get this part right. mkdep.c gets this _right_. Linus ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
On Fri, 28 Dec 2001, Eric S. Raymond wrote: Legacy Fishtank [EMAIL PROTECTED]: Note I am specifically NOT talking about MAINTAINERS and CREDITS. -PLEASE- don't obscure my point by mentioning them. It's the same problem, Jeff. Same solution, too. It's not. We already have pre-file credits information - people can put stuff in there for their own (C) messages etc. The MAINTAINERS file is a much higher-level thing which is there to be human-readable. Note that I do _not_ want to mess up source files with magic comments. I absolutely detest those. They only detract from the real job of coding, and do not make anybody happier. We have a hierarchical filesystem. Most drivers already have driver.c driver.h (in fact _very_ few drivers are single-file) and some have a subdirectory of their own. So why not just have driver.conf and be done with it. No point in messing up the C file with stuff that doesn't add any information to either the programmer _or_ the compiler. Then you can make the config file _truly_ readable, and make the format something like define tristate CONFIG_SCSI_AIC7XXX: Adaptec AIC7xxx support This driver supports all of Adaptec's PCI based SCSI controllers (not the hardware RAID controllers though) as well as the aic7770 based EISA and VLB SCSI controllers (the 274x and 284x series). This is an Adaptec sponsored driver written by Justin Gibbs. It is intended to replace the previous aic7xxx driver maintained by Doug Ledford since Doug is no longer maintaining that driver. depends on CONFIG_SCSI depends on CONFIG_PCI depends on ... define integer CONFIG_AIC7XXX_CMDS_PER_DEVICE: Maximum number of TCQ commands per device depends on CONFIG_SCSI_AIC7XXX default value 253 define integer CONFIG_AIC7XXX_RESET_DELAY_MS: Initial bus reset delay in milli-seconds depends on CONFIG_SCSI_AIC7XXX default value 15000 and it's readable and probably trivially parseable into both the existing format (ie some find . -name '*.conf' plus sed-scripts) and into cml2 or whatever. Linus ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
Linus Torvalds [EMAIL PROTECTED]: On Fri, 28 Dec 2001, Eric S. Raymond wrote: I'm not certain what you're objecting to, and I want to understand it. There are rules that use architecture symbols to suppress things like bus types. I presume that's not a problem for you, but tell me if it is. It _is_ a problem for me, because I want to do diffstat on a patch from a PPC maintainer, and if I see anything non-PPC, loud ringing noises go off in my head. I want that diffstat to say _only_ arch/ppc/... include/asm-ppc/... and nothing else. That way I know that I don't have to worry. Perhaps we're talking past each other. I don't understand your objection yet, and I want to so I can design (or redesign) to meet it. When I talk about rules that use architecture symbols to suppress things like bus types I have in mind things like this: unless X86 suppress dependent MCA EISA unless MIPS32 suppress dependent TC unless (PCI and (X86 or SUPERH)) suppress pci_access unless (ISA or PCI) suppress dependent IDE unless PCI suppress dependent USB HOTPLUG_PCI unless (X86 or ALPHA or MIPS32 or PPC) suppress usb unless (X86 and PCI and EXPERIMENTAL) or PPC or ARM or SPARC suppress dependent IEEE1394 unless (M68K or ALL_PPC) suppress MACINTOSH_DRIVERS unless SPARC suppress dependent FC4 unless ARCH_S390==n suppress buses It seems to me *extremely* unlikely that a typical patch from a PPC maintainer would mess with any of these! They're rules that are likely to be written once at the time a new port is added to the tree and seldom or ever changed afterwards. Thus I really don't think you have to worry about spurious spikes in your diffstat. The root rules.cml file will not change very often -- I know this is true, because I can look at the RCS history since I broke it out in response to your request at the Kernel Summit and *see* that changes have been few and sparse. In contrast, if it starts talking about Documentation/Configure.help and the main config file, I start worrying. Rightly so in the latter case. Configure.help patches shouldn't worry you, I don't think. It's not like they can actually break anything. For example, that MATHEMU thing is just ugly. It was perfectly fine in the per-architecture version, now it suddenly has magic dependencies just because different architectures call it different things, and different architectures have different rules on when it's needed. It sounds to me like you're agreeing that it *shouldn't* be called different things, and thus with my goal of cleaning this mess up the rest of the way. Yes? No? Guidance, please. I am, as ever, willing to meet your concerns. But I have to understand clearly what they are in order to do that. -- a href=http://www.tuxedo.org/~esr/;Eric S. Raymond/a ...The Bill of Rights is a literal and absolute document. The First Amendment doesn't say you have a right to speak out unless the government has a 'compelling interest' in censoring the Internet. The Second Amendment doesn't say you have the right to keep and bear arms until some madman plants a bomb. The Fourth Amendment doesn't say you have the right to be secure from search and seizure unless some FBI agent thinks you fit the profile of a terrorist. The government has no right to interfere with any of these freedoms under any circumstances. -- Harry Browne, 1996 USA presidential candidate, Libertarian Party ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
Frankly, I find it very amusing that advocates of i18n efforts tend to be either British or USAnians. Folks, get real - your languages are too close to show where the problems are. I can see how doing that gives you a warm fuzzy feeling, but could you please listen to those of us who have to deal with the resulting mess for real? The biggest advocates I see are from the Middle-East and Japan. We already have people providing translations for Configure.help in several languages. Alan ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
On Fri, 28 Dec 2001, Alan Cox wrote: It would certainly fit nicely with the existing metadata. We already rip out code comments via kernel-doc, and extending it to rip out - Help text - Web site ... No no no. The comments can at least be helpful to programmers, whether ripped out or not. Extra stuff is not helpful to anybody, and is just really irritating. I personally despise source trees that start out with one page of copyright statement crap, it just detracts from the real _point_ of the .c file, which is to contain C code. Making it a comment requirement is - stupid: we have a filesystem, guys - slow: we don't need to parse every C file we encounter when we can just open another file based on filename - nonsensical: many config options are _not_ limited to one C file - hard to parse and read: why limit ourself to C comments, when just keeping the thing logically separated means that we don't have to. Having per-function comment blocks, in contrast, makes sense to have inline: - you read the comment when you read the function - you might even update the comment when you update the function - you have a reasonable 1:1 relationship. _None_ of those are sensible for config file entries. Linus ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
Re: [kbuild-devel] Re: State of the new config build system
Linus Torvalds [EMAIL PROTECTED]: Eric, this is the _wrong_approach_. I want /local/ files, not global ones. I hear you. There are some problems with that, however. First: where should the prompt-string definitions for capability symbols that occur in multiple port trees live? (This is an important question. Right now, most options are low-level and platform-specific, which makes it easy to decide what directory their symbol declaration(s) should live in. But that's not good; there are lots of excellent reasons we want there to be *more* cross-platform capability symbols rather than fewer. So the percentage of roving symbols without an obvious home is likely to go up over time.) Second: Forward references, and references across the tree, mean that there is a class of symbols that have theoretically natural home directories but which would have to be declared elsewhere in order to be defined at the point of first reference. (A potential solution to this would be to improve the CML2 compiler's handling of forward references.) Third: I could hack my installer to break Configure.help up into a bunch of little component CML files distributed through the tree... but Configure.help doesn't currently contain any markup that says where to direct each entry to. (The logical time to split up symbols.cml would be immediately after CML2 goes into the tree, because at that point Configure.help won't be an issue any more.) Fourth: There's still the localization issue. If it's your ukase that this is not an important problem, then I'll accept that -- but I haven't heard you say that yet, so I'm not sure you've considered it enough. So, I can and will put this in the transition plan if that's what you direct. But you need to be aware that it's not a snap-of-the-fingers change, and not something best done before CML1 goes away. -- a href=http://www.tuxedo.org/~esr/;Eric S. Raymond/a As to the species of exercise, I advise the gun. While this gives [only] moderate exercise to the body, it gives boldness, enterprise, and independence to the mind. Games played with the ball and others of that nature, are too violent for the body and stamp no character on the mind. Let your gun, therefore, be the constant companion to your walks. -- Thomas Jefferson, writing to his teenaged nephew. ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
Alan Cox [EMAIL PROTECTED]: I'd be happy to take another swing at this problem once the kbuild-2.5/CML2 transition is done. But I don't think we should let it block us from having the good results we can get from that change. It would certainly fit nicely with the existing metadata. We already rip out code comments via kernel-doc, and extending it to rip out - Help text - Web site - Version information - Man page for the driver - Module options etc, shouldn't be too challenging. Ok so kernel-doc is in perl and ugly perl but if someone hates it enough to rewrite it in python thats a win too 8) I've been thinking about doing that very thing anyway. Part of my master plan to reduce the tree's external dependencies. -- a href=http://www.tuxedo.org/~esr/;Eric S. Raymond/a I do not find in orthodox Christianity one redeeming feature. -- Thomas Jefferson ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
On Fri, 28 Dec 2001, Linus Torvalds wrote: On Fri, 28 Dec 2001, Legacy Fishtank wrote: I think one thing to note is that dependencies is that if you are smart about it, dependencies -really- do not even change when your .config changes. Absolutely. I detest gcc -MD, exactly because it doesn't get this part right. mkdep.c gets this _right_. Well, -MD gets this right. The dependencies it generates will cause a recompile when necessary. Unfortunately, though, it's too good, because the dependency on include/linux/autoconf.h will cause lots of unnecessary recompiles. But yes, it seems possible to replace the -MD dependency file, which depends on a specific config, with a generic dependency file, which knows about our #ifdef CONFIG_XXX and translates them to the corresponding ifeq(CONFIG_,) Makefile syntax. It'd make an interesting project, but it effectively means re-implementing a C preprocessor. I don't think you can blame gcc -MD for not knowing about the kernel's CONFIG_ system, though ;-) From --- #ifdef CONFIG_XXX #include linux/xxx.h #endif #ifdef CONFIG_YYY const int nr = 10; #else const int nr = 100; #endif --- you'd have to generate --- ifeq(CONFIG_XXX,y) DEPS += include/linux/xxx.h endif DEPS += include/config/yyy --- i.e. the include/config trick has to stay any way. I don't think the above is necessary, though, the following does work pretty good (I did it this way, inspired by mec, and I think kbuild-2.5 does it similarly): Generate dependencies for a .o file when compiling it. [ Doing make dep in advance is unnessary. Actually, it's pretty stupid to generate dependencies for *all* possible object files which you are never going to compile (think arch/*). If you don't have the object yet, you don't need to know the dependencies, dependencies only make sense for recompiles. It's also cheaper to generate dependencies during the compile, as you need to read the file anyway. Also, dependencies on generated files cannot be found correctly until these files have been generated. ] The generated dependencies will always include linux/autoconf.h, which is correct, but will cause too many recompiles. So, replace linux/autoconf.h with linux/config/xxx, where xxx are all the config options which appear in all of the files used to build the object file (which is what -MD gave you). The result is still dependencies which are 100% correct. It's that simple. The object file gcc generates depends on the command line and all the files it reads during the compile. Why make it more complex? --Kai ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
cc: list trimmed. On Fri, 28 Dec 2001 12:01:04 -0800, Larry McVoy [EMAIL PROTECTED] wrote: On Fri, Dec 28, 2001 at 08:42:44PM +1100, Keith Owens wrote: All I need to do is have one server process that reads the big list once and the other client processes talk to the server. Much less data involved means faster conversion from absolute to standardized names. Actually, if you use the mdbm code, you can have a server process which reads the data, stashes it in the db, touchs ./i_am_done, and exits. client processes do a while (!exists(i_am_done)) usleep(10); m = mdbm_open(db, O_RDONLY, 0, 0); val = mdbm_fetch_str(m, key); etc. No sockets, no back and forth, runs at mmap speed. If you tell me what the data looks like that needs to be in the DB, i.e., how to get it, I'll code you up the server side. I also want updates from the dependency back end code, to remove the phase 5 processing. The extract dependency code runs after each compile step so there can be multiple updates running in parallel. My gut feeling is that it will be faster to have one database server and all the back ends talk to that server. Otherwise each compile will have overhead for lock, open, mmap, update, close, write back, unlock. A single threading server removes the need for lock/unlock and can sync the data to disk after n compiles instead of being forced to do it after every compile. If your experience says that doing updates from each compile step without a server process would not be too slow, let me know. ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
I also want updates from the dependency back end code, to remove the phase 5 processing. The extract dependency code runs after each compile step so there can be multiple updates running in parallel. My gut feeling is that it will be faster to have one database server and all the back ends talk to that server. Otherwise each compile will have overhead for lock, open, mmap, update, close, write back, unlock. A single threading server removes the need for lock/unlock and can sync the data to disk after n compiles instead of being forced to do it after every compile. If your experience says that doing updates from each compile step without a server process would not be too slow, let me know. You certainly don't need a server process. And as was pointed out earlier, it's nice not to have them, then you don't have to worry about them still being there. I can write you up a multi writer version using in file locks (which work over NFS, we had do that for BK and I'm pretty sure it is platform independent, I can't break it). We have to do this sort of multi reader/writer crud in BK all the time and have lots of experience with locking, breaking locks, waiting, NFS, etc. Much more experience than we ever wanted :-) You don't need to sync to disk at all, let the data sit in memory, that's why mmap is cool. Give me a spec for what you want, I'll crank out some code. Maybe I'll finally actually be useful to the kernel after all these years... -- --- Larry McVoy lm at bitmover.com http://www.bitmover.com/lm ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
On Fri, 28 Dec 2001 14:17:24 -0800 (PST), Linus Torvalds [EMAIL PROTECTED] wrote: On Fri, 28 Dec 2001, Legacy Fishtank wrote: I think one thing to note is that dependencies is that if you are smart about it, dependencies -really- do not even change when your .config changes. Absolutely. I detest gcc -MD, exactly because it doesn't get this part right. mkdep.c gets this _right_. Sorry, it does not. Everybody is attacking little bits of the dependency problem, any solution that does not fix _all_ 9 problems in http://prdownloads.sourceforge.net/kbuild/kbuild-2.5-history.tar.bz2, makefile-2.5_make_dep.html is not a complete fix. Yes, some of the problems with mkdep can be fixed in the current design but there is one problem that is inherently unfixable. make dep is a manual process so it relies on users knowing when they have to rerun make dep AND THEY DON'T DO IT! Please do not say I always run make dep after a change, I guarantee that you are the exception. Users apply patches and do not run make dep, then wonder why their kernel is broken. Dependencies _do_ change when your .config changes, the list of files that are included varies. gcc -MD gets this exactly right, gcc knows which files it read. mkdep does an incorrect approximation, see tyhe bug list in makefile-2.5_make_dep.html. The errors in mkdep were acceptable as long as only kernel hackers built their own kernels, they could be relied upon to manually run commands when necessary. The target population has changed, more and more beginners are building kernels and too many are getting it wrong. I am aiming at the entire population, not that small subset who have been building kernels since the year dot. Any build system that silently fails when users forget to run a command is a broken system. kbuild 2.5 fixes _all_ 9 problems with mkdep, it also positions us for correct modversion handling. kbuild 2.4 is faster, inaccurate and manual, kbuild 2.5 is slower, accurate and totally automatic. I know how to speed up 2.5. What I don't have is time to rewrite the code for speed, I am too busy tracking kernel changes because kbuild 2.5 is not in the kernel yet. Linus, you have a choice between a known broken build system and a clean and reliable system, which is slightly slower in mark 1. Please add kbuild 2.5 to the kernel, then I will have time to rewrite the core programs for speed. Mark 2 of the core code will be significantly faster. ps. I don't want mail discussing individual bug fixes to mkdep. Code that does not fix _all_ 9 bugs listed in makefile-2.5_make_dep.html is pointless. ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
On Fri, 28 Dec 2001 16:16:03 -0500, Legacy Fishtank [EMAIL PROTECTED] wrote: I think one thing to note is that dependencies is that if you are smart about it, dependencies -really- do not even change when your .config changes. What about a system where Linus runs make deps -once- before he releases a tarball. This in turn generates dependency information (perhaps not in purely make format) which includes 'ifdef CONFIG_xxx' information embedded within. We know that make can support ifeq CONFIG_xxx for example... Then people apply patches and break. Please read the list of mkdep bugs before suggesting partial fixes. http://prdownloads.sourceforge.net/kbuild/kbuild-2.5-history.tar.bz2, makefile-2.5_make_dep.html. I want a system that fixes _all_ the bugs, not just some of them. ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
Unlike bio, kbuild 2.5 works, it just needs to be a bit faster. Put kbuild 2.5 in the kernel and I will have a faster version within 2 weeks. Ok. I was assuming from what you had said that we were talking about months before it got up to a sane speed. If its 2 weeks then I have absolutely no problems with that. Alan ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
Larry McVoy wrote: On Fri, Dec 28, 2001 at 12:41:48PM +1100, Keith Owens wrote: On Thu, 27 Dec 2001 17:37:39 -0800, Larry McVoy [EMAIL PROTECTED] wrote: A couple of questions: a) will 2.5 be as fast as the current system? Faster? At the moment kbuild 2.5 ranges from 10% faster on small builds to 100% slower on a full kernel build. I don't understand why it would be slower. Thank's go to basically to python and other excessfull overengineering there. ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
dammit, didn't hit reply all grr On Saturday, December 29, 2001, at 05:02 AM, Linus Torvalds wrote: snip My pet peeve is centralized knowledge. I absolutely detested the first versions of cml2 for having a single config file, and quite frankly I don't think Eric has even _yet_ separated things out enough - why does the main rules.cml file have architecture-specific info, for example? agreed - it's something that really irritates me too. As Linux is running on so many different architectures (some of which are purely virtual, such as Usermode Linux and my whacky idea of running it ontop of MacOS X) so it seems that keeping all the options for architectures separate would make a lot of sense. I've never seen a cross-platform binary kernel (although have had scary dreams of one) snip So if somebody really wants to help this, make scripts that generate config files AND Configure.help files from a distributed set. And once you do that, you could even imagine creating the old-style config files (without the automatic checking and losing some information) from the information. This shouldn't be too hard should it? In each module directory have a config and Configure.help file, then just find . |grep config and then cat all the files together. If I have some spare time today I'll see if I can hack something up :) -- Stewart Smith [EMAIL PROTECTED] Ph: +61 4 3884 4332 ICQ: 6734154 ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
that if there is reusable code in BK, we're willing to let people use it under whatever license they want. It would be nice if that actually happened after all the yelling and screaming. mdbm is one I've not seen. The timings I've done are with db2/db3/tdb when I was playing with a fast UDP server that had to do a db lookup per packet. Alan ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
Linus Torvalds wrote: On Fri, 28 Dec 2001, Alan Cox wrote: It would certainly fit nicely with the existing metadata. We already rip out code comments via kernel-doc, and extending it to rip out - Help text - Web site ... No no no. The comments can at least be helpful to programmers, whether ripped out or not. Extra stuff is not helpful to anybody, and is just really irritating. I personally despise source trees that start out with one page of copyright statement crap, it just detracts from the real _point_ of the .c file, which is to contain C code. Making it a comment requirement is - stupid: we have a filesystem, guys Not quite... It is making moving patches through e-mail around easier... ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
On Fri, Dec 28, 2001 at 10:02:01AM -0800, Linus Torvalds wrote: [ Btw, Jeff, any reason why you changed your name to Legacy Fishtank? It took a few mails before I noticed that it also said garzik in the fine print;] Away-from-home account and a long story :) Jeff ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
On Fri, Dec 28, 2001 at 02:27:37PM -0800, Linus Torvalds wrote: and it's readable and probably trivially parseable into both the existing format (ie some find . -name '*.conf' plus sed-scripts) and into cml2 or whatever. It's even doable within the .c file (and preferable for small drivers). Something like: /* mydriver.c header blah blah */ config_requires(CONFIG_INET); config_option(CONFIG_MY_FAST_CHIP, Help info for this); which gets picked out of the .c files during depend phase, and nullified during compile by means of -Iconfig_system.h would even let us get rid of Makefiles for drivers. Wouldn't being able to just drop a .c file (or a bunch of .c files) into the tree in the right place be great? Eliminating makefiles means eliminating more conflicts, which might mean more time to respond to other issues... -ben -- Fish. ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
I think one thing to note is that dependencies is that if you are smart about it, dependencies -really- do not even change when your .config changes. What about a system where Linus runs make deps -once- before he releases a tarball. This in turn generates dependency information (perhaps not in purely make format) which includes 'ifdef CONFIG_xxx' information embedded within. We know that make can support ifeq CONFIG_xxx for example... Jeff ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
On Fri, Dec 28, 2001 at 03:45:37PM -0500, Eric S. Raymond wrote: Legacy Fishtank [EMAIL PROTECTED]: For single-file drivers, I like Becker's (correct credit?) system... about 10 lines of metadata is embedded in a C comment, and it includes the Config.in and Configure.help info. I proposed implementing something like this about a year ago (to replace the nasty centralized knowledge in the MAINTAINERS and CREDITS files) and was shot down. Note I am specifically NOT talking about MAINTAINERS and CREDITS. -PLEASE- don't obscure my point by mentioning them. Dealing with MAINTAINERS and CREDITS in an automated fashion seems more like pointless masturbation to me. If you want to find out who needs to be CC'd on patches, use your brain like I do. Jeff ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
Linus Torvalds wrote: [ Btw, Jeff, any reason why you changed your name to Legacy Fishtank? It took a few mails before I noticed that it also said garzik in the fine print;] One thing that this big flame-war has brought up is that different people like different things. There may be a simpler solution to this: have the core dependency files generated from some other file format. My pet peeve is centralized knowledge. I absolutely detested the first versions of cml2 for having a single config file, and quite frankly I don't think Eric has even _yet_ separated things out enough - why does the main rules.cml file have architecture-specific info, for example? That's a big step backwards as far as I'm concerned - we didn't use to have those stupid global files, and each architecture could do it's own config rules. Eric never got the point that to me, modularity is _the_ most important thing for maintenance. Something I also asked for the config system at least a year ago was to have Configure.help split up. Never happened. It's still one large ugly file. Driver or architecture maintainers still can't just change _their_ small fragment, they have to touch a global file that they don't own. So if somebody really wants to help this, make scripts that generate config files AND Configure.help files from a distributed set. And once you do that, you could even imagine creating the old-style config files (without the automatic checking and losing some information) from the information. If you go thus far... then I think, that the Configure.help stuff should be embedded inside the driver source code itself. Like for example the postfix MTA code is embedding whole *man* pages there. And *man* pages would be anyway a more appriopriate and classical place where the current Configure.help information should be. Just lift the code over from there (The extraction is even proper awk insead of some perl crap...) and be nearly done. ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
Larry McVoy writes: On Fri, Dec 28, 2001 at 08:42:44PM +1100, Keith Owens wrote: All I need to do is have one server process that reads the big list once and the other client processes talk to the server. Much less data involved means faster conversion from absolute to standardized names. Actually, if you use the mdbm code, you can have a server process which reads the data, stashes it in the db, touchs ./i_am_done, and exits. client processes do a while (!exists(i_am_done)) usleep(10); m = mdbm_open(db, O_RDONLY, 0, 0); val = mdbm_fetch_str(m, key); etc. No sockets, no back and forth, runs at mmap speed. That sounds like a better approach. I got a bit nervous when Keith talked about a server process. Made me think I'm going to have to install some daemon, or I'm going to have a pile of background processes being left behind (no matter how careful you are, you always end up with some leakage of stale processes). Regards, Richard Permanent: [EMAIL PROTECTED] Current: [EMAIL PROTECTED] ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
On Fri, Dec 28, 2001 at 10:02:01AM -0800, Linus Torvalds wrote: Something I also asked for the config system at least a year ago was to have Configure.help split up. Never happened. It's still one large ugly file. Driver or architecture maintainers still can't just change _their_ small fragment, they have to touch a global file that they don't own. So if somebody really wants to help this, make scripts that generate config files AND Configure.help files from a distributed set. And once you do that, you could even imagine creating the old-style config files (without the automatic checking and losing some information) from the information. For single-file drivers, I like Becker's (correct credit?) system... about 10 lines of metadata is embedded in a C comment, and it includes the Config.in and Configure.help info. Jeff ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
On Sat, Dec 29, 2001 at 12:26:49PM +1100, Keith Owens wrote: On Fri, 28 Dec 2001 16:16:03 -0500, Legacy Fishtank [EMAIL PROTECTED] wrote: I think one thing to note is that dependencies is that if you are smart about it, dependencies -really- do not even change when your .config changes. What about a system where Linus runs make deps -once- before he releases a tarball. This in turn generates dependency information (perhaps not in purely make format) which includes 'ifdef CONFIG_xxx' information embedded within. We know that make can support ifeq CONFIG_xxx for example... Then people apply patches and break. s/break/update dependencies/ I assumed this was blindingly obvious, but I guess not. Jeff ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
On Sat, Dec 29, 2001 at 12:27:24PM +1100, Keith Owens wrote: Dependencies _do_ change when your .config changes, the list of files that are included varies. 1) #ifdef CONFIG_FOO #include ... is usually wrong and a bug. But that is a tangent and I digress. 2) Such changes can be expressed without regenerating all dependencies. Linus, you have a choice between a known broken build system and a clean and reliable system, which is slightly slower in mark 1. Please add kbuild 2.5 to the kernel, Your system is known broken because it is 100% slower. My kernel builds work just fine now, your changes gain me nothing, while COSTING me productivity. I see no gains, only costs, with your kbuild-2.5 system as it exists. Keith the target audience is Linus and Alan and ME etc. We are the kernel hackers that perform kernel -development-. Making end-user builds easier is NOT a primary nor secondary nor tertiary goal here. Make my life easier first. Fuck Aunt Tillie. Aunt Tillie can get her kernels from a vendor. Jeff ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
Dave Jones [EMAIL PROTECTED]: Maybe keep them both in the tree until this issue is worked out ? That way those who want to play with kbuild can do so, and those who build a few dozen kernels a day don't have to twiddle thumbs. That is such an unutterably horrible concept that the very tentacles of Cthulhu himself must twitch in dread at the thought. The last thing anyone sane wants to do is have to maintain two parallel build systems at the same time. -- a href=http://www.tuxedo.org/~esr/;Eric S. Raymond/a You know why there's a Second Amendment? In case the government fails to follow the first one. -- Rush Limbaugh, in a moment of unaccustomed profundity 17 Aug 1993 ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
On Thu, Dec 27, 2001 at 07:57:38PM -0500, Eric S. Raymond wrote: Dave Jones [EMAIL PROTECTED]: Maybe keep them both in the tree until this issue is worked out ? That way those who want to play with kbuild can do so, and those who build a few dozen kernels a day don't have to twiddle thumbs. That is such an unutterably horrible concept that the very tentacles of Cthulhu himself must twitch in dread at the thought. The last thing anyone sane wants to do is have to maintain two parallel build systems at the same time. Then it does seem reasonable to ask that the new one is at least as fast as the old one. -- --- Larry McVoy lm at bitmover.com http://www.bitmover.com/lm ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
Re: [kbuild-devel] Re: State of the new config build system
On Fri, 28 Dec 2001 01:54:42 +0100 (CET), Dave Jones [EMAIL PROTECTED] wrote: On Thu, 27 Dec 2001, Eric S. Raymond wrote: ..., and Keith's stuff is stable enough that he's now adding features like kernel-image type selection that were obviously way down his to-do list. How far down the list was make it not take twice as long to build the kernel as kbuild 2.4 ? Keith mentioned O(n^2) effects due to each compile operation needing to reload the dependancies etc. I had to choose between helping other architectures to convert and rewriting the core code to speed everything up. I chose to get other architectures converted, finding some interesting features at the same time. The core code is stable and I will not change it right now, I want stable code to go to Linus. Once Linus takes kbuild 2.5 then I can start on the rewrite, without affecting anybody else. ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
On Thu, 27 Dec 2001 17:15:45 -0800, Larry McVoy [EMAIL PROTECTED] wrote: [talking about kbuild 2.5 speed] Then it does seem reasonable to ask that the new one is at least as fast as the old one. kbuild 2.4 is fast but inaccurate, kbuild 2.5 is slower but accurate. Pick one. I am sure that I can speed up kbuild 2.5 with a rewrite of the core code but I am staying on stable code to send to Linus. ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
Re: [kbuild-devel] Re: State of the new config build system
On Fri, Dec 28, 2001 at 02:22:01AM +0100, Dave Jones wrote: On Thu, 27 Dec 2001, Eric S. Raymond wrote: That is such an unutterably horrible concept that the very tentacles of Cthulhu himself must twitch in dread at the thought. The last thing anyone sane wants to do is have to maintain two parallel build systems at the same time. Funny, I could have sworn I read this was Keith's intention at least for a few pre's. Maybe I misinterpreted his intentions. I think Keith wanted a very small time window tho (~24 hrs, barring big supprises). But if we're going to be worried about the build time, kbuild-2.5 and cml2 aren't co-dependant, yes? I know kbuild-2.5 works w/o cml2, and last I tried (ages ago admitedly) cml2 didn't need kbuild-2.5. So we could, in theory dump cml1 quickly but leave the old Makefiles for a bit longer. Or if Keith thinks he can start on the speed problems soon, just plod along for a few releases. :) -- Tom Rini (TR1265) http://gate.crashing.org/~trini/ ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
On Fri, 28 Dec 2001 02:22:01 +0100 (CET), Dave Jones [EMAIL PROTECTED] wrote: On Thu, 27 Dec 2001, Eric S. Raymond wrote: That is such an unutterably horrible concept that the very tentacles of Cthulhu himself must twitch in dread at the thought. The last thing anyone sane wants to do is have to maintain two parallel build systems at the same time. Funny, I could have sworn I read this was Keith's intention at least for a few pre's. Maybe I misinterpreted his intentions. Only long enough to prove that kbuild 2.5 built kernels that worked. And to give unconverted architectures a kernel that had both old and new kbuild in it for their conversion. Once kbuild 2.5 has proved it works, kbuild 2.4 is coming out. ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
On Thu, 27 Dec 2001 17:37:39 -0800, Larry McVoy [EMAIL PROTECTED] wrote: A couple of questions: a) will 2.5 be as fast as the current system? Faster? At the moment kbuild 2.5 ranges from 10% faster on small builds to 100% slower on a full kernel build. But that is using slow core code which I know I can rewrite to make it significantly faster. b) what's the eta on 2.5? kbuild 2.5 is ready now. I am not even going to start on the core rewrite until Linus takes the existing kbuild 2.5 code. The existing code works and is stable, doing a complete core rewrite just before includeing in the kernel strikes me as stupid. ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
On Fri, Dec 28, 2001 at 12:35:50PM +1100, Keith Owens wrote: On Thu, 27 Dec 2001 17:15:45 -0800, Larry McVoy [EMAIL PROTECTED] wrote: [talking about kbuild 2.5 speed] Then it does seem reasonable to ask that the new one is at least as fast as the old one. kbuild 2.4 is fast but inaccurate, kbuild 2.5 is slower but accurate. Pick one. I am sure that I can speed up kbuild 2.5 with a rewrite of the core code but I am staying on stable code to send to Linus. A couple of questions: a) will 2.5 be as fast as the current system? Faster? b) what's the eta on 2.5? -- --- Larry McVoy lm at bitmover.com http://www.bitmover.com/lm ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
On Fri, Dec 28, 2001 at 12:41:48PM +1100, Keith Owens wrote: On Thu, 27 Dec 2001 17:37:39 -0800, Larry McVoy [EMAIL PROTECTED] wrote: A couple of questions: a) will 2.5 be as fast as the current system? Faster? At the moment kbuild 2.5 ranges from 10% faster on small builds to 100% slower on a full kernel build. I don't understand why it would be slower. Maybe I'm clueless but I thought you were moving more towards a single makefile system, kind of like what BSD had about 15 years ago, you went to /sys/MYMACHINE and typed make and it did the build completely in that directory. You did different configs by running a configure tool that made /sys/MYMACHINE /sys/YOURMACHINE, etc. If this is the general approach, shouldn't this be a lot faster than the current approach? The current approach stats stuff many times. Linux is really good at making stats cheap, but nothing is as good as not doing it twice. Am I completely misunderstanding what kbuild is all about? My apologies if so... -- --- Larry McVoy lm at bitmover.com http://www.bitmover.com/lm ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
Re: [kbuild-devel] Re: State of the new config build system
Tom Rini [EMAIL PROTECTED]: I think Keith wanted a very small time window tho (~24 hrs, barring big supprises). But if we're going to be worried about the build time, kbuild-2.5 and cml2 aren't co-dependant, yes? I know kbuild-2.5 works w/o cml2, and last I tried (ages ago admitedly) cml2 didn't need kbuild-2.5. That's right. CML2 and kbuild-2.5 do not require each other So we could, in theory dump cml1 quickly but leave the old Makefiles for a bit longer. Or if Keith thinks he can start on the speed problems soon, just plod along for a few releases. :) As Keith has pointed out, old kbuild achieves its speed by being broken. That's an argument for plod along, IMHO. -- a href=http://www.tuxedo.org/~esr/;Eric S. Raymond/a (Those) who are trying to read the Second Amendment out of the Constitution by claiming it's not an individual right (are) courting disaster by encouraging others to use the same means to eliminate portions of the Constitution they don't like. -- Alan Dershowitz, Harvard Law School ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
[kbuild-devel] Re: State of the new config build system
Unlike the broken make dep, kbuild 2.5 extracts accurate dependencies by using the -MD option of cpp and post processing the cpp list. The post processing code is slow because the current design requires every compile to read a complete list of all the files, giving O(n^2) effects. Mark 2 of the core code will use a shared database with concurrent update so post processing is limited to looking up just the required files, instead of reading the complete list every time. Ah, OK, I get it. Hey, would it help to have a dbm interface compat library which uses mmap instead of building the db in brk() space? We've got a small, fast one that you can have under any license you like, GPL, LGPL, whatever. We use it all over the BK code. -- --- Larry McVoy lm at bitmover.com http://www.bitmover.com/lm ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
