Re: [kbuild-devel] Re: State of the new config build system
Hi Keith, The main reason is to convert absolute dependency names to $(xxx) followed by a relative name, where xxx is one of the KBUILD_OBJTREE or KBUILD_SRCTREE_nnn variables. This conversion allows users to rename their source and object trees and to compile on one machine and install on another over NFS without being bitten by absolute dependencies. I really need to do that conversion using the current values of the kbuild variables, the variables might have changed on the next make. Yes, I understand, but this could be done as well at the start of the make run, couldn't it? My new design for module symbol versions requires that the version data be stored immediately after the compile. That also requires processing after each compile using the current environment. This sounds worse ... damned modversions, I still think it was one of the biggest mistakes in Linux history and an one which will be probably very hard to get rid of. Anyway, why do you need to process it immediately? Have a nice fortnight -- Martin `MJ' Mares [EMAIL PROTECTED] http://atrey.karlin.mff.cuni.cz/~mj/ Faculty of Math and Physics, Charles University, Prague, Czech Rep., Earth Minimalist definition of maximalism: `more!'. ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
Re: [kbuild-devel] Re: State of the new config build system
Hi Keith,
That is exactly what kbuild 2.5 does. The slowdown occurs when
massaging the -MD dependencies from absolute names to relative path
names. To support separate source and object trees, renaming of trees,
different names in local and NFS mode etc., the massage code needs a
list of where all the files are before it can convert the absolute
dependencies produced by gcc. Reading and indexing that file for every
compile is _slow_.
I didn't follow the thread from the very beginning nor did I study
your makefiles carefully, because I don't have much time for kernel
hacking these days, but maybe I won't miss the pond :)
Is there any reason for processing all the files for each compile
instead of merging them to a single file once at the start of the make?
I use it in one of my projects, there is the relevant part of the
Makefile:
# Black magic with dependencies. It would be more correct to make depend.new
# a prerequisite for depend, but depend.new often has the same timestamp
# as depend which would confuse make a lot and either force remaking anyway
# or (as in current versions of GNU make) erroneously skipping the remaking.
-include depend
depend: force
if [ -s depend.new ] ; then build/mergedeps depend depend.new ; depend.new ;
fi
force:
# Implicit rules
obj/%.o: %.c
DEPENDENCIES_OUTPUT=depend.new $@ $(CC) $(CFLAGS) -c -o $@ $
The DEPENDENCIES_OUTPUT mode is much more convenient than gcc -Mx as it
avoids scattering the relevant information over many files. The mergedeps
script is a simple Perl script which takes care of merging the dependencies
gathered during the previous run of make to the depend file for the next
run. It can do a lot of fixups and translations, here is a trivial
example:
#!/usr/bin/perl
@ARGV == 2 or die Usage: mergedeps base update;
foreach $a (@ARGV) {
open F, $a or next;
$t = ;
while (F) {
$t .= $_;
if (! /\\$/) {
($t =~ /^(.*):/) || die Parse error at $t;
$rules{$1} = $t;
$t = ;
}
}
close F;
}
open(F, . $ARGV[0]) || die Unable to write output file;
foreach $a (sort keys %rules) {
print F $rules{$a};
}
close F;
Have a nice fortnight
--
Martin `MJ' Mares [EMAIL PROTECTED] http://atrey.karlin.mff.cuni.cz/~mj/
Faculty of Math and Physics, Charles University, Prague, Czech Rep., Earth
Light-year? One-third less calories than a regular year.
___
kbuild-devel mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/kbuild-devel
Re: [kbuild-devel] Re: State of the new config build system
On Sun, 6 Jan 2002 09:55:49 +0100, Martin Mares [EMAIL PROTECTED] wrote: Is there any reason for processing all the files for each compile instead of merging them to a single file once at the start of the make? The main reason is to convert absolute dependency names to $(xxx) followed by a relative name, where xxx is one of the KBUILD_OBJTREE or KBUILD_SRCTREE_nnn variables. This conversion allows users to rename their source and object trees and to compile on one machine and install on another over NFS without being bitten by absolute dependencies. I really need to do that conversion using the current values of the kbuild variables, the variables might have changed on the next make. My new design for module symbol versions requires that the version data be stored immediately after the compile. That also requires processing after each compile using the current environment. ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
Re: [kbuild-devel] Re: State of the new config build system
[EMAIL PROTECTED] (Peter Samuelson) wrote on 31.12.01 in
[EMAIL PROTECTED]:
[Alan Cox]
find $TOPDIR -name *.cf -exec cat {} \; Configure.help
[Horst von Brand]
cat `find $TOPDIR -name *.cf` Configure.help #;-)
[Arnaldo Carvalho de Melo]
whatever is faster, do you have trustable benchmark numbers? ;)
Fewer forks vs. increased parallelism ... depends on the nature of your
What parallelism? Neither version seems to have any.
bottlenecks, I guess, and cold vs. hot cache. Or you could have it
both ways:
find $TOPDIR -name \*.cf | xargs -n10 cat Configure.help
...where 10 is tuned by benchmarking. (:
Not *there* you actually do have parallelism.
Yes, its a joke, have a nice 2002 all!
Yeah, same from me..
Yeah, yeah, AOL that stuff ...
MfG Kai
___
kbuild-devel mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/kbuild-devel
Re: [kbuild-devel] Re: State of the new config build system
On Saturday 29 December 2001 05:43 pm, Eric S. Raymond wrote: Tom Rini [EMAIL PROTECTED]: unless (ISA or PCI) suppress dependent IDE Just a minor point, but what about non-PCI/ISA ide? The CML1 rules seem to imply that this set is empty. There are, apparently, paralell port IDE devices. I've never seen one, but we've got drivers for them. See PARIDE and paride_devices. Rob ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
Re: [kbuild-devel] Re: State of the new config build system
Just a minor point, but what about non-PCI/ISA ide? The CML1 rules seem to imply that this set is empty. There are, apparently, paralell port IDE devices. I've never seen one, but we've got drivers for them. See PARIDE and paride_devices. There are IDE drives on just about every conceivable bus or interface. ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
Re: [kbuild-devel] Re: State of the new config build system
On Fri, Dec 28, 2001 at 03:39:02PM -0500, Eric S. Raymond wrote: It may be that the reason our experiences have been different is because we focus on different target languages. But I think my experience is an existence proof that there *is* demand for localization and that meeting it can have useful results. Is your native language something different thæn english or Al's? Localization for technical messages sucks. badly. Just take a look at a european computer magazine, you will find lots of english words in the text because there is no german/frensh/whatever one. Trying to use different grammar doesn't help the understanding. Christoph -- Of course it doesn't work. We've performed a software upgrade. ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
Re: [kbuild-devel] Re: State of the new config build system
[EMAIL PROTECTED] said: unless (ISA or PCI) suppress dependent IDE Just a minor point, but what about non-PCI/ISA ide? Eric is merely representing the _existing_ rules. Changing the behaviour can come later - that shouldn't be done at the same time as introducing CML2. -- dwmw2 ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
Re: [kbuild-devel] Re: State of the new config build system
On Sun, Dec 30, 2001 at 05:14:22PM +, David Woodhouse wrote: [EMAIL PROTECTED] said: unless (ISA or PCI) suppress dependent IDE Just a minor point, but what about non-PCI/ISA ide? Eric is merely representing the _existing_ rules. Changing the behaviour can come later - that shouldn't be done at the same time as introducing CML2. Yes, but what I was getting at was that these constraints will change (either because they were incorrect or no longer aplicable). Either way, why not fix bugs now? (since there are non-PCI/ISA ide, which is why I kept that example to start with). -- Tom Rini (TR1265) http://gate.crashing.org/~trini/ ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
Re: [kbuild-devel] Re: State of the new config build system
On Sun, Dec 30, 2001 at 05:14:22PM +, David Woodhouse wrote: [EMAIL PROTECTED] said: unless (ISA or PCI) suppress dependent IDE Just a minor point, but what about non-PCI/ISA ide? Eric is merely representing the _existing_ rules. Changing the behaviour can come later - that shouldn't be done at the same time as introducing CML2. Existing rules allow non-PCI/ISA IDE. Its a bug, not a change of behaviour. -- Russell King ([EMAIL PROTECTED])The developer of ARM Linux http://www.arm.linux.org.uk/personal/aboutme.html ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
Re: [kbuild-devel] Re: State of the new config build system
On Sun, 30 Dec 2001, Christoph Hellwig wrote: On Fri, Dec 28, 2001 at 03:39:02PM -0500, Eric S. Raymond wrote: It may be that the reason our experiences have been different is because we focus on different target languages. But I think my experience is an existence proof that there *is* demand for localization and that meeting it can have useful results. Is your native language something different thæn english or Al's? Localization for technical messages sucks. badly. Just take a look at a european computer magazine, you will find lots of english words in the text because there is no german/frensh/whatever one. Trying to use different grammar doesn't help the understanding. For some people it helps when the text is in e.g. German although the technical words are still English. The most important point I see is: If the tanslation works similar to gettext, IOW there's a seperate directory that contains the complete translations I can't see problem for the normal kernel hacker: You don't have to care about the translations but if someone wants to provide a translation to e.g. Esperanto he can always do so by adding a file with the translated texts. People like you and me who prefer the English version can always use it but other people who prefer the translated messages can use them instead. Christoph cu Adrian ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
Re: [kbuild-devel] Re: State of the new config build system
Eric S. Raymond wrote: Linus Torvalds [EMAIL PROTECTED]: Eric, this is the _wrong_approach_. I want /local/ files, not global ones. First: where should the prompt-string definitions for capability symbols that occur in multiple port trees live? Proposal: the main cml script in linux root dir should be as little as possible. it will include all the arch/*/ cml files (for really specifific options) and you move other item into the subdir drivers/ (the natural place, all file who should not be in driver/ are by definition arch specific). Second: Forward references, and references across the tree, mean that there is a class of symbols that have theoretically natural home directories but which would have to be declared elsewhere in order to be defined at the point of first reference. (A potential solution to this would be to improve the CML2 compiler's handling of forward references.) No. CML2 could be improved to handle che forward references, but not user that will use line config. Third: I could hack my installer to break Configure.help up into a bunch of little component CML files distributed through the tree... but Configure.help doesn't currently contain any markup that says where to direct each entry to. The Makefile should help you. Fourth: There's still the localization issue. If it's your ukase that this is not an important problem, then I'll accept that -- but I haven't heard you say that yet, so I'm not sure you've considered it enough. PROPOSAL: You add a tool to build a big file from the sparse symbols.cml. Translator will use this file as references, adn your CML2 will use translated big files or the default sparse little files. This should not be a problem, because a translator will read documentation (unlike the most user), so you can explain how to do this work. (And the 'diff' could be a friend to the translators). kbuild-2.5 have already support for 'clean' driver (clean: driver that don't touch existing files). I like it. If CML2 could handle natively also these change it would great. The problem is the use of multiple sources dir. I think you and Keith should coordinate this work. And I find clean if also configuration files go into makefiles. giacomo PS: Keith: How you handle the obsolete files? (foo.c in the main source. the patch in source src1 will remove this file). Actually I have create a shell script kpatch: a new implementation of scripts/patch-kernel, that handles normal, testing and testing/incr patches, dont-use patches, multiple sources and new (and clean) destination dir). ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
Re: [kbuild-devel] Re: State of the new config build system
On Fri, Dec 28, 2001 at 05:31:51PM -0500, Eric S. Raymond wrote: When I talk about rules that use architecture symbols to suppress things like bus types I have in mind things like this: [snip] unless (ISA or PCI) suppress dependent IDE Just a minor point, but what about non-PCI/ISA ide? unless PCI suppress dependent USB HOTPLUG_PCI And there's hope this will die soon too (USB) ... unless (X86 or ALPHA or MIPS32 or PPC) suppress usb or SPARC or SPARC64 (iirc) or ARM (once !pci usb is allowed)... unless (X86 and PCI and EXPERIMENTAL) or PPC or ARM or SPARC suppress dependent IEEE1394 Wouldn't the experimental be global? And maybe the PCI too? It seems to me *extremely* unlikely that a typical patch from a PPC maintainer would mess with any of these! They're rules that are likely to be written once at the time a new port is added to the tree and seldom or ever changed afterwards. But they will be modified for new arch X, or when constraint X (like PCI) is removed. -- Tom Rini (TR1265) http://gate.crashing.org/~trini/ ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
Re: [kbuild-devel] Re: State of the new config build system
On Sat, Dec 29, 2001 at 05:43:54PM -0500, Eric S. Raymond wrote: Tom Rini [EMAIL PROTECTED]: unless (ISA or PCI) suppress dependent IDE Just a minor point, but what about non-PCI/ISA ide? The CML1 rules seem to imply that this set is empty. RiscPC: CONFIG_PCI=n CONFIG_ISA=n CONFIG_ARCH_ACORN=y Yet, we have in drivers/ide: if [ $CONFIG_ARCH_ACORN = y ]; then dep_bool 'ICS IDE interface support' CONFIG_BLK_DEV_IDE_ICSIDE $CONFIG_ARCH_ACORN dep_bool ' ICS DMA support' CONFIG_BLK_DEV_IDEDMA_ICS $CONFIG_BLK_DEV_IDE_ICSIDE dep_bool 'Use ICS DMA by default' CONFIG_IDEDMA_ICS_AUTO $CONFIG_BLK_DEV_IDEDMA_ICS define_bool CONFIG_BLK_DEV_IDEDMA $CONFIG_BLK_DEV_IDEDMA_ICS dep_bool 'RapIDE interface support' CONFIG_BLK_DEV_IDE_RAPIDE $CONFIG_ARCH_ACORN fi So I guess I've found a bug. -- Russell King ([EMAIL PROTECTED])The developer of ARM Linux http://www.arm.linux.org.uk/personal/aboutme.html ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
Re: [kbuild-devel] Re: State of the new config build system
Russell King [EMAIL PROTECTED]: On Sat, Dec 29, 2001 at 05:43:54PM -0500, Eric S. Raymond wrote: Tom Rini [EMAIL PROTECTED]: unless (ISA or PCI) suppress dependent IDE Just a minor point, but what about non-PCI/ISA ide? The CML1 rules seem to imply that this set is empty. RiscPC: CONFIG_PCI=n CONFIG_ISA=n CONFIG_ARCH_ACORN=y Yet, we have in drivers/ide: if [ $CONFIG_ARCH_ACORN = y ]; then dep_bool 'ICS IDE interface support' CONFIG_BLK_DEV_IDE_ICSIDE $CONFIG_ARCH_ACORN dep_bool ' ICS DMA support' CONFIG_BLK_DEV_IDEDMA_ICS $CONFIG_BLK_DEV_IDE_ICSIDE dep_bool 'Use ICS DMA by default' CONFIG_IDEDMA_ICS_AUTO $CONFIG_BLK_DEV_IDEDMA_ICS define_bool CONFIG_BLK_DEV_IDEDMA $CONFIG_BLK_DEV_IDEDMA_ICS dep_bool 'RapIDE interface support' CONFIG_BLK_DEV_IDE_RAPIDE $CONFIG_ARCH_ACORN fi So I guess I've found a bug. I have removed the constraint in question. -- a href=http://www.tuxedo.org/~esr/;Eric S. Raymond/a Let us hope our weapons are never needed --but do not forget what the common people knew when they demanded the Bill of Rights: An armed citizenry is the first defense, the best defense, and the final defense against tyranny. If guns are outlawed, only the government will have guns. Only the police, the secret police, the military, the hired servants of our rulers. Only the government -- and a few outlaws. I intend to be among the outlaws. -- Edward Abbey, Abbey's Road, 1979 ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
Re: [kbuild-devel] Re: State of the new config build system
Linus Torvalds [EMAIL PROTECTED]: Eric, this is the _wrong_approach_. I want /local/ files, not global ones. I hear you. There are some problems with that, however. First: where should the prompt-string definitions for capability symbols that occur in multiple port trees live? (This is an important question. Right now, most options are low-level and platform-specific, which makes it easy to decide what directory their symbol declaration(s) should live in. But that's not good; there are lots of excellent reasons we want there to be *more* cross-platform capability symbols rather than fewer. So the percentage of roving symbols without an obvious home is likely to go up over time.) Second: Forward references, and references across the tree, mean that there is a class of symbols that have theoretically natural home directories but which would have to be declared elsewhere in order to be defined at the point of first reference. (A potential solution to this would be to improve the CML2 compiler's handling of forward references.) Third: I could hack my installer to break Configure.help up into a bunch of little component CML files distributed through the tree... but Configure.help doesn't currently contain any markup that says where to direct each entry to. (The logical time to split up symbols.cml would be immediately after CML2 goes into the tree, because at that point Configure.help won't be an issue any more.) Fourth: There's still the localization issue. If it's your ukase that this is not an important problem, then I'll accept that -- but I haven't heard you say that yet, so I'm not sure you've considered it enough. So, I can and will put this in the transition plan if that's what you direct. But you need to be aware that it's not a snap-of-the-fingers change, and not something best done before CML1 goes away. -- a href=http://www.tuxedo.org/~esr/;Eric S. Raymond/a As to the species of exercise, I advise the gun. While this gives [only] moderate exercise to the body, it gives boldness, enterprise, and independence to the mind. Games played with the ball and others of that nature, are too violent for the body and stamp no character on the mind. Let your gun, therefore, be the constant companion to your walks. -- Thomas Jefferson, writing to his teenaged nephew. ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
Re: [kbuild-devel] Re: State of the new config build system
On Fri, 28 Dec 2001 01:54:42 +0100 (CET), Dave Jones [EMAIL PROTECTED] wrote: On Thu, 27 Dec 2001, Eric S. Raymond wrote: ..., and Keith's stuff is stable enough that he's now adding features like kernel-image type selection that were obviously way down his to-do list. How far down the list was make it not take twice as long to build the kernel as kbuild 2.4 ? Keith mentioned O(n^2) effects due to each compile operation needing to reload the dependancies etc. I had to choose between helping other architectures to convert and rewriting the core code to speed everything up. I chose to get other architectures converted, finding some interesting features at the same time. The core code is stable and I will not change it right now, I want stable code to go to Linus. Once Linus takes kbuild 2.5 then I can start on the rewrite, without affecting anybody else. ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
Re: [kbuild-devel] Re: State of the new config build system
On Fri, Dec 28, 2001 at 02:22:01AM +0100, Dave Jones wrote: On Thu, 27 Dec 2001, Eric S. Raymond wrote: That is such an unutterably horrible concept that the very tentacles of Cthulhu himself must twitch in dread at the thought. The last thing anyone sane wants to do is have to maintain two parallel build systems at the same time. Funny, I could have sworn I read this was Keith's intention at least for a few pre's. Maybe I misinterpreted his intentions. I think Keith wanted a very small time window tho (~24 hrs, barring big supprises). But if we're going to be worried about the build time, kbuild-2.5 and cml2 aren't co-dependant, yes? I know kbuild-2.5 works w/o cml2, and last I tried (ages ago admitedly) cml2 didn't need kbuild-2.5. So we could, in theory dump cml1 quickly but leave the old Makefiles for a bit longer. Or if Keith thinks he can start on the speed problems soon, just plod along for a few releases. :) -- Tom Rini (TR1265) http://gate.crashing.org/~trini/ ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
Re: [kbuild-devel] Re: State of the new config build system
Tom Rini [EMAIL PROTECTED]: I think Keith wanted a very small time window tho (~24 hrs, barring big supprises). But if we're going to be worried about the build time, kbuild-2.5 and cml2 aren't co-dependant, yes? I know kbuild-2.5 works w/o cml2, and last I tried (ages ago admitedly) cml2 didn't need kbuild-2.5. That's right. CML2 and kbuild-2.5 do not require each other So we could, in theory dump cml1 quickly but leave the old Makefiles for a bit longer. Or if Keith thinks he can start on the speed problems soon, just plod along for a few releases. :) As Keith has pointed out, old kbuild achieves its speed by being broken. That's an argument for plod along, IMHO. -- a href=http://www.tuxedo.org/~esr/;Eric S. Raymond/a (Those) who are trying to read the Second Amendment out of the Constitution by claiming it's not an individual right (are) courting disaster by encouraging others to use the same means to eliminate portions of the Constitution they don't like. -- Alan Dershowitz, Harvard Law School ___ kbuild-devel mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/kbuild-devel
