Re: kconfig/kbuild rewite (Re: What's up with CONFIG_BLK_DEV?)
On Sunday 02 September 2007 6:51:50 am Sam Ravnborg wrote: > As for Kconfig the low hanging fruits are not in the tools but in the > structure of the Kconfig files. There are a lot that can be improved > with a decent effort but nobody has stepped up doing so. > The tools could be better too but if the root problem is the structure > of the Kconfig files this is where we should focus and not on the tools. On a semi-related note, I recently wrote a dumb little minimal python parser that converted all the menuconfig help to html: http://kernel.org/doc/menuconfig http://kernel.org/doc/make/menuconfig2html.py I did this by ignoring half of the structure of the files (I was only interested in the help text), but it occurs to me that my current script to create miniconfig files by repeatedly calling "allnoconfig": http://landley.net/hg/firmware/file/fe0e5b641cb4/sources/toys/miniconfig.sh Could probably be replaced by a python script to read the .config, parse the kconfig, understand the dependencies, and spit out the miniconfig, without _too_ much effort. I'll throw it on the todo heap after the other 12 projects I hope to get to this month... > For Kbuild I fail to see anything that demand a rewrite from a structure > view of the Kbuild files. > The Kbuild internal stuff is antoehr story - here a rewrite to a saner > language then GNU make syntax could improve hackability a lot. I agree about getting away from make, but I arrived at the conclusion from a different perspective. I believe make is starting to outlive its usefulness. Rampant opinion follows: Incremental builds are a developer convenience. Users who download the source code to open source projects but who aren't modifying the project tend to do "make all", and nothing else. Source build systems like gentoo generally don't have any "rebuild several variants of the same package incrementally" option, and for many packages changing configuration requires a "make clean" anyway. (Since make doesn't handle configuration dependencies, anybody who _does_ make that work without an intervening make clean implemented extensive infrastructure of their own, on top of make.) As far as release versions are concerned, all make provides is an expected user interface (./configure; make; make install). The infrastructure to calculate dependencies (make's reason to exist) is essentially useless during deployment of release versions. For 90% of the software packages out there, "make all" takes less than 10 seconds on modern hardware. Sometimes the ./configure step takes longer to run than the actual build. (The kernel is not one of these packages, but the kernel is probably the largest open source software development effort in history, at least in terms of the number of developers involved if not absolute code size.) So for all but the largest and most complicated software packages, make doesn't even significantly improve the lives of developers. And those large software packages tend to either reimplement make (XFree86 had ibuild, KDE did cmake, Apache has ant...) because for _large_ packages, make sucks. Kbuild can be seen as yet another such reimplementation, in this case built on top of gnu make rather than by replacing it. The most efficient way to build software these days is to feed all the .c files to gcc in one go, so the optimizer can work on the entire program in one big tree. This can give up about 10% smaller and faster code, assuming you have a few hundred megs of ram which essentially all new development systems do. It's also faster to do this than to do a normal "make all" because you don't re-exec gcc lots of times, and can stay cache-hot more. So for deployment builds, eliminating the granularity of make and batching the compile into larger chunks is functionally superior. This reduces make's job to "call gcc once for each output binary, then do any fancy linker stuff". Intermediate levels of granularity are available, for example the linux kernel source code already produces one .o file per directory (built-in.o). It could compile a directory at a time rather than a file at a time, and check that this one .o file is newer than every other file in the directory or else rebuild it, improving efficiency and reducing build complexity without requiring full 4-minute rebuilds. This is the same kind of "more intelligent batching" optimization people were doing back in the days of reel-to-reel tape. Ask Maddog about it sometime, he's got great stories. :) Using a faster non-optimizing compiler (like tcc) can build even large projects like the entire Linux kernel in the 10 second range. (For example, http://fabrice.bellard.free.fr/tcc/tccboot.html took 15 seconds to compile the linux kernel on a Pentium 4. A modern 64-bit core 2 duo is noticeably faster than this.) The resulting code has some downsides (inefficient, and tcc isn't finished yet: I'm still working on getting tcc
Re: kconfig/kbuild rewite (Re: What's up with CONFIG_BLK_DEV?)
On Sun, Sep 02, 2007 at 01:51:50PM +0200, Sam Ravnborg wrote: [] > Then as now you have not yet expalined what you are trying to do. > Nevertheless I look forward for a minmal set of patches that improve > whatever you are working with. Yes, because it's LKML, that wants not-hand-waving stuff in first place. But recent kevent, scheduler stuff shows inflexibility/agendas, though. That examples were bad WRT kernel production infrastructure. Even then i doubt, i must waste everyone's time with details, only goals. > As for Kconfig the low hanging fruits are not in the tools but in the > structure of the Kconfig files. There are a lot that can be improved > with a decent effort but nobody has stepped up doing so. > The tools could be better too but if the root problem is the structure > of the Kconfig files this is where we should focus and not on the tools. In my view this all interconnected. Designing flexible and easy configuration system yields dramatic changes to the build one as well. * profiles non/debug, non/production * per file, per algorithm tuning * efficiency * choosing structure sizes * selecting fast/slow paths * per case choosing need/dead code * various parameters * optimization * per compiler/version * option profiles * feature/warning sets * linker * is there anything alternative? * distributed development * open possibility to work in any part of the tree * making changes and quickly having * config (dependency, etc.) set/UI ready * per profile/option test builds (e.g. making return->goto or loop change and quickly getting -O0, -O2, -Os images; check size; have userspace testing skeleton -> have runtime test) * integration with quilt-like source/patch managers ``here'' * allow per architecture development * small source tree * developer's profiles, that will have exact feature/tuning/build config options results for everybody within given source tree version (for easy testing, but not "send me your .config; what binutils?..") * base set of tools to have easy to configure alternatives * shell to use basic POSIX (plus accepted, not NIH like in bash) features (i have some examples; unfortunately even basic set behaves differently and buggy) * make stat() wrapper executing shell everywhere; of course there are some features, but anyway, interface for it and the like is needed * perl/python/ruby establish text processing rules * coreutils/busybox/etc non is perfect, having replacement mechanism allows faster debug and enhancement of their own development and testing * UI (maybe next time) Only one thing. I don't have time and will to study all that ncurses/slang/qt/gtk/AJAX/whatever stuff. I wanted to do basic terminal or text/stream editor friendly user interface. As for the former, i just upset about software capabilities of the todays terminal emulators. I'm fine with exchanging escape sequences, but all that inherited TEKTRONIX 4010, APL, HP2645, Microterm ACT-IV, Ann Arbor 4080, LSI ADM-3a (man terminfo) legacy without even a hint of progress last 20 years is just dead. I likely to end up with shell script generation, that will be available for everybody who knows shell and have ordinary text editor. autoconf/configure inside out? Maybe, but at least from the new sheet of paper, with good background in history and basic text processing tools. Just in case anybody cares about how ugly modern software development is (INA software industry dude, and may be just crazy, of course). Well, recent Rusty's gig may give a clue, how things may look different. > For Kbuild I fail to see anything that demand a rewrite from a structure > view of the Kbuild files. > The Kbuild internal stuff is antoehr story - here a rewrite to a saner > language then GNU make syntax could improve hackability a lot. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: kconfig/kbuild rewite (Re: What's up with CONFIG_BLK_DEV?)
On Sun, Sep 02, 2007 at 01:51:50PM +0200, Sam Ravnborg wrote: [] Then as now you have not yet expalined what you are trying to do. Nevertheless I look forward for a minmal set of patches that improve whatever you are working with. Yes, because it's LKML, that wants not-hand-waving stuff in first place. But recent kevent, scheduler stuff shows inflexibility/agendas, though. That examples were bad WRT kernel production infrastructure. Even then i doubt, i must waste everyone's time with details, only goals. As for Kconfig the low hanging fruits are not in the tools but in the structure of the Kconfig files. There are a lot that can be improved with a decent effort but nobody has stepped up doing so. The tools could be better too but if the root problem is the structure of the Kconfig files this is where we should focus and not on the tools. In my view this all interconnected. Designing flexible and easy configuration system yields dramatic changes to the build one as well. * profiles non/debug, non/production * per file, per algorithm tuning * efficiency * choosing structure sizes * selecting fast/slow paths * per case choosing need/dead code * various parameters * optimization * per compiler/version * option profiles * feature/warning sets * linker * is there anything alternative? * distributed development * open possibility to work in any part of the tree * making changes and quickly having * config (dependency, etc.) set/UI ready * per profile/option test builds (e.g. making return-goto or loop change and quickly getting -O0, -O2, -Os images; check size; have userspace testing skeleton - have runtime test) * integration with quilt-like source/patch managers ``here'' * allow per architecture development * small source tree * developer's profiles, that will have exact feature/tuning/build config options results for everybody within given source tree version (for easy testing, but not send me your .config; what binutils?..) * base set of tools to have easy to configure alternatives * shell to use basic POSIX (plus accepted, not NIH like in bash) features (i have some examples; unfortunately even basic set behaves differently and buggy) * make stat() wrapper executing shell everywhere; of course there are some features, but anyway, interface for it and the like is needed * perl/python/ruby establish text processing rules * coreutils/busybox/etc non is perfect, having replacement mechanism allows faster debug and enhancement of their own development and testing * UI (maybe next time) Only one thing. I don't have time and will to study all that ncurses/slang/qt/gtk/AJAX/whatever stuff. I wanted to do basic terminal or text/stream editor friendly user interface. As for the former, i just upset about software capabilities of the todays terminal emulators. I'm fine with exchanging escape sequences, but all that inherited TEKTRONIX 4010, APL, HP2645, Microterm ACT-IV, Ann Arbor 4080, LSI ADM-3a (man terminfo) legacy without even a hint of progress last 20 years is just dead. I likely to end up with shell script generation, that will be available for everybody who knows shell and have ordinary text editor. autoconf/configure inside out? Maybe, but at least from the new sheet of paper, with good background in history and basic text processing tools. Just in case anybody cares about how ugly modern software development is (INA software industry dude, and may be just crazy, of course). Well, recent Rusty's gig may give a clue, how things may look different. For Kbuild I fail to see anything that demand a rewrite from a structure view of the Kbuild files. The Kbuild internal stuff is antoehr story - here a rewrite to a saner language then GNU make syntax could improve hackability a lot. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: kconfig/kbuild rewite (Re: What's up with CONFIG_BLK_DEV?)
On Sunday 02 September 2007 6:51:50 am Sam Ravnborg wrote: As for Kconfig the low hanging fruits are not in the tools but in the structure of the Kconfig files. There are a lot that can be improved with a decent effort but nobody has stepped up doing so. The tools could be better too but if the root problem is the structure of the Kconfig files this is where we should focus and not on the tools. On a semi-related note, I recently wrote a dumb little minimal python parser that converted all the menuconfig help to html: http://kernel.org/doc/menuconfig http://kernel.org/doc/make/menuconfig2html.py I did this by ignoring half of the structure of the files (I was only interested in the help text), but it occurs to me that my current script to create miniconfig files by repeatedly calling allnoconfig: http://landley.net/hg/firmware/file/fe0e5b641cb4/sources/toys/miniconfig.sh Could probably be replaced by a python script to read the .config, parse the kconfig, understand the dependencies, and spit out the miniconfig, without _too_ much effort. I'll throw it on the todo heap after the other 12 projects I hope to get to this month... For Kbuild I fail to see anything that demand a rewrite from a structure view of the Kbuild files. The Kbuild internal stuff is antoehr story - here a rewrite to a saner language then GNU make syntax could improve hackability a lot. I agree about getting away from make, but I arrived at the conclusion from a different perspective. I believe make is starting to outlive its usefulness. Rampant opinion follows: Incremental builds are a developer convenience. Users who download the source code to open source projects but who aren't modifying the project tend to do make all, and nothing else. Source build systems like gentoo generally don't have any rebuild several variants of the same package incrementally option, and for many packages changing configuration requires a make clean anyway. (Since make doesn't handle configuration dependencies, anybody who _does_ make that work without an intervening make clean implemented extensive infrastructure of their own, on top of make.) As far as release versions are concerned, all make provides is an expected user interface (./configure; make; make install). The infrastructure to calculate dependencies (make's reason to exist) is essentially useless during deployment of release versions. For 90% of the software packages out there, make all takes less than 10 seconds on modern hardware. Sometimes the ./configure step takes longer to run than the actual build. (The kernel is not one of these packages, but the kernel is probably the largest open source software development effort in history, at least in terms of the number of developers involved if not absolute code size.) So for all but the largest and most complicated software packages, make doesn't even significantly improve the lives of developers. And those large software packages tend to either reimplement make (XFree86 had ibuild, KDE did cmake, Apache has ant...) because for _large_ packages, make sucks. Kbuild can be seen as yet another such reimplementation, in this case built on top of gnu make rather than by replacing it. The most efficient way to build software these days is to feed all the .c files to gcc in one go, so the optimizer can work on the entire program in one big tree. This can give up about 10% smaller and faster code, assuming you have a few hundred megs of ram which essentially all new development systems do. It's also faster to do this than to do a normal make all because you don't re-exec gcc lots of times, and can stay cache-hot more. So for deployment builds, eliminating the granularity of make and batching the compile into larger chunks is functionally superior. This reduces make's job to call gcc once for each output binary, then do any fancy linker stuff. Intermediate levels of granularity are available, for example the linux kernel source code already produces one .o file per directory (built-in.o). It could compile a directory at a time rather than a file at a time, and check that this one .o file is newer than every other file in the directory or else rebuild it, improving efficiency and reducing build complexity without requiring full 4-minute rebuilds. This is the same kind of more intelligent batching optimization people were doing back in the days of reel-to-reel tape. Ask Maddog about it sometime, he's got great stories. :) Using a faster non-optimizing compiler (like tcc) can build even large projects like the entire Linux kernel in the 10 second range. (For example, http://fabrice.bellard.free.fr/tcc/tccboot.html took 15 seconds to compile the linux kernel on a Pentium 4. A modern 64-bit core 2 duo is noticeably faster than this.) The resulting code has some downsides (inefficient, and tcc isn't finished yet: I'm still working on getting tcc to build an unmodified
Re: kconfig/kbuild rewite (Re: What's up with CONFIG_BLK_DEV?)
On Sun, Sep 02, 2007 at 02:02:37AM +0200, Oleg Verych wrote: > * Sun, 26 Aug 2007 01:08:28 -0500 > * Organization: Boundaries Unlimited > > > [] > > Also "here's a symbol, show me a menu containing everything else that is > > either required by or enabled by this symbol..." That sounds like a more > > powerful abstraction, since the previous one is "show me everything that > > depends on CONFIG_BLOCK". > > > > (I wonder if this would be a largeish rewrite of the menuconfig > > infrastructure? Hmmm...) > > Yess. I'm doing this, actually. Sam, Andrew and Linus have got an email > half a year ago about my intent. Tried to release 2.6.20-j4f, but that > was just a dream. Now i did some training in non-kernel related stuff and > always catch (30k, 14k) LKML backlogs to stay in tune. Some bits i > get are here ftp://flower.upol.cz/Linux/info-LKML/tools/ > > So, ideas: UI, organization, efficiency, simplicity, etc. are welcome. > > Maybe in one month i'll get something to show (base is 2.6.22). > Imagination plays very well, the thing must be ground shaking, but i'll > see how it will fit reality. Yours one in consideration from very > beginning, don't bother :) Then as now you have not yet expalined what you are trying to do. Nevertheless I look forward for a minmal set of patches that improve whatever you are working with. As for Kconfig the low hanging fruits are not in the tools but in the structure of the Kconfig files. There are a lot that can be improved with a decent effort but nobody has stepped up doing so. The tools could be better too but if the root problem is the structure of the Kconfig files this is where we should focus and not on the tools. For Kbuild I fail to see anything that demand a rewrite from a structure view of the Kbuild files. The Kbuild internal stuff is antoehr story - here a rewrite to a saner language then GNU make syntax could improve hackability a lot. Sam - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: kconfig/kbuild rewite (Re: What's up with CONFIG_BLK_DEV?)
On Sun, Sep 02, 2007 at 02:02:37AM +0200, Oleg Verych wrote: * Sun, 26 Aug 2007 01:08:28 -0500 * Organization: Boundaries Unlimited [] Also here's a symbol, show me a menu containing everything else that is either required by or enabled by this symbol... That sounds like a more powerful abstraction, since the previous one is show me everything that depends on CONFIG_BLOCK. (I wonder if this would be a largeish rewrite of the menuconfig infrastructure? Hmmm...) Yess. I'm doing this, actually. Sam, Andrew and Linus have got an email half a year ago about my intent. Tried to release 2.6.20-j4f, but that was just a dream. Now i did some training in non-kernel related stuff and always catch (30k, 14k) LKML backlogs to stay in tune. Some bits i get are here ftp://flower.upol.cz/Linux/info-LKML/tools/ So, ideas: UI, organization, efficiency, simplicity, etc. are welcome. Maybe in one month i'll get something to show (base is 2.6.22). Imagination plays very well, the thing must be ground shaking, but i'll see how it will fit reality. Yours one in consideration from very beginning, don't bother :) Then as now you have not yet expalined what you are trying to do. Nevertheless I look forward for a minmal set of patches that improve whatever you are working with. As for Kconfig the low hanging fruits are not in the tools but in the structure of the Kconfig files. There are a lot that can be improved with a decent effort but nobody has stepped up doing so. The tools could be better too but if the root problem is the structure of the Kconfig files this is where we should focus and not on the tools. For Kbuild I fail to see anything that demand a rewrite from a structure view of the Kbuild files. The Kbuild internal stuff is antoehr story - here a rewrite to a saner language then GNU make syntax could improve hackability a lot. Sam - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
kconfig/kbuild rewite (Re: What's up with CONFIG_BLK_DEV?)
* Sun, 26 Aug 2007 01:08:28 -0500 * Organization: Boundaries Unlimited > [] > Also "here's a symbol, show me a menu containing everything else that is > either required by or enabled by this symbol..." That sounds like a more > powerful abstraction, since the previous one is "show me everything that > depends on CONFIG_BLOCK". > > (I wonder if this would be a largeish rewrite of the menuconfig > infrastructure? Hmmm...) Yess. I'm doing this, actually. Sam, Andrew and Linus have got an email half a year ago about my intent. Tried to release 2.6.20-j4f, but that was just a dream. Now i did some training in non-kernel related stuff and always catch (30k, 14k) LKML backlogs to stay in tune. Some bits i get are here ftp://flower.upol.cz/Linux/info-LKML/tools/ So, ideas: UI, organization, efficiency, simplicity, etc. are welcome. Maybe in one month i'll get something to show (base is 2.6.22). Imagination plays very well, the thing must be ground shaking, but i'll see how it will fit reality. Yours one in consideration from very beginning, don't bother :) Another problem that i have to solve before any publishing (it's completely another tree, logic, interface), is the work tracking system, i was describing in June. This is important, because i sick of current chaos and manual organizing work in-tree or in regression tracking. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
kconfig/kbuild rewite (Re: What's up with CONFIG_BLK_DEV?)
* Sun, 26 Aug 2007 01:08:28 -0500 * Organization: Boundaries Unlimited [] Also here's a symbol, show me a menu containing everything else that is either required by or enabled by this symbol... That sounds like a more powerful abstraction, since the previous one is show me everything that depends on CONFIG_BLOCK. (I wonder if this would be a largeish rewrite of the menuconfig infrastructure? Hmmm...) Yess. I'm doing this, actually. Sam, Andrew and Linus have got an email half a year ago about my intent. Tried to release 2.6.20-j4f, but that was just a dream. Now i did some training in non-kernel related stuff and always catch (30k, 14k) LKML backlogs to stay in tune. Some bits i get are here ftp://flower.upol.cz/Linux/info-LKML/tools/ So, ideas: UI, organization, efficiency, simplicity, etc. are welcome. Maybe in one month i'll get something to show (base is 2.6.22). Imagination plays very well, the thing must be ground shaking, but i'll see how it will fit reality. Yours one in consideration from very beginning, don't bother :) Another problem that i have to solve before any publishing (it's completely another tree, logic, interface), is the work tracking system, i was describing in June. This is important, because i sick of current chaos and manual organizing work in-tree or in regression tracking. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
