Re: Change to kernel+modules build approach
On 15-Aug-2003 M. Warner Losh wrote: In message: [EMAIL PROTECTED] John Baldwin [EMAIL PROTECTED] writes: : : On 14-Aug-2003 Andrew Gallatin wrote: : : John Baldwin writes: : :On 14-Aug-2003 Ruslan Ermilov wrote: : On Thu, Aug 14, 2003 at 02:10:19AM -0600, Scott Long wrote: : Luoqi Chen wrote: : [...] : On the other hand, all modules should create all the opt_*.h files : it needs when built individually. Add opt_ddb.h to nullfs's Makefile : should fix the breakage. : : Our kernel build system isn't set up to handle passing config options : to modules. Various solutions to this have been proposed, but nothing : has appeared yet. In 5.x, we document that modules will not work with : PAE. : : How does the below look? This is basically a more generic implementation : of Luoqi's idea, but for -CURRENT: : :I would prefer something far more radical that would involve moving :all the module metadata to sys/conf (i.e. removing sys/modules) and :building all the modules based on a single kernel config file. : : Would this tie modules to that kernel config? If so, would it mean : the end of the ability of 3rd party developers to ship binary drivers : and expect them to work with any kernel? : : Well, yes, but, one could always build generic modules by using : a kernel config containing 'options KLD_MODULE' or some such. : This would allow one to compile optimized modules if they wanted to, : but still provide the ability to build fully generic modules. This sounds like an either or choice. I don't care too much if the third party drivers aren't hyper optimzied for my kernel. But to force users of them to use some generic kernel would be a big support nightmare. No, generic modules would always work with all kernels except for exceptional cases like PAE (unavoidable, really), and MUTEX_PROFILING (this is a debugging thing, so ISV's wouldn't need to ship modules with that turned on). All this would add is the ability to build modules optimized for your current kernel. If this is not super desired (which I wouldn't mind), then I think we should take the modules out of /boot/kernel and put them in /boot/modules or some such. I do want to get the metadata down to one copy somehow though. -- John Baldwin [EMAIL PROTECTED]http://www.FreeBSD.org/~jhb/ Power Users Use the Power to Serve! - http://www.FreeBSD.org/ ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Change to kernel+modules build approach
John Baldwin writes: No, generic modules would always work with all kernels except for exceptional cases like PAE (unavoidable, really), and MUTEX_PROFILING (this is a debugging thing, so ISV's wouldn't need to ship modules with that turned on). All this would add is the ability to build modules optimized for your current kernel. If this is not super desired (which I wouldn't mind), then I think we should take the modules out of /boot/kernel and put them in /boot/modules or some such. I do want to get the metadata down to one copy somehow though. YES! YES! I'd be very much in favor of totally decoupling the modules from the kernel. In fact, once we've done that, we can move the kernel back to /kernel where it belongs, and /boot/modules can become /modules ;) BTW, what, exactly, changes size with PAE? Everything? Or would a driver that just used things like busdma, mutexes, interrupts, etc, be OK assuming the busdma interface were made so that they were always 64-bit? Drew ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Change to kernel+modules build approach
In message: [EMAIL PROTECTED] Andrew Gallatin [EMAIL PROTECTED] writes: : : John Baldwin writes: : : No, generic modules would always work with all kernels except for : exceptional cases like PAE (unavoidable, really), and MUTEX_PROFILING : (this is a debugging thing, so ISV's wouldn't need to ship modules : with that turned on). All this would add is the ability to build : modules optimized for your current kernel. If this is not super : desired (which I wouldn't mind), then I think we should take the : modules out of /boot/kernel and put them in /boot/modules or some such. : I do want to get the metadata down to one copy somehow though. : : YES! YES! I'd be very much in favor of totally decoupling the : modules from the kernel. : : In fact, once we've done that, we can move the kernel back to /kernel : where it belongs, and /boot/modules can become /modules ;) That would be somewhat difficult. It would make it a lot harder to keep a 2 or 4 week old kernel around for testing since you couldn't load current modules with an old kernel (generally, but sometimes it works). Warner ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Change to kernel+modules build approach
On Fri, 15 Aug 2003, M. Warner Losh wrote: In message: [EMAIL PROTECTED] Andrew Gallatin [EMAIL PROTECTED] writes: : : John Baldwin writes: : : No, generic modules would always work with all kernels except for : exceptional cases like PAE (unavoidable, really), and MUTEX_PROFILING : (this is a debugging thing, so ISV's wouldn't need to ship modules : with that turned on). All this would add is the ability to build : modules optimized for your current kernel. If this is not super : desired (which I wouldn't mind), then I think we should take the : modules out of /boot/kernel and put them in /boot/modules or some such. : I do want to get the metadata down to one copy somehow though. : : YES! YES! I'd be very much in favor of totally decoupling the : modules from the kernel. : : In fact, once we've done that, we can move the kernel back to /kernel : where it belongs, and /boot/modules can become /modules ;) That would be somewhat difficult. It would make it a lot harder to keep a 2 or 4 week old kernel around for testing since you couldn't load current modules with an old kernel (generally, but sometimes it works). Has anyone in this discussion looked at what Matt has done with Dragonfly? He's re-arranged the kernel tree and moved each driver/module into its own directory. Each directory has a Makefile. thus a traversal of the kernel tree make hierarchy generates the modules. The modules subdirectory is going away.. (I think he's in the middle of doing that now) Warner ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED] ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Change to kernel+modules build approach
In message: [EMAIL PROTECTED] Julian Elischer [EMAIL PROTECTED] writes: : : : On Fri, 15 Aug 2003, M. Warner Losh wrote: : : In message: [EMAIL PROTECTED] : Andrew Gallatin [EMAIL PROTECTED] writes: : : : : John Baldwin writes: : : : : No, generic modules would always work with all kernels except for : : exceptional cases like PAE (unavoidable, really), and MUTEX_PROFILING : : (this is a debugging thing, so ISV's wouldn't need to ship modules : : with that turned on). All this would add is the ability to build : : modules optimized for your current kernel. If this is not super : : desired (which I wouldn't mind), then I think we should take the : : modules out of /boot/kernel and put them in /boot/modules or some such. : : I do want to get the metadata down to one copy somehow though. : : : : YES! YES! I'd be very much in favor of totally decoupling the : : modules from the kernel. : : : : In fact, once we've done that, we can move the kernel back to /kernel : : where it belongs, and /boot/modules can become /modules ;) : : That would be somewhat difficult. It would make it a lot harder to : keep a 2 or 4 week old kernel around for testing since you couldn't : load current modules with an old kernel (generally, but sometimes it : works). : : Has anyone in this discussion looked at what Matt has done with : Dragonfly? He's re-arranged the kernel tree and moved each driver/module : into its own directory. Each directory has a Makefile. thus : a traversal of the kernel tree make hierarchy generates the modules. : : The modules subdirectory is going away.. (I think he's in the middle : of doing that now) That's certainly an interesting concept. One that would make it easier to deal with modules since you have the Makefile right where you need it. If that is all that he's done, then that wouldn't answer John's objection to the current state of affairs: meta data in two places (module Makefile and conf/files*). If he's done something else in addition to the movement, then that would be interesting to look at. Warner ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Change to kernel+modules build approach
:: Has anyone in this discussion looked at what Matt has done with :: Dragonfly? He's re-arranged the kernel tree and moved each driver/module :: into its own directory. Each directory has a Makefile. thus :: a traversal of the kernel tree make hierarchy generates the modules. :: :: The modules subdirectory is going away.. (I think he's in the middle :: of doing that now) : :That's certainly an interesting concept. One that would make it :easier to deal with modules since you have the Makefile right where :you need it. If that is all that he's done, then that wouldn't answer :John's objection to the current state of affairs: meta data in two :places (module Makefile and conf/files*). If he's done something :else in addition to the movement, then that would be interesting to :look at. : :Warner Yes, I've done away with the 'modules' directory and have been reincorporating the modules Makefiles into the main part of the kernel tree. It turns out not to be all that difficult. Most module Makefile's can be plopped into the proper directory with very few changes. On half of them I only had to remove the .PATH directive. Subdirectories are glued together with the standard bsd.subdir.mk. Some surgery is required, but nothing difficult. For example, the netgraph modules necessitated moving each /usr/src/sys/netgraph/* element into a subdirectory to accomodate the Makefile for that netgraph module. There are a few areas like that... mainly changes which force partitionable entities into their own subdirectories which I consider to be good for the structure of the system. It is still a work in progress but I am very close to getting all the ducks honking properly in regards to config based kernel builds, make buildkernel, separate module builds, and module builds with and without 'make obj'. I heartily recommend that -current investigate and implement the refolding of the module build into the main kernel source tree. Eventually my goal is to make the entire kernel sufficiently modular that 'config' can be gotten rid of (or, at least, relegated to just generating the various .h files from the config options). -- I have also done additional (and very extensive) reorganization of the kernel source tree, but it is probably too extensive for -current to consider (read: about 40 dillon hours of work plus another 40 dillon hours to cleanup the build issues afterwords). Not only did I completely reorganize filesystems, network subsystems, and device drivers, I also moved driver-specific architecture-specific files out of e.g. i386/ and into appropriate_driver/i386, and I also changed config to generate use_*.h instead *.h files, which allowed me to remove the -I- sillyness from the Makefiles, which in turn allows relative #include file paths to be used again in the kernel source (in the many places where they make sense, which is just about everywhere). This greatly improves our ability to modularize of the kernel. But it was a huge amount of work. If I were to pick *one* thing to recommend that -current adopt it would be to change all the config generated *.h files to use_*.h (plus the same thing in those module makefiles which generated *.h files), and get rid of the -I- compiler option, then incrementally fix all the #include's that can be trivially relative to being trivially relative. -Matt Matthew Dillon [EMAIL PROTECTED] ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Change to kernel+modules build approach
In message: [EMAIL PROTECTED] John Baldwin [EMAIL PROTECTED] writes: : : On 14-Aug-2003 Ruslan Ermilov wrote: : On Thu, Aug 14, 2003 at 02:10:19AM -0600, Scott Long wrote: : Luoqi Chen wrote: : [...] : On the other hand, all modules should create all the opt_*.h files : it needs when built individually. Add opt_ddb.h to nullfs's Makefile : should fix the breakage. : : Our kernel build system isn't set up to handle passing config options : to modules. Various solutions to this have been proposed, but nothing : has appeared yet. In 5.x, we document that modules will not work with : PAE. : : How does the below look? This is basically a more generic implementation : of Luoqi's idea, but for -CURRENT: : : I would prefer something far more radical that would involve moving : all the module metadata to sys/conf (i.e. removing sys/modules) and : building all the modules based on a single kernel config file. Does that mean that we're abandoning the idea that modules will work with all kernels? I don't disagree with the metadata move, since it is duplicated in two places now and allows for some more interesting subsettting, but I'm concerned that our third party ISVs will need to generate N different modules for the system... Warner ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Change to kernel+modules build approach
In message: [EMAIL PROTECTED] John Baldwin [EMAIL PROTECTED] writes: : : On 14-Aug-2003 Andrew Gallatin wrote: : : John Baldwin writes: : :On 14-Aug-2003 Ruslan Ermilov wrote: : On Thu, Aug 14, 2003 at 02:10:19AM -0600, Scott Long wrote: : Luoqi Chen wrote: : [...] : On the other hand, all modules should create all the opt_*.h files : it needs when built individually. Add opt_ddb.h to nullfs's Makefile : should fix the breakage. : : Our kernel build system isn't set up to handle passing config options : to modules. Various solutions to this have been proposed, but nothing : has appeared yet. In 5.x, we document that modules will not work with : PAE. : : How does the below look? This is basically a more generic implementation : of Luoqi's idea, but for -CURRENT: : :I would prefer something far more radical that would involve moving :all the module metadata to sys/conf (i.e. removing sys/modules) and :building all the modules based on a single kernel config file. : : Would this tie modules to that kernel config? If so, would it mean : the end of the ability of 3rd party developers to ship binary drivers : and expect them to work with any kernel? : : Well, yes, but, one could always build generic modules by using : a kernel config containing 'options KLD_MODULE' or some such. : This would allow one to compile optimized modules if they wanted to, : but still provide the ability to build fully generic modules. This sounds like an either or choice. I don't care too much if the third party drivers aren't hyper optimzied for my kernel. But to force users of them to use some generic kernel would be a big support nightmare. Warner ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Change to kernel+modules build approach
M. Warner Losh wrote: In message: [EMAIL PROTECTED] John Baldwin [EMAIL PROTECTED] writes: : : On 14-Aug-2003 Ruslan Ermilov wrote: : On Thu, Aug 14, 2003 at 02:10:19AM -0600, Scott Long wrote: : Luoqi Chen wrote: : [...] : On the other hand, all modules should create all the opt_*.h files : it needs when built individually. Add opt_ddb.h to nullfs's Makefile : should fix the breakage. : : Our kernel build system isn't set up to handle passing config options : to modules. Various solutions to this have been proposed, but nothing : has appeared yet. In 5.x, we document that modules will not work with : PAE. : : How does the below look? This is basically a more generic implementation : of Luoqi's idea, but for -CURRENT: : : I would prefer something far more radical that would involve moving : all the module metadata to sys/conf (i.e. removing sys/modules) and : building all the modules based on a single kernel config file. Does that mean that we're abandoning the idea that modules will work with all kernels? I don't disagree with the metadata move, since it is duplicated in two places now and allows for some more interesting subsettting, but I'm concerned that our third party ISVs will need to generate N different modules for the system... Warner I can tell you first hand that this is painful. However, in the case of PAE, it's somewhat neccessary since certain fundamental types change size. I can envision solutions for this, but I'm not sure if they are less painful than just dealing with multiple module builds. Scott ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Change to kernel+modules build approach
In message: [EMAIL PROTECTED] Scott Long [EMAIL PROTECTED] writes: : I can tell you first hand that this is painful. However, in the case of : PAE, it's somewhat neccessary since certain fundamental types change : size. I can envision solutions for this, but I'm not sure if they are : less painful than just dealing with multiple module builds. Personally I think that pae should be its own port, much like pc98 is its own port, although much smaller in scope. It has a few, fundamental differences from the stock i386 port. Since the data type sizes are different with PAE there's no way around that. Warner ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: Change to kernel+modules build approach
John Baldwin writes: On 14-Aug-2003 Ruslan Ermilov wrote: On Thu, Aug 14, 2003 at 02:10:19AM -0600, Scott Long wrote: Luoqi Chen wrote: [...] On the other hand, all modules should create all the opt_*.h files it needs when built individually. Add opt_ddb.h to nullfs's Makefile should fix the breakage. Our kernel build system isn't set up to handle passing config options to modules. Various solutions to this have been proposed, but nothing has appeared yet. In 5.x, we document that modules will not work with PAE. How does the below look? This is basically a more generic implementation of Luoqi's idea, but for -CURRENT: I would prefer something far more radical that would involve moving all the module metadata to sys/conf (i.e. removing sys/modules) and building all the modules based on a single kernel config file. Would this tie modules to that kernel config? If so, would it mean the end of the ability of 3rd party developers to ship binary drivers and expect them to work with any kernel? Drew ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: Change to kernel+modules build approach
On 14-Aug-2003 Andrew Gallatin wrote: John Baldwin writes: On 14-Aug-2003 Ruslan Ermilov wrote: On Thu, Aug 14, 2003 at 02:10:19AM -0600, Scott Long wrote: Luoqi Chen wrote: [...] On the other hand, all modules should create all the opt_*.h files it needs when built individually. Add opt_ddb.h to nullfs's Makefile should fix the breakage. Our kernel build system isn't set up to handle passing config options to modules. Various solutions to this have been proposed, but nothing has appeared yet. In 5.x, we document that modules will not work with PAE. How does the below look? This is basically a more generic implementation of Luoqi's idea, but for -CURRENT: I would prefer something far more radical that would involve moving all the module metadata to sys/conf (i.e. removing sys/modules) and building all the modules based on a single kernel config file. Would this tie modules to that kernel config? If so, would it mean the end of the ability of 3rd party developers to ship binary drivers and expect them to work with any kernel? Well, yes, but, one could always build generic modules by using a kernel config containing 'options KLD_MODULE' or some such. This would allow one to compile optimized modules if they wanted to, but still provide the ability to build fully generic modules. -- John Baldwin [EMAIL PROTECTED]http://www.FreeBSD.org/~jhb/ Power Users Use the Power to Serve! - http://www.FreeBSD.org/ ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: Change to kernel+modules build approach
On 14-Aug-2003 Ruslan Ermilov wrote: On Thu, Aug 14, 2003 at 02:10:19AM -0600, Scott Long wrote: Luoqi Chen wrote: [...] On the other hand, all modules should create all the opt_*.h files it needs when built individually. Add opt_ddb.h to nullfs's Makefile should fix the breakage. Our kernel build system isn't set up to handle passing config options to modules. Various solutions to this have been proposed, but nothing has appeared yet. In 5.x, we document that modules will not work with PAE. How does the below look? This is basically a more generic implementation of Luoqi's idea, but for -CURRENT: I would prefer something far more radical that would involve moving all the module metadata to sys/conf (i.e. removing sys/modules) and building all the modules based on a single kernel config file. -- John Baldwin [EMAIL PROTECTED]http://www.FreeBSD.org/~jhb/ Power Users Use the Power to Serve! - http://www.FreeBSD.org/ ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: Change to kernel+modules build approach
John Baldwin writes: On 14-Aug-2003 Andrew Gallatin wrote: John Baldwin writes: On 14-Aug-2003 Ruslan Ermilov wrote: On Thu, Aug 14, 2003 at 02:10:19AM -0600, Scott Long wrote: Luoqi Chen wrote: [...] On the other hand, all modules should create all the opt_*.h files it needs when built individually. Add opt_ddb.h to nullfs's Makefile should fix the breakage. Our kernel build system isn't set up to handle passing config options to modules. Various solutions to this have been proposed, but nothing has appeared yet. In 5.x, we document that modules will not work with PAE. How does the below look? This is basically a more generic implementation of Luoqi's idea, but for -CURRENT: I would prefer something far more radical that would involve moving all the module metadata to sys/conf (i.e. removing sys/modules) and building all the modules based on a single kernel config file. Would this tie modules to that kernel config? If so, would it mean the end of the ability of 3rd party developers to ship binary drivers and expect them to work with any kernel? Well, yes, but, one could always build generic modules by using a kernel config containing 'options KLD_MODULE' or some such. This would allow one to compile optimized modules if they wanted to, but still provide the ability to build fully generic modules. My concern is that if we do such a thing, we'll gradually gain more options which break modules. Right now, the list includes MUTEX_PROFILING and PAE. I'm afraid it can only grow if modules are further coupled with the kernel build. Drew ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
