Re: kernel module parallel build?
On Dec 6, 2012, at 1:28 AM, Garrett Cooper yaneg...@gmail.com wrote: On Wed, Dec 5, 2012 at 3:51 PM, Damien Fleuriot m...@my.gd wrote: ... All trolling aside, I believe an awesome fix to be setting module override in /etc/make.conf to only build the 4-5 specific modules one needs. To be honest I think this configuration tweak should be advertised a bit more as it definitely speeds up kernel builds. I would be happy to check if this is advertised in the handbook in the rebuilding kernel section and enhance its visibility if required. I can provide en_US and fr_FR. +1. Please write it up if you can; it's much quicker/better than the kitchen sink approach if you know what you're doing and don't have to build for a large set of platforms. Thanks! -Garrett I've had a look at the handbook and the MODULES_OVERRIDE bit is already well written and definitely visible: http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-building.html I believe no rewrite is necessary. However, I fail to see any mention of nextboot. It might be a good idea to let people know they can boot their new kernel just once to see if it works, and revert to their old kernel if boot should fail. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: kernel module parallel build?
On Wednesday, December 05, 2012 6:51:17 pm Damien Fleuriot wrote: On 5 Dec 2012, at 18:39, Warner Losh i...@bsdimp.com wrote: On Dec 5, 2012, at 9:42 AM, John Baldwin wrote: On Tuesday, December 04, 2012 2:41:32 pm Ryan Stone wrote: On Tue, Dec 4, 2012 at 10:52 AM, John Baldwin j...@freebsd.org wrote: Hmm, I certainly see the module directories being built in parallel. Some of the make jobs may not be as obvious since links are silent (no output unless there is an error). This is definitely not the behaviour that I see trying to build any version of FreeBSD. I see the same behaviour as Andre: the depend and all targets both iterate through the module directories sequentially. It never builds two module subdirectories concurrently. Hmm, I think I was confused by seeing kernel builds intermingle with the associated modules. sys/modules/Makefile uses bsd.subdir.mk. I think I see similar things in world builds where I will see parallel builds of bin vs sbin vs usr.bin vs usr.sbin, but within each of those directories the builds go sequentially. I think you would need to change bsd.subdir.mk if you want to fix this. The builds are in parallel, just that the parallelism is low because it is only parallel within the module being built. Would love to see a fix. Warner All trolling aside, I believe an awesome fix to be setting module override in /etc/make.conf to only build the 4-5 specific modules one needs. To be honest I think this configuration tweak should be advertised a bit more as it definitely speeds up kernel builds. I would be happy to check if this is advertised in the handbook in the rebuilding kernel section and enhance its visibility if required. I can provide en_US and fr_FR. Better than doing it in /etc/make.conf (or /etc/src.conf) is doing it direclty in the kernel config file itself via makeoptions MODULES_OVERRIDE=foo You can use multiple of these (with +=) in a config file as well. -- John Baldwin ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: kernel module parallel build?
On Tuesday, December 04, 2012 2:41:32 pm Ryan Stone wrote: On Tue, Dec 4, 2012 at 10:52 AM, John Baldwin j...@freebsd.org wrote: Hmm, I certainly see the module directories being built in parallel. Some of the make jobs may not be as obvious since links are silent (no output unless there is an error). This is definitely not the behaviour that I see trying to build any version of FreeBSD. I see the same behaviour as Andre: the depend and all targets both iterate through the module directories sequentially. It never builds two module subdirectories concurrently. Hmm, I think I was confused by seeing kernel builds intermingle with the associated modules. sys/modules/Makefile uses bsd.subdir.mk. I think I see similar things in world builds where I will see parallel builds of bin vs sbin vs usr.bin vs usr.sbin, but within each of those directories the builds go sequentially. I think you would need to change bsd.subdir.mk if you want to fix this. -- John Baldwin ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: kernel module parallel build?
On Wed, Dec 5, 2012 at 8:42 AM, John Baldwin j...@freebsd.org wrote:
On Tuesday, December 04, 2012 2:41:32 pm Ryan Stone wrote:
On Tue, Dec 4, 2012 at 10:52 AM, John Baldwin j...@freebsd.org wrote:
Hmm, I certainly see the module directories being built in parallel. Some
of
the make jobs may not be as obvious since links are silent (no output
unless
there is an error).
This is definitely not the behaviour that I see trying to build any version
of FreeBSD. I see the same behaviour as Andre: the depend and all targets
both iterate through the module directories sequentially. It never builds
two module subdirectories concurrently.
Hmm, I think I was confused by seeing kernel builds intermingle with the
associated modules. sys/modules/Makefile uses bsd.subdir.mk. I think I see
similar things in world builds where I will see parallel builds of bin vs sbin
vs usr.bin vs usr.sbin, but within each of those directories the builds go
sequentially. I think you would need to change bsd.subdir.mk if you want to
fix this.
Correct:
45 @${_+_}for entry in ${SUBDIR}; do \
^^^ This is where things get serialized
46 if test -d ${.CURDIR}/$${entry}.${MACHINE_ARCH}; then \
Same thing applies for buildkernel building modules because it
just wraps around bsd.subdir.mk in sys/modules/Makefile . Enhancing it
to be parallel would introduce potential races. Some of the work sjg's
doing with meta make will make this unnecessary from a buildworld
perspective, but I'm not sure about buildkernel.
Thanks,
-Garrett
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: kernel module parallel build?
On Dec 5, 2012, at 9:42 AM, John Baldwin wrote: On Tuesday, December 04, 2012 2:41:32 pm Ryan Stone wrote: On Tue, Dec 4, 2012 at 10:52 AM, John Baldwin j...@freebsd.org wrote: Hmm, I certainly see the module directories being built in parallel. Some of the make jobs may not be as obvious since links are silent (no output unless there is an error). This is definitely not the behaviour that I see trying to build any version of FreeBSD. I see the same behaviour as Andre: the depend and all targets both iterate through the module directories sequentially. It never builds two module subdirectories concurrently. Hmm, I think I was confused by seeing kernel builds intermingle with the associated modules. sys/modules/Makefile uses bsd.subdir.mk. I think I see similar things in world builds where I will see parallel builds of bin vs sbin vs usr.bin vs usr.sbin, but within each of those directories the builds go sequentially. I think you would need to change bsd.subdir.mk if you want to fix this. The builds are in parallel, just that the parallelism is low because it is only parallel within the module being built. Would love to see a fix. Warner ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: kernel module parallel build?
On 5 Dec 2012, at 18:39, Warner Losh i...@bsdimp.com wrote: On Dec 5, 2012, at 9:42 AM, John Baldwin wrote: On Tuesday, December 04, 2012 2:41:32 pm Ryan Stone wrote: On Tue, Dec 4, 2012 at 10:52 AM, John Baldwin j...@freebsd.org wrote: Hmm, I certainly see the module directories being built in parallel. Some of the make jobs may not be as obvious since links are silent (no output unless there is an error). This is definitely not the behaviour that I see trying to build any version of FreeBSD. I see the same behaviour as Andre: the depend and all targets both iterate through the module directories sequentially. It never builds two module subdirectories concurrently. Hmm, I think I was confused by seeing kernel builds intermingle with the associated modules. sys/modules/Makefile uses bsd.subdir.mk. I think I see similar things in world builds where I will see parallel builds of bin vs sbin vs usr.bin vs usr.sbin, but within each of those directories the builds go sequentially. I think you would need to change bsd.subdir.mk if you want to fix this. The builds are in parallel, just that the parallelism is low because it is only parallel within the module being built. Would love to see a fix. Warner All trolling aside, I believe an awesome fix to be setting module override in /etc/make.conf to only build the 4-5 specific modules one needs. To be honest I think this configuration tweak should be advertised a bit more as it definitely speeds up kernel builds. I would be happy to check if this is advertised in the handbook in the rebuilding kernel section and enhance its visibility if required. I can provide en_US and fr_FR. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: kernel module parallel build?
On Wed, Dec 5, 2012 at 3:51 PM, Damien Fleuriot m...@my.gd wrote: ... All trolling aside, I believe an awesome fix to be setting module override in /etc/make.conf to only build the 4-5 specific modules one needs. To be honest I think this configuration tweak should be advertised a bit more as it definitely speeds up kernel builds. I would be happy to check if this is advertised in the handbook in the rebuilding kernel section and enhance its visibility if required. I can provide en_US and fr_FR. +1. Please write it up if you can; it's much quicker/better than the kitchen sink approach if you know what you're doing and don't have to build for a large set of platforms. Thanks! -Garrett ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: kernel module parallel build?
On Sunday, November 04, 2012 2:53:02 pm Andre Oppermann wrote: On 22.10.2012 15:28, John Baldwin wrote: On Sunday, October 21, 2012 7:11:10 am Andre Oppermann wrote: What's keeping kernel modules from building in parallel with make -j8? They don't for you? They do for me either via 'make buildkernel' or the old method. They do, but only partially. Within a module the files are built in parallel. However the module directories seem to be serialized. I'm a Makefile noob though. Hmm, I certainly see the module directories being built in parallel. Some of the make jobs may not be as obvious since links are silent (no output unless there is an error). -- John Baldwin ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: kernel module parallel build?
On Tue, Dec 4, 2012 at 10:52 AM, John Baldwin j...@freebsd.org wrote: Hmm, I certainly see the module directories being built in parallel. Some of the make jobs may not be as obvious since links are silent (no output unless there is an error). This is definitely not the behaviour that I see trying to build any version of FreeBSD. I see the same behaviour as Andre: the depend and all targets both iterate through the module directories sequentially. It never builds two module subdirectories concurrently. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: kernel module parallel build?
On 22.10.2012 15:28, John Baldwin wrote: On Sunday, October 21, 2012 7:11:10 am Andre Oppermann wrote: What's keeping kernel modules from building in parallel with make -j8? They don't for you? They do for me either via 'make buildkernel' or the old method. They do, but only partially. Within a module the files are built in parallel. However the module directories seem to be serialized. I'm a Makefile noob though. -- Andre ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: kernel module parallel build?
On Sunday, October 21, 2012 7:11:10 am Andre Oppermann wrote: What's keeping kernel modules from building in parallel with make -j8? They don't for you? They do for me either via 'make buildkernel' or the old method. -- John Baldwin ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
kernel module parallel build?
What's keeping kernel modules from building in parallel with make -j8? -- Andre ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
