Re: -ffunction-sections, -fdata-sections and -Wl,--gc-sections
On Tue, Sep 17, 2013 at 11:45:19PM +0200, Ed Schouten wrote:
Hi Matthew,
2013/9/16 Matthew Fleming m...@freebsd.org:
Would it be possible to enable this only for devd, init, and clang binaries?
Or is it a matter of enabling it for library builds that are linked
statically with the mentioned binaries?
For it to have effect, it has to be enabled for both the libraries and
the binaries. The libraries need to be built with
-f{data,function}-sections. The binaries can also be built with those
flags, but it is actually more important to link with --gc-sections.
Could init/devd be made smaller by
finding out which functions they do/don't use and separating those into
separate .c files?
Also trying to answer Tim and Adrian's questions at the same time.
I've just taken a look at init and devd to see why the difference in
size is so big:
init seems to pull in the following things:
- Sun RPC,
- XDR,
- YP,
- res_*,
- All of the jemalloc profiling/stats code,
- Some widechar functions,
- malloc-related utility functions that are not used (posix_memalign,
aligned_alloc),
- Some stdio bloat,
- All sorts of termios tc* functions.
devd seems to pull in these:
- A very big pile of C++ symbols, as libc++ places many functions in a
single file.
- jemalloc profiling/stats again,
- A big pile of pthread,
- Maybe *_l() functions, which are of course rarely used.
Honestly, I think we can assume we'll never reach the point where all
the components listed above will properly have all functions
partitioned over separate compilation units.
I suspect that it would make a lot of sense to at least enable these
build flags for our core libraries (libc, libc++, libpthread,
libcompiler_rt, libcxxrt, etc). We could also enable it on
INTERNALLIBs (libraries that are not installed into /usr/lib), as for
these libraries, it would of course not come at any cost.
Would that sound okay?
I think this is a wrong direction. First, the split should be done at
the source level, as it was usually done forever. One of the offender
there was you, AFAIR.
Second, I would rather see init and devd, and in fact all other statically
linked binaries from our base system, to become dynamically linked. At
least I added a knob for building toolchain dynamic, but avoided the
fight of making this default.
pgpf4ASNxyWm0.pgp
Description: PGP signature
Re: -ffunction-sections, -fdata-sections and -Wl,--gc-sections
On Sun, 15 Sep 2013 19:30:01 -0700 Tim Kientzle kient...@freebsd.org wrote: On Sep 15, 2013, at 2:24 PM, Ed Schouten e...@80386.nl wrote: GCC and Clang support the -ffunction-sections and -fdata-sections flags. Essentially, these flags force the compiler to put every function and variable in its own section. Though this will blow up the …. - devd suddenly becomes 500 KB in size, instead of a megabyte, - init's size drops from 900 KB to 600 KB, Can you figure out what functions are getting omitted when you make this change? You can add -Wl,--print-gc-sections to LDFLAGS to print which sections the linker is removing. Andrew ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: -ffunction-sections, -fdata-sections and -Wl,--gc-sections
(Copy for the list, wrong from address problem ...) On 18/09/2013, at 4:22 PM, Konstantin Belousov kostik...@gmail.com wrote: On Tue, Sep 17, 2013 at 11:45:19PM +0200, Ed Schouten wrote: [...] Honestly, I think we can assume we'll never reach the point where all the components listed above will properly have all functions partitioned over separate compilation units. I suspect that it would make a lot of sense to at least enable these build flags for our core libraries (libc, libc++, libpthread, libcompiler_rt, libcxxrt, etc). We could also enable it on INTERNALLIBs (libraries that are not installed into /usr/lib), as for these libraries, it would of course not come at any cost. Would that sound okay? I think this is a wrong direction. First, the split should be done at the source level, as it was usually done forever. One of the offender there was you, AFAIR. Second, I would rather see init and devd, and in fact all other statically linked binaries from our base system, to become dynamically linked. At least I added a knob for building toolchain dynamic, but avoided the fight of making this default. Why do things by hand when there are good tools? Note ... as it was usually done forever doesn't contain a good argument, and compilers and linkers on other platforms have been doing it like this for an awfully long time. Adding the flags has a benefit in the case where there are many functions in a source file and minimal cost when everything is perfect. Not having the flags means paying a bigger price when things are not perfect. And things are very rarely perfect. Having the structure of your source code driven by link-time considerations when there is a choice seems silly to me. Larger source files gives the compiler more scope for optimisation, and you can structure the code in a way useful to people working in the codebase. If you have a moral argument about how code should be structured, I think that is separate discussion. Adding the flags has a benefit, regardless of how the code is structured. I can see all upside, and I am having trouble seeing a problem with adding them at all. On the static linking vs. dynamic linking argument: I am strongly on the static linking side. But that is also a different discussion. Regards, Jan Mikkelsen ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: -ffunction-sections, -fdata-sections and -Wl,--gc-sections
On Wed, Sep 18, 2013 at 05:36:40PM +1000, Jan Mikkelsen wrote: On 18/09/2013, at 4:22 PM, Konstantin Belousov kostik...@gmail.com wrote: On Tue, Sep 17, 2013 at 11:45:19PM +0200, Ed Schouten wrote: [ ... ] Honestly, I think we can assume we'll never reach the point where all the components listed above will properly have all functions partitioned over separate compilation units. I suspect that it would make a lot of sense to at least enable these build flags for our core libraries (libc, libc++, libpthread, libcompiler_rt, libcxxrt, etc). We could also enable it on INTERNALLIBs (libraries that are not installed into /usr/lib), as for these libraries, it would of course not come at any cost. Would that sound okay? I think this is a wrong direction. First, the split should be done at the source level, as it was usually done forever. One of the offender there was you, AFAIR. Second, I would rather see init and devd, and in fact all other statically linked binaries from our base system, to become dynamically linked. At least I added a knob for building toolchain dynamic, but avoided the fight of making this default. Why do things by hand when there are good tools? Note ... as it was usually done forever doesn't contain a good argument, and compilers and linkers on other platforms have been doing it like this for an awfully long time. Tools are not good. Using of this feature locks us to the toolchain. And, the implementation of at least gc in the linker is known to be buggy even in recent binutils. Also, even perfect linker cannot always guess the correct value of garbage, so we have to hack in other direction. With the current set-up, it is easy to guarantee that some symbol is always present if other symbol is linked in. Adding the flags has a benefit in the case where there are many functions in a source file and minimal cost when everything is perfect. Not having the flags means paying a bigger price when things are not perfect. And things are very rarely perfect. Having the structure of your source code driven by link-time considerations when there is a choice seems silly to me. Larger source files gives the compiler more scope for optimisation, and you can structure the code in a way useful to people working in the codebase. If you have a moral argument about how code should be structured, I think that is separate discussion. Adding the flags has a benefit, regardless of how the code is structured. I can see all upside, and I am having trouble seeing a problem with adding them at all. On the static linking vs. dynamic linking argument: I am strongly on the static linking side. But that is also a different discussion. Yeah, make the things like nss, pam or iconv work or fully functional with the statically linked binaries first. pgp6v0lniNU29.pgp Description: PGP signature
Re: -ffunction-sections, -fdata-sections and -Wl,--gc-sections
On 18 Sep 2013, at 07:22, Konstantin Belousov kostik...@gmail.com wrote: I think this is a wrong direction. First, the split should be done at the source level, as it was usually done forever. Until we are all using toolchains that support LTO (which requires importing a new linker and will make people who complain about buildworld memory usage even more unhappy), this approach will typically result in worse code being generated, as the compiler has significantly reduced scope for interprocedural analysis. One of the offender there was you, AFAIR. This is irrelevant to the discussion. Second, I would rather see init and devd, and in fact all other statically linked binaries from our base system, to become dynamically linked. At least I added a knob for building toolchain dynamic, but avoided the fight of making this default. In my (very informal) testing, a dynamically linked clang showed about a 5% slowdown over a statically linked one. Spending some time profiling rtld may let us improve this, but I suspect that people would complain if compilation suddenly got slower[1], especially if the only win is a small reduction in disk usage (which is most important on the kinds of platforms where you're not likely to be doing a lot of compilation). David [1] Note that once we have a linker that does LTO, these numbers may change, as we'd end up sharing the LLVM optimisation libraries between clang and the linker so the increased code sharing might offset the extra cost of the extra relocations. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: -ffunction-sections, -fdata-sections and -Wl,--gc-sections
On 18/09/2013, at 4:22 PM, Konstantin Belousov kostik...@gmail.com wrote: On Tue, Sep 17, 2013 at 11:45:19PM +0200, Ed Schouten wrote: [ ... ] Honestly, I think we can assume we'll never reach the point where all the components listed above will properly have all functions partitioned over separate compilation units. I suspect that it would make a lot of sense to at least enable these build flags for our core libraries (libc, libc++, libpthread, libcompiler_rt, libcxxrt, etc). We could also enable it on INTERNALLIBs (libraries that are not installed into /usr/lib), as for these libraries, it would of course not come at any cost. Would that sound okay? I think this is a wrong direction. First, the split should be done at the source level, as it was usually done forever. One of the offender there was you, AFAIR. Second, I would rather see init and devd, and in fact all other statically linked binaries from our base system, to become dynamically linked. At least I added a knob for building toolchain dynamic, but avoided the fight of making this default. Why do things by hand when there are good tools? Note ... as it was usually done forever doesn't contain a good argument, and compilers and linkers on other platforms have been doing it like this for an awfully long time. Adding the flags has a benefit in the case where there are many functions in a source file and minimal cost when everything is perfect. Not having the flags means paying a bigger price when things are not perfect. And things are very rarely perfect. Having the structure of your source code driven by link-time considerations when there is a choice seems silly to me. Larger source files gives the compiler more scope for optimisation, and you can structure the code in a way useful to people working in the codebase. If you have a moral argument about how code should be structured, I think that is separate discussion. Adding the flags has a benefit, regardless of how the code is structured. I can see all upside, and I am having trouble seeing a problem with adding them at all. On the static linking vs. dynamic linking argument: I am strongly on the static linking side. But that is also a different discussion. Regards, Jan Mikkelsen ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: -ffunction-sections, -fdata-sections and -Wl,--gc-sections
... I'd rather see if we can actually separate out things some more so these builds can shrink. Eg, if there's malloc related functions that aren't used, maybe we should break malloc down into a directory full of functions. I'm not surprised libc++ is doing .. that. It's likely better on the compile time to have things bundled up in one big source file. Sigh. -adrian ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: -ffunction-sections, -fdata-sections and -Wl,--gc-sections
Hi Matthew,
2013/9/16 Matthew Fleming m...@freebsd.org:
Would it be possible to enable this only for devd, init, and clang binaries?
Or is it a matter of enabling it for library builds that are linked
statically with the mentioned binaries?
For it to have effect, it has to be enabled for both the libraries and
the binaries. The libraries need to be built with
-f{data,function}-sections. The binaries can also be built with those
flags, but it is actually more important to link with --gc-sections.
Could init/devd be made smaller by
finding out which functions they do/don't use and separating those into
separate .c files?
Also trying to answer Tim and Adrian's questions at the same time.
I've just taken a look at init and devd to see why the difference in
size is so big:
init seems to pull in the following things:
- Sun RPC,
- XDR,
- YP,
- res_*,
- All of the jemalloc profiling/stats code,
- Some widechar functions,
- malloc-related utility functions that are not used (posix_memalign,
aligned_alloc),
- Some stdio bloat,
- All sorts of termios tc* functions.
devd seems to pull in these:
- A very big pile of C++ symbols, as libc++ places many functions in a
single file.
- jemalloc profiling/stats again,
- A big pile of pthread,
- Maybe *_l() functions, which are of course rarely used.
Honestly, I think we can assume we'll never reach the point where all
the components listed above will properly have all functions
partitioned over separate compilation units.
I suspect that it would make a lot of sense to at least enable these
build flags for our core libraries (libc, libc++, libpthread,
libcompiler_rt, libcxxrt, etc). We could also enable it on
INTERNALLIBs (libraries that are not installed into /usr/lib), as for
these libraries, it would of course not come at any cost.
Would that sound okay?
--
Ed Schouten e...@80386.nl
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: -ffunction-sections, -fdata-sections and -Wl,--gc-sections
Adrian Chadd wrote this message on Tue, Sep 17, 2013 at 14:56 -0700: ... I'd rather see if we can actually separate out things some more so these builds can shrink. Eg, if there's malloc related functions that aren't used, maybe we should break malloc down into a directory full of functions. I'm not surprised libc++ is doing .. that. It's likely better on the compile time to have things bundled up in one big source file. Sigh. Depends.. one big file can't be parallelized over multiple cores, but the startup of parsing all those headers will be a big penalty though... -- John-Mark Gurney Voice: +1 415 225 5579 All that I will do, has been done, All that I have, has not. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: -ffunction-sections, -fdata-sections and -Wl,--gc-sections
On Tue, 2013-09-17 at 14:56 -0700, Adrian Chadd wrote: ... I'd rather see if we can actually separate out things some more so these builds can shrink. Eg, if there's malloc related functions that aren't used, maybe we should break malloc down into a directory full of functions. Why is that better than using this automated solution? -- Ian ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: -ffunction-sections, -fdata-sections and -Wl,--gc-sections
On 17 September 2013 16:22, Ian Lepore i...@freebsd.org wrote: On Tue, 2013-09-17 at 14:56 -0700, Adrian Chadd wrote: ... I'd rather see if we can actually separate out things some more so these builds can shrink. Eg, if there's malloc related functions that aren't used, maybe we should break malloc down into a directory full of functions. Why is that better than using this automated solution? Not everyone is going to run clang on their target development platform? :-) Personally I'd rather structure my work to behave better instead of doing something that relies on a specific tool/suite to be able to optimise. -adrian ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: -ffunction-sections, -fdata-sections and -Wl,--gc-sections
On Sep 17, 2013 4:31 PM, Adrian Chadd adr...@freebsd.org wrote: On 17 September 2013 16:22, Ian Lepore i...@freebsd.org wrote: On Tue, 2013-09-17 at 14:56 -0700, Adrian Chadd wrote: ... I'd rather see if we can actually separate out things some more so these builds can shrink. Eg, if there's malloc related functions that aren't used, maybe we should break malloc down into a directory full of functions. Why is that better than using this automated solution? Not everyone is going to run clang on their target development platform? :-) Personally I'd rather structure my work to behave better instead of doing something that relies on a specific tool/suite to be able to optimise. Gcc also supports this. I used it when generating builds with newlib for microcontrollers at a previous job. -Justin ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: -ffunction-sections, -fdata-sections and -Wl,--gc-sections
On Tue, 2013-09-17 at 16:31 -0700, Adrian Chadd wrote: On 17 September 2013 16:22, Ian Lepore i...@freebsd.org wrote: On Tue, 2013-09-17 at 14:56 -0700, Adrian Chadd wrote: ... I'd rather see if we can actually separate out things some more so these builds can shrink. Eg, if there's malloc related functions that aren't used, maybe we should break malloc down into a directory full of functions. Why is that better than using this automated solution? Not everyone is going to run clang on their target development platform? :-) Personally I'd rather structure my work to behave better instead of doing something that relies on a specific tool/suite to be able to optimise. The original mail began describing the feature with GCC and Clang support the -ffunction-sections and -fdata-sections flags. I would much rather have the tools perform this mundane task, and keep the source code maintainable rather than artificially broken into a zillion tiny pieces. I'm interested in it for $work. We have lots and lots of .a libraries that get linked to make an app that also uses shared libs at runtime (libc et. al.). We often have large .o files with lots of C++ methods of a class in it, and only 2 or 3 of the methods may be actually used in a given app.I haven't tested it yet, but the way I'm picturing this working, we should get good savings because unused functions from all our .a libs won't get linked in, even though the overall app isn't static-linked. -- Ian ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: -ffunction-sections, -fdata-sections and -Wl,--gc-sections
In message CAJOYFBBGY0GosPwG1B=1mkyapchetx-o97r2zhxpgs8o7wo...@mail.gmail.com , Ed Schouten writes: Thoughts? Jordan and I would have killed for that 19 years ago... -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 p...@freebsd.org | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: -ffunction-sections, -fdata-sections and -Wl,--gc-sections
On Sep 16, 2013, at 03:08, Adrian Chadd adr...@freebsd.org wrote: The results are interesting. On amd64: - devd suddenly becomes 500 KB in size, instead of a megabyte, - init's size drops from 900 KB to 600 KB, - clang becomes a megabyte smaller. .. so, I'd like to know specific information as to why these three are now smaller. So what's going on? This is simply unreferenced code and data being eliminated. It is quite normal for libraries (libc, the LLVM libs, etc) to contain functions that aren't called by the program you are linking. However, any .o file with multiple functions in it will be linked in as a whole, even if just one of the functions is called. I really think functions and data should always be separately packaged in object files, but for some reason this has never been the default for GNU tools, and everybody has apparently copied the behavior. :-) -Dimitry signature.asc Description: Message signed with OpenPGP using GPGMail
Re: -ffunction-sections, -fdata-sections and -Wl,--gc-sections
On 16 Sep 2013, at 07:52, Dimitry Andric d...@freebsd.org wrote: On Sep 16, 2013, at 03:08, Adrian Chadd adr...@freebsd.org wrote: The results are interesting. On amd64: - devd suddenly becomes 500 KB in size, instead of a megabyte, - init's size drops from 900 KB to 600 KB, - clang becomes a megabyte smaller. .. so, I'd like to know specific information as to why these three are now smaller. So what's going on? This is simply unreferenced code and data being eliminated. It is quite normal for libraries (libc, the LLVM libs, etc) to contain functions that aren't called by the program you are linking. However, any .o file with multiple functions in it will be linked in as a whole, even if just one of the functions is called. I really think functions and data should always be separately packaged in object files, but for some reason this has never been the default for GNU tools, and everybody has apparently copied the behavior. :-) Apple's ld, lld, and MCLinker all prefer to use a finer-grained model (atoms or fragments) than segments, and so are intended to do this sort of thing by default. This isn't an issue for dlsym(), as long as the functions that you may wish to resolve using dlsym() are explicitly marked (typically by including them in the version script) as not being eligible for deletion - only functions and data reachable from the public entry points should be retained. Of course, if you're building a shared library and there is unreachable code or data, you probably have a bug... Even in the shared library case, it's useful as it allows the linker to rearrange functions and data to perform optimisations to minimise PLT usage and improve cache and TLB usage by keeping functions that call each other together and data with the functions that use it (profiling can also help here, to prioritise hot functions). David ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: -ffunction-sections, -fdata-sections and -Wl,--gc-sections
On Sun, Sep 15, 2013 at 2:24 PM, Ed Schouten e...@80386.nl wrote: Hi all, Today I did a tiny experiment and I am not entirely sure what to do. Throw away the patch or eventually push it into the tree. GCC and Clang support the -ffunction-sections and -fdata-sections flags. Essentially, these flags force the compiler to put every function and variable in its own section. Though this will blow up the size of the .o files a lot, it has a nice advantage. In combination with ld's --gc-sections flag, it can garbage collect functions and variables during the linking phase more accurately, thereby reducing the size of the resulting binary. This will of course not work for binaries where you want to use dlsym(), so this would only be safely usable in cases where you want to do static linking. I've written a tiny patch for share/mk, crunchgen and /rescue, to set these flags, only for .o files that are used for static linking (e.g. for .a files) and when linking statically linked executables. http://80386.nl/pub/gc-sections.txt The results are interesting. On amd64: - devd suddenly becomes 500 KB in size, instead of a megabyte, - init's size drops from 900 KB to 600 KB, - clang becomes a megabyte smaller. There is a downside, though. The total size of the system becomes 8 MB larger, because the .a files in /usr/lib are a bit bigger than before. Still, I think that this can be justified: - Systems short on disk-space are typically not used for software development. As /usr/include and /usr/lib/*.a together already account for ~30% of the disk space used, these files will likely be removed on a disk-space constrained system anyway. I can confirm that at least one vendor who is constrained on the size of the root partition does not install /usr/include or any *.a files. On shipping hardware. We do use the hardware for developer builds where we do need include files (though not .a files as we only use shared libraries). - init and devd are so fundamental that they will likely be installed on basically any embedded FreeBSD system. Reducing the size of these binaries makes sense. - It could also reduce the size of statically linked binaries outside of the base system, depending on base libraries. What I also like about this, is that at least for the base system, it will become less important to spread out functions in libraries over separate source files, in an attempt to reduce the size of the resulting binary. We may now leave functions that are related to each other in the same source file, thus improving readability. Would it be possible to enable this only for devd, init, and clang binaries? Or is it a matter of enabling it for library builds that are linked statically with the mentioned binaries? Could init/devd be made smaller by finding out which functions they do/don't use and separating those into separate .c files? Thanks, matthew ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: -ffunction-sections, -fdata-sections and -Wl,--gc-sections
The results are interesting. On amd64: - devd suddenly becomes 500 KB in size, instead of a megabyte, - init's size drops from 900 KB to 600 KB, - clang becomes a megabyte smaller. .. so, I'd like to know specific information as to why these three are now smaller. So what's going on? -adrian ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: -ffunction-sections, -fdata-sections and -Wl,--gc-sections
On Sep 15, 2013, at 2:24 PM, Ed Schouten e...@80386.nl wrote: GCC and Clang support the -ffunction-sections and -fdata-sections flags. Essentially, these flags force the compiler to put every function and variable in its own section. Though this will blow up the …. - devd suddenly becomes 500 KB in size, instead of a megabyte, - init's size drops from 900 KB to 600 KB, Can you figure out what functions are getting omitted when you make this change? Can you extract a linkage map from a build done this way and compare it to one done the regular way? That big of a difference suggests we have some badly-factored code in our libraries. That is, some library is putting functions into a single source file that shouldn't be combined. Tim ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
