Re: Prebind from OpenBSD
Quoting Jesse Smith jessefrgsm...@yahoo.ca (from Sun, 27 Mar 2011 17:29:20 -0300): Thanks very much for the reply. I had thought this was a project/port that was in progress, rather than something waiting on research. Do you know if that's also the case with other projects on the Ideas list? A lot of them don't have many details or contact information associated with them. I'm especially interested in the OpenBSD xlint port. Here's what I told a student who wants to participate in the GSoC about the OpenBSD lint port entry: ---snip--- The project is about having a look at what OpenBSD did to their lint. So first analyze what we have, then analyze what they have, and then see if stuff they have but we haven't can be ported. This assumes that there is a common ancestor to both lints, and that the code didn't diverge too much so that we do not have an either this or that situation. IF the code diverged to much or if there is no common ancestor, there is nothing to do regarding this specific project (it would be a new project to improve our lint). I do not know the code of both lints, so I do not know much improvements there are which we don't have, and how much work it is to port this over. In the worst case there is not enough or too much to do this as a GSoC project. Should be easy to determine by reading the history of lint on both version control systems to see if there is a common starting point, and if yes to see how much they derived from there. Regarding the lint project itself... I (= personal opinion, not an official word from FreeBSD) do not think this is a project which would gain a lot of interest (= highly rated during the GSoC evaluation). If you want to send in a proposal for this (after researching if it is feasible, see above), I suggest to send in more than one proposal. This should improve your change to be chosen. In the last GSoCs we had students which send in several proposals, some of the proposals didn't gain enough interest, but another one from the same person was chosen. Seeing multiple proposals from the same person gave us a hint about his technical knowledge (sane expectations in the proposals or not). ---snip--- Bye, Alexander. -- Snakes. Why did it have to be snakes? http://www.Leidinger.netAlexander @ Leidinger.net: PGP ID = B0063FE7 http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137 ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Prebind from OpenBSD
On 2011-Mar-27 20:54:18 +0100, Robert Watson rwat...@freebsd.org wrote: part of rc.d. I'd also investigate large applications like Firefox, Chrome, KDE, Gnome, etc. KDE already integrates prebinding tricks in its design, but I don't think the others do. Improving startup time for large, infrequently started executables will enhance the user experience but not do a great deal for overall system performance. (And note that things like OOo, emacs and browsers have significant amounts of code in embedded scripting languages and it's unlikely that pre-binding will help much there). I suspect a bigger overall win would be gained by speeding up small but very frequently started executables - /bin/sh is the most obvious candidate here, though there are probably other candidates in /bin and /usr/bin. In this case, you need to measure how frequenctly it is started as well as the per-startup savings. For some of these executables, it's easy to get a reasonably accurate estimate of the potential pre-binding savings by comparing the speed of the existing dynamically-linked executable in /bin with the same statically-linked executable in /rescue. One thing that I'm not sure if you've take into account is process- initiated library loading (using dlopen(3) and friends). Note that even /bin/sh can do this through things like locale handling. -- Peter Jeremy pgpj6ptimMgB2.pgp Description: PGP signature
Re: Prebind from OpenBSD
On Sat, 26 Mar 2011, Jesse Smith wrote: I'm interested in working on the Port prebind from OpenBSD project mentioned on the FreeBSD Ideas page. ( http://wiki.freebsd.org/IdeasPage#head-d28cdd95ca1755d5afe63d653cb4926d4bdc99de ) There isn't much to go on from the project description and I'm curious what FreeBSD devs are looking for specifically. For example, should the entire ldconfig program be ported from OpenBSD (it looks like it's close enough to FreeBSD's to make that suitable), or should just the prebind code be merged into FreeBSD's ldcnfig? Once the project is complete, who should the work be submitted to? Has anyone else worked on this and made any progress? Hi Jesse: I think the intent of the ideas list entry is more a research project than a direct-to-commit project: the question is whether prebinding of some sort would observably help performance for important FreeBSD applications or, for example, the boot process. If so, then certainly the OpenBSD prebinding code is a possible model -- Mac OS X also has prebinding, of course, and it's done quite differently (and probably less reusably from our perspective as they use Mach-O rather than ELF); however, there might be interesting ideas as well. I think therefore I'd structure a project along the following lines: first, you want to establish to what extent synchronous waiting on linkage at run-time is a significant problem. It could be that some combination of hwpmc and DTrace would be the right tools for this. I'd especially pay attention to boot time, since we know that quite a lot of executing takes place then as part of rc.d. I'd also investigate large applications like Firefox, Chrome, KDE, Gnome, etc. KDE already integrates prebinding tricks in its design, but I don't think the others do. Next, I'd dig a bit more into the areas where it's hurting performance -- can you add up all the time spent waiting and cut 10 seconds from boot, or 5 seconds from Firefox startup? Or is the best win going to be .2 seconds in Firefox? Does the OpenBSD optimisation actually address the problem we're experiencing? Perhaps perform some experiments with prebinding-like behaviour, working up to an implementation. It's worth remembering that prebinding comes with some baggage as well, of course. Perhaps less relevant in the world of 64-bit address spaces, but there are some design trade-offs in this department... Robert ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Prebind from OpenBSD
On Sun, Mar 27, 2011 at 08:54:18PM +0100, Robert Watson wrote: On Sat, 26 Mar 2011, Jesse Smith wrote: I'm interested in working on the Port prebind from OpenBSD project mentioned on the FreeBSD Ideas page. ( http://wiki.freebsd.org/IdeasPage#head-d28cdd95ca1755d5afe63d653cb4926d4bdc99de ) There isn't much to go on from the project description and I'm curious what FreeBSD devs are looking for specifically. For example, should the entire ldconfig program be ported from OpenBSD (it looks like it's close enough to FreeBSD's to make that suitable), or should just the prebind code be merged into FreeBSD's ldcnfig? Once the project is complete, who should the work be submitted to? Has anyone else worked on this and made any progress? Hi Jesse: I think the intent of the ideas list entry is more a research project than a direct-to-commit project: the question is whether prebinding of some sort would observably help performance for important FreeBSD applications or, for example, the boot process. If so, then certainly the OpenBSD prebinding code is a possible model -- Mac OS X also has prebinding, of course, and it's done quite differently (and probably less reusably from our perspective as they use Mach-O rather than ELF); however, there might be interesting ideas as well. I think therefore I'd structure a project along the following lines: first, you want to establish to what extent synchronous waiting on linkage at run-time is a significant problem. It could be that some combination of hwpmc and DTrace would be the right tools for this. I'd especially pay attention to boot time, since we know that quite a lot of executing takes place then as part of rc.d. I'd also investigate large applications like Firefox, Chrome, KDE, Gnome, etc. KDE already integrates prebinding tricks in its design, but I don't think the others do. Next, I'd dig a bit more into the areas where it's hurting performance -- can you add up all the time spent waiting and cut 10 seconds from boot, or 5 seconds from Firefox startup? Or is the best win going to be .2 seconds in Firefox? Does the OpenBSD optimisation actually address the problem we're experiencing? Perhaps perform some experiments with prebinding-like behaviour, working up to an implementation. It's worth remembering that prebinding comes with some baggage as well, of course. Perhaps less relevant in the world of 64-bit address spaces, but there are some design trade-offs in this department... The most serious issue with prebind is a consistency. It is very easy to get prebind data out of date, and this is esp. easy in the FreeBSD where buildworld and source port upgrades are everyday activity. Before this goes much further, yes, we need a benchmarks that demonstrate that system spends significant time in the symbol resolution for often started images [*], and second, we need to have a workable model for source upgrades. * - so that Firefox, OpenOffice etc do not qualify, IMHO. pgpamzbTnhOCD.pgp Description: PGP signature
Re: Prebind from OpenBSD
On 27 Mar 2011, at 21:04, Kostik Belousov wrote: The most serious issue with prebind is a consistency. It is very easy to get prebind data out of date, and this is esp. easy in the FreeBSD where buildworld and source port upgrades are everyday activity. Before this goes much further, yes, we need a benchmarks that demonstrate that system spends significant time in the symbol resolution for often started images [*], and second, we need to have a workable model for source upgrades. * - so that Firefox, OpenOffice etc do not qualify, IMHO. As I recall, Apple actually does prebinding/etc only for a small set of commonly used libraries -- their libSystem, and then a dynamically determined set of other libraries. Their model puts the libraries into a common shared memory segment that can then be mapped at the same address for all dynamically linked processes in the same chroot. This amortised not only the cost of linking, but also the cost of managing memory mappings (I believe it's a shared submap, but my VM terminology is ... ageing). The downside to their approach is that it requires special kernel support, and probably encodes things that we would prefer not to. However, you could imagine a daemon providing a similar service, in which a file descriptor is made available via a UNIX domain socket to instanced of rtld around the system, which can consistently map it in the same place on each exec. This would also give us an upgrade model that might work better -- the daemon actually prepares the contents of that memory and makes them independent from on-disk storage: when an upgrade is done, you kick the daemon to introduce a new shared memory object, closing the previous one, which causes it to be GC'd when all outstanding processes have exited. Robert___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Prebind from OpenBSD
-Original Message- From: Robert Watson rwat...@freebsd.org To: Jesse Smith jessefrgsm...@yahoo.ca Cc: freebsd-hackers@freebsd.org Subject: Re: Prebind from OpenBSD Date: Sun, 27 Mar 2011 20:54:18 +0100 (BST) On Sat, 26 Mar 2011, Jesse Smith wrote: I'm interested in working on the Port prebind from OpenBSD project mentioned on the FreeBSD Ideas page. ( http://wiki.freebsd.org/IdeasPage#head-d28cdd95ca1755d5afe63d653cb4926d4bdc99de ) There isn't much to go on from the project description and I'm curious what FreeBSD devs are looking for specifically. For example, should the entire ldconfig program be ported from OpenBSD (it looks like it's close enough to FreeBSD's to make that suitable), or should just the prebind code be merged into FreeBSD's ldcnfig? Once the project is complete, who should the work be submitted to? Has anyone else worked on this and made any progress? Hi Jesse: I think the intent of the ideas list entry is more a research project than a direct-to-commit project: the question is whether prebinding of some sort would observably help performance for important FreeBSD applications or, for example, the boot process. If so, then certainly the OpenBSD prebinding code is a possible model -- Mac OS X also has prebinding, of course, and it's done quite differently (and probably less reusably from our perspective as they use Mach-O rather than ELF); however, there might be interesting ideas as well. I think therefore I'd structure a project along the following lines: first, you want to establish to what extent synchronous waiting on linkage at run-time is a significant problem. It could be that some combination of hwpmc and DTrace would be the right tools for this. I'd especially pay attention to boot time, since we know that quite a lot of executing takes place then as part of rc.d. I'd also investigate large applications like Firefox, Chrome, KDE, Gnome, etc. KDE already integrates prebinding tricks in its design, but I don't think the others do. Next, I'd dig a bit more into the areas where it's hurting performance -- can you add up all the time spent waiting and cut 10 seconds from boot, or 5 seconds from Firefox startup? Or is the best win going to be .2 seconds in Firefox? Does the OpenBSD optimisation actually address the problem we're experiencing? Perhaps perform some experiments with prebinding-like behaviour, working up to an implementation. It's worth remembering that prebinding comes with some baggage as well, of course. Perhaps less relevant in the world of 64-bit address spaces, but there are some design trade-offs in this department... Robert Robert, Thanks very much for the reply. I had thought this was a project/port that was in progress, rather than something waiting on research. Do you know if that's also the case with other projects on the Ideas list? A lot of them don't have many details or contact information associated with them. I'm especially interested in the OpenBSD xlint port. I appreciate you time, Jesse ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
