Re: My personal pkgsrc FAQ
Hi! 2008/12/18 Justin C. Sherrill jus...@shiningsilence.com: On Thu, December 18, 2008 2:16 am, Robert Luciani wrote: The advantage of using a vkernel (or at least keeping your chroot around for a long while) is that it allows you to keep rebuilding packages that were tagged with vulnerabilities, from the same environment, for the entire lifespan of the package set. Otherwise, security updates render a stable package set obsolete very quickly. This was also why I mentioned pkg_chk and that it needs to be fixed. Because now, updating packages is so arduous that people just leave firefox-3 as an old version even though it might have multiple security problems. I'd say stick with a chroot; it'll accomplish the same thing without the overhead. I suppose trying and timing both strategies with the same pkgsrc release would provide an interesting benchmark on just how much overhead the virtualized kernel introduces... /me wishes DFBSD has cluster support already. I'd be glad to share some CPU cycles for package building. :) Having a packed vkernel environment will let me to easily install a little cluster block with limited access to other system stuff. PS. Yes, I can imagine the amount of work to be done to achieve that goal. Treat this as a dreaming-rumbling-mumbling aloud. :) -- Dennis Melentyev
Re: My personal pkgsrc FAQ
On Wed, Dec 17, 2008 at 09:07:23AM +0100, Matthias Schmidt wrote: - IIRC it should be possible to build pkgsrc packages in parallel. Does that mode works well? If yes, we could distribute the builds between multiple machines to reduce the time needed to run a full build. And yes, a have a machine to offer :) It works well, but if you want to distribute over a non-local network you get some interesting issues with redistribution of the packages during the build. I don't think it is worth the effort. - How about renting a colocation machine? Renting a fast machine with a fast traffic flat is not that expensive nowadays. We even have some SoC money left. Donating some money is not a big problem either. Last time I check, SMP scalability in fork/exec heavy work load was not that well, so it was more effective to use vmware than native SMP. Otherwise it can bring down the build time a lot. Joerg
Re: My personal pkgsrc FAQ
Another thought I had for some time: Pkg_radd uses the http://pkgbox.dragonflybsd.org/All redirection if I remember correctly. Can't we have a version number in the /All redirection mechanism so that we can have different pkg_radd's (HEAD, release, etc.) use different sets of packages? We need 2 package sets. The first one, the system default PKG_PATH, would be based on a stable set which gets rebuilt only for every DragonFly release (like the upcoming DF 2.2). For this, using a quarterly release is most appropriate because it would allow for security updates to be added to the package set without worries of dependency problems from other random updated packages. Is this something that you could oversee, Justin? The second package set would be based on the current pkgsrc. I volunteer for this task and am willing to spend the time to get it working reliably and with updates in a timely fashion. The package set would need to be rebuilt _at least_ every other week, but hopefully more than once a week depending on whether my machine can handle bi-weekly updates. I'm going to set up a vkernel for this which would just run all day, and try to get the process as automated as possible. I'm wondering though, if it would be more prudent send the built packages straight to chlamydia and let them propagate from there, or if the updates are small enough when done frequently that they should be uploaded to Matt's pkgbox? Either way, to switch from the stable to the current repository one would simply point PKG_PATH to http://pkgbox.dragonflybsd.org/packages/current to start grabbing new packages. After installing the packages comes the question of keeping them up to date. Something reliable, autonomous, that uses binaries, and works does not really exist for pkgsrc as far as I know. There is however a rudimentary program, pkgtools/pkg_chk, which _almost_ works. It seems to detect packages that need updating correctly when checking against an updated /usr/pkgsrc but when checking against a URL like (for instance): http://pkgbox.dragonflybsd.org/packages/DragonFly-2.0/pkgsrc-current/All/ it gets confused like: www/apache2 - apache-2.0.63nb4 apache-2.0.63nb2 (binary package available) or databases/php-pgsql - php4-pgsql-4.4.9nb1 missing (binary package available) even though I have php5-pgsql-5.2.6nb1 installed. But it comes pretty close. If this program worked we'd almost have a usable package system! I'd appreciate it if someone wants to take a peek at it, otherwise I'll check it out during the holiday break. :) -- Robert Luciani Chalmers University of Technology, SWE Department of Computer Science and Engineering http://www.rluciani.com/rluciani.asc
Re: My personal pkgsrc FAQ
On Wed, December 17, 2008 3:36 am, Sascha Wildner wrote: Please don't forget the LiveDVD which should be rebuildable by the user using the official set of packages without having to resort to home-rolled packages or packages from some non-standard location. To ensure this, we definitely need the packages earlier than right at release time. pkgbox has up to date packages now that are downloadable, though the pkg_radd redirect hasn't been pointed at them yet - will this work? Another thought I had for some time: Pkg_radd uses the http://pkgbox.dragonflybsd.org/All redirection if I remember correctly. Can't we have a version number in the /All redirection mechanism so that we can have different pkg_radd's (HEAD, release, etc.) use different sets of packages? Yeah, this would work nicely. We'd have to be careful about versioning, though, since our releases and pkgsrc quarterly releases aren't synced. Each quarterly release of pkgsrc stops development on the previous one, so that's an easy answer: always build the latest. Should I do the same for DragonFly? We also need to lock pkgbox.dragonflybsd.org to the latest release version; it's at 2.1.1 now which was somewhat unavoidable because of the Hammer changes Matt had to add, but it in general should match if we're going to make release-specific sets.
Re: My personal pkgsrc FAQ
On Wed, December 17, 2008 6:26 pm, Robert Luciani wrote: The second package set would be based on the current pkgsrc. I volunteer for this task and am willing to spend the time to get it working reliably and with updates in a timely fashion. The package set would need to be rebuilt _at least_ every other week, but hopefully more than once a week depending on whether my machine can handle bi-weekly updates. I'm going to set up a vkernel for this which would just run all day, and try to get the process as automated as possible. Using a vkernel may slow it down too much - not that vkernels impose a lot of overhead, but when the process already takes multiple days, even a small percentage increase may end up being a lot. Uploading to chlamydia may work, but I don't know if all our mirrors are running from there. (Aho isn't? Speak up, please) Would you be building this on the current DragonFly release, or following HEAD?
Re: My personal pkgsrc FAQ
Justin C. Sherrill wrote: On Wed, December 17, 2008 6:26 pm, Robert Luciani wrote: The second package set would be based on the current pkgsrc. I volunteer for this task and am willing to spend the time to get it working reliably and with updates in a timely fashion. The package set would need to be rebuilt _at least_ every other week, but hopefully more than once a week depending on whether my machine can handle bi-weekly updates. I'm going to set up a vkernel for this which would just run all day, and try to get the process as automated as possible. Using a vkernel may slow it down too much - not that vkernels impose a lot of overhead, but when the process already takes multiple days, even a small percentage increase may end up being a lot. I'm thinking that in my case it will be alright since a complete build from scratch takes between 4-5 days. Since I'm just building the updates all the time, it shouldn't take more than a day (worst case) to get in sync. The advantage of using a vkernel (or at least keeping your chroot around for a long while) is that it allows you to keep rebuilding packages that were tagged with vulnerabilities, from the same environment, for the entire lifespan of the package set. Otherwise, security updates render a stable package set obsolete very quickly. This was also why I mentioned pkg_chk and that it needs to be fixed. Because now, updating packages is so arduous that people just leave firefox-3 as an old version even though it might have multiple security problems. I think if we want a stable binary package set with security updates, using quarterly releases is the simplest choice. Then in conjunction, we have also have the current package set for users that don't mind updating 10 additional packages in order to satisfy new dependencies for some newly installed GTK app that was built against the latest pkgsrc. I think the versioning of the /All redirection is a fine idea if anyone wants to implement it but since pkg_radd uses PKG_PATH we don't really _need_ to change anything in the mirroring scripts except maybe agree on an official directory layout. So a person wanting to use the current packages can just manually set PKG_PATH to packages/current.
