We've talked about ways of making new repo tasks faster, and one of the big speedups is no longer making a tree of hardlinks to run createrepo over.
Recently createrepo gained[1] the ability to read from an include list. Paired with the option to specify a baseurl to make everything relative from, we can now run createrepo directly on the packages/ dir, using the include file (generated from the list of packages we /would/ have hardlinked), use an http url as the baseurl (so that static-repos continue to work) and skip the hardlink step all together. An added bonus is that when a static-repo expires, the metadata you may be working with will continue to reference valid package URLS so your transaction will continue without failure. While I was working with this code, I also noticed that we can stop creating a buildsys-build rpm and instead rely on the groupdata in the repodata and just use 'groupinstall buildsys-build' as are chroot init command. Add to that we were unnecessarily linking comps.xml files into each arch directory of a repo directory, when instead we can just reference the one we create in groups/ when creating repodata. Since we no longer need to make the buildgroups rpm, the PrepRepo task in kojid is no longer needed. I took the one part that it was still usefully doing (asking the hub to prep the repo dirs) and move it up to the NewRepo task. This saves a task creation and pickup/handoff. I introduced a config item in kojid.conf, which is pkgurl. This is used as the baseurl component to the repodata and should be the url which leads to the packages/ directory. Finally, since the repodata itself will continue to live in the same expected locations, builders can continue to consume the repodata via nfs. But since the repodata itself references http, yum will use http to download the packages to populate the buildroot. We just have to ensure that the url used in pkgurl is not only valid in the outside world (so that static-repos continue to work) but valid within the buildsystem structure as well. I've committed my changes to a git tree (cloned from tip today) here: http://jkeating.fedorapeople.org/git/koji/ Feedback would be welcome. We're going to get the createrepo change into RHEL5.2 (along with --update support) with the idea that when we refresh the Fedora buildsystem machines in early December, we'll go with RHEL5 and perhaps a newer createrepo package. Ideally we'd get the changes I've made into koji beforehand so that we can release a new build of koji during the refresh. [1] https://lists.dulug.duke.edu/pipermail/rpm-metadata/2007-November/000812.html -- Jesse Keating Fedora -- All my bits are free, are yours?
signature.asc
Description: PGP signature
-- Fedora-buildsys-list mailing list [email protected] https://www.redhat.com/mailman/listinfo/fedora-buildsys-list
