Bug#610689: [buildd-tools-devel] Bug#610689: sbuild: cross build support
On Sat, 26 Mar 2011 20:06:21 + Roger Leigh wrote: > On Fri, Mar 25, 2011 at 05:06:02PM +, Hector Oron wrote: > > 2011/3/10 Roger Leigh : > > > For the cross-packages, you can add these to the core build > > > dependencies list, so they will be installed along with build-essential > > > etc. Or (better) you can add a separate XAPT_DEPENDS variable which can > > > be appended to CORE_DEPENDS in Build.pm prior to installing the core > > > dependencies. Does xapt not bring in these packages directly, or do > > > they always need installing by hand? > > > > xapt downloads the foreign architecture packages using APT with -o > > APT::Architecture=$HOST_ARCH into a directory controlled by xapt > > (/var/lib/xapt/, irrc), then throws dpkg-cross into that directory to > > convert those packages and installs them. > > This is the part that I think can be significantly improved. Not until we have Multi-Arch-Cross. Indeed, there's no point making special changes to sbuild ahead of Multi-Arch as the entire cross-dependency resolution mechanism will be completely thrown away at that stage. What you're trying to create in this bug report is a Multi-Arch-Cross resolver for sbuild and then hacking it to handle dpkg-cross. That is the wrong way around. Use the hacks which already work with the dpkg-cross hacks and then implement a *clean* Multi-Arch-Cross resolver when that becomes possible. dpkg-cross is fundamentally broken and is completely alien to Multi-Arch. Anything based on dpkg-cross will need to be completely re-written for Multi-Arch-Cross, so use the tools which exist (and which will disappear alongside dpkg-cross) to retain as much isolation as possible between the sbuild code and the broken dpkg-cross methods. > sbuild creates a local APT archive to install dependency packages from. > We can use the same mechanism to allow the dpkg-cross packages to be > installed. It could even be used to cache generated packages between > builds. We can quite easily refactor the code to allow multiple > archives (if required). The problems in this area are: 0: The setup_apt_archive code uses the Dpkg bindings to resolve the dependencies but these are inherently native (despite taking an arch, it is difficult to get this right when there are alternatives and architecture-specific dependencies). These have been re-written in embuilddeps but all that code will be completely pointless once Multi-Arch-Cross arrives. There is no point implementing yet another broken version of it in an XaptResolver for sbuild. 1: embuilddeps has tried to solve these problems but has a fundamentally different approach to actually installing the packages. Sbuild::AptResolver and Sbuild::AptitudeResolver use the dummy-package method to actually do the work of resolving the dependencies within the specified constraints. embuilddeps resolves the constraints directly. e.g. constraints of foo (>= 1.2.3) [armel] | bar [!linux-any] | baz [i386| amd64] are particularly hard to resolve correctly when running on i386 to cross-build for armel. 2: Cross-dependency resolution requires re-doing the dpkg deps_parse calculation with an empty dpkg status file and then chasing the dependencies all the way down because -cross packages do not exist in the archives. This is why I'm looking at Multi-Arch being the preferred solution here. 3: dummy-package methods won't work for cross because we must resolve all the constraints ahead of passing a list of packages to apt so that these can be downloaded *ahead* of being installed, then converted using dpkg-cross and installed as packagename-$arch-cross packages. libfoo-dev_1.2.3_armel.deb becomes libfoo-dev-armel-cross_1.2.3_all.deb 4: The converted packages do not (indeed cannot) exist in any network-accessible archive - these are all generated on-the-fly until Multi-Arch makes the whole conversion avoidable. 5: apt and aptitude consistently treat -$arch-cross packages as "local or obsolete" and consistently get removal parsing wrong when upgrading existing packages. This means that the list of -$arch-cross packages to be removed needs to be carefully managed, especially if the cross toolchain is not to be removed upon every build. Much easier is for xapt|embuilddeps to put the downloaded and the converted packages into a directory which sbuild can manage, or for sbuild to become aware of the current defaults which are /var/lib/xapt/output for -cross packages and /var/lib/xapt/archives for the foreign architecture originals. This will also solve problems of package removal after the build, albeit that some extra processing is required to skip certain -cross packages which would otherwise cause the removal of the cross-building toolchain. (libc6-dev-$arch-cross is a frequent problem here.) This list of packages does need to be kept updated to cope with changes in gcc and in eglibc - e.g. the Squeeze chroots will have a libc-bin-$ARCH-cross package which doesn't exist in Lenny chroots. Whilst it
Bug#610689: [buildd-tools-devel] Bug#610689: sbuild: cross build support
On Fri, Mar 25, 2011 at 05:06:02PM +, Hector Oron wrote: > 2011/3/10 Roger Leigh : > > > This part can be done by the xapt resolver directly, without any > > special user setup required. > > True. > > > For the cross-packages, you can add these to the core build > > dependencies list, so they will be installed along with build-essential > > etc. Or (better) you can add a separate XAPT_DEPENDS variable which can > > be appended to CORE_DEPENDS in Build.pm prior to installing the core > > dependencies. Does xapt not bring in these packages directly, or do > > they always need installing by hand? > > xapt downloads the foreign architecture packages using APT with -o > APT::Architecture=$HOST_ARCH into a directory controlled by xapt > (/var/lib/xapt/, irrc), then throws dpkg-cross into that directory to > convert those packages and installs them. This is the part that I think can be significantly improved. sbuild creates a local APT archive to install dependency packages from. We can use the same mechanism to allow the dpkg-cross packages to be installed. It could even be used to cache generated packages between builds. We can quite easily refactor the code to allow multiple archives (if required). > > Well, when we install the build dependencies the package isn't yet > > unpacked. The unpacking is done in the build() function. But there's > > no reason we can't reorder it if we need to. The unpacking could > > be split out of build() into a separate unpack_source() function, > > which could be run earlier if xapt/embuilddeps is in use. > > > > What information does embuilddeps require from the unpacked source > > package? Is this information also available in the .dsc? (We do > > have the .dsc information directly to hand). > > It is possible to use .dsc information, but embuilddeps does not know > how (yet). :-) OK. For the purpose of actually implementing this, I would like to have a description of exactly which fields we need for cross building from which files in the source or DSC, and what they do and why. Is this all documented somewhere? > > xapt is doing two jobs it seems: > > 1) from the package list, make a list containing all the cross packages > > needed > > That is given with simulation option (-n). This is with embuilddeps, not xapt? This option appears to give a package list, prefixed with "/usr/bin/apt-get -y". This just isn't suitable. - it's omitted packages which were already installed; this is not necessary and involves poking around under /var/lib/dpkg. - we need to strip off the "/usr/bin/apt-get -y" which might be subject to change and then cause breakage, at some point in the future. All we want is the /complete/ package list, without any removal of packages or addition of other bits. > > 2) install them (but it doesn't list them in the list of packages to > > install, which makes removal harder potentially) > > > > From the resolver POV, what would be really good if it could just do > > (1). This way, we can just give the expanded list back to the > > resolver for it to install. Because we need to clean up after > > ourselves, the apt and aptitude resolvers build a dummy > > "dependency package" from the dependency list, and then get apt/ > > aptitude to install it. > > This is how old pbuilder used to do it, but it does not work with > cross packages as adding dependencies on > $foreign_package-$HOST_ARCH-cross won't work, as those packages are > output from dpkg-cross and not available from repositories. Right. But with sbuild, as I mentioned above, we can install the dpkg-cross-generated packages into a local archive, which *is* accessible to apt-get/aptitude, and hence you will then be able to use cross packages in the dependencies /directly/. This will permit proper cleanup after a build using the normal mechanisms sbuild uses for all builds. This is to say the resolver can do the following: - get the cross dependencies from the dsc or unpacked source - download the debs - run dpkg-cross - install the dpkg-cross-generated debs into the local archive - add the cross-dependencies to the package build dependencies ... - run dpkg-buildpackage with the necessary options to cross build ... We already allow various things to add to the package build deps, and so adding cross-deps will be trivial. All we really need is the necessary information for what to parse from the dsc or source package, and then how to use dpkg-cross. Just to be clear: none of the above uses xapt/embuilddeps, because sbuild can, I think, do a better job with its local archive and which will integrate much better with our existing cleanup code when it comes to removing build deps. > > We then reverse the process to clean up > > after the build. So ideally, we don't really want xapt to do any > > installation: we just want it to give us the package list. If > > this is possible, it might not even be require to have a separate > > XaptResolver--we just expand the cros
