Re: History of Debian bootstrapping/porting efforts
Since yesterday, my tools can now finally turn the whole dependency graph Does this whole dependency graph include the implicit build-dependency every package has on build-essential? The above case for example has no alternative solution as the cycle is of length two and has no other way of braking it than building pkg-config without libglib2.0-dev. Since this is unlikely to be possible I don't see why it would be impossible to hack up the glib source package to not rely on pkg-config. Whether that is a good idea or not is another matter. and since the assumption is that only build dependencies might be dropped when necessary but not binary dependencies, a possible solution might be cross compilation. It seems pretty clear to me that there is a core of software that will need to be cross-built as the first stage of bootstrapping a port. Obviously essential and build-essential fall into this category but while i'm sure there are ways one could hack away the cycles and make things like pkg-config and debhelper natively bootstrapable I don't think there is much point in doing so. What i'd ideally like to see is for a tool to be able to generate a directed acyclic graph of build jobs (some cross, some native, there should be an option in the tool as to whether to preffer native or cross-build jobs) that takes the user from having no packages for the target architecture to having a set of bootstrap packages that can be used to seed the regular building process. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/50aef1f4.6040...@p10link.net
Re: History of Debian bootstrapping/porting efforts
Johannes Schauer dixit: If you were ever involved in porting Debian to a new architecture or know how a port was done, please do not hesitate to either write me an For reviving m68k: - did you cross compile parts of Debian? How much? How hard was it? Only to test kernels and a few userspace issues. I was lucky I could use packages from both etch-m68k (a snapshot of mostly sid at the time when etch was released) and unstable (mostly broken by then) as basis. - did you natively compile on another distribution (openembedded, gentoo) No, just on old Debian, peu à peu, until I could get cowbuilder running and create a clean build chroot, after which I rebuilt all not-cleanly-built packages. (I prefer cowbuilder over sbuild a lot!) - how did you go about finding reduced build dependencies? What were the heuristics you used? How did you find dependency cycles to break and how did you pick a solution? Fully manually. I mostly drew ASCII files like this: sourcepkg1 sourcepkg2 sourcepkg3 sourcepkg4 sourcepkg5 sourcepkg3dup sourcepkg6 sourcepkg2cyc The web view on packages.debian.org was, and still is, a *huge* help for drawing those, especially as it also includes packages from debian-ports unstable in their sid view (I just wish they’d also include d-p unreleased suite in it, even without a link but just an inclusion in the version table would be superb). Then I worked on them from right to left, occasionally patching a huge dependency out or breaking a B-D loop by hand, sometimes also installing older versions of B-Ds manually, or even building versions older than sid but newer than what I had. Sometimes, package maintainers were a huge help in tracking down optional dependencies or things I could turn off for bootstrapping; too bad we still don’t have build profiles in Debian proper. Other times, I did that using human judgment ;) - how long did the port take you? After about a year, doing this “on the side”, i.e. not concentrating on it a lot, and considering the very slow speed of the systems, most things were working. One has to consider I learned about how Debian really works during that time, too… - what were the most common bugs you filed when introducing the new architecture? Dropping of B-D or versioning them proper or making them arch-specific. For some packages (mostly toolchain, some device drivers like libdri) also patch inclusion. Almost all maintainers and the kernel team were very helpful and happy to include arch-specific patches that did not touch MI parts, and some accepted MI patches as well. For some, I went upstream, which took months(!) longer in most cases. Build-Depends: huge (= 1.0) [i386 arm] !embedded !bootstrap, tiny Yeah, waiting for it… As for the rest, we already talked on the mailing list. bye, //mirabilos -- Sometimes they [people] care too much: pretty printers [and syntax highligh- ting, d.A.] mechanically produce pretty output that accentuates irrelevant detail in the program, which is as sensible as putting all the prepositions in English text in bold font. -- Rob Pike in Notes on Programming in C -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/pine.bsm.4.64l.1211202112490.29...@herc.mirbsd.org
Re: History of Debian bootstrapping/porting efforts
Hi, On Tue, Nov 20, 2012 at 09:22:40PM +, Thorsten Glaser wrote: For reviving m68k: Thanks for your detailed explanations! I added a summary of it to the m68k section of the wiki page [1], extending the notes entered there by Ingo Jürgensmann. Thanks to both of you! - how did you go about finding reduced build dependencies? What were the heuristics you used? How did you find dependency cycles to break and how did you pick a solution? Fully manually. I mostly drew ASCII files like this: sourcepkg1 sourcepkg2 sourcepkg3 sourcepkg4 sourcepkg5 sourcepkg3dup sourcepkg6 sourcepkg2cyc [...] Then I worked on them from right to left, occasionally patching a huge dependency out or breaking a B-D loop by hand, sometimes also installing older versions of B-Ds manually, or even building versions older than sid but newer than what I had. Since yesterday, my tools can now finally turn the whole dependency graph into an acyclic graph by braking only a small number of build dependencies (using an approximative solution to the minimal feedback arc set problem) and output a final build order to build all of Debian starting from an initial minimal build system (priority:essential plus build-essential plus debhelper). Detailed explanations can be found in the email I wrote to our debian-bootstrap mailinglist: http://lists.mister-muffin.de/pipermail/debian-bootstrap/2012-November/000425.html A summary: Using the results I got from Daniel Scheplers x32 bootstrapping work, droppable build dependencies from Thorsten Glaser, Patrick McDermott and my Gentoo work, I was able to reduce the original 923 nodes SCC into one SCC with 159 nodes and another with 42 nodes: http://mister-muffin.de/p/NRTs.png http://mister-muffin.de/p/myFd.png Those two graphs were easily breakable into a DAG by just removing 4 and 2 build dependencies respectively using the feedback arc set algorithm I wrote over the weekend. The same algorithm can also be applied to the full 923 node SCC for Debian Sid, resulting in 90 build dependencies to be dropped to make the graph acyclic and making a final build order possible with (close to) minimal changes. For this to happen, only 50 source packages have to be modified. Build-Depends: huge (= 1.0) [i386 arm] !embedded !bootstrap, tiny Yeah, waiting for it… As build profiles do not exist yet, there might be instances where my algorithm chooses build dependencies that are not droppable in practice. One obvious example is the dependency cycle: src:pkg-config -- libglib2.0-dev So I'm now implementing a facility that allows to explicitly mark build dependencies as unbreakable so that the algorithm finds an alternative solution or throws an error. The above case for example has no alternative solution as the cycle is of length two and has no other way of braking it than building pkg-config without libglib2.0-dev. Since this is unlikely to be possible and since the assumption is that only build dependencies might be dropped when necessary but not binary dependencies, a possible solution might be cross compilation. Looking at the cross build dependency graph and braking its dependency cycles is the next step I will take. cheers, josch [1] http://wiki.debian.org/DebianBootstrap/History -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20121120225523.GA18533@hoothoot
Re: History of Debian bootstrapping/porting efforts
I read your recent post to debian-devel with great interest, as I've done some bootstrapping efforts in the past, and I'm currently in the middle of a port for the x32 ABI. In the past, what I've done (mostly privately) was to develop a script I called pbuildd which essentially just runs through the list of currently unbuilt packages and tries running pbuilder on them all, then installs anything that succeeds into a local repository and starts up the loop again. Then, when things got stuck, I just did a manual inspection of the unsatisfied dependencies to find the cycles, and chose one to break. In fact, I've just started uploading my current iteration of this to http://87.98.215.228/debian/ -- you might want to especially look at scripts/pbuildd which is the central script to run this loop. (And over time, it's gathered various optimizations to speed up the installation into local repository step, try to avoid invoking pbuilder if it can easily determine that certain Build-Depends aren't present at all, etc.) Initially, when I needed to break a cycle, I would just build something by hand and stick it into the partial directory, but over time I started developing automated cycle-breaker scripts, which are currently under scripts/cb.inactive (the pbuildd script looks for them under scripts/cb). The scripts tend to become outdated over time, though, with a moving target, and I'm sure the current state is no exception. My personal heuristics for what I preferred were: first, prefer cycle-breaking which just removes Build-Depends which are there to build documentation. Then, prefer cycle-breaking which ignores Build-Depends on one or a few libraries which provide purely optional features. If I couldn't find anything of this sort, I'd just try to find the cycle-breaking point which would be (fuzzily) least invasive and least likely to break the resulting packages, at least as far as packages that Build-Depend on them. In the past, pbuildd was mainly geared towards trying to build all of Debian (including the binary-indep packages) starting from a minimal chroot and with minimal extra package downloads, but on an established architecture. It was only recently that I started applying it to bootstrapping x32. The way I started that was actually: I started off mainly following the instructions from Linux From Scratch, though of course adjusting it to cross-building to x32 as necessary. I also inserted dpkg into the process as soon as possible after the first LFS stage creating the chroot with /tools, and from then on ran installs into temporary directories, and built dummy dpkg packages with no dependencies. Then, after the LFS builds were over, I started building real Debian packages from the actual .dsc source packages, and eventually had enough packages built in this way that I was able to do a debootstrap, and start the pbuildd process. (So yes, x32 might be a special case in that it runs on already widely available hardware, and I could develop from an existing Debian installation. But I'd imagine that was probably the case as well, though possibly to a smaller degree, for the amd64 and ia64 ports, and possibly other recent ports to new ABIs on existing hardware (armhf?).) As for specifics on the x32 port -- currently the most common issues I see (roughly from most common to least common) are: * Outdated libtool, which causes it to want to ld -r with the wrong -m target type. (And also ld -r more often than usual, because getconf ARG_MAX outputs undefined on x32 in certain circumstances, which the outdated libtool can't handle properly.) Of course this isn't an issue for packages that autoreconf at build time. * Issues which will hit every architecture once we switch to eglibc 2.16. Out of this, by far the most common is gnulib (other than very recent git versions) wanting to attach a warning to the gets prototype, which is no longer exported by default in eglibc 2.16. * Code still using sysctl(2), which is no longer supported in x32. * Code which unconditionally uses 64-bit asm snippets if __x86_64__ or __amd64__ is defined -- which causes assembler failures if one of the inputs or outputs is a long or a pointer type, and the asm snippet uses explicit q sizing suffixes (or there are other mismatches). -- Daniel Schepler -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/CADf0C45igv-SnQcEz8v_A0vH=VKRJ+eFc7cwt2j5U=8rcyf...@mail.gmail.com
History of Debian bootstrapping/porting efforts
Dear list, I am the student that did the Port bootstrap build-ordering tool Google Summer of Code project this summer [1]. I am still continuing that work and will turn it into my Master thesis. The tools I developed are currently in use by wookey for doing the arm64 port [2]. A long list of results of my programs as well as outstanding problems can be found in the Debian wiki [3]. For the introduction of my thesis, I want to give a summary of how past ports were accomplished. Since there does not seem to be a mailinglist for general ports (but only for each specific port), I am addressing this email to debian-devel. I created a wiki page to collect the information so that it would also be available to everybody else: http://wiki.debian.org/DebianBootstrap/History If you were ever involved in porting Debian to a new architecture or know how a port was done, please do not hesitate to either write me an email or directly enter the information in the linked wiki page linked above. It assumes that arch labels are established and that gcc, glibc, binutils etc understand the arch. I basically would like to know things like: - did you cross compile parts of Debian? How much? How hard was it? - did you natively compile on another distribution (openembedded, gentoo) - how did you go about finding reduced build dependencies? What were the heuristics you used? How did you find dependency cycles to break and how did you pick a solution? - how long did the port take you? - what were the most common bugs you filed when introducing the new architecture? Having an overview that answers above questions (and anything else noteworthy or quantifiable) can also be useful for future porters to get an overview or ideas how to go about it technically. At the very best, I will get more ideas for heuristics that allow my tools to get better at breaking build dependency cycles. Thanks for your help! cheers, josch [1] http://wiki.debian.org/SummerOfCode2012/StudentApplications/JohannesSchauer [2] http://wiki.debian.org/Arm64Port [3] http://wiki.debian.org/DebianBootstrap/TODO -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20121113201924.GA16619@hoothoot
Re: History of Debian bootstrapping/porting efforts
I took OpenIndiana and started to rebuild Debian packages ignoring formal dependencies. Each new package was install via dpkg-deb -x :-)
