Re: Rebuilding the entire Debian archive twice on arm64 hardware for fun and proft
On Mon, Jan 07, 2019 at 11:46:41PM +, Luke Kenneth Casson Leighton wrote: > On Tuesday, January 8, 2019, Mike Hommey wrote: > > > . > > > > Note that Firefox is built with --no-keep-memory > > --reduce-memory-overheads, and that was still not enough for 32-bts > > builds. GNU gold instead of BFD ld was also given a shot. That didn't > > work either. Presently, to make things link at all on 32-bits platforms, > > debug info is entirely disabled. I still need to figure out what minimal > > debug info can be enabled without incurring too much memory usage > > during linking. > > > Dang. Yes, removing debug symbols was the only way I could get webkit to > link without thrashing, it's a temporary fix though. > > So the removal of the algorithm in ld Dr Stallman wrote, dating back to the > 1990s, has already resulted in a situation that's worse than I feared. > > At some point apps are going to become so insanely large that not even > disabling debug info will help. That's less likely, I'd say. Debug info *is* getting incredibly more and more complex for the same amount of executable weight, and linking that is making things worse and worse. But having enough code to actually be a problem without debug info is probably not so close. There are solutions to still keep full debug info, but the Debian packaging side doesn't support that presently: using split-dwarf. It would probably be worth investing in supporting that. Mike
Re: Rebuilding the entire Debian archive twice on arm64 hardware for fun and proft
On Mon, Jan 07, 2019 at 10:28:31AM +, Luke Kenneth Casson Leighton wrote: > On Sun, Jan 6, 2019 at 11:46 PM Steve McIntyre wrote: > > > > [ Please note the cross-post and respect the Reply-To... ] > > > > Hi folks, > > > > This has taken a while in coming, for which I apologise. There's a lot > > of work involved in rebuilding the whole Debian archive, and many many > > hours spent analysing the results. You learn quite a lot, too! :-) > > > > I promised way back before DC18 that I'd publish the results of the > > rebuilds that I'd just started. Here they are, after a few false > > starts. I've been rebuilding the archive *specifically* to check if we > > would have any problems building our 32-bit Arm ports (armel and > > armhf) using 64-bit arm64 hardware. I might have found other issues > > too, but that was my goal. > > very cool. > > steve, this is probably as good a time as any to mention a very > specific issue with binutils (ld) that has been slowly and inexorably > creeping up on *all* distros - both 64 and 32 bit - where the 32-bit > arches are beginning to hit the issue first. > > it's a 4GB variant of the "640k should be enough for anyone" problem, > as applied to linking. > > i spoke with dr stallman a couple of weeks ago and confirmed that in > the original version of ld that he wrote, he very very specifically > made sure that it ONLY allocated memory up to the maximum *physical* > resident available amount (i.e. only went into swap as an absolute > last resort), and secondly that the number of object files loaded into > memory was kept, again, to the minimum that the amount of spare > resident RAM could handle. > > some... less-experienced people, somewhere in the late 1990s, ripped > all of that code out [what's all this crap, why are we not just > relying on swap, 4GB swap will surely be enough for anybody"] > > by 2008 i experienced a complete melt-down on a 2GB system when > compiling webkit. i tracked it down to having accidentally enabled > "-g -g -g" in the Makefile, which i had done specifically for one > file, forgot about it, and accidentally recompiled everything. > > that resulted in an absolute thrashing meltdown that nearly took out > the entire laptop. > > the problem is that the linker phase in any application is so heavy > on cross-references that the moment the memory allocated by the linker > goes outside of the boundary of the available resident RAM it is > ABSOLUTELY GUARANTEED to go into permanent sustained thrashing. > > i cannot emphasise enough how absolutely critical that this is to > EVERY distribution to get this fixed. > > resources world-wide are being completely wasted (power, time, and the > destruction of HDDs and SSDs) because systems which should only really > take an hour to do a link are instead often taking FIFTY times longer > due to swap thrashing. > > not only that, but the poor design of ld is beginning to stop certain > packages from even *linking* on 32-bit systems! firefox i heard now > requires SEVEN GIGABYTES during the linker phase! > > and it's down to this very short-sighted decision to remove code > written by dr stallman, back in the late 1990s. > > it would be extremely useful to confirm that 32-bit builds can in fact > be completed, simply by adding "-Wl no-keep-memory" to any 32-bit > builds that are failing at the linker phase due to lack of memory. Note that Firefox is built with --no-keep-memory --reduce-memory-overheads, and that was still not enough for 32-bts builds. GNU gold instead of BFD ld was also given a shot. That didn't work either. Presently, to make things link at all on 32-bits platforms, debug info is entirely disabled. I still need to figure out what minimal debug info can be enabled without incurring too much memory usage during linking. Mike
Re: armel *and* armhf qualification for Wheezy
On Fri, May 18, 2012 at 05:32:08PM +0100, Steve McIntyre wrote: [ dropped -release, added direct mail to Mike; dunno if you're subscribed to d-arm... ] I'm not. OK, cool. Building with ld and gold on a panda right now, to see how they compare. Something I didn't expect - it seems the toolchain in sid armhf is already configured to use gold by default using symlinks. I've done a gold build, now running with ld.bfd for comparison. Using gold, build time of ~8 hours. Another thing that may help, is to use the stabs format instead of dwarf, if the arm toolchain supports that. If you want to try that, you can add: ac_add_options --enable-debug-symbols=-gstabs to both debian/xulrunner.mozconfig and debian/iceweasel.mozconfig, and change ASFLAGS in debian/rules to -gstabs. (I should probably make that easier) Cheers, Mike -- To UNSUBSCRIBE, email to debian-arm-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120518164308.ga16...@glandium.org
Re: armel *and* armhf qualification for Wheezy
Hi Steve, On Wed, May 16, 2012 at 04:26:10PM +0100, Steve McIntyre wrote: In terms of raw buildd CPU right now, I think we're doing OK, but memory is more of a limiting factor with bigger C++ builds. As maintainer of such a package that pushes buildds limits, I have a question. Isn't memory really only a problem when linking C++ with big DWARF info? Would it be worth trying to link with gold for these? Mike -- To UNSUBSCRIBE, email to debian-arm-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120516154410.ga15...@glandium.org
Re: armel *and* armhf qualification for Wheezy
On Wed, May 16, 2012 at 04:56:39PM +0100, Steve McIntyre wrote: On Wed, May 16, 2012 at 05:44:10PM +0200, Mike Hommey wrote: Hi Steve, Hey Mike, On Wed, May 16, 2012 at 04:26:10PM +0100, Steve McIntyre wrote: In terms of raw buildd CPU right now, I think we're doing OK, but memory is more of a limiting factor with bigger C++ builds. As maintainer of such a package that pushes buildds limits, I have a question. Isn't memory really only a problem when linking C++ with big DWARF info? Honestly, I'm not 100% sure where all the memory is going. I do know that at current rates of usage increase we'll struggle to link some large programs (like browsers) on any 32-bit platform soon. Would it be worth trying to link with gold for these? It might be, yes. I can try that with iceweasel on an imx53 or Panda with 1GB if you like. Are there any non-obvious patches needed to the packaging? Apart from whatever is needed for gcc to use gold, there shouldn't be. Mike -- To UNSUBSCRIBE, email to debian-arm-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120516191838.ga17...@glandium.org
Re: Bug#648233: SIGILL in rsvg-convert on armel
reassign 648233 release.debian.org thanks nmu libxau_1:1.0.6-4 . armel . -m Rebuild for armv4t nmu libxcb_1.7-4 . armel . -m Rebuild for armv4t On Wed, Nov 09, 2011 at 09:15:13PM +0100, Mike Hommey wrote: Package: librsvg2-bin Justification: renders package unusable Severity: grave Iceweasel 8.0-3 failed to build on armel: On Wed, Nov 09, 2011 at 06:45:04PM +, Debian buildds wrote: * Source package: iceweasel * Version: 8.0-3 * Architecture: armel * State: failed * Suite: sid * Builder: alwyn.debian.org * Build log: https://buildd.debian.org/fetch.cgi?pkg=iceweaselarch=armelver=8.0-3stamp=1320864081file=log Please note that these notifications do not necessarily mean bug reports in your package but could also be caused by other packages, temporary uninstallabilities and arch-specific breakages. A look at the build log despite this disclaimer would be appreciated however. The build failure looks like this: rsvg-convert -w 16 -h 16 -o default16.png ../../../debian/branding/iceweasel_icon.svg make[5]: *** [default16.png] Illegal instruction Which suggests rsvg-convert or one of the libs it uses is compiled for armv5t, making it not work on armv5t (what our buildds actually are) or armv4t (what we actually target) So, it turns out 8.0-1 has been built with the same version of librsvg2-bin on a different buildd, which is supposedly the same hardware as the other failing ones. So the difference in installed packages versions are: ok fail man-db 2.6.0.2-2 2.6.0.2-3 libxau-dev 1:1.0.6-3 1:1.0.6-4 libxau6 1:1.0.6-3 1:1.0.6-4 libxcb-render0 1.7-3 1.7-4 libxcb-render0-dev 1.7-3 1.7-4 libxcb-shape0 1.7-3 1.7-4 libxcb-shm0 1.7-3 1.7-4 libxcb-shm0-dev 1.7-3 1.7-4 I doubt man-db is involved, here, so i checked two almost random packages in this list: libxcb-shape0 and libxau6. Bingo: On the files in version 1.7-4 and 1:1.0.6-4 (resp.): Attribute Section: aeabi File Attributes Tag_CPU_name: 7-A Tag_CPU_arch: v7 Tag_CPU_arch_profile: Application Tag_ARM_ISA_use: Yes Tag_THUMB_ISA_use: Thumb-2 Tag_VFP_arch: VFPv3-D16 Tag_ABI_PCS_wchar_t: 4 Tag_ABI_FP_denormal: Needed Tag_ABI_FP_exceptions: Needed Tag_ABI_FP_number_model: IEEE 754 Tag_ABI_align8_needed: Yes Tag_ABI_align8_preserved: Yes, except leaf SP Tag_ABI_enum_size: int Tag_ABI_HardFP_use: SP and DP Tag_ABI_optimization_goals: Aggressive Speed Tag_DIV_use: Not allowed On the files in version 1.7-3 and 1:1.0.6-3 (resp.): Attribute Section: aeabi File Attributes Tag_CPU_name: ARM9TDMI Tag_CPU_arch: v4T Tag_ARM_ISA_use: Yes Tag_THUMB_ISA_use: Thumb-1 Tag_ABI_PCS_wchar_t: 4 Tag_ABI_FP_denormal: Needed Tag_ABI_FP_exceptions: Needed Tag_ABI_FP_number_model: IEEE 754 Tag_ABI_align8_needed: Yes Tag_ABI_align8_preserved: Yes, except leaf SP Tag_ABI_enum_size: int Tag_ABI_optimization_goals: Aggressive Speed Tag_DIV_use: Not allowed Both packages in their new version were *not* autobuilt, while the old one was. Please double check the armel packages you upload. Cheers, Mike -- To UNSUBSCRIBE, email to debian-arm-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/2009233009.ga10...@glandium.org
Re: GCC 4.2 transition
On Fri, Jul 20, 2007 at 11:33:01AM +0200, Johannes Berg [EMAIL PROTECTED] wrote: On Fri, 2007-07-20 at 10:16 +0200, Matthias Klose wrote: Does any port still need to stick with GCC 4.1 for a while? Feedback from hppa, mips*, s390, powerpc, amd64, i386 porters doesn't show objections against the transition. I have objections :) http://bugs.debian.org/433629 Yes, it's pretty odd, but recompiling the whole kernel tree with gcc 4.2 causes my usbhid to totally not work. I have another objection. I'd like all mozilla security updates to be built before gcc 4.2 becomes the default, because they don't build correctly yet, and I am (still) waiting for an upstream comment on how to fix it. Mike -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: arm release issues
On Mon, Aug 28, 2006 at 01:01:00PM -0700, Steve Langasek [EMAIL PROTECTED] wrote: Yes, it can be and is configured by source package. If you can get some numbers to the buildd admin (James) about how long it should take for these packages to build, it should be straightforward to fix up that config AFAIK. Unfortunately, not being on owner of an arm, I don't have any numbers. Does anyone know what happened to make ld so much slower on arm all of a sudden? Probably the switch to providing debugging symbols, but i'm somehow surprised the same doesn't happen seem to happen with firefox, though the build is not done the exact same way. The main big part of firefox is built as a static binary while in xulrunner it is a big dynamic library. Maybe that makes some kind of difference. Anyways, on my x86, it already takes quite some time to build this library, though again, I don't have timings. But it doesn't take more than a few minutes. The main thing is that it takes a huge amount of memory. It takes more than 500MB on my build machine. Which means on slow machines, with few amounts of memory (arm buildd have 64MB, i think), it can take long time, and depending on how ld plays with its memory, it can be even worse. Mike -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Please hint xulrunner
On Sun, Jul 02, 2006 at 11:27:19AM +0900, Kenshi Muto [EMAIL PROTECTED] wrote: Hi, At Sat, 1 Jul 2006 13:27:56 +0200, Mike Hommey wrote: On Sat, Jul 01, 2006 at 12:55:30PM +0200, Andreas Barth [EMAIL PROTECTED] wrote: * Mike Hommey ([EMAIL PROTECTED]) [060701 12:06]: Important security fixes for testing are waiting xulrunner to be built on arm, which seem to be not likely to happen soonish :( Could you hint xulrunner in, despite arm not being built yet ? Hm, I'm not so happy with that for obvious reasons. Did you contact debian-arm why the build failed? Arm porters, any hints for http://buildd.debian.org/fetch.php?pkg=xulrunnerver=1.8.0.4-1arch=armstamp=1150438876file=logas=raw ? How good are our chances to get this issue resolved soon? I contacted the buildd admin first, without any answer. Then asked for a porter build on debian-arm without much more success, though at least, the message got answers. Note that it's not the first time xulrunner fails in such a manner. It's just that the buildd is too slow and doesn't wait long enough for the link to finish. I've put the job to my ARM unstable (primary for non-free) buildd yesterday. Because my sbuild configuration is $staled_pkg_timeout=10080(min.) and it has 256MB memory, I hope this build will succeed. Thanks, Thanks a lot, Muto-san. Mike -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Please hint xulrunner
On Sat, Jul 01, 2006 at 12:55:30PM +0200, Andreas Barth [EMAIL PROTECTED] wrote: * Mike Hommey ([EMAIL PROTECTED]) [060701 12:06]: Important security fixes for testing are waiting xulrunner to be built on arm, which seem to be not likely to happen soonish :( Could you hint xulrunner in, despite arm not being built yet ? Hm, I'm not so happy with that for obvious reasons. Did you contact debian-arm why the build failed? Arm porters, any hints for http://buildd.debian.org/fetch.php?pkg=xulrunnerver=1.8.0.4-1arch=armstamp=1150438876file=logas=raw ? How good are our chances to get this issue resolved soon? I contacted the buildd admin first, without any answer. Then asked for a porter build on debian-arm without much more success, though at least, the message got answers. Note that it's not the first time xulrunner fails in such a manner. It's just that the buildd is too slow and doesn't wait long enough for the link to finish. Mike -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Please hint xulrunner
On Sat, Jul 01, 2006 at 01:27:56PM +0200, Mike Hommey [EMAIL PROTECTED] wrote: On Sat, Jul 01, 2006 at 12:55:30PM +0200, Andreas Barth [EMAIL PROTECTED] wrote: * Mike Hommey ([EMAIL PROTECTED]) [060701 12:06]: Important security fixes for testing are waiting xulrunner to be built on arm, which seem to be not likely to happen soonish :( Could you hint xulrunner in, despite arm not being built yet ? Hm, I'm not so happy with that for obvious reasons. Did you contact debian-arm why the build failed? Arm porters, any hints for http://buildd.debian.org/fetch.php?pkg=xulrunnerver=1.8.0.4-1arch=armstamp=1150438876file=logas=raw ? How good are our chances to get this issue resolved soon? I contacted the buildd admin first, without any answer. Then asked for a porter build on debian-arm without much more success, though at least, the message got answers. Note that it's not the first time xulrunner fails in such a manner. It's just that the buildd is too slow and doesn't wait long enough for the link to finish. After thinking about it, i wonder if i've not seen the timeout thing on another package... Anyways, I'm not surprised building a huge .so file on a buildd with 60MB RAM (iirc) is pretty slow. It takes 500MB+ on my computer when I build it... Mike -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Please hint xulrunner
On Sat, Jul 01, 2006 at 09:02:53AM -0500, Bill Gatliff [EMAIL PROTECTED] wrote: I have physical ownership of leisner.debian.org (armel), though I'm not a DD and not the buildd admin. Is there something I can do to restart the build? Top says that leisner is idling. As a DD, I have access to leisner. It's not a buildd, though. I was more hoping for a reschedule on an arm buildd or a build on something faster than leisner, but since it doesn't seem to happen, i'll just contact the debian-admins to get the necessary build deps installed on leisner and use the old DIY method. Cheers Mike -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Need build for xulrunner
Hi all, xulrunner 1.8.0.4-1 is in need for a build on arm. The buildd failed it because it took too much time, and elmo seems to be away. This release contains security fixes that need to propagate to testing. Would someone be kind to build and upload an arm binary ? (Be aware that building xulrunner requires disk space, memory and time) Thanks, Mike -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
