Re: PowerPC daily install CDs? [Was: Re: Netinst for testing?]
Ian Campbell wrote: Is there a way to contact the build machine maintainers? Other than hoping they read this thread? The overview shows who they are. Ping on IRC, private mail, ... Note that the powerpc issue is not for the buildd admin (although it's possible that needs updating too). -- To UNSUBSCRIBE, email to debian-boot-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: PowerPC daily install CDs? [Was: Re: Netinst for testing?]
On Sun, 2009-10-25 at 15:00 +0100, Frans Pop wrote: On Sunday 25 October 2009, Ian Campbell wrote: I was unable to reproduce in an uptodate i386 sid chroot. I'm not sure which environment these builds occur, maybe it's a testing thing? No, daily images get build in a sid environment. Thanks. I still cannot reproduce so I am somewhat clutching at straws. I compared the logs for a local (successful) i386 run of the daily-build script vs. the logs from http://people.debian.org/~joeyh/d-i/build-logs.html I've been concentrating on the netboot-gtk log and the differences are really rather small up until the failure point. The output from apt and/or dpkg differs a little -- I wonder if the build environment is not completely up to date and has a buggy (or just slightly differing in behaviour) version of one or the other? Seems unlikely to have effected so many build servers though (seemingly admin'd by several different people). FWIW I'm running apt 0.7.24 and dpkg 1.15.4.1. The Object: and Objects: lines printed by library reduction pass 1 seem to be the same in both the failed and local successful builds, although the ordering is different. The symbol _nss_files_parse_sgent is provided by /lib/libc.so.6 and required by /lib/libnss_files.so.2. Both libraries are part of libc. Comparing a manual make build_netboot-gtk with the daily-build logs (either my local ones or the build server ones), I see this which seems odd: cp -a `find ./tmp/netboot-gtk/tree/extraudebs-tmp/lib -name '*.so.*'` ./tmp/netboot-gtk/udeblibs +find: `./tmp/netboot-gtk/tree/extraudebs-tmp/lib': No such file or directory +cp: missing destination file operand after `./tmp/netboot-gtk/udeblibs' +Try `cp --help' for more information. +make[2]: [stamps/tree-netboot-gtk-stamp] Error 1 (ignored) Immediately before this the extraudebs-tmp directory is created by installing a bunch of stuff into it: dpkg --force-overwrite --log=/dev/null --root=./tmp/netboot-gtk/tree/extraudebs-tmp --unpack \ udebs/cdebconf-newt-entropy.udeb udebs/cdebconf-text-entropy.udeb udebs/cdebconf-gtk-entropy.udeb Selecting previously deselected package cdebconf-newt-entropy. (Reading database ... 0 files and directories currently installed.) Unpacking cdebconf-newt-entropy (from .../cdebconf-newt-entropy.udeb) ... Selecting previously deselected package cdebconf-text-entropy. Unpacking cdebconf-text-entropy (from .../cdebconf-text-entropy.udeb) ... Selecting previously deselected package cdebconf-gtk-entropy. Unpacking cdebconf-gtk-entropy (from .../cdebconf-gtk-entropy.udeb) ... which does sort of suggest a dpkg issue, or perhaps corrupted udebs? Sorry that I can't provide any more concrete ideas what might be going on. Perhaps it would be useful if mklibs printed a bit more detail about where and why a symbol is thought to be required if an error occurs? The current version of http://people.debian.org/~joeyh/d-i/build-logs.html seems to suggest even bigger issues though, every arch failed to download summary log Ian. -- Ian Campbell You'll wish that you had done some of the hard things when they were easier to do. signature.asc Description: This is a digitally signed message part
Re: PowerPC daily install CDs? [Was: Re: Netinst for testing?]
Thanks for making an effort to look into this. On Monday 26 October 2009, Ian Campbell wrote: Thanks. I still cannot reproduce so I am somewhat clutching at straws. I compared the logs for a local (successful) i386 run of the daily-build script vs. the logs from http://people.debian.org/~joeyh/d-i/build-logs.html In the past I've always been able to reproduce mklibs errors. I've been concentrating on the netboot-gtk log and the differences are really rather small up until the failure point. The output from apt and/or dpkg differs a little -- I wonder if the build environment is not completely up to date and has a buggy (or just slightly differing in behaviour) version of one or the other? Seems unlikely to have effected so many build servers though (seemingly admin'd by several different people). FWIW I'm running apt 0.7.24 and dpkg 1.15.4.1. If you cannot reproduce it with an up-to-date sid build environment then it's quite possible that the buildds are simply outdated. apt and dpkg are not really relevant here. More relevant are versions of mklibs itself and packages like binutils. The symbol _nss_files_parse_sgent is provided by /lib/libc.so.6 and required by /lib/libnss_files.so.2. Both libraries are part of libc. Does libc.so.6 provide the exact symbol looked for (I'd expect it does)? In that case this looks like an mklibs/binutils issue. Comparing a manual make build_netboot-gtk with the daily-build logs (either my local ones or the build server ones), I see this which seems odd: [...] That is indeed strange and should be looked into, but I doubt it is related to the mklibs failure as neither libc6 nor libnss-files are extraudebs. Perhaps it would be useful if mklibs printed a bit more detail about where and why a symbol is thought to be required if an error occurs? It can do. The usual way to do that is this change in build/config/common to make it more verbose: -MKLIBS = mklibs +MKLIBS = mklibs -v -v But that is only really useful if you can reproduce it locally. The current version of http://people.debian.org/~joeyh/d-i/build-logs.html seems to suggest even bigger issues though, every arch failed to download summary log Probably just a temporary network problem somewhere. I'll see if I can do a test build myself today or tomorrow. Cheers, FJP -- To UNSUBSCRIBE, email to debian-boot-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: PowerPC daily install CDs? [Was: Re: Netinst for testing?]
On Monday 26 October 2009, Frans Pop wrote: If you cannot reproduce it with an up-to-date sid build environment then it's quite possible that the buildds are simply outdated. apt and dpkg are not really relevant here. More relevant are versions of mklibs itself and packages like binutils. There was an upload of binutils a bit more than a week ago. You could check if you can reproduce the failure after downgrading to binutils from testing. -- To UNSUBSCRIBE, email to debian-boot-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: PowerPC daily install CDs? [Was: Re: Netinst for testing?]
On Mon, 2009-10-26 at 15:55 +0100, Frans Pop wrote: Thanks for making an effort to look into this. On Monday 26 October 2009, Ian Campbell wrote: Thanks. I still cannot reproduce so I am somewhat clutching at straws. I compared the logs for a local (successful) i386 run of the daily-build script vs. the logs from http://people.debian.org/~joeyh/d-i/build-logs.html In the past I've always been able to reproduce mklibs errors. It does seem like the sort of thing which ought to fail fairly reliably if it fails. I've been concentrating on the netboot-gtk log and the differences are really rather small up until the failure point. The output from apt and/or dpkg differs a little -- I wonder if the build environment is not completely up to date and has a buggy (or just slightly differing in behaviour) version of one or the other? Seems unlikely to have effected so many build servers though (seemingly admin'd by several different people). FWIW I'm running apt 0.7.24 and dpkg 1.15.4.1. If you cannot reproduce it with an up-to-date sid build environment then it's quite possible that the buildds are simply outdated. apt and dpkg are not really relevant here. More relevant are versions of mklibs itself and packages like binutils. I tried downgrading binutils, as you suggested in your next mail, to 2.19.91.20091006-1 from testing and still saw no issues. The issue seems to have started between http://people.debian.org/~joeyh/d-i/images/20090930-19:15 and http://people.debian.org/~joeyh/d-i/images/20091024-21:59/ which is unfortunately a rather large range. Apart from the version currently in testing binutils seems to have had one other version uploaded in that period (2.19.91.20091003-1) but in the absence of snapshot.debian.net I don't think I can get hold of it. mklibs doesn't seem to have changed since early August which seems to rule it out. The symbol _nss_files_parse_sgent is provided by /lib/libc.so.6 and required by /lib/libnss_files.so.2. Both libraries are part of libc. Does libc.so.6 provide the exact symbol looked for (I'd expect it does)? In that case this looks like an mklibs/binutils issue. I believe it does: $ objdump -T tmp/netboot-gtk/tree/lib/libc.so.6 | grep nss_files_parse 00056a30 gDF .text 02ed GLIBC_PRIVATE _nss_files_parse_pwent 0007c880 gDF .text 03cf GLIBC_PRIVATE _nss_files_parse_sgent 00055900 gDF .text 030e GLIBC_PRIVATE _nss_files_parse_grent 0007c3f0 gDF .text 03d3 GLIBC_PRIVATE _nss_files_parse_spent Comparing a manual make build_netboot-gtk with the daily-build logs (either my local ones or the build server ones), I see this which seems odd: [...] That is indeed strange and should be looked into, but I doubt it is related to the mklibs failure as neither libc6 nor libnss-files are extraudebs. Originally I wondered if one of the extra udebs was introducing an extra dependency on the symbol which I wasn't finding because the extraudebs dir is deleted after it is used. This turned out not to be the case though. The current version of http://people.debian.org/~joeyh/d-i/build-logs.html seems to suggest even bigger issues though, every arch failed to download summary log Probably just a temporary network problem somewhere. Indeed, it seems to have resolved itself now. Thanks, Ian. -- Ian Campbell 'Scuse me, while I kiss the sky! -- Robert James Marshall (Jimi) Hendrix signature.asc Description: This is a digitally signed message part
Re: PowerPC daily install CDs? [Was: Re: Netinst for testing?]
On Monday 26 October 2009, Ian Campbell wrote: I tried downgrading binutils, as you suggested in your next mail, to 2.19.91.20091006-1 from testing and still saw no issues. Hmmm. Another option is that it's due to a libc version skew. During a build libc6 is copied from the host system (because we need the corresponding -pic), while libnss-* is unpacked from udebs. So it could be that the buildds have a version of libc which really doesn't provide the symbol. /me tests this theory by firing up his armel box, which promptly reproduces the error After updating the system the problem disappears. So, that confirms the reason is outdated buildd environments after an API change in libc (2.9 - 2.10). The first 2.10 upload was 18 Okt, so the problem should be only about a week old. Cheers, FJP -- To UNSUBSCRIBE, email to debian-boot-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: PowerPC daily install CDs? [Was: Re: Netinst for testing?]
On Monday 26 October 2009, Frans Pop wrote: After updating the system the problem disappears. So, that confirms the reason is outdated buildd environments after an API change in libc (2.9 - 2.10). The first 2.10 upload was 18 Okt, so the problem should be only about a week old. s/API/ABI/ probably. -- To UNSUBSCRIBE, email to debian-boot-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: PowerPC daily install CDs? [Was: Re: Netinst for testing?]
On Mon, 2009-10-26 at 18:14 +0100, Frans Pop wrote: On Monday 26 October 2009, Ian Campbell wrote: I tried downgrading binutils, as you suggested in your next mail, to 2.19.91.20091006-1 from testing and still saw no issues. Hmmm. Another option is that it's due to a libc version skew. During a build libc6 is copied from the host system (because we need the corresponding -pic), while libnss-* is unpacked from udebs. So it could be that the buildds have a version of libc which really doesn't provide the symbol. /me tests this theory by firing up his armel box, which promptly reproduces the error After updating the system the problem disappears. So, that confirms the reason is outdated buildd environments after an API change in libc (2.9 - 2.10). Nice one, thanks! I downgraded my chroot libc to 2.9-25 and reproduced the issue also. That libc has no _nss_files_parse_sgent, as expected: $ objdump -T /lib/libc.so.6 | grep nss_files_parse 00096c60 gDF .text 02d0 GLIBC_PRIVATE _nss_files_parse_pwent 00095810 gDF .text 02f5 GLIBC_PRIVATE _nss_files_parse_grent 000d1c60 gDF .text 03d2 GLIBC_PRIVATE _nss_files_parse_spent The first 2.10 upload was 18 Okt, so the problem should be only about a week old. Is there a way to contact the build machine maintainers? Other than hoping they read this thread? Ian. -- Ian Campbell Quod erat demonstrandum. [Thus it is proven. For those who wondered WTF QED means.] signature.asc Description: This is a digitally signed message part
Re: PowerPC daily install CDs? [Was: Re: Netinst for testing?]
On Sun, 2009-10-25 at 06:32 +0100, Frans Pop wrote: Looks to me that the general state of builds is rather pathetic ATM: Sorry. I should have just written Looks like there are quite a few problems with builds ATM. Does not change the facts or the likelyhood of a successful upload/release any time soon though. I noticed the i386 failures yesterday. I've no idea when they started, seems like quite a while ago. Would it be worth having a notification of build failures sent to the list? (or even just an occasional summary) i386 is failing with: 3366 symbols, 654 unresolved Traceback (most recent call last): File /usr/bin/mklibs, line 538, in module raise No library provides non-weak %s % name No library provides non-weak _nss_files_parse_sg...@glibc_private@libc.so.6 make[2]: *** [stamps/tree-cdrom_gtk-stamp] Error 1 make[1]: *** [_build] Error 2 make: *** [build_cdrom_gtk] Error 2 I only checked the first failure for each arch but this same failure seems to have effected mips (build_r4k-ip22_cdrom), s390 (build_generic) and armel (uild_iop32x_netboot_glantank). The symbol in the mips case is __gnu_local_gp while the others are _nss_files_parse_sg...@glibc_private@libc. I was unable to reproduce in an uptodate i386 sid chroot. I'm not sure which environment these builds occur, maybe it's a testing thing? I'm installing a squeeze VM now to check. powerpc is failing differently with E: Couldn't find package firewire-core-modules-2.6.30-1-powerpc-di or E: Couldn't find package floppy-modules-2.6.30-1-powerpc64-di depending on the flavour. Ian. -- Ian Campbell Q: Why did the astrophysicist order three hamburgers? A: Because he was hungry. signature.asc Description: This is a digitally signed message part
Re: PowerPC daily install CDs? [Was: Re: Netinst for testing?]
On Sunday 25 October 2009, Ian Campbell wrote: I was unable to reproduce in an uptodate i386 sid chroot. I'm not sure which environment these builds occur, maybe it's a testing thing? No, daily images get build in a sid environment. powerpc is failing differently with E: Couldn't find package firewire-core-modules-2.6.30-1-powerpc-di or E: Couldn't find package floppy-modules-2.6.30-1-powerpc64-di depending on the flavour. Looks like someone forgot to update the kernel ABI. -- To UNSUBSCRIBE, email to debian-boot-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: PowerPC daily install CDs? [Was: Re: Netinst for testing?]
On Sunday 25 October 2009, Ian Campbell wrote: I only checked the first failure for each arch but this same failure seems to have effected mips (build_r4k-ip22_cdrom), s390 (build_generic) and armel (uild_iop32x_netboot_glantank). For mips and mipsel there's a much bigger problem: there have not been any builds at all since early July. It looks as if some of the central D-I buildds are simply not being managed. -- To UNSUBSCRIBE, email to debian-boot-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: PowerPC daily install CDs? [Was: Re: Netinst for testing?]
Frans Pop wrote: On Sunday 25 October 2009, Ian Campbell wrote: I only checked the first failure for each arch but this same failure seems to have effected mips (build_r4k-ip22_cdrom), s390 (build_generic) and armel (uild_iop32x_netboot_glantank). For mips and mipsel there's a much bigger problem: there have not been any builds at all since early July. It looks as if some of the central D-I buildds are simply not being managed. Wrong, ssh is/was broken on mips* so the results did not get uploaded, but the builds were not interrupted. Cheers Luk -- To UNSUBSCRIBE, email to debian-boot-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: PowerPC daily install CDs? [Was: Re: Netinst for testing?]
On Sunday 25 October 2009, Luk Claes wrote: Frans Pop wrote: For mips and mipsel there's a much bigger problem: there have not been any builds at all since early July. It looks as if some of the central D-I buildds are simply not being managed. Wrong, ssh is/was broken on mips* so the results did not get uploaded, but the builds were not interrupted. Would have been nice to see something like that mentioned on this list. Avoids misunderstandings. -- To UNSUBSCRIBE, email to debian-boot-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: PowerPC daily install CDs? [Was: Re: Netinst for testing?]
On Sunday 25 October 2009, Frans Pop wrote: On Sunday 25 October 2009, Luk Claes wrote: Frans Pop wrote: For mips and mipsel there's a much bigger problem: there have not been any builds at all since early July. It looks as if some of the central D-I buildds are simply not being managed. Wrong, ssh is/was broken on mips* so the results did not get uploaded, but the builds were not interrupted. Would have been nice to see something like that mentioned on this list. Avoids misunderstandings. And documented on http://wiki.debian.org/DebianInstaller/Today for users (and team members). -- To UNSUBSCRIBE, email to debian-boot-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: PowerPC daily install CDs? [Was: Re: Netinst for testing?]
On Sunday 25 October 2009, Luk Claes wrote: Wrong, ssh is/was broken on mips* so the results did not get uploaded, but the builds were not interrupted. I guess you're referring to #538313? That bug was fixed on Okt 5, but the current situation, 20 days later, is that we still don't have daily builds. From your use of the past tense I suspect you'll probably have updated the chroots now, after my message? Thanks. -- To UNSUBSCRIBE, email to debian-boot-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: PowerPC daily install CDs? [Was: Re: Netinst for testing?]
On Sunday 25 October 2009, Rick Thomas wrote: On Oct 24, 2009, at 10:03 PM, Frans Pop wrote: Links to current images are available from: http://www.debian.org/devel/debian-installer/ Cheers, FJP Hmmm... If I follow that link, then click on • netinst ... and businesscard ... CD images ... [powerpc] I get taken to a directory that claims This build finished at Thu Oct 1 23:28:01 UTC 2009. Is it possible that PowerPC CD builds have been down for over three weeks and nobody noticed? That's very likely the case. Looks to me that the general state of builds is rather pathetic ATM: http://people.debian.org/~joeyh/d-i/build-logs.html Hard to see how people expect to be able to do a D-I release (as mentioned in the logs from the last team meeting) given that fact (amongst others). -- To UNSUBSCRIBE, email to debian-boot-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: PowerPC daily install CDs? [Was: Re: Netinst for testing?]
Looks to me that the general state of builds is rather pathetic ATM: Sorry. I should have just written Looks like there are quite a few problems with builds ATM. Does not change the facts or the likelyhood of a successful upload/release any time soon though. -- To UNSUBSCRIBE, email to debian-boot-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
