Bug#779616: debian-cd: broken debian-testing-$arch-netinst.iso generation?
On Thu, Mar 05, 2015 at 12:25:50PM +0100, Cyril Brulebois wrote: Steve McIntyre st...@einval.com (2015-03-05): We've had this discussion befire, surely? On pettersson we rsync things (over ssh) directly from d-i.d.o (dillon), so we don't depend on http or git. Or am I missing something new? (I can't see #746967 atm...) There's also an explicit warning in debian-cd if you grab things via plain http or ftp: echo WARNING WARNING WARNING WARNING WARNING WARNING echo $0: insecure download for d-i build: $DI_WWW_HOME echo WARNING WARNING WARNING WARNING WARNING WARNING The rsync part covers the former, but there's still a broken trust chain because of the latter. Right, I've just had a chance to look at that now. Ugh... :-( OK. Assuming amd64 is working OK, I should check the rsync has not broken - that seems to be the most likely cause right now. Thinking: it would also be lovely to verify the versions of vmlinuz and the kernel udebs in debian-cd at build time, and (optionally?) abort the build if we know we're going to be making broken images. What do you think? There's no way for us to know a breakage is going to happen. In this particular case the ABI is the same, say 3.16.0-4-amd64; the ABI is embedded in kernel udeb package names, so if the ABI matches, udebs are supposed to be compatible with the kernel. I'm not sure why that is not the case here (maybe the ABI doesn't cover or not completely udebs?). We really shouldn't enforce using the very same revision for kernel and module udebs, IMHO. ACK, I see your logic. In the past, we had the two different sets of daily images: * use the most recent testing version of d-i in tha archive * use the daily d-i builds from d-i.d.o (and other hosts before that) and then we'd switch the weekly images from one source to the other depending on how working/broken we thought each source was likely to be. Since the kernel team got much more active a few years back and kernel ABI changes have been much more common during the life of testing, using the most recent d-i release from the archive has had a much shorter working life than we used to have. Hence, we switched over to using the daily d-i builds as a base. This can be YA argument for more regular d-i uploads to the archive to fix that problem, of course. But we know how much work that involves. At the moment I feel what we have is just about the best thing we can get without that massive additional effort. It normally works for people, modulo the odd breakage from time to time like we've seen here. I guess we could stick to this once the trust issue is resolved. Daily builds will have to be changed one way or another anyway since it can't be implemented as done previously starting with jessie hosts… Right, yes. Anyone have any bright ideas yet on how to do that? -- Steve McIntyre, Cambridge, UK.st...@einval.com There's no sensation to compare with this Suspended animation, A state of bliss -- To UNSUBSCRIBE, email to debian-cd-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20150308134848.go6...@einval.com
Bug#779616: debian-cd: broken debian-testing-$arch-netinst.iso generation?
On Fri, Mar 06, 2015 at 06:51:46PM +0100, Cyril Brulebois wrote: Ian Campbell i...@debian.org (2015-03-05): I think the ABI only guarantees you can load old modules into a newer vmlinuz. In particular I think it doesn't guarantee that you can load newer modules into an older vmlinuz (even if the ABI is the same). The ABI is there to stop your existing modules from breaking when you update the kernel. IOW it is permissible for a module to gain a dependency on a new symbol provided by a newer vmlinuz. Right, that helps explain. Thanks Ian! This is why local netboot pxe infra sometimes breaks -- they have a vmlinuz downloaded locally but are pulling udebs off the network, which might therefore be newer. Please reread my caveat now though ;-) Why do I forget about this all the time? :( Thinking a bit more about this: I think having weekly testing with everything from testing makes the most sense. OK... Are we going to do more frequent d-i uploads to make sure that works, though? It looks like the changes for (daily) builds are going to force quite a lot of work on us here. I'm quite prepared to help with that when I'm back (and have more time!) in a week or so... We could have a separate, not-labelled-testing but rather “daily” or bleeding-edge or something in that spirit (but neither experimental nor sid) that picks d-i on d-i.d.o and udebs from sid. I'm not sure it would be reasonable to expect such a build on a daily basis (maybe by reducing the generated images to the bare minimum?), but then we would have a clear winner as far as the name goes… Right. -- Steve McIntyre, Cambridge, UK.st...@einval.com We don't need no education. We don't need no thought control. -- To UNSUBSCRIBE, email to debian-cd-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20150308135605.gp6...@einval.com
Bug#779616: debian-cd: broken debian-testing-$arch-netinst.iso generation?
Steve McIntyre st...@einval.com (2015-03-08): On Fri, Mar 06, 2015 at 06:51:46PM +0100, Cyril Brulebois wrote: Thinking a bit more about this: I think having weekly testing with everything from testing makes the most sense. OK... Are we going to do more frequent d-i uploads to make sure that works, though? I can't promise anything. Mraw, KiBi. signature.asc Description: Digital signature
Bug#779616: debian-cd: broken debian-testing-$arch-netinst.iso generation?
Ian Campbell i...@debian.org (2015-03-05): On Thu, 2015-03-05 at 12:25 +0100, Cyril Brulebois wrote: OK. Assuming amd64 is working OK, I should check the rsync has not broken - that seems to be the most likely cause right now. Thinking: it would also be lovely to verify the versions of vmlinuz and the kernel udebs in debian-cd at build time, and (optionally?) abort the build if we know we're going to be making broken images. What do you think? There's no way for us to know a breakage is going to happen. In this particular case the ABI is the same, say 3.16.0-4-amd64; the ABI is embedded in kernel udeb package names, so if the ABI matches, udebs are supposed to be compatible with the kernel. I'm not sure why that is not the case here (maybe the ABI doesn't cover or not completely udebs?). We really shouldn't enforce using the very same revision for kernel and module udebs, IMHO. Caveat: I might be talking out my a**e here... I think the ABI only guarantees you can load old modules into a newer vmlinuz. In particular I think it doesn't guarantee that you can load newer modules into an older vmlinuz (even if the ABI is the same). The ABI is there to stop your existing modules from breaking when you update the kernel. IOW it is permissible for a module to gain a dependency on a new symbol provided by a newer vmlinuz. This is why local netboot pxe infra sometimes breaks -- they have a vmlinuz downloaded locally but are pulling udebs off the network, which might therefore be newer. Please reread my caveat now though ;-) Why do I forget about this all the time? :( Thinking a bit more about this: I think having weekly testing with everything from testing makes the most sense. We could have a separate, not-labelled-testing but rather “daily” or bleeding-edge or something in that spirit (but neither experimental nor sid) that picks d-i on d-i.d.o and udebs from sid. I'm not sure it would be reasonable to expect such a build on a daily basis (maybe by reducing the generated images to the bare minimum?), but then we would have a clear winner as far as the name goes… Mraw, KiBi. signature.asc Description: Digital signature
Bug#779616: debian-cd: broken debian-testing-$arch-netinst.iso generation?
On Thu, 2015-03-05 at 12:25 +0100, Cyril Brulebois wrote: OK. Assuming amd64 is working OK, I should check the rsync has not broken - that seems to be the most likely cause right now. Thinking: it would also be lovely to verify the versions of vmlinuz and the kernel udebs in debian-cd at build time, and (optionally?) abort the build if we know we're going to be making broken images. What do you think? There's no way for us to know a breakage is going to happen. In this particular case the ABI is the same, say 3.16.0-4-amd64; the ABI is embedded in kernel udeb package names, so if the ABI matches, udebs are supposed to be compatible with the kernel. I'm not sure why that is not the case here (maybe the ABI doesn't cover or not completely udebs?). We really shouldn't enforce using the very same revision for kernel and module udebs, IMHO. Caveat: I might be talking out my a**e here... I think the ABI only guarantees you can load old modules into a newer vmlinuz. In particular I think it doesn't guarantee that you can load newer modules into an older vmlinuz (even if the ABI is the same). The ABI is there to stop your existing modules from breaking when you update the kernel. IOW it is permissible for a module to gain a dependency on a new symbol provided by a newer vmlinuz. This is why local netboot pxe infra sometimes breaks -- they have a vmlinuz downloaded locally but are pulling udebs off the network, which might therefore be newer. Please reread my caveat now though ;-) Ian. -- To UNSUBSCRIBE, email to debian-cd-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/1425568787.25940.233.ca...@debian.org
Bug#779616: debian-cd: broken debian-testing-$arch-netinst.iso generation?
Steve McIntyre st...@einval.com (2015-03-05): [ Writing this off-line again while travelling so I can't check things directly on pettersson etc. ] ACK. On Tue, Mar 03, 2015 at 03:47:12PM +0100, Cyril Brulebois wrote: Steve McIntyre st...@einval.com (2015-03-03): Hmmm. That's very odd. The weekly builds are currently set up to use daily d-i builds rather than what's in the archive, and they've been that way for a long time. I don't think that's a good idea. For one thing, there's no trust chain here (from d-i.d.o because http; and from git, because #746967). We've had this discussion befire, surely? On pettersson we rsync things (over ssh) directly from d-i.d.o (dillon), so we don't depend on http or git. Or am I missing something new? (I can't see #746967 atm...) There's also an explicit warning in debian-cd if you grab things via plain http or ftp: echo WARNING WARNING WARNING WARNING WARNING WARNING echo $0: insecure download for d-i build: $DI_WWW_HOME echo WARNING WARNING WARNING WARNING WARNING WARNING The rsync part covers the former, but there's still a broken trust chain because of the latter. We've been using the daily d-i builds for a long time, to guard against exactly this kind of breakage with kernel version mismatches. Has something changed? Some archs are missing daily builds, because autobuilding in jessie is broken, but AFAICT missing builds are only: - arm64 - ppc64el - sparc (See http://d-i.debian.org/daily-images/daily-build-overview.html) OK. Assuming amd64 is working OK, I should check the rsync has not broken - that seems to be the most likely cause right now. Thinking: it would also be lovely to verify the versions of vmlinuz and the kernel udebs in debian-cd at build time, and (optionally?) abort the build if we know we're going to be making broken images. What do you think? There's no way for us to know a breakage is going to happen. In this particular case the ABI is the same, say 3.16.0-4-amd64; the ABI is embedded in kernel udeb package names, so if the ABI matches, udebs are supposed to be compatible with the kernel. I'm not sure why that is not the case here (maybe the ABI doesn't cover or not completely udebs?). We really shouldn't enforce using the very same revision for kernel and module udebs, IMHO. We probably should start by figuring out what d-i we want to embed (see first paragraph). Having (some, maybe not the whole set of) images with a daily d-i (if we can fix the trust chain) would be nice. But that really shouldn't be labelled “official testing images” as currently done. I'm definitely open to suggestions on what else we should be providing. I'm not 100% sure the best way to go, though... In the past, we had the two different sets of daily images: * use the most recent testing version of d-i in tha archive * use the daily d-i builds from d-i.d.o (and other hosts before that) and then we'd switch the weekly images from one source to the other depending on how working/broken we thought each source was likely to be. Since the kernel team got much more active a few years back and kernel ABI changes have been much more common during the life of testing, using the most recent d-i release from the archive has had a much shorter working life than we used to have. Hence, we switched over to using the daily d-i builds as a base. This can be YA argument for more regular d-i uploads to the archive to fix that problem, of course. But we know how much work that involves. At the moment I feel what we have is just about the best thing we can get without that massive additional effort. It normally works for people, modulo the odd breakage from time to time like we've seen here. I guess we could stick to this once the trust issue is resolved. Daily builds will have to be changed one way or another anyway since it can't be implemented as done previously starting with jessie hosts… Being able to spin some weekly build (possibly manually) with what's in *sid* (i.e. not dailies) is nice to figure out whether we're going to end up with breakages if/when d-i/sid migrates to testing, before we build official release images. Not sure it gains us much compared to first migrating d-i/sid to testing and building images, though. Yeah :-/ We've done that in the past, and it's not that difficult to configure debian-cd to do this. But IIRC it never really gave us much. OK… Mraw, KiBi. signature.asc Description: Digital signature
Bug#779616: debian-cd: broken debian-testing-$arch-netinst.iso generation?
[ Writing this off-line again while travelling so I can't check things directly on pettersson etc. ] Hi KiBi, On Tue, Mar 03, 2015 at 03:47:12PM +0100, Cyril Brulebois wrote: Steve McIntyre st...@einval.com (2015-03-03): Hmmm. That's very odd. The weekly builds are currently set up to use daily d-i builds rather than what's in the archive, and they've been that way for a long time. I don't think that's a good idea. For one thing, there's no trust chain here (from d-i.d.o because http; and from git, because #746967). We've had this discussion befire, surely? On pettersson we rsync things (over ssh) directly from d-i.d.o (dillon), so we don't depend on http or git. Or am I missing something new? (I can't see #746967 atm...) There's also an explicit warning in debian-cd if you grab things via plain http or ftp: echo WARNING WARNING WARNING WARNING WARNING WARNING echo $0: insecure download for d-i build: $DI_WWW_HOME echo WARNING WARNING WARNING WARNING WARNING WARNING As such, we also use sid udebs for debian-installer, as that should match what we'll be getting from the daily d-i builds which are built using sid. Are you sure that the kernel there is old? If so, that's the bug I think. It's difficult for me to check exactly right now what that kernel version is. I extracted the kernel from the iso, and looked at its version string. The guy who reported that issue to me also mentioned that (even if non publicly, and I still haven't seen him open a proper bug report). OK, thanks for checking that. We've been using the daily d-i builds for a long time, to guard against exactly this kind of breakage with kernel version mismatches. Has something changed? Some archs are missing daily builds, because autobuilding in jessie is broken, but AFAICT missing builds are only: - arm64 - ppc64el - sparc (See http://d-i.debian.org/daily-images/daily-build-overview.html) OK. Assuming amd64 is working OK, I should check the rsync has not broken - that seems to be the most likely cause right now. Thinking: it would also be lovely to verify the versions of vmlinuz and the kernel udebs in debian-cd at build time, and (optionally?) abort the build if we know we're going to be making broken images. What do you think? I certainly didn't touch anything on pettersson, or anything remotely involving debian-cd as far as I can remember. Yup. ACK. That's old wording, and has always been confusing. Given we don't tend to upload to the archive like normal packages anyway, it's best to just remove the sid mention altogether I think. Ditto the mentions of sid etc. in the daily-builds area coule do with fixing up I think? We probably should start by figuring out what d-i we want to embed (see first paragraph). Having (some, maybe not the whole set of) images with a daily d-i (if we can fix the trust chain) would be nice. But that really shouldn't be labelled “official testing images” as currently done. I'm definitely open to suggestions on what else we should be providing. I'm not 100% sure the best way to go, though... In the past, we had the two different sets of daily images: * use the most recent testing version of d-i in tha archive * use the daily d-i builds from d-i.d.o (and other hosts before that) and then we'd switch the weekly images from one source to the other depending on how working/broken we thought each source was likely to be. Since the kernel team got much more active a few years back and kernel ABI changes have been much more common during the life of testing, using the most recent d-i release from the archive has had a much shorter working life than we used to have. Hence, we switched over to using the daily d-i builds as a base. This can be YA argument for more regular d-i uploads to the archive to fix that problem, of course. But we know how much work that involves. At the moment I feel what we have is just about the best thing we can get without that massive additional effort. It normally works for people, modulo the odd breakage from time to time like we've seen here. Being able to spin some weekly build (possibly manually) with what's in *sid* (i.e. not dailies) is nice to figure out whether we're going to end up with breakages if/when d-i/sid migrates to testing, before we build official release images. Not sure it gains us much compared to first migrating d-i/sid to testing and building images, though. Yeah :-/ We've done that in the past, and it's not that difficult to configure debian-cd to do this. But IIRC it never really gave us much. -- Steve McIntyre, Cambridge, UK.st...@einval.com When C++ is your hammer, everything looks like a thumb. -- Steven M. Haflich -- To UNSUBSCRIBE, email to debian-cd-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20150305090550.gc6...@einval.com
Bug#779616: debian-cd: broken debian-testing-$arch-netinst.iso generation?
Hi KiBi, On Tue, Mar 03, 2015 at 05:25:11AM +0100, Cyril Brulebois wrote: Package: debian-cd Severity: important Hi, it's been reported to me that debian-testing-amd64-netinst.iso[1] was broken, in that it features d-i using an “old” kernel (3.16.7-ckt4-3), but kernel udebs apparently from sid, as can be seen in the list file[2]: they're at version 3.16.7-ckt7-1. 1. http://cdimage.debian.org/cdimage/weekly-builds/amd64/iso-cd/debian-testing-amd64-netinst.iso 2. http://cdimage.debian.org/cdimage/weekly-builds/amd64/list-cd/debian-testing-amd64-netinst.list.gz Given the kernel ABI hadn't been bumped, one might argue that the udebs should still work and not break due to some unknown symbols, but that's another topic. I really don't expect us to fetch bits from unstable in testing images. Hmmm. That's very odd. The weekly builds are currently set up to use daily d-i builds rather than what's in the archive, and they've been that way for a long time. As such, we also use sid udebs for debian-installer, as that should match what we'll be getting from the daily d-i builds which are built using sid. Are you sure that the kernel there is old? If so, that's the bug I think. It's difficult for me to check exactly right now what that kernel version is. We've been using the daily d-i builds for a long time, to guard against exactly this kind of breakage with kernel version mismatches. Has something changed? I find the doc in the parent directory[3] quite confusing anyway: | These images are produced every week, normally on a Monday, but this may | vary. They include: | * The latest debian-installer build (currently the sid daily builds of the installer) 3. http://cdimage.debian.org/cdimage/weekly-builds/ It's either a daily build (from d-i.debian.org), or d-i from sid. ACK. That's old wording, and has always been confusing. Given we don't tend to upload to the archive like normal packages anyway, it's best to just remove the sid mention altogether I think. Ditto the mentions of sid etc. in the daily-builds area coule do with fixing up I think? -- Steve McIntyre, Cambridge, UK.st...@einval.com Further comment on how I feel about IBM will appear once I've worked out whether they're being malicious or incompetent. Capital letters are forecast. Matthew Garrett, http://www.livejournal.com/users/mjg59/30675.html -- To UNSUBSCRIBE, email to debian-cd-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20150303133547.gx6...@einval.com
Bug#779616: debian-cd: broken debian-testing-$arch-netinst.iso generation?
Hi, Steve McIntyre st...@einval.com (2015-03-03): Hmmm. That's very odd. The weekly builds are currently set up to use daily d-i builds rather than what's in the archive, and they've been that way for a long time. I don't think that's a good idea. For one thing, there's no trust chain here (from d-i.d.o because http; and from git, because #746967). As such, we also use sid udebs for debian-installer, as that should match what we'll be getting from the daily d-i builds which are built using sid. Are you sure that the kernel there is old? If so, that's the bug I think. It's difficult for me to check exactly right now what that kernel version is. I extracted the kernel from the iso, and looked at its version string. The guy who reported that issue to me also mentioned that (even if non publicly, and I still haven't seen him open a proper bug report). We've been using the daily d-i builds for a long time, to guard against exactly this kind of breakage with kernel version mismatches. Has something changed? Some archs are missing daily builds, because autobuilding in jessie is broken, but AFAICT missing builds are only: - arm64 - ppc64el - sparc (See http://d-i.debian.org/daily-images/daily-build-overview.html) I certainly didn't touch anything on pettersson, or anything remotely involving debian-cd as far as I can remember. I find the doc in the parent directory[3] quite confusing anyway: | These images are produced every week, normally on a Monday, but this may | vary. They include: | * The latest debian-installer build (currently the sid daily builds of the installer) 3. http://cdimage.debian.org/cdimage/weekly-builds/ It's either a daily build (from d-i.debian.org), or d-i from sid. ACK. That's old wording, and has always been confusing. Given we don't tend to upload to the archive like normal packages anyway, it's best to just remove the sid mention altogether I think. Ditto the mentions of sid etc. in the daily-builds area coule do with fixing up I think? We probably should start by figuring out what d-i we want to embed (see first paragraph). Having (some, maybe not the whole set of) images with a daily d-i (if we can fix the trust chain) would be nice. But that really shouldn't be labelled “official testing images” as currently done. Being able to spin some weekly build (possibly manually) with what's in *sid* (i.e. not dailies) is nice to figure out whether we're going to end up with breakages if/when d-i/sid migrates to testing, before we build official release images. Not sure it gains us much compared to first migrating d-i/sid to testing and building images, though. Mraw, KiBi. signature.asc Description: Digital signature
Bug#779616: debian-cd: broken debian-testing-$arch-netinst.iso generation?
Package: debian-cd Severity: important Hi, it's been reported to me that debian-testing-amd64-netinst.iso[1] was broken, in that it features d-i using an “old” kernel (3.16.7-ckt4-3), but kernel udebs apparently from sid, as can be seen in the list file[2]: they're at version 3.16.7-ckt7-1. 1. http://cdimage.debian.org/cdimage/weekly-builds/amd64/iso-cd/debian-testing-amd64-netinst.iso 2. http://cdimage.debian.org/cdimage/weekly-builds/amd64/list-cd/debian-testing-amd64-netinst.list.gz Given the kernel ABI hadn't been bumped, one might argue that the udebs should still work and not break due to some unknown symbols, but that's another topic. I really don't expect us to fetch bits from unstable in testing images. I find the doc in the parent directory[3] quite confusing anyway: | These images are produced every week, normally on a Monday, but this may | vary. They include: | * The latest debian-installer build (currently the sid daily builds of the installer) 3. http://cdimage.debian.org/cdimage/weekly-builds/ It's either a daily build (from d-i.debian.org), or d-i from sid. Mraw, KiBi. -- To UNSUBSCRIBE, email to debian-cd-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20150303042511.5742.77863.report...@arya.home.mraw.org
