Bug#749991: Wrong kernel in debian-installer package

[Wolfgang Schweer]

On Thu, Apr 06, 2017 at 09:46:55AM +0200, Cyril Brulebois wrote:
> Alexander Sosedkin  (2017-04-06):
> > That's one way to think about it. Got it, keeping old modules is hard.
> > But I wasn't asking about keeping old modules, I see no point in this.
> > I was asking about generating and publishing a matching
> > dists/testing/main/installer-/current on kernel upload.
> > Why is _that_ hard?
> 
> Because what's below testing/ was copied over from unstable/, and what's
> below unstable/ is being copied from the results of the debian-installer
> upload, which fetches its components from testing (that's our release
> cycle works: we hammer testing into shape until it gets released as a
> new stable). So a new kernel in unstable isn't sufficient to have all
> the pieces together, that's why we have daily builds, and that's why you
> keep being pointed at them.

Maybe people could be pointed to stable instead.

As the debian-installer netboot packages are updated with each point 
release, kernel module mismatch/missing issues should be avoidable.

Besides installing dist=stable by default, also testing and sid are 
supported.

Just adding 'mirror/suite=testing' as additional param to the kernel 
command line is enough.

(Someone wanting unstable would replace 'testing' with 'sid'.)

Wolfgang


signature.asc
Description: PGP signature

Bug#749991: Wrong kernel in debian-installer package

[Cyril Brulebois]

Alexander Sosedkin  (2017-04-06):
> That's one way to think about it. Got it, keeping old modules is hard.
> But I wasn't asking about keeping old modules, I see no point in this.
> I was asking about generating and publishing a matching
> dists/testing/main/installer-/current on kernel upload.
> Why is _that_ hard?

Because what's below testing/ was copied over from unstable/, and what's
below unstable/ is being copied from the results of the debian-installer
upload, which fetches its components from testing (that's our release
cycle works: we hammer testing into shape until it gets released as a
new stable). So a new kernel in unstable isn't sufficient to have all
the pieces together, that's why we have daily builds, and that's why you
keep being pointed at them.


KiBi.


signature.asc
Description: Digital signature

Bug#749991: Wrong kernel in debian-installer package

[Wouter Verhelst]

On Thu, Apr 06, 2017 at 10:01:41AM +0700, Alexander Sosedkin wrote:
> On Thu, 6 Apr 2017 00:05:17 +0200
> Cyril Brulebois  wrote:
> 
> > That's unfortunate, yes, but there's no easy way to keep old packages
> > around in a given repository.
> 
> That's one way to think about it. Got it, keeping old modules is hard.
> But I wasn't asking about keeping old modules, I see no point in this.
> I was asking about generating and publishing a matching
> dists/testing/main/installer-/current on kernel upload.
> Why is _that_ hard?
> 
> I'm asking less of "why isn't it equal to
> https://d-i.debian.org/daily-images tree",
> and more of "why not d-i stretch rc 2 rebuilt with new kernel".
> And https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=749991#59
> doesn't quite cut it. Triggering is hard? Maybe, why not cron then?

We actually do cron. That's what daily-images is.

What's in the repository, however, is a whole different beast, and
"cron" is so far away from the solution that it's not funny.

-- 
< ron> I mean, the main *practical* problem with C++, is there's like a dozen
   people in the world who think they really understand all of its rules,
   and pretty much all of them are just lying to themselves too.
 -- #debian-devel, OFTC, 2016-02-12

Bug#749991: Wrong kernel in debian-installer package

[Alexander Sosedkin]

On Thu, 6 Apr 2017 00:05:17 +0200
Cyril Brulebois  wrote:

> That's unfortunate, yes, but there's no easy way to keep old packages
> around in a given repository.

That's one way to think about it. Got it, keeping old modules is hard.
But I wasn't asking about keeping old modules, I see no point in this.
I was asking about generating and publishing a matching
dists/testing/main/installer-/current on kernel upload.
Why is _that_ hard?

I'm asking less of "why isn't it equal to
https://d-i.debian.org/daily-images tree",
and more of "why not d-i stretch rc 2 rebuilt with new kernel".
And https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=749991#59
doesn't quite cut it. Triggering is hard? Maybe, why not cron then?

Bug#749991: Wrong kernel in debian-installer package

[Cyril Brulebois]

Alexander Sosedkin  (2017-04-05):
> > Added tag(s) stretch-ignore.
> 
> Whatever it means, it sounds very wrong.
> 
> I see you guys like discussing workarounds, but how about, you know,
> actually publishing the correct kernel for netboot in the repo?
> Somebody used to do that, why isn't it the case now?
> 
> http://debian.backend.mirrors.debian.org/debian/dists/stretch/main/installer-amd64/
> 
> Netboot's been broken for weeks, how is that normal? And please, don't
> bring up snapshots.debian.org again. Modules and kernel are in the
> same repository tree, how exactly it happens that keep mismatching?
> Can my virt-install just work, huh?

That's unfortunate, yes, but there's no easy way to keep old packages
around in a given repository. So you've got to wait for a release (and
there won't be one for each and every linux upload), or you've got to
adapt your workflow. Testing is work in progress, and this issue won't
be present in released distributions anyway.


KiBi.


signature.asc
Description: Digital signature

Bug#749991: Wrong kernel in debian-installer package

[Alexander Sosedkin]

> Added tag(s) stretch-ignore.

Whatever it means, it sounds very wrong.

I see you guys like discussing workarounds, but how about, you know,
actually publishing the correct kernel for netboot in the repo?
Somebody used to do that, why isn't it the case now?

http://debian.backend.mirrors.debian.org/debian/dists/stretch/main/installer-amd64/

Netboot's been broken for weeks, how is that normal? And please, don't
bring up snapshots.debian.org again. Modules and kernel are in the same
repository tree, how exactly it happens that keep mismatching?
Can my virt-install just work, huh?

Bug#749991: Wrong kernel in debian-installer package

[Holger Levsen]

On Mon, Mar 27, 2017 at 12:54:18PM +0100, Ian Campbell wrote:
> > > One can always use http://snapshot.debian.org/ as one's mirror and
> > specify a dated URL that matches the ISO's creation date.
> I think (based on the last few paragraphs in the "Usage" section of
> that URL) that one would also need to preseed some stuff to cause it to
> accept the expired signatures on that repo (with all that implies wrt
> security), not sure if/how that can be done in practice though.
 
if accepting expired signatures in this case were made the default, I'd
consider this as much worse than the status quo. if there was a question
explaining this is dangerous and this question has a "proceed (y/N)" question
default to "no" it might be acceptable in my book…


-- 
cheers,
Holger


signature.asc
Description: Digital signature

Bug#749991: Wrong kernel in debian-installer package

[Ian Campbell]

On Mon, 2017-03-27 at 13:32 +0200, Philip Hands wrote:
> > Alexander Sosedkin  writes:
> 
> > On Mon, 27 Mar 2017 12:43:40 +0200
> > > > Philipp Kern  wrote:
> > 
> > > Even if we'd leave the old kernel udebs in testing for a while, you'd 
> > > still hit a point where we'd need to drop them and old installers
> > > would break.
> > 
> > I can see that it's impossible to support downloading modules for all
> > old ISOs.
> 
> > One can always use http://snapshot.debian.org/ as one's mirror and
> specify a dated URL that matches the ISO's creation date.

I think (based on the last few paragraphs in the "Usage" section of
that URL) that one would also need to preseed some stuff to cause it to
accept the expired signatures on that repo (with all that implies wrt
security), not sure if/how that can be done in practice though.

Ian.

Bug#749991: Wrong kernel in debian-installer package

[Alexander Sosedkin]

On Mon, 27 Mar 2017 13:32:46 +0200
Philip Hands  wrote:

> Alexander Sosedkin  writes:
> 
> > On Mon, 27 Mar 2017 12:43:40 +0200
> > Philipp Kern  wrote:
> >  
> >> Even if we'd leave the old kernel udebs in testing for a while,
> >> you'd still hit a point where we'd need to drop them and old
> >> installers would break.  
> >
> > I can see that it's impossible to support downloading modules for
> > all old ISOs.  
> 
> One can always use http://snapshot.debian.org/ as one's mirror and
> specify a dated URL that matches the ISO's creation date.

My point is that for ISOs that makes at least some sense,
but having an old netboot kernel lying in-tree with no modules to
complement it a grave bug. I shouldn't hunt for a snapshot to make my
virt-install invocation work.

Bug#749991: Wrong kernel in debian-installer package

[Philip Hands]

Alexander Sosedkin  writes:

> On Mon, 27 Mar 2017 12:43:40 +0200
> Philipp Kern  wrote:
>
>> Even if we'd leave the old kernel udebs in testing for a while, you'd 
>> still hit a point where we'd need to drop them and old installers
>> would break.
>
> I can see that it's impossible to support downloading modules for all
> old ISOs.

One can always use http://snapshot.debian.org/ as one's mirror and
specify a dated URL that matches the ISO's creation date.

Cheers, Phil.

P.S. I suppose we could build-in such a snapshot URL, as a fallback in
case mirrors have dropped the files we're after, but I think doing so is
asking for trouble.  It would be the sort of code that gets used rarely
enough to end up rotting, and would also have the potential to mask bugs
in the normal code path.  Generally, if you want to survive this, just
use a CD image that includes the matching modules.
-- 
|)|  Philip Hands  [+44 (0)20 8530 9560]  HANDS.COM Ltd.
|-|  http://www.hands.com/http://ftp.uk.debian.org/
|(|  Hugo-Klemm-Strasse 34,   21075 Hamburg,GERMANY


signature.asc
Description: PGP signature

Bug#749991: Wrong kernel in debian-installer package

[Alexander Sosedkin]

On Mon, 27 Mar 2017 12:43:40 +0200
Philipp Kern  wrote:

> Even if we'd leave the old kernel udebs in testing for a while, you'd 
> still hit a point where we'd need to drop them and old installers
> would break.

I can see that it's impossible to support downloading modules for all
old ISOs.

But how about supporting at least one target: current netboot kernel?
Both module udebs and netboot kernel seem to belong in a single
repository hierarchy, why do they desync then?

Bug#749991: Wrong kernel in debian-installer package

[Philipp Kern]

On 2017-03-27 11:56, Nye Liu wrote:

The reason I ask is because I have custom  grub menu configs that I
don't want to constantly hand edit or patch on a cron job along with a
cron to download the dailies... and then have no idea if the cron will
do the right thing, or if the daily even works. I'd like a "known
good" net boot install server for testing and this is making it
difficult.


Well, no-one can give you that guarantee with testing (rather than 
stable, but even there you might need to re-download the installer). One 
way would be to mirror a snapshot of testing, get an installer to work 
with it and then let the install process upgrade everything 
post-install. There isn't necessarily a need to install the most current 
at all times.


Even if we'd leave the old kernel udebs in testing for a while, you'd 
still hit a point where we'd need to drop them and old installers would 
break.


So if you are serious about the "even works", which is also a function 
of the hardware you install on, then maybe you need to run your own 
installer qualification process once in a while.



If you think this might be useful I'd be willing to help develop a
patch and/or help test one.


Well, if you have ideas that work within the current framework, we can 
see about that. Thanks for the offer. :)


Kind regards
Philipp Kern

Bug#749991: Wrong kernel in debian-installer package

[Nye Liu]

Thanks for your response.

On Sun, 26 Mar 2017 18:31:45 +0200 Philipp Kern  wrote:

> IOW: Is there a way to guarantee that
> (dist)/main/installer-amd64/current/images/netboot/netboot.tar.gz does not
> contain a kernel that has no modules package IN THAT SAME mirror?
>
> Or maybe even an automated way to update netboot.tar.gz every time a
> dists linux-image-(arch).deb is updated, a new netboot.tar.gz can be
> created from 
(dist)/main/installer-amd64/current/images/netboot/netboot.tar.gz?

Unfortunately such a mechanism does not currently exist. You can try to
peruse the daily builds hosted on https://d-i.debian.org for this. They
are rebuilt daily and should be able to install testing.


Understood.

Another crazy idea then: have a way to specify where the "current" 
kernel .deb (for whatever kernel is in netboot.tar.gz) is via preseed or 
something?


Or keep a copy of the that .deb for a given netboot.tar.gz in the same 
dir (main/installer-amd64/current/images/netboot?) or other known 
location on a mirror?


The reason I ask is because I have custom  grub menu configs that I 
don't want to constantly hand edit or patch on a cron job along with a 
cron to download the dailies... and then have no idea if the cron will 
do the right thing, or if the daily even works. I'd like a "known good" 
net boot install server for testing and this is making it difficult.


If you think this might be useful I'd be willing to help develop a patch 
and/or help test one.


Thanks for your time, as usual, Ben.

Bug#749991: Wrong kernel in debian-installer package

[Philipp Kern]

On 03/14/2017 10:34 PM, Nye Liu wrote:
> On Tue, Mar 14, 2017 at 08:39:31PM +, Ben Hutchings wrote:
>> On Tue, 2017-03-14 at 11:36 -0700, Nye Liu wrote:
>>> The only apparent solution is to have the kernel maintainers coordinate 
>>> with the d-i maintainers so that whatever kernel is used in d-i is NOT 
>>> removed from the package repository and its mirrors.
>> The kernel maintainers already coordinate with the d-i maintainers,
>> thanks.  We don't remove any packages; that happens automatically.
> Is there a mechanism to insure that when packages are removed from a repo
> are reflected in netboot.tar.gz?
> 
> IOW: Is there a way to guarantee that
> (dist)/main/installer-amd64/current/images/netboot/netboot.tar.gz does not
> contain a kernel that has no modules package IN THAT SAME mirror?
>  
> Or maybe even an automated way to update netboot.tar.gz every time a
> dists linux-image-(arch).deb is updated, a new netboot.tar.gz can be
> created from 
> (dist)/main/installer-amd64/current/images/netboot/netboot.tar.gz?

Unfortunately such a mechanism does not currently exist. You can try to
peruse the daily builds hosted on https://d-i.debian.org for this. They
are rebuilt daily and should be able to install testing.

It's mostly driven by some complexity in ensuring consistency. There is
currently not a single continuous build and upload process that deals
with new source uploads to the main archive. We'd need to trigger builds
whenever testing changes and then auto-upload a corresponding build.
It's both a technical and political problem to make that happen.

Kind regards
Philipp Kern



signature.asc
Description: OpenPGP digital signature

Bug#749991: Wrong kernel in debian-installer package

[Nye Liu]

On Tue, Mar 14, 2017 at 08:39:31PM +, Ben Hutchings wrote:
> On Tue, 2017-03-14 at 11:36 -0700, Nye Liu wrote:
> > The only apparent solution is to have the kernel maintainers coordinate 
> > with the d-i maintainers so that whatever kernel is used in d-i is NOT 
> > removed from the package repository and its mirrors.
> 
> The kernel maintainers already coordinate with the d-i maintainers,
> thanks.  We don't remove any packages; that happens automatically.

Is there a mechanism to insure that when packages are removed from a repo
are reflected in netboot.tar.gz?

IOW: Is there a way to guarantee that
(dist)/main/installer-amd64/current/images/netboot/netboot.tar.gz does not
contain a kernel that has no modules package IN THAT SAME mirror?
 
Or maybe even an automated way to update netboot.tar.gz every time a
dists linux-image-(arch).deb is updated, a new netboot.tar.gz can be
created from (dist)/main/installer-amd64/current/images/netboot/netboot.tar.gz?

Bug#749991: Wrong kernel in debian-installer package

[Ben Hutchings]

On Tue, 2017-03-14 at 11:36 -0700, Nye Liu wrote:
> The only apparent solution is to have the kernel maintainers coordinate 
> with the d-i maintainers so that whatever kernel is used in d-i is NOT 
> removed from the package repository and its mirrors.

The kernel maintainers already coordinate with the d-i maintainers,
thanks.  We don't remove any packages; that happens automatically.

Ben.

-- 
Ben Hutchings
The obvious mathematical breakthrough [to break modern encryption]
would be
development of an easy way to factor large prime numbers. - Bill Gates



signature.asc
Description: This is a digitally signed message part

Bug#749991: Wrong kernel in debian-installer package

[Nye Liu]

The only apparent solution is to have the kernel maintainers coordinate 
with the d-i maintainers so that whatever kernel is used in d-i is NOT 
removed from the package repository and its mirrors.

Bug#749991: Wrong kernel in debian-installer package

[Nye Liu]

This really should be marked grave or serious because it means net
install is completely broken.