Bug#902226: debian-installer-netboot-images: FTBFS in stretch, even when allowing network access

[Cyril Brulebois]

Hi Didier,

Didier 'OdyX' Raboud  (2018-06-27):
> Given my failed attempts, I suspect your patches are the lesser evil
> for solving this. But I'm not convinced that solving this is better
> than ensuring we only ever build "pure-stretch" or
> "pure-stretch-proposed-updates" d-i-n-i's.

No argument here, I'm totally with you.

> > I'll let others comment on this bug report plus proposed solution;
> > Didier maybe?
> 
> Thanks for the explicit ping; I'm not on top of Debian things these
> days.

Thanks for your valuable input. Based on the fact you've made several
attempts already, I decided patches were good enough to include them in
the dini upload for this point release, instead of spending more time
to get a perfect solution (esp. with the next point release coming up in
a few days).

I've opened #903618 to keep track of the improvement over this set of
patches.


Cheers,
-- 
Cyril Brulebois (k...@debian.org)
D-I release manager -- Release team member -- Freelance Consultant


signature.asc
Description: PGP signature

Bug#902226: debian-installer-netboot-images: FTBFS in stretch, even when allowing network access

[Didier 'OdyX' Raboud]

Hi theres,

Le lundi, 25 juin 2018, 01.33:38 h CEST Cyril Brulebois a écrit :
> Cyril Brulebois  (2018-06-24):
> > At first glance, it seems to me this bug could be addressed in two
> > different ways, which don't seem to be too convoluted. The first way
> > would be to try the s-p-u download and fall back to s download, for each
> > and every download. But this could (probably only theoretically) lead to
> > inconsistent downloads, mixing bits and pieces from s-p-u and from s.
> > Plus plenty of errors when the default location isn't the right one.

Exactly. If a pure s-p-u build fails, it's because the s-p-u debian-installer 
isn't built on all architectures, so the d-i-n-i s-p-u build should really 
fail. (acronyms ftw)

> > I suppose a better way would be to figure out with an early test if the
> > target version is available in s-p-u or in s, and then pick the right
> > suite for all downloads. Patches for this second approach are welcome.

That seems more fool-proof and consistent: download from a single suite: 
either from s-p-u or from stretch only, and for all archs.

> I've pushed a prospective branch (pu/fix-ftbfs-in-stretch) with two commits:
> https://salsa.debian.org/installer-team/debian-installer-netboot-images/com
> mit/86f910f8e1e60e308747a7f53045862705b0a132
> https://salsa.debian.org/installer-team/debian-installer-netboot-images/com
> mit/eb2e5b3fb437b288c4c83427fb1c0d213f7b5a5e

Looks good to me, given that strategy.

> Only checked with the first few architectures (still on limited bandwidth),
> but that seems to do the trick. Slightly not happy about having that check
> and fallback done for each and every architecture (instead of once and for
> all), which could again lead to bits and pieces from both sources mixed
> together); but I guess that's a reasonable compromise (no big changes needed
> in the code).

I've tried for some time to (ab)use make targets to modify DISTRIBUTION 
depending on partial calls to get_images, but failed.

Given my failed attempts, I suspect your patches are the lesser evilfor 
solving this. But I'm not convinced that solving this is better than ensuring 
we only ever build "pure-stretch" or "pure-stretch-proposed-updates" d-i-n-
i's.

> I'll let others comment on this bug report plus proposed solution; Didier
> maybe?

Thanks for the explicit ping; I'm not on top of Debian things these days.

Cheers,
OdyX

Bug#902226: debian-installer-netboot-images: FTBFS in stretch, even when allowing network access

[Cyril Brulebois]

Control: tag -1 patch

Cyril Brulebois  (2018-06-24):
> At first glance, it seems to me this bug could be addressed in two
> different ways, which don't seem to be too convoluted. The first way
> would be to try the s-p-u download and fall back to s download, for each
> and every download. But this could (probably only theoretically) lead to
> inconsistent downloads, mixing bits and pieces from s-p-u and from s.
> Plus plenty of errors when the default location isn't the right one.
> 
> I suppose a better way would be to figure out with an early test if the
> target version is available in s-p-u or in s, and then pick the right
> suite for all downloads. Patches for this second approach are welcome.
> (Currently travelling, not easy to test things needed network access.)

I've pushed a prospective branch (pu/fix-ftbfs-in-stretch) with two commits:
  
https://salsa.debian.org/installer-team/debian-installer-netboot-images/commit/86f910f8e1e60e308747a7f53045862705b0a132
  
https://salsa.debian.org/installer-team/debian-installer-netboot-images/commit/eb2e5b3fb437b288c4c83427fb1c0d213f7b5a5e

Only checked with the first few architectures (still on limited bandwidth),
but that seems to do the trick. Slightly not happy about having that check
and fallback done for each and every architecture (instead of once and for
all), which could again lead to bits and pieces from both sources mixed
together); but I guess that's a reasonable compromise (no big changes needed
in the code).

I'll let others comment on this bug report plus proposed solution; Didier
maybe?

(Of course testing needs to happen when there's a version in s-p-u again.)


Cheers,
-- 
Cyril Brulebois (k...@debian.org)
D-I release manager -- Release team member -- Freelance Consultant


signature.asc
Description: PGP signature

Bug#902226: debian-installer-netboot-images: FTBFS in stretch, even when allowing network access

[Cyril Brulebois]

Hi Santiago,

Santiago Vila  (2018-06-23):
> Without using any special flag or environment variable, what I would
> expect is that it downloads things from stretch as well, as every
> point release of stretch should ideally be "self-contained" for
> building purposes.
> 
> I see this line in debian/rules:
> 
> export DISTRIBUTION?=stretch-proposed-updates
> 
> Maybe that's the problem.

Right, that allows us to build from s-p-u when catching up with d-i
changes staged into s-p-u, which happens for each and every point
release where d-i gets updated (I don't remember any where it was
skipped, but my memory is not to be trusted), starting right after r0.

> I'm putting all my build logs here for you to see:
> 
> https://people.debian.org/~sanvila/build-logs/debian-installer-netboot-images/
> 
> You will see that version 20170615 of this package used to build ok in
> the past.

Right, see commit 5a0d0847c8a12e2dd3f13ce86aff797f4472dbd6 for stretch;
previous releases should have something similar.

> [ Note: I fear that this could be another wontfix bug, like the one
>   complaining about network access, so I will not bother to set a
>   severity, but I still believe that for consistency, packages in
>   stretch should be buildable in stretch by default, even when they
>   need network access ].

Well, I've seen a fair share of “complaints”, but no solutions, so I
fear this isn't going to be fixed any time soon indeed. Needless to say,
I'm not a huge fan of this specific requirement / policy violation, but
I have yet to see actual, viable solutions. From my rather self-centered
point of view, d-i releases are a sufficient burden already that I don't
wish to make this worse by adding some extra layers of complexity or
indirections.

>   If, after all, you consider that the cure is worse than the disease,
>   then please consider this as a documentation bug and if possible
>   explain somewhere why we are not expected to build this package
>   without failures ].

At first glance, it seems to me this bug could be addressed in two
different ways, which don't seem to be too convoluted. The first way
would be to try the s-p-u download and fall back to s download, for each
and every download. But this could (probably only theoretically) lead to
inconsistent downloads, mixing bits and pieces from s-p-u and from s.
Plus plenty of errors when the default location isn't the right one.

I suppose a better way would be to figure out with an early test if the
target version is available in s-p-u or in s, and then pick the right
suite for all downloads. Patches for this second approach are welcome.
(Currently travelling, not easy to test things needed network access.)


Cheers,
-- 
Cyril Brulebois (k...@debian.org)
D-I release manager -- Release team member -- Freelance Consultant


signature.asc
Description: PGP signature

Bug#902226: debian-installer-netboot-images: FTBFS in stretch, even when allowing network access

[Santiago Vila]

Package: src:debian-installer-netboot-images
Version: 20170615+deb9u3
Tags: ftbfs

Dear Debian Installer people:

Even when we allow network access in the autobuilder, building this
package no longer works since version 20170615+deb9u1.

---
Connecting to cdn-fastly.deb.debian.org
(cdn-fastly.deb.debian.org)|151.101.132.204|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 20 [application/x-gzip]
Saving to: 'amd64.Packages.gz'

 0K   100% 7.00M=0s

2017-07-22 15:38:16 (7.00 MB/s) - 'amd64.Packages.gz' saved [20/20]

amd64.Packages.gz: OK
Building 20170615+deb9u1, but stretch-proposed-updates has , failing the build
debian/rules:19: recipe for target 'get-images-amd64' failed
make[1]: *** [get-images-amd64] Error 1
---

Without using any special flag or environment variable, what I would
expect is that it downloads things from stretch as well, as every
point release of stretch should ideally be "self-contained" for
building purposes.

I see this line in debian/rules:

export DISTRIBUTION?=stretch-proposed-updates

Maybe that's the problem.

I'm putting all my build logs here for you to see:

https://people.debian.org/~sanvila/build-logs/debian-installer-netboot-images/

You will see that version 20170615 of this package used to build ok in
the past.


[ Note: I fear that this could be another wontfix bug, like the one
  complaining about network access, so I will not bother to set a
  severity, but I still believe that for consistency, packages in
  stretch should be buildable in stretch by default, even when they
  need network access ].

  If, after all, you consider that the cure is worse than the disease,
  then please consider this as a documentation bug and if possible
  explain somewhere why we are not expected to build this package
  without failures ].

Thanks.