Bug#989473: choose-mirror: switch mirror list from salsa to mirror-master

[Peter Palfrader]

On Sat, 05 Jun 2021, Philip Hands wrote:

>  c) filter the old masterlist to only include entries that are also in
> the new list, and then use the result of that, perhaps with a tweak
> to promote deb.d.o
> 
> c) is a bit of a cludge, but seems like the only one that's got a chance
> of happening before the release, and gets most of the benefit of the new
> list.

The Type info in the salsa Masterlist is also probably not correct in
a lot of cases.

-- 
|  .''`.   ** Debian **
  Peter Palfrader   | : :' :  The  universal
 https://www.palfrader.org/ | `. `'  Operating System
|   `-https://www.debian.org/

Bug#989473: choose-mirror: switch mirror list from salsa to mirror-master

[Peter Palfrader]

On Sat, 05 Jun 2021, Philip Hands wrote:

> Philip Hands  writes:
> 
> ...
> >  c) filter the old masterlist to only include entries that are also in
> > the new list, and then use the result of that, perhaps with a tweak
> > to promote deb.d.o
> 
> BTW Promoting deb.d.o can be done thus:
> 
>   
> https://salsa.debian.org/philh/choose-mirror/-/commit/70caed09fbf4bfbcc9eca82168cf3936868d8394
> 
> which produces this menu ordering:
> 
>   https://openqa.debian.net/tests/6101#step/mirror_selection/2

And then add a line that maybe gives some number (6, 7?) to other things
matching ~/\.debian\.org$/?


-- 
|  .''`.   ** Debian **
  Peter Palfrader   | : :' :  The  universal
 https://www.palfrader.org/ | `. `'  Operating System
|   `-https://www.debian.org/

Bug#989473: choose-mirror: switch mirror list from salsa to mirror-master

[Philip Hands]

Philip Hands  writes:

...
>  c) filter the old masterlist to only include entries that are also in
> the new list, and then use the result of that, perhaps with a tweak
> to promote deb.d.o

BTW Promoting deb.d.o can be done thus:

  
https://salsa.debian.org/philh/choose-mirror/-/commit/70caed09fbf4bfbcc9eca82168cf3936868d8394

which produces this menu ordering:

  https://openqa.debian.net/tests/6101#step/mirror_selection/2

Cheers, Phil.
-- 
|)|  Philip Hands  [+44 (0)20 8530 9560]  HANDS.COM Ltd.
|-|  http://www.hands.com/http://ftp.uk.debian.org/
|(|  Hugo-Klemm-Strasse 34,   21075 Hamburg,GERMANY

Bug#989473: choose-mirror: switch mirror list from salsa to mirror-master

[Philip Hands]

Holger Wansing  writes:

> Hi,
>
> Cyril Brulebois  wrote (Fri, 04 Jun 2021 18:39:44 +0200):
>> Filing this on behalf of Peter Palfrader who suggested we switched from
>> the manually curated mirror list hosted on salsa[1] to I suppose a live
>> one published on mirror-master[2], that would be more representative of
>> the current state of mirrors when release time approaches.
>> 
>>  1. 
>> https://salsa.debian.org/mirror-team/masterlist/raw/master/Mirrors.masterlist
>>  2. https://mirror-master.debian.org/status/Mirrors.masterlist
>
> Hmm, what exactly is the benefit?
> Sourcing a file from a git repo on Salsa seems reasonable to me, given that
> we have revision tracking there, which would be a good basis for debugging
> d-i's mirror list in case of problems or questions...

This is just changing the Makefile target that allows one to then run
`make Mirrors.masterlist` in order to update the copy of the list in the
choose-mirrors repo, so the version you care about is tracked in git.
You can see this with:

   git log | grep Update\ Mirrors.masterlist

>> Having attempted that, I'm not comfortable updating choose-mirror with
>> this change right now, as that would mean shuffling all entries.
>> 
>> A quick brainstorming on IRC resulted in what appeared to be a good idea
>> to everyone involved: the list might be random, but let's make sure we
>> show deb.d.o at the top (default entry anyway), followed by ftp.CC.d.o
>> (if there's such an entry for the given country).
>> 
>> Happy to take opinions, and possibly patches for that. :)
>
> Having deb.debian.org at the top would be an improvement, as well as having 
> ftp.CC.d.o next to it.
> However, the difference is only marginal IMHO ...

Reading the IRC chat I had assumed that reordering the masterlist might
achieve that result, but it doesn't, as a quick test showed:

  https://openqa.debian.net/tests/5976#step/mirror_selection/2

Looking at the code, the ordering is determined mostly by looking at the
Type: of the mirror:

  
https://salsa.debian.org/installer-team/choose-mirror/-/blob/master/mirrorlist#L99

which would be easy to add a line like this to:

  $rating=9 if $data[$id]->{site} eq 'deb.debian.org';

except for the slight problem that the new file is lacking almost all of
the Type: fields (it has none of the Push-Primary/Push-Secondary) stuff,
so the order you end up with is mostly based on the perl hash table
order if you use the new file, which strikes me as sub-optimal (and
explains the mess in that screenshot).

I guess we either need do one of:

 a) reinstate the Type information somehow,
which I understand is probably a problem, since the code seems to
rely on the CC mirrors having Type: Push-Primary, but since that's
actually a CNAME that may wander about maintaining that information
isn't completely straight-forward.

 b) rewrite the logic to be something like deb.d.o first, then
ftp.CC.d.o, then the rest in alphabetical order, say.

 c) filter the old masterlist to only include entries that are also in
the new list, and then use the result of that, perhaps with a tweak
to promote deb.d.o

c) is a bit of a cludge, but seems like the only one that's got a chance
of happening before the release, and gets most of the benefit of the new
list.

Cheers, Phil.
-- 
|)|  Philip Hands  [+44 (0)20 8530 9560]  HANDS.COM Ltd.
|-|  http://www.hands.com/http://ftp.uk.debian.org/
|(|  Hugo-Klemm-Strasse 34,   21075 Hamburg,GERMANY

Bug#989473: choose-mirror: switch mirror list from salsa to mirror-master

[Holger Wansing]

Hi,

Cyril Brulebois  wrote (Fri, 04 Jun 2021 18:39:44 +0200):
> Filing this on behalf of Peter Palfrader who suggested we switched from
> the manually curated mirror list hosted on salsa[1] to I suppose a live
> one published on mirror-master[2], that would be more representative of
> the current state of mirrors when release time approaches.
> 
>  1. 
> https://salsa.debian.org/mirror-team/masterlist/raw/master/Mirrors.masterlist
>  2. https://mirror-master.debian.org/status/Mirrors.masterlist

Hmm, what exactly is the benefit?
Sourcing a file from a git repo on Salsa seems reasonable to me, given that
we have revision tracking there, which would be a good basis for debugging
d-i's mirror list in case of problems or questions...

> Having attempted that, I'm not comfortable updating choose-mirror with
> this change right now, as that would mean shuffling all entries.
> 
> A quick brainstorming on IRC resulted in what appeared to be a good idea
> to everyone involved: the list might be random, but let's make sure we
> show deb.d.o at the top (default entry anyway), followed by ftp.CC.d.o
> (if there's such an entry for the given country).
> 
> Happy to take opinions, and possibly patches for that. :)

Having deb.debian.org at the top would be an improvement, as well as having 
ftp.CC.d.o next to it.
However, the difference is only marginal IMHO ...


Holger


-- 
Holger Wansing 
PGP-Fingerprint: 496A C6E8 1442 4B34 8508  3529 59F1 87CA 156E B076

Bug#989473: choose-mirror: switch mirror list from salsa to mirror-master

[Peter Palfrader]

On Fri, 04 Jun 2021, Holger Wansing wrote:

> Hi,
> 
> Cyril Brulebois  wrote (Fri, 04 Jun 2021 18:39:44 +0200):
> > Filing this on behalf of Peter Palfrader who suggested we switched from
> > the manually curated mirror list hosted on salsa[1] to I suppose a live
> > one published on mirror-master[2], that would be more representative of
> > the current state of mirrors when release time approaches.
> > 
> >  1. 
> > https://salsa.debian.org/mirror-team/masterlist/raw/master/Mirrors.masterlist
> >  2. https://mirror-master.debian.org/status/Mirrors.masterlist
> 
> Hmm, what exactly is the benefit?
> Sourcing a file from a git repo on Salsa seems reasonable to me, given that
> we have revision tracking there, which would be a good basis for debugging
> d-i's mirror list in case of problems or questions...

on salsa you get all the down and broken mirrors.  on mirror-master,
they are filtered out.
-- 
|  .''`.   ** Debian **
  Peter Palfrader   | : :' :  The  universal
 https://www.palfrader.org/ | `. `'  Operating System
|   `-https://www.debian.org/

Bug#989473: choose-mirror: switch mirror list from salsa to mirror-master

[Cyril Brulebois]

Package: choose-mirror
Severity: wishlist
X-Debbugs-Cc: wea...@debian.org

Filing this on behalf of Peter Palfrader who suggested we switched from
the manually curated mirror list hosted on salsa[1] to I suppose a live
one published on mirror-master[2], that would be more representative of
the current state of mirrors when release time approaches.

 1. 
https://salsa.debian.org/mirror-team/masterlist/raw/master/Mirrors.masterlist
 2. https://mirror-master.debian.org/status/Mirrors.masterlist

Having attempted that, I'm not comfortable updating choose-mirror with
this change right now, as that would mean shuffling all entries.

A quick brainstorming on IRC resulted in what appeared to be a good idea
to everyone involved: the list might be random, but let's make sure we
show deb.d.o at the top (default entry anyway), followed by ftp.CC.d.o
(if there's such an entry for the given country).

Happy to take opinions, and possibly patches for that. :)

The trivial change to apply before running `make Mirrors.masterlist` is
in the Makefile:


-MIRRORLISTURL=https://salsa.debian.org/mirror-team/masterlist/raw/master/Mirrors.masterlist
+MIRRORLISTURL=https://mirror-master.debian.org/status/Mirrors.masterlist


Cheers,
-- 
Cyril Brulebois (k...@debian.org)
D-I release manager -- Release team member -- Freelance Consultant