Re: RHEL9 migration

[Adrian Reber]

On Mon, Sep 26, 2022 at 02:56:18PM -0700, Kevin Fenzi wrote:
> Some of them need applications/packages built for rhel9 and we can't do them 
> until
> that is sorted out:
> 
> badges (hopefully now ongoing?)
> notifs (ongoing)
> mm- (is mirrormanager2 ready to branch/build for rhel9?)

We were never able to build mirrormanager for rhel8 because some
dependencies were missing in EPEL.

Missing dependencies in EPEL is probably the only thing stopping us
using mirrormanager on rhel8 or rhel9 as far as I know. I am not aware
of any other problems.

Adrian
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Upcoming MirrorManager changes (removal of Internet2 support)

[Adrian Reber]

I will remove Internet2 support from MirrorManager in the next few
weeks.

The main reason is, that the source for the Internet2 routing tables
does not exist any more: http://routes.net.internet2.edu/

It changed a couple of years ago and it seems Fedora was one of the only
users of that data. They fixed it for us, but the latest ticket I opened
to access this information was closed with "This service was decomd".

I do not think that the removal of Internet2 support for our users will
be noticed at all. Removing Internet2 will remove some complexity from
the code and using ASN and GeoIP should still provide a reasonably close
mirror for most of out users.

I will start to update the different components to not expose and rely
on Internet2 information any more in the next couple of weeks.

Let me know if you see any major problems in removing Internet2 support
from MirrorManager.

Adrian
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: FBR: Enable CentOS Stream in MirrorManager

[Adrian Reber]

On Mon, Sep 06, 2021 at 08:01:40PM +0200, Adrian Reber wrote:
> Over the last weeks we prepared adding CentOS Stream to Fedora's
> MirrorManager instance and are now at a point where we would like to
> push the changes to ansible.
> 
> The current state can be seen at:
> 
> http://mirrors.stg.centos.org/metalink?repo=centos-baseos-9-stream=x86_64
> 
> (https just broke over the weekend)
> 
> To enable CentOS Stream in MirrorManager not only configuration file
> changes are necessary, but it also requires an update of all software
> components. This is mainly due to the fact that CentOS Stream is using
> an empty topdir. (topdir in MirrorManager are things like 'epel/' or
> 'fedora/linux' or 'fedora-secondary/').
> 
> Unfortunately all code assumed that topdir is not '' and hard-coded the
> removal of a slash all over the place.
> 
> All corresponding projects have been update to handle empty topdirs.
> 
> To apply https://pagure.io/fedora-infra/ansible/pull-request/775 for
> prod I need this FBR.
> 
> There are risks doing code changes like this during a freeze. So far I
> have not seen any problems in staging, but staging is not using using
> MirrorManager as thoroughly as prod. I don't expect any major problems
> with this change.
> 
> I hope someone from the CentOS team can weigh if this is very time
> critical to get running or if we can wait until after the freeze.

This is now active and I am able to use it on my CentOS Stream 9 VM
with:

metalink=https://mirrors.centos.org/metalink?repo=centos-baseos-9-stream=x86_64
metalink=https://mirrors.centos.org/metalink?repo=centos-appstream-9-stream=x86_64
metalink=https://mirrors.centos.org/metalink?repo=centos-crb-9-stream=x86_64

Let me know if something does not work as expected.

Adrian
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

FBR: Enable CentOS Stream in MirrorManager

[Adrian Reber]

Over the last weeks we prepared adding CentOS Stream to Fedora's
MirrorManager instance and are now at a point where we would like to
push the changes to ansible.

The current state can be seen at:

http://mirrors.stg.centos.org/metalink?repo=centos-baseos-9-stream=x86_64

(https just broke over the weekend)

To enable CentOS Stream in MirrorManager not only configuration file
changes are necessary, but it also requires an update of all software
components. This is mainly due to the fact that CentOS Stream is using
an empty topdir. (topdir in MirrorManager are things like 'epel/' or
'fedora/linux' or 'fedora-secondary/').

Unfortunately all code assumed that topdir is not '' and hard-coded the
removal of a slash all over the place.

All corresponding projects have been update to handle empty topdirs.

To apply https://pagure.io/fedora-infra/ansible/pull-request/775 for
prod I need this FBR.

There are risks doing code changes like this during a freeze. So far I
have not seen any problems in staging, but staging is not using using
MirrorManager as thoroughly as prod. I don't expect any major problems
with this change.

I hope someone from the CentOS team can weigh if this is very time
critical to get running or if we can wait until after the freeze.

Adrian
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: epel-next and MirrorManager

[Adrian Reber]

On Wed, Jun 02, 2021 at 09:21:05AM -0400, Neal Gompa wrote:
> On Wed, Jun 2, 2021 at 9:17 AM Adrian Reber  wrote:
> >
> > Recently a bug was opened to add additional repositories to
> > MirrorManager for epel-next. One of the problems with adding new
> > repository definitions to MirrorManager is that the definitions are part
> > of the code. For each new/changed repository a new release is needed.
> >
> > I was thinking about changing this for a long time. Have the repository
> > definition in a configuration file.
> >
> 
> Making this a configuration file would also make it easier to adopt in
> other projects. For example, it's still bandied about for openSUSE.
> The hardwired definitions in the codebase are a problem, though.
> 
> > During the last months I have been rewriting umdl in Rust after the
> > first two MirrorManager tools rewritten in Rust proofed to be really
> > reliable during the last 1.5 years.
> >
> > The new tool, scan-primary-mirror, is already used in RPM Fusion for a
> > few weeks and now with the need to have epel-next repositories I
> > replaced umdl only for EPEL also in Fedora.
> >
> > So far it seems to work correctly. The epel-next repositories have been
> > detected and all other metalinks have also been updated correctly.
> >
> > Please let me know if something around EPEL metalinks is not working as
> > expected.
> >
> 
> Are these tools packaged in Fedora yet?

The mirrorlist-server part is packaged in Fedora which also contains the
second tool, generate-mirrorlist-cache. The new, scan-primary-mirror, is
not.

Adrian
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

epel-next and MirrorManager

[Adrian Reber]

Recently a bug was opened to add additional repositories to
MirrorManager for epel-next. One of the problems with adding new
repository definitions to MirrorManager is that the definitions are part
of the code. For each new/changed repository a new release is needed.

I was thinking about changing this for a long time. Have the repository
definition in a configuration file.

During the last months I have been rewriting umdl in Rust after the
first two MirrorManager tools rewritten in Rust proofed to be really
reliable during the last 1.5 years.

The new tool, scan-primary-mirror, is already used in RPM Fusion for a
few weeks and now with the need to have epel-next repositories I
replaced umdl only for EPEL also in Fedora.

So far it seems to work correctly. The epel-next repositories have been
detected and all other metalinks have also been updated correctly.

Please let me know if something around EPEL metalinks is not working as
expected.

Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Another Rust MirrorManager experiment

[Adrian Reber]

On Tue, Oct 06, 2020 at 10:38:40AM -0400, Stephen John Smoogen wrote:
> On Tue, 6 Oct 2020 at 03:46, Adrian Reber  wrote:
> 
> > On Mon, Oct 05, 2020 at 08:30:12AM -0400, Stephen John Smoogen wrote:
> > > On Mon, 5 Oct 2020 at 02:24, Adrian Reber  wrote:
> > >
> >
> > > We are not wanting to deploy new EL7 systems but would probably install
> > an
> > > EL8 box for this. Does this change the need for moving to Fedora on it?
> >
> > I just asked on #fedora-rust, but it seems it is not easily possible to
> > build the Fedora Rust packages for EL8. If I am understanding it
> > correctly it seems we need to run the Rust based mirrorlist cache
> > generation on a Fedora host. If we have a second mm-backend system
> > (mm-backend02) that is Fedora based to generate the mirrorlist cache we
> > could decrease the amount of RAM (32GB) on mm-backend01 to something
> > like 8GB.
> >
> >
> OK that makes sense. This will be something that needs upgrading every 6
> months like our proxies, but it is what it is.

I have also seen that countme is deployed from ansible directly from
git. I could do that also for mirrorlist cache generation code on RHEL 7
which would mean we do not need any other hosts. We could just run it
directly on mm-backend01 as it is right now and switch between the
existing Python based code and the new Rust based code just as we need
it.

Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Mirrorlist bad entry for ftp.icm.edu.pl resulting 404

[Adrian Reber]

On Tue, Oct 27, 2020 at 02:11:49PM +0100, Rafal Maszkowski wrote:
> On Tue, Oct 27, 2020 at 10:20:08AM +0100, Adrian Reber wrote:
> > On Tue, Oct 27, 2020 at 06:29:27AM -, Mariusz Brzezik wrote:
> > > Dear community, recently i we faced a errors during machines creation 
> > > phase due to bad entry for ftp.icm.edu.pl within EPEL mirrorlist. Checked 
> > > for CentOS 7 and 8.
> > > Centos 7 entry giving 404 error: 
> > > https://ftp.icm.edu.pl/pub/Linux/fedora/linux/epel/7/x86_64/
> > > Working link for CentOS7:  
> > > https://ftp.icm.edu.pl/pub/Linux/dist/fedora-epel/7/x86_64/
> > > Centos8 entry giving 404 error: 
> > > http://ftp.icm.edu.pl/pub/Linux/fedora/linux/epel/8/Everything/x86_64/
> > > Working link for CentOS8: 
> > > https://ftp.icm.edu.pl/pub/Linux/dist/fedora-epel/8/Everything/x86_64/
> > > Who should i contact with to issue those mirrorlist errors?
> > > Thank You in advance for You help!
> > Thanks for letting us know. The best place to report mirror problems is
> > probably: https://pagure.io/fedora-infrastructure/issues
> > I had a look at that mirror and it seems it changed URLs and I just
> > updated our mirror entry to use the new URLs.
> > It takes about one hour until the changes are live so it should soon
> > point to the correct URLs.
> 
> I prepare to use quick-fedora-mirror finally. It looks to me that with
> that type of mirroring I can mirror only all subdirectories of
> fedora/linux, including core and extra which I did not mirror until now
> but I can no longer keep epel there. But the official paths registered
> long ago in https://admin.fedoraproject.org/mirrormanager/mirrors/EPEL
> are
> rsync://ftp.icm.edu.pl/pub/Linux/dist/epel
> https://ftp.icm.edu.pl/pub/Linux/dist/fedora-epel
> http://ftp.icm.edu.pl/pub/Linux/dist/fedora-epel
> (I just try to unify them to …/epel) and they are available.
> 
> Despite that I already got a report that
> curl -Lv http://download.fedoraproject.org/pub/epel/7/x86_64/
> returns a location
> https://ftp.icm.edu.pl/pub/Linux/fedora/linux/epel/7/x86_64/
> from X-Fedora-ProxyServer: proxy04.fedoraproject.org
> 
> I have tried it myself and I get
> http://ftp.icm.edu.pl/pub/Linux/dist/fedora-epel (not https) but from
> other proxies. My hypotesis is that this proxy (at ibiblio) has stale
> information. Should I fill a report on that in
> https://pagure.io/fedora-infrastructure/issues ?

I do not think that the proxy is out of date:

$ curl -s 
"http://proxy04.fedoraproject.org/mirrorlist?repo=epel-7=x86_64=pl=1;
 -H 'Host: mirrors.fedoraproject.org' -I | grep location
location: http://ftp.icm.edu.pl/pub/Linux/dist/fedora-epel/7/x86_64/

But maybe when you looked the data was still old. It always takes some
time. Anyway, if you think that something is not correct please open a
ticket. As far as I see it, everything looks correct now.

Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Mirrorlist bad entry for ftp.icm.edu.pl resulting 404

[Adrian Reber]

On Tue, Oct 27, 2020 at 06:29:27AM -, Mariusz Brzezik wrote:
> Dear community, recently i we faced a errors during machines creation phase 
> due to bad entry for ftp.icm.edu.pl within EPEL mirrorlist. Checked for 
> CentOS 7 and 8.
> 
> Centos 7 entry giving 404 error: 
> https://ftp.icm.edu.pl/pub/Linux/fedora/linux/epel/7/x86_64/
> Working link for CentOS7:  
> https://ftp.icm.edu.pl/pub/Linux/dist/fedora-epel/7/x86_64/
> 
> Centos8 entry giving 404 error: 
> http://ftp.icm.edu.pl/pub/Linux/fedora/linux/epel/8/Everything/x86_64/
> Working link for CentOS8: 
> https://ftp.icm.edu.pl/pub/Linux/dist/fedora-epel/8/Everything/x86_64/
> 
> Who should i contact with to issue those mirrorlist errors?
> Thank You in advance for You help!

Thanks for letting us know. The best place to report mirror problems is
probably: https://pagure.io/fedora-infrastructure/issues

I had a look at that mirror and it seems it changed URLs and I just
updated our mirror entry to use the new URLs.

It takes about one hour until the changes are live so it should soon
point to the correct URLs.

Adrian
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Another Rust MirrorManager experiment

[Adrian Reber]

On Mon, Oct 05, 2020 at 08:30:12AM -0400, Stephen John Smoogen wrote:
> On Mon, 5 Oct 2020 at 02:24, Adrian Reber  wrote:
> 
> > On Wed, Jul 01, 2020 at 09:16:05AM -0700, Kevin Fenzi wrote:
> > > On Tue, Jun 30, 2020 at 02:17:30PM +0200, Adrian Reber wrote:
> > > > On Mon, Jun 15, 2020 at 03:36:23PM -0700, Kevin Fenzi wrote:
> > > > > On Wed, Jun 10, 2020 at 11:09:49AM +0200, Adrian Reber wrote:
> > > > > >
> > > > > > Then I just have to wait a bit. No problem.
> > > > > >
> > > > > > > > Having the possibility to generate the mirrorlist input data
> > in about a
> > > > > > > > minute would significantly reduce the load on the database
> > server and
> > > > > > > > enable us to react much faster if broken protobuf data has
> > been synced
> > > > > > > > to the mirrorlist servers on the proxies.
> > > > > > >
> > > > > > > Yeah, and I wonder if it would let us revisit the entire
> > sequence from
> > > > > > > 'update push finished' to updated mirrorlist server.
> > > > >
> > > > > This would help us with the case of:
> > > > > - updates push/rawhide finishes, master mirror is updated.
> > > > > - openqa/other internal thing tries to get images or updates in that
> > > > >   change and gets a metalink with the old checksum so it can't get
> > the
> > > > >   new stuff.
> > > > > - mm-backend01 generates and pushes out a new protobuf.
> > > > > >
> > > > > > Probably. As the new code will not run on the current RHEL 7 based
> > > > > > mm-backend01 would it make sense to run a short running service
> > like
> > > > > > this on Fedora's OpenShift? We could also create a new read-only
> > (SELECT
> > > > > > only) database account for this.
> > > > >
> > > > > We could, or as smooge suggests make a mm-backend02?
> > > > >
> > > > > But I guess now mm-backend02 just generates new proobuf files and
> > copies
> > > > > them to mirrorlists? If thats all it's doing, perhaps we could indeed
> > > > > replace it with an openshift project.
> > > >
> > > > We need a system to run the tool and copy the data to all proxies.
> > > >
> > > > I would like to see a new MirrorManager database user who can only do
> > > > selects as that is all we need.
> > > >
> > > > Currently we copy the files via SSH to the proxies, if we continue
> > doing
> > > > it that way, then we would also need the existing SSH key to copy the
> > > > data to the proxies.
> > > >
> > > > Easiest would probably be a small Fedora 32 based VM with 2GB of
> > memory.
> > >
> > > I'm not sure f32 will work with 2gb memory anymore. I dont think it
> > > installs at any rate.
> > >
> > > I do like the idea of just making it an openshift pod. Perhaps this
> > > could even fit with pingous 'toddlers' setup. ie:
> >
> > I tried to create a toddler, but that setup is too complicated for me.
> > Especially if something is not working it will be almost impossible for
> > me to debug it if it is running somewhere I cannot reach via SSH.
> >
> > I just tried to build the generate-mirrorlist-cache on RHEL 7 (using
> > Rust from EPEL) and it works fine. Instead of 20 minutes it needs 30
> > seconds to generate the mirrorlist cache file on mm-backend01.
> >
> > Although a RPM is available in Fedora I am not sure the RPM can be made
> > available in EPEL 7.
> >
> > RPM Fusion is using the Rust based generate-mirrorlist-cache for some
> > months already and I do not see any problems with it.
> >
> >
> We are not wanting to deploy new EL7 systems but would probably install an
> EL8 box for this. Does this change the need for moving to Fedora on it?

I just asked on #fedora-rust, but it seems it is not easily possible to
build the Fedora Rust packages for EL8. If I am understanding it
correctly it seems we need to run the Rust based mirrorlist cache
generation on a Fedora host. If we have a second mm-backend system
(mm-backend02) that is Fedora based to generate the mirrorlist cache we
could decrease the amount of RAM (32GB) on mm-backend01 to something
like 8GB.

Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Another Rust MirrorManager experiment

[Adrian Reber]

On Wed, Jul 01, 2020 at 09:16:05AM -0700, Kevin Fenzi wrote:
> On Tue, Jun 30, 2020 at 02:17:30PM +0200, Adrian Reber wrote:
> > On Mon, Jun 15, 2020 at 03:36:23PM -0700, Kevin Fenzi wrote:
> > > On Wed, Jun 10, 2020 at 11:09:49AM +0200, Adrian Reber wrote:
> > > > 
> > > > Then I just have to wait a bit. No problem.
> > > > 
> > > > > > Having the possibility to generate the mirrorlist input data in 
> > > > > > about a
> > > > > > minute would significantly reduce the load on the database server 
> > > > > > and
> > > > > > enable us to react much faster if broken protobuf data has been 
> > > > > > synced
> > > > > > to the mirrorlist servers on the proxies.
> > > > > 
> > > > > Yeah, and I wonder if it would let us revisit the entire sequence from
> > > > > 'update push finished' to updated mirrorlist server. 
> > > 
> > > This would help us with the case of: 
> > > - updates push/rawhide finishes, master mirror is updated. 
> > > - openqa/other internal thing tries to get images or updates in that
> > >   change and gets a metalink with the old checksum so it can't get the
> > >   new stuff.
> > > - mm-backend01 generates and pushes out a new protobuf.
> > > > 
> > > > Probably. As the new code will not run on the current RHEL 7 based
> > > > mm-backend01 would it make sense to run a short running service like
> > > > this on Fedora's OpenShift? We could also create a new read-only (SELECT
> > > > only) database account for this.
> > > 
> > > We could, or as smooge suggests make a mm-backend02? 
> > > 
> > > But I guess now mm-backend02 just generates new proobuf files and copies
> > > them to mirrorlists? If thats all it's doing, perhaps we could indeed
> > > replace it with an openshift project. 
> > 
> > We need a system to run the tool and copy the data to all proxies.
> > 
> > I would like to see a new MirrorManager database user who can only do
> > selects as that is all we need.
> > 
> > Currently we copy the files via SSH to the proxies, if we continue doing
> > it that way, then we would also need the existing SSH key to copy the
> > data to the proxies.
> > 
> > Easiest would probably be a small Fedora 32 based VM with 2GB of memory.
> 
> I'm not sure f32 will work with 2gb memory anymore. I dont think it
> installs at any rate. 
> 
> I do like the idea of just making it an openshift pod. Perhaps this
> could even fit with pingous 'toddlers' setup. ie: 

I tried to create a toddler, but that setup is too complicated for me.
Especially if something is not working it will be almost impossible for
me to debug it if it is running somewhere I cannot reach via SSH.

I just tried to build the generate-mirrorlist-cache on RHEL 7 (using
Rust from EPEL) and it works fine. Instead of 20 minutes it needs 30
seconds to generate the mirrorlist cache file on mm-backend01.

Although a RPM is available in Fedora I am not sure the RPM can be made
available in EPEL 7. 

RPM Fusion is using the Rust based generate-mirrorlist-cache for some
months already and I do not see any problems with it.

I would like to use this also for Fedora and a dedicated Fedora based VM
or building from source on RHEL 7 is both possible. Building from source
does not sound like something used in Fedora Infrastructure so setting
up a Fedora based VM would be necessary. Would that be possible?

Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Another Rust MirrorManager experiment

[Adrian Reber]

On Thu, Jul 02, 2020 at 09:44:10AM +0200, Pierre-Yves Chibon wrote:
> On Wed, Jul 01, 2020 at 06:25:47PM +0200, Adrian Reber wrote:
> > On Wed, Jul 01, 2020 at 09:16:05AM -0700, Kevin Fenzi wrote:
> > > On Tue, Jun 30, 2020 at 02:17:30PM +0200, Adrian Reber wrote:
> > > > On Mon, Jun 15, 2020 at 03:36:23PM -0700, Kevin Fenzi wrote:
> > > > > On Wed, Jun 10, 2020 at 11:09:49AM +0200, Adrian Reber wrote:
> > > > > > 
> > > > > > Then I just have to wait a bit. No problem.
> > > > > > 
> > > > > > > > Having the possibility to generate the mirrorlist input data in 
> > > > > > > > about a
> > > > > > > > minute would significantly reduce the load on the database 
> > > > > > > > server and
> > > > > > > > enable us to react much faster if broken protobuf data has been 
> > > > > > > > synced
> > > > > > > > to the mirrorlist servers on the proxies.
> > > > > > > 
> > > > > > > Yeah, and I wonder if it would let us revisit the entire sequence 
> > > > > > > from
> > > > > > > 'update push finished' to updated mirrorlist server. 
> > > > > 
> > > > > This would help us with the case of: 
> > > > > - updates push/rawhide finishes, master mirror is updated. 
> > > > > - openqa/other internal thing tries to get images or updates in that
> > > > >   change and gets a metalink with the old checksum so it can't get the
> > > > >   new stuff.
> > > > > - mm-backend01 generates and pushes out a new protobuf.
> > > > > > 
> > > > > > Probably. As the new code will not run on the current RHEL 7 based
> > > > > > mm-backend01 would it make sense to run a short running service like
> > > > > > this on Fedora's OpenShift? We could also create a new read-only 
> > > > > > (SELECT
> > > > > > only) database account for this.
> > > > > 
> > > > > We could, or as smooge suggests make a mm-backend02? 
> > > > > 
> > > > > But I guess now mm-backend02 just generates new proobuf files and 
> > > > > copies
> > > > > them to mirrorlists? If thats all it's doing, perhaps we could indeed
> > > > > replace it with an openshift project. 
> > > > 
> > > > We need a system to run the tool and copy the data to all proxies.
> > > > 
> > > > I would like to see a new MirrorManager database user who can only do
> > > > selects as that is all we need.
> > > > 
> > > > Currently we copy the files via SSH to the proxies, if we continue doing
> > > > it that way, then we would also need the existing SSH key to copy the
> > > > data to the proxies.
> > > > 
> > > > Easiest would probably be a small Fedora 32 based VM with 2GB of memory.
> > > 
> > > I'm not sure f32 will work with 2gb memory anymore. I dont think it
> > > installs at any rate. 
> > > 
> > > I do like the idea of just making it an openshift pod. Perhaps this
> > > could even fit with pingous 'toddlers' setup. ie: 
> > > 
> > > * listen for message saying a repo has updated
> > > * update the db
> > > * create the protobuf
> > > * push out to proxies
> > > 
> > > The only weird part of putting it in openshift is that we would need to
> > > have fedora_ftp (ro) there available as a volume, but that is doable... 
> > 
> > No, this part only needs to talk read-only to the database. This is not
> > touching anything on the disk besides writing the output. I guess you
> > were thinking about the umdl (update-master-directory-listing) part.
> > That would need ro access to the file-system.
> > 
> > The part I am talking about just reads the database and creates a
> > protobuf snapshot of the database which is then used by the mirrorlist
> > servers on the proxies.
> > 
> > Currently it runs once every hour. Which works pretty good so far.
> > Triggering it on a message makes only limited sense as it depends on the
> > results of the crawler. We could run it twice an hour to have newer
> > database snapshots on the proxies.
> > 
> > How can I prepare it for running in openshift. Can I use the
> > configuration for toddlers? Where can I find that?
> 
> Toddlers has rece

Re: Another Rust MirrorManager experiment

[Adrian Reber]

On Wed, Jul 01, 2020 at 09:16:05AM -0700, Kevin Fenzi wrote:
> On Tue, Jun 30, 2020 at 02:17:30PM +0200, Adrian Reber wrote:
> > On Mon, Jun 15, 2020 at 03:36:23PM -0700, Kevin Fenzi wrote:
> > > On Wed, Jun 10, 2020 at 11:09:49AM +0200, Adrian Reber wrote:
> > > > 
> > > > Then I just have to wait a bit. No problem.
> > > > 
> > > > > > Having the possibility to generate the mirrorlist input data in 
> > > > > > about a
> > > > > > minute would significantly reduce the load on the database server 
> > > > > > and
> > > > > > enable us to react much faster if broken protobuf data has been 
> > > > > > synced
> > > > > > to the mirrorlist servers on the proxies.
> > > > > 
> > > > > Yeah, and I wonder if it would let us revisit the entire sequence from
> > > > > 'update push finished' to updated mirrorlist server. 
> > > 
> > > This would help us with the case of: 
> > > - updates push/rawhide finishes, master mirror is updated. 
> > > - openqa/other internal thing tries to get images or updates in that
> > >   change and gets a metalink with the old checksum so it can't get the
> > >   new stuff.
> > > - mm-backend01 generates and pushes out a new protobuf.
> > > > 
> > > > Probably. As the new code will not run on the current RHEL 7 based
> > > > mm-backend01 would it make sense to run a short running service like
> > > > this on Fedora's OpenShift? We could also create a new read-only (SELECT
> > > > only) database account for this.
> > > 
> > > We could, or as smooge suggests make a mm-backend02? 
> > > 
> > > But I guess now mm-backend02 just generates new proobuf files and copies
> > > them to mirrorlists? If thats all it's doing, perhaps we could indeed
> > > replace it with an openshift project. 
> > 
> > We need a system to run the tool and copy the data to all proxies.
> > 
> > I would like to see a new MirrorManager database user who can only do
> > selects as that is all we need.
> > 
> > Currently we copy the files via SSH to the proxies, if we continue doing
> > it that way, then we would also need the existing SSH key to copy the
> > data to the proxies.
> > 
> > Easiest would probably be a small Fedora 32 based VM with 2GB of memory.
> 
> I'm not sure f32 will work with 2gb memory anymore. I dont think it
> installs at any rate. 
> 
> I do like the idea of just making it an openshift pod. Perhaps this
> could even fit with pingous 'toddlers' setup. ie: 
> 
> * listen for message saying a repo has updated
> * update the db
> * create the protobuf
> * push out to proxies
> 
> The only weird part of putting it in openshift is that we would need to
> have fedora_ftp (ro) there available as a volume, but that is doable... 

No, this part only needs to talk read-only to the database. This is not
touching anything on the disk besides writing the output. I guess you
were thinking about the umdl (update-master-directory-listing) part.
That would need ro access to the file-system.

The part I am talking about just reads the database and creates a
protobuf snapshot of the database which is then used by the mirrorlist
servers on the proxies.

Currently it runs once every hour. Which works pretty good so far.
Triggering it on a message makes only limited sense as it depends on the
results of the crawler. We could run it twice an hour to have newer
database snapshots on the proxies.

How can I prepare it for running in openshift. Can I use the
configuration for toddlers? Where can I find that?

Adrian
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Another Rust MirrorManager experiment

[Adrian Reber]

On Mon, Jun 15, 2020 at 03:36:23PM -0700, Kevin Fenzi wrote:
> On Wed, Jun 10, 2020 at 11:09:49AM +0200, Adrian Reber wrote:
> > 
> > Then I just have to wait a bit. No problem.
> > 
> > > > Having the possibility to generate the mirrorlist input data in about a
> > > > minute would significantly reduce the load on the database server and
> > > > enable us to react much faster if broken protobuf data has been synced
> > > > to the mirrorlist servers on the proxies.
> > > 
> > > Yeah, and I wonder if it would let us revisit the entire sequence from
> > > 'update push finished' to updated mirrorlist server. 
> 
> This would help us with the case of: 
> - updates push/rawhide finishes, master mirror is updated. 
> - openqa/other internal thing tries to get images or updates in that
>   change and gets a metalink with the old checksum so it can't get the
>   new stuff.
> - mm-backend01 generates and pushes out a new protobuf.
> > 
> > Probably. As the new code will not run on the current RHEL 7 based
> > mm-backend01 would it make sense to run a short running service like
> > this on Fedora's OpenShift? We could also create a new read-only (SELECT
> > only) database account for this.
> 
> We could, or as smooge suggests make a mm-backend02? 
> 
> But I guess now mm-backend02 just generates new proobuf files and copies
> them to mirrorlists? If thats all it's doing, perhaps we could indeed
> replace it with an openshift project. 

We need a system to run the tool and copy the data to all proxies.

I would like to see a new MirrorManager database user who can only do
selects as that is all we need.

Currently we copy the files via SSH to the proxies, if we continue doing
it that way, then we would also need the existing SSH key to copy the
data to the proxies.

Easiest would probably be a small Fedora 32 based VM with 2GB of memory.

Adrian
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: mirrorlist-server fix is in testing

[Adrian Reber]

On Thu, Jun 25, 2020 at 04:02:35PM -0700, Kevin Fenzi wrote:
> On Wed, Jun 24, 2020 at 06:06:26PM +0200, Adrian Reber wrote:
> > Currently there is a bug open against our mirrorlist servers that
> > certain locations have trouble, especially with the
> > fedora-cisco-openh264 repository, getting a list of up to date mirrors.
> > 
> > https://bugzilla.redhat.com/show_bug.cgi?id=1844087
> > 
> > This has been fixed upstream and is available from testing:
> > 
> > https://bodhi.fedoraproject.org/updates/FEDORA-2020-f86d3b3196
> > 
> > As Fedora Infrastructure is, so far, the only user of this package
> > nobody besides us can/will test it.
> > 
> > Can we install it on one of our proxies and see if it still works and if
> > it actually fixes the problems which have been reported?
> 
> Yep. 
> 
> > I cannot install anything on the proxies myself, so somebody would need
> > to install that package and restart the corresponding services and let
> > me know. There is no staging, but maybe there are proxies which are
> > currently not active which could be used for a test like this.
> 
> I installed it on proxy05 (since thats a slower/lower resource one) 
> lets let it run there for a few days at least. 
> 
> It seems to be running normally from what I can see so far.. 

Thanks. Seems to work as before and the error is also fixed:

$ curl 
"127.0.0.1:18081/mirrorlist?repo=fedora-cisco-openh264-32=x86_64=141.58.0.0"
# repo = fedora-cisco-openh264-32 arch = x86_64 Using ASN 553 country = global 
https://codecs.fedoraproject.org/openh264/32/x86_64/

The update will reach stable in one day according to bodhi, so we can
install it soon on all proxies.

Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

mirrorlist-server fix is in testing

[Adrian Reber]

Currently there is a bug open against our mirrorlist servers that
certain locations have trouble, especially with the
fedora-cisco-openh264 repository, getting a list of up to date mirrors.

https://bugzilla.redhat.com/show_bug.cgi?id=1844087

This has been fixed upstream and is available from testing:

https://bodhi.fedoraproject.org/updates/FEDORA-2020-f86d3b3196

As Fedora Infrastructure is, so far, the only user of this package
nobody besides us can/will test it.

Can we install it on one of our proxies and see if it still works and if
it actually fixes the problems which have been reported?

I cannot install anything on the proxies myself, so somebody would need
to install that package and restart the corresponding services and let
me know. There is no staging, but maybe there are proxies which are
currently not active which could be used for a test like this.

Adrian
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Another Rust MirrorManager experiment

[Adrian Reber]

On Thu, Jun 04, 2020 at 07:02:41PM -0700, Kevin Fenzi wrote:
> On Mon, Jun 01, 2020 at 04:18:35PM +0200, Adrian Reber wrote:
> > Our MirrorManager setup exports the current state of all mirrors every
> > hour at :30 to a protobuf based file which is then used by the
> > mirrorlist servers to answer the requests from yum and dnf.
> > 
> > The Python script requires up to 10GB of memory and takes between 35 and
> > 50 minutes. The script does a lot of SQL queries and also some really
> > big SQL queries joining up to 6 large MirrorManager tables.
> > 
> > I have rewritten this Python script in Rust and now it only needs around
> > 1 minute instead of 35 to 50 minutes and only 600MB instead of 10GB.
> 
> Wow. nice!
> 
> > I think the biggest difference is that I am almost not doing any joins
> > in my SQL request. I download all the tables once and then I do a lot of
> > loops over the downloaded tables and this seems to be massively faster.
> > 
> > As the mirrorlist-server in Rust has proven to be extremely stable over
> > the last months we have been using it I would also like to replace the
> > mirrorlist protbuf input generation with my new Rust based code.
> > 
> > I am planing to try out the new protobuf file in staging in the next
> > days and would then try to get my new protobuf generation program into
> > Fedora. Once it is packaged I would discuss here how and if we want to
> > deploy in Fedora's infrastructure.
> 
> Cool. You will need to hurry as staging goes off on monday, and back in
> a few weeks. :) 

Then I just have to wait a bit. No problem.

> > Having the possibility to generate the mirrorlist input data in about a
> > minute would significantly reduce the load on the database server and
> > enable us to react much faster if broken protobuf data has been synced
> > to the mirrorlist servers on the proxies.
> 
> Yeah, and I wonder if it would let us revisit the entire sequence from
> 'update push finished' to updated mirrorlist server. 

Probably. As the new code will not run on the current RHEL 7 based
mm-backend01 would it make sense to run a short running service like
this on Fedora's OpenShift? We could also create a new read-only (SELECT
only) database account for this.

Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Another Rust MirrorManager experiment

[Adrian Reber]

Our MirrorManager setup exports the current state of all mirrors every
hour at :30 to a protobuf based file which is then used by the
mirrorlist servers to answer the requests from yum and dnf.

The Python script requires up to 10GB of memory and takes between 35 and
50 minutes. The script does a lot of SQL queries and also some really
big SQL queries joining up to 6 large MirrorManager tables.

I have rewritten this Python script in Rust and now it only needs around
1 minute instead of 35 to 50 minutes and only 600MB instead of 10GB.

I think the biggest difference is that I am almost not doing any joins
in my SQL request. I download all the tables once and then I do a lot of
loops over the downloaded tables and this seems to be massively faster.

As the mirrorlist-server in Rust has proven to be extremely stable over
the last months we have been using it I would also like to replace the
mirrorlist protbuf input generation with my new Rust based code.

I am planing to try out the new protobuf file in staging in the next
days and would then try to get my new protobuf generation program into
Fedora. Once it is packaged I would discuss here how and if we want to
deploy in Fedora's infrastructure.

Having the possibility to generate the mirrorlist input data in about a
minute would significantly reduce the load on the database server and
enable us to react much faster if broken protobuf data has been synced
to the mirrorlist servers on the proxies.

Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: IAD2 datacenter testing/application validation

[Adrian Reber]

On Tue, May 26, 2020 at 03:22:02PM -0700, Kevin Fenzi wrote:
> Greetings everyone. 
> 
> We now have most things deployed in our new datacenter (IAD2). 
> 
> Accordingly, I would like to get some testing and validation started.
> 
> Please see the following document:
> 
> https://hackmd.io/op6N_nIaR7aMzw9Ib-sDAQ
> 
> Please feel free to ask questions here and add/check off services that
> appear to be running as expected. Application owners (people in
> sysadmin-whatever groups) should check that they have the expected level
> of access as before, that their playbooks run (idempotently!) and the
> logs and any interfaces show the application appears to be running fine. 
> 
> We have this week and next to get everything in good shape for migration
> week (20202-06-08) when we will moving everything out of phx2. 

I had a look at mm-backend01.iad2 and it seems work. Unfortunately it
does seem to work better than it should, because it is syncing out data
to the proxies. So now we have production data synced to the proxies
from phx2 but also data synced out from iad2, which is probably not as
up to date as phx2. I stopped cron on mm-backend01.iad2 for now to avoid
this.

Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Issue with Fedora GeoIP service

[Adrian Reber]

On Wed, Dec 18, 2019 at 01:29:03PM +0100, Martin Kolman wrote:
> On Fri, 2019-12-13 at 09:21 -0800, Kevin Fenzi wrote:
> > On Fri, Dec 13, 2019 at 08:00:01AM +0100, Adrian Reber wrote:
> > > On Mon, Dec 09, 2019 at 12:19:17PM -0800, Kevin Fenzi wrote:
> > > > On Sat, Dec 07, 2019 at 09:39:46AM -0500, Kevin Sandy wrote:
> > > > > I have an updated copy working with geoip2, with the exception of 
> > > > > area code data (which isn’t included in the
> > > > > new database). But if I’m understanding it’s purpose correctly, 
> > > > > telephone area code isn’t needed. I need to do
> > > > > some testing with IPs around the world to ensure that the data is 
> > > > > sane outside the US; once I’ve had a chance to
> > > > > do that I’ll submit a PR on GitHub.
> > > > 
> > > > Wow. Awesome work!
> > > > 
> > > > Thanks for diving in and getting this going. 
> > > > 
> > > > In another email thread someone, Adrian said he had some code to do
> > > > this, but if you already have it working, perhaps he could review your
> > > > code/setup and we could then push it into ansible?
> > > 
> > > I had a look at the PR 
> > > (https://github.com/fedora-infra/geoip-city-wsgi/pull/1)
> > > and it works and the output is almost the same. Maybe one of the known
> > > data consumers can have a short look if those subtle changes in the
> > > output are not a problem.
> > 
> > The main user I know of is anaconda and it sounds like the changes work
> > for them. 
> > 
> > I guess I'd say we should pick a date (likely after the new year now), 
> > announce we are going to be changing it then a few times, and then on
> > that date cut over to the new system.
> Sounds good to me (as an Anaconda developer)! :)
> 
> BTW, will the new API be available somewhere before that (stagin, etc.) ? 
> 
> We could patch Anaconda to talk to the new API & do some smoke testing,
> just in case. :)

We finally pushed it to Fedora's staging environment. Please try it
using https://geoip.stg.fedoraproject.org/city if you have a chance.

Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Update geoip-city

[Adrian Reber]

We tried it and it failed because python-iso3116 (a new dependency) is
not available from EPEL. Will retry once we have a solution for iso3116.

Adrian

On Tue, Mar 31, 2020 at 07:23:56AM -0400, Stephen John Smoogen wrote:
> Let me know when you want to run the playbook.
> 
> On Tue, 31 Mar 2020 at 05:45, Adrian Reber  wrote:
> 
> > Attached is the patch to take the upstream update of geoip-city to
> > Fedora infrastructure.
> >
> > I would like to try this first in staging and need some help to run the
> > corresponding playbook to see if it works in our environment.
> >
> > Adrian
> > ___
> > infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> > To unsubscribe send an email to
> > infrastructure-le...@lists.fedoraproject.org
> > Fedora Code of Conduct:
> > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
> > https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
> >
> 
> 
> -- 
> Stephen J Smoogen.

> ___
> infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
> Fedora Code of Conduct: 
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org



signature.asc
Description: PGP signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Update geoip-city

[Adrian Reber]

Attached is the patch to take the upstream update of geoip-city to
Fedora infrastructure.

I would like to try this first in staging and need some help to run the
corresponding playbook to see if it works in our environment.

Adrian
>From 59d639cbdc360d81ccb28f617aa58250e79202ef Mon Sep 17 00:00:00 2001
From: Adrian Reber 
Date: Tue, 31 Mar 2020 11:40:40 +0200
Subject: [PATCH] geoip-city-wsgi: update geoip-city-wsgi

The geoip-city-wsgi script has been updated upstream to support geoip2
and Python 3. This pulls the update to Fedora infrastructure.

Signed-off-by: Adrian Reber 
---
 .../geoip-city-wsgi/app/files/geoip-city.wsgi | 42 +++
 roles/geoip-city-wsgi/app/tasks/main.yml  |  7 +---
 2 files changed, 35 insertions(+), 14 deletions(-)

diff --git a/roles/geoip-city-wsgi/app/files/geoip-city.wsgi 
b/roles/geoip-city-wsgi/app/files/geoip-city.wsgi
index 94358205a..dc5822c42 100755
--- a/roles/geoip-city-wsgi/app/files/geoip-city.wsgi
+++ b/roles/geoip-city-wsgi/app/files/geoip-city.wsgi
@@ -11,14 +11,14 @@
 #  accelerator) in front of the application server running this WSGI,
 #  to avoid looking "behind" the real client's own forward HTTP proxy.
 
-from string import zfill, atoi, strip, replace
 from paste.wsgiwrappers import *
-import GeoIP
+from iso3166 import countries
+import geoip2.database
+import geoip2.errors
 import json
 
 global gi
-gi = GeoIP.open("/usr/share/GeoIP/GeoLiteCity.dat", GeoIP.GEOIP_STANDARD)
-gi.set_charset(GeoIP.GEOIP_CHARSET_UTF8)
+gi = geoip2.database.Reader("/usr/share/GeoIP/GeoLite2-City.mmdb")
 
 
 def real_client_ip(xforwardedfor):
@@ -32,18 +32,18 @@ def get_client_ip(environ, request):
 request_data = request.GET
 
 if 'ip' in request_data:
-client_ip = strip(request_data['ip'])
+client_ip = request_data['ip'].strip()
 elif 'X-Forwarded-For' in request.headers and 'geoip_city.noreverseproxy' 
not in environ:
-client_ip = real_client_ip(strip(request.headers['X-Forwarded-For']))
+client_ip = real_client_ip(request.headers['X-Forwarded-For'].strip())
 else:
 client_ip = request.environ['REMOTE_ADDR']
 
-client_ip = unicode(client_ip, 'utf8', 'replace')
 return client_ip
 
 def application(environ, start_response):
 request = WSGIRequest(environ)
 response = WSGIResponse()
+results = {}
 code = 500
 
 try:
@@ -51,15 +51,39 @@ def application(environ, start_response):
 if client_ip is None:
 code = 400
 raise Exception
-results =  gi.record_by_addr(client_ip)
-if results is None:
+data = gi.city(client_ip)
+if data is None:
 code = 404
 raise Exception
+except geoip2.errors.AddressNotFoundError:
+response.status_code = 404
+return response(environ, start_response)
 except: 
 response.status_code=code
 return response(environ, start_response)
 
 results['ip'] = client_ip
+
+# map geoip2 data to a structure that matches the prior geoip format
+results['city'] = data.city.name
+results['region_name']  = data.subdivisions.most_specific.name
+results['region']   = data.subdivisions.most_specific.iso_code
+results['postal_code']  = data.postal.code
+results['country_name'] = data.country.name
+results['country_code'] = data.country.iso_code
+results['time_zone']= data.location.time_zone
+results['latitude'] = data.location.latitude
+results['longitude']= data.location.longitude
+results['metro_code']   = data.location.metro_code
+results['dma_code'] = data.location.metro_code
+
+# geoip2 no longer includes country_code3, so it has to be pulled
+# from iso3166.countries
+if data.country.iso_code in countries:
+results['country_code3'] = countries[data.country.iso_code].alpha3
+else:
+results['country_code3'] = None
+
 results = json.dumps(results)
 response.headers['Content-Length'] = str(len(results))
 response.write(results)
diff --git a/roles/geoip-city-wsgi/app/tasks/main.yml 
b/roles/geoip-city-wsgi/app/tasks/main.yml
index 3f0fea39e..29bdcd865 100644
--- a/roles/geoip-city-wsgi/app/tasks/main.yml
+++ b/roles/geoip-city-wsgi/app/tasks/main.yml
@@ -11,16 +11,13 @@
   - geoip-city-wsgi
   - geoip-city-wsgi/app
 
-- name: install GeoIP-data
-  package: name=GeoIP-data state=present
+- name: install geolite2-city
+  package: name=geolite2-city state=present
   tags:
   - packages
   - geoip-city-wsgi
   - geoip-city-wsgi/app
 
-- name: link the data file to the one we use
-  file: state=link dest=/usr/share/GeoIP/GeoIPCity.dat 
src=/usr/share/GeoIP/GeoLiteCity.dat
-
 - name: install geoip-city-wsgi.conf file
   copy: >
 src="geoip-city-wsgi.conf"
-- 
2.24.1

___
infrastructure mailing list -- infrastructure@lists.

Re: [PATCH] s3-mirror: sync s3-sync-path script with ideas from s3.sh

[Adrian Reber]

On Fri, Mar 27, 2020 at 05:14:48PM -0700, Kevin Fenzi wrote:
> On Fri, Mar 27, 2020 at 12:06:22PM +0100, Pavel Raiskup wrote:
> > On Friday, March 27, 2020 11:34:28 AM CET Adrian Reber wrote:
> > > The global 'exclude' has '--exclude "*/repodata/*"' and you are using
> > > "${excludes[@]}" everywhere. In all three syncs. This looks like
> > > 'repodata/*' will never synced.
> > 
> > Good catch!  Thank you for the review.  I am attaching updated patch.
> > 
> > Pavel
> 
> Looks ok to me from a quick glance... lets give it a try. :) 
> 
> I think also we should remove the test releases one. It's currently
> failing the invalidate step (because it's repodata is not in the same
> place as the updates repos) and the only reason we would want say
> 32_Beta to be there is so we could point people to download isos from
> it. The repodata will never change. 

Just to confirm from the MirrorManager side. MirrorManager is not aware
of any repomd.xml files under releases/test. For 32 MirrorManager points
to development/32 or updates/32 and updates/testing/32.

So MirrorManager will not redirect any clients to releases/test except
for ISOs.

Adrian
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: [PATCH] s3-mirror: sync s3-sync-path script with ideas from s3.sh

[Adrian Reber]

On Fri, Mar 27, 2020 at 05:14:48PM -0700, Kevin Fenzi wrote:
> On Fri, Mar 27, 2020 at 12:06:22PM +0100, Pavel Raiskup wrote:
> > On Friday, March 27, 2020 11:34:28 AM CET Adrian Reber wrote:
> > > The global 'exclude' has '--exclude "*/repodata/*"' and you are using
> > > "${excludes[@]}" everywhere. In all three syncs. This looks like
> > > 'repodata/*' will never synced.
> > 
> > Good catch!  Thank you for the review.  I am attaching updated patch.
> > 
> > Pavel
> 
> Looks ok to me from a quick glance... lets give it a try. :) 
> 
> I think also we should remove the test releases one. It's currently
> failing the invalidate step (because it's repodata is not in the same
> place as the updates repos) and the only reason we would want say
> 32_Beta to be there is so we could point people to download isos from
> it. The repodata will never change. 

Accessing cloudfront I can now see the correct header:

cache-control: max-age=60

and the age header never gets larger than 60

I can see 'age: 59', but each request after that gives me an

x-cache: RefreshHit from cloudfront

From the header returned by cloudfront it seems we are now never seeing
repomd.xml files older than 1 minute.

I would like to turn on the cloudfront mirror again to see if COPR still
breaks. Any objections?

Adrian
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: [PATCH] s3-mirror: sync s3-sync-path script with ideas from s3.sh

[Adrian Reber]

On Fri, Mar 27, 2020 at 12:06:22PM +0100, Pavel Raiskup wrote:
> On Friday, March 27, 2020 11:34:28 AM CET Adrian Reber wrote:
> > The global 'exclude' has '--exclude "*/repodata/*"' and you are using
> > "${excludes[@]}" everywhere. In all three syncs. This looks like
> > 'repodata/*' will never synced.
> 
> Good catch!  Thank you for the review.  I am attaching updated patch.

Looks good. I have not verified everything again, but the excludes are
looking correct now.

Looking forward to see if this improves the caching behaviour of repomd.xml.

Adrian

> >From 9bad0d6a2dcf48b1d611290e11e1ad38c27a6eff Mon Sep 17 00:00:00 2001
> From: Pavel Raiskup 
> Date: Fri, 27 Mar 2020 09:16:12 +0100
> Subject: [PATCH] s3-mirror: sync s3-sync-path script with ideas from s3.sh
> 
> 1. sync everything except for repomd.xml
> 2. then sync repomd.xml files only, and invalidate caches
> 3. gently wait a bit to give current downloads a chance
> 4. delete outdated RPMs and metadata, shouldn't be needed
> 
> Also make the sleep/cache configurable.
> ---
>  roles/s3-mirror/files/s3-sync-path.sh | 98 ++-
>  roles/s3-mirror/files/s3.sh   | 19 --
>  2 files changed, 64 insertions(+), 53 deletions(-)
> 
> diff --git a/roles/s3-mirror/files/s3-sync-path.sh 
> b/roles/s3-mirror/files/s3-sync-path.sh
> index 79b4d63eb..4913767f7 100644
> --- a/roles/s3-mirror/files/s3-sync-path.sh
> +++ b/roles/s3-mirror/files/s3-sync-path.sh
> @@ -9,58 +9,64 @@ if [[ "$1" == "" ]] || [[ $1 != /pub* ]] || [[ $1 != */ ]]; 
> then
>exit 1
>  fi
>  
> +aws_sync=( aws s3 sync --no-follow-symlinks )
> +
>  # first run do not delete anything or copy the repodata.
> -CMD1="aws s3 sync   \
> -  --exclude */repodata/* \
> -  --exclude *.snapshot/*  \
> -  --exclude *source/* \
> -  --exclude *SRPMS/*  \
> -  --exclude *debug/*  \
> -  --exclude *beta/*   \
> -  --exclude *ppc/*\
> -  --exclude *ppc64/*  \
> -  --exclude *repoview/*   \
> -  --exclude *Fedora/* \
> -  --exclude *EFI/*\
> -  --exclude *core/*   \
> -  --exclude *extras/* \
> -  --exclude *LiveOS/* \
> -  --exclude *development/rawhide/* \
> -  --no-follow-symlinks\
> -  --only-show-errors  \
> -  "
> -  #--dryrun \
> +exclude=(
> +  --exclude "*.snapshot/*"
> +  --exclude "*source/*"
> +  --exclude "*SRPMS/*"
> +  --exclude "*debug/*"
> +  --exclude "*beta/*"
> +  --exclude "*ppc/*"
> +  --exclude "*ppc64/*"
> +  --exclude "*repoview/*"
> +  --exclude "*Fedora/*"
> +  --exclude "*EFI/*"
> +  --exclude "*core/*"
> +  --exclude "*extras/*"
> +  --exclude "*LiveOS/*"
> +  --exclude "*development/rawhide/*"
> +  --only-show-errors
> +)
>  
> -# second we delete old content and also copy the repodata
> -CMD2="aws s3 sync   \
> -  --delete \
> -  --exclude *.snapshot/*  \
> -  --exclude *source/* \
> -  --exclude *SRPMS/*  \
> -  --exclude *debug/*  \
> -  --exclude *beta/*   \
> -  --exclude *ppc/*\
> -  --exclude *ppc64/*  \
> -  --exclude *repoview/*   \
> -  --exclude *Fedora/* \
> -  --exclude *EFI/*\
> -  --exclude *core/*   \
> -  --exclude *extras/* \
> -  --exclude *LiveOS/* \
> -  --exclude *development/rawhide/* \
> -  --no-follow-symlinks\
> -  --only-show-errors  \
> -  "
> -  #--dryrun \
> +S3_MIRROR=s3-mirror-us-west-1-02.fedoraproject.org
> +DIST_ID=E2KJMDC0QAJDMU
> +MAX_CACHE_SEC=60
> +DNF_GENTLY_TIMEOUT=120
> +
> +# First run this command that syncs, but does not delete.
> +# It also excludes repomd.xml.
> +CMD1=( "${aws_sync[@]}" "${excludes[@]}" --exclude "*/repomd.xml" )
> +
> +# Next we run this command which syncs repomd.xml files.  Include must 
> precede
> +# the large set of excludes.  Make sure that the 'max-age' isn't too large so
> +# we know that we can start removing old data ASAP.
> +CMD2=( "${aws_sync[@]}" --exclude "*" --include "*/repomd.xml" 
> "${excludes[@]}"
> +--cache-control "max-age=$MAX_CACHE_SEC" )
> 

Re: [PATCH] s3-mirror: sync s3-sync-path script with ideas from s3.sh

[Adrian Reber]

repodata/
> +"${CMD1[@]}" "/srv$1" "s3://$S3_MIRROR$1"
> +"${CMD2[@]}" "/srv$1" "s3://$S3_MIRROR$1"
> +
>  # Always do the invalidations because they are quick and prevent issues
>  # depending on which path is synced.
> -for file in $(echo $1/repodata/* ); do
> -  aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU 
> --paths "$file" > /dev/null
> +for file in $(echo $1/repodata/repomd.xml ); do
> +  aws cloudfront create-invalidation --distribution-id $DIST_ID --paths 
> "$file" > /dev/null
>  done
> -$CMD2 /srv$1 s3://s3-mirror-us-west-1-02.fedoraproject.org$1
> +
> +SLEEP=$(( MAX_CACHE_SEC + DNF_GENTLY_TIMEOUT ))
> +echo "Ready $1 sync, giving dnf downloads ${SLEEP}s before delete, at 
> $(date)" >> /var/log/s3-mirror/timestamps
> +
> +# Consider some DNF processes started downloading metadata before we 
> invalidated
> +# caches, and started with outdated repomd.xml file.  Give it few more 
> seconds
> +# so they have chance to download the rest of metadata and RPMs.
> +sleep $SLEEP
> +
> +"${CMD3[@]}" "/srv$1" "s3://$S3_MIRROR$1"
> +
>  echo "Ending $1 sync at $(date)" >> /var/log/s3-mirror/timestamps
> diff --git a/roles/s3-mirror/files/s3.sh b/roles/s3-mirror/files/s3.sh
> index c157b0cdb..df58ac153 100644
> --- a/roles/s3-mirror/files/s3.sh
> +++ b/roles/s3-mirror/files/s3.sh
> @@ -88,6 +88,11 @@ excludes=(
>--exclude "*/updates/testing/29/*"
>  )
>  
> +S3_MIRROR=s3-mirror-us-west-1-02.fedoraproject.org
> +DIST_ID=E2KJMDC0QAJDMU
> +MAX_CACHE_SEC=60
> +DNF_GENTLY_TIMEOUT=120
> +
>  # First run this command that syncs, but does not delete.
>  # It also excludes repomd.xml.
>  CMD1=( "${aws_sync[@]}" "${excludes[@]}" --exclude "*/repomd.xml" )
> @@ -95,14 +100,12 @@ CMD1=( "${aws_sync[@]}" "${excludes[@]}" --exclude 
> "*/repomd.xml" )
>  # Next we run this command which syncs repomd.xml files.  Include must 
> precede
>  # the large set of excludes.  Make sure that the 'max-age' isn't too large so
>  # we know that we can start removing old data ASAP.
> -CMD2=( "${aws_sync[@]}" --exclude "*" --include "*/repomd.xml" 
> "${excludes[@]}" --cache-control max-age=300 )
> +CMD2=( "${aws_sync[@]}" --exclude "*" --include "*/repomd.xml" 
> "${excludes[@]}"
> +--cache-control "max-age=$MAX_CACHE_SEC" )
>  
>  # Then we delete old RPMs and old metadata (but after invalidating caches).
>  CMD3=( "${aws_sync[@]}" "${excludes[@]}" --delete )
>  
> -S3_MIRROR=s3-mirror-us-west-1-02.fedoraproject.org
> -DIST_ID=E2KJMDC0QAJDMU
> -
>  # Sync EPEL
>  #echo $CMD /srv/pub/epel/ s3://$S3_MIRROR/pub/epel/
>  echo "Starting EPEL sync at $(date)" >> /var/log/s3-mirror/timestamps
> @@ -132,10 +135,12 @@ for file in $(echo 
> /srv/pub/fedora/linux/updates/*/*/*/repodata/repomd.xml | sed
>aws cloudfront create-invalidation --distribution-id "$DIST_ID" --paths 
> "$file"
>  done
>  
> +SLEEP=$(( MAX_CACHE_SEC + DNF_GENTLY_TIMEOUT ))
> +
>  # Consider some DNF processes started downloading metadata before we 
> invalidated
> -# caches, and started with outdated repomd.xml file.  Give it 10 minutes so 
> they
> -# have chance to download the rest of metadata and RPMs.
> -sleep 600
> +# caches, and started with outdated repomd.xml file.  Give it few more 
> seconds
> +# so they have chance to download the rest of metadata and RPMs.
> +sleep $SLEEP
>  
>  "${CMD3[@]}" /srv/pub/epel/ "s3://$S3_MIRROR/pub/epel/"
>  "${CMD3[@]}" /srv/pub/fedora/ s3://$S3_MIRROR/pub/fedora/
> -- 
> 2.25.1
> ___
> infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
> Fedora Code of Conduct: 
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Adrian

-- 
Adrian Reber http://lisas.de/~adrian/
QOTD:
"My life is a soap opera, but who gets the movie rights?"
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: [PATCH] s3-mirror: avoid potential unexpected shell-expansion, dedup

[Adrian Reber]

e )
> > +
> > +S3_MIRROR=s3-mirror-us-west-1-02.fedoraproject.org
> > +DIST_ID=E2KJMDC0QAJDMU
> >  
> >  # Sync EPEL
> > -#echo $CMD /srv/pub/epel/ 
> > s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/epel/
> > +#echo $CMD /srv/pub/epel/ s3://$S3_MIRROR/pub/epel/
> >  echo "Starting EPEL sync at $(date)" >> /var/log/s3-mirror/timestamps
> > -$CMD1 /srv/pub/epel/ 
> > s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/epel/
> > -$CMD2 /srv/pub/epel/ 
> > s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/epel/
> > +"${CMD1[@]}" /srv/pub/epel/ "s3://$S3_MIRROR/pub/epel/"
> > +"${CMD2[@]}" /srv/pub/epel/ "s3://$S3_MIRROR/pub/epel/"
> >  echo "Ending EPEL sync at $(date)" >> /var/log/s3-mirror/timestamps
> >  for file in $(echo /srv/pub/epel/6/*/repodata/repomd.xml | sed 
> > 's#/srv##g'); do
> > -  aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU 
> > --paths "$file"
> > +  aws cloudfront create-invalidation --distribution-id "$DIST_ID" --paths 
> > "$file"
> >  done
> >  
> >  for file in $(echo /srv/pub/epel/7/*/repodata/repomd.xml | sed 
> > 's#/srv##g'); do
> > -  aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU 
> > --paths "$file"
> > +  aws cloudfront create-invalidation --distribution-id "$DIST_ID" --paths 
> > "$file"
> >  done
> >  
> >  for file in $(echo /srv/pub/epel/8/*/repodata/repomd.xml | sed 
> > 's#/srv##g'); do
> > -  aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU 
> > --paths "$file"
> > +  aws cloudfront create-invalidation --distribution-id "$DIST_ID" --paths 
> > "$file"
> >  done
> > -$CMD2 --delete /srv/pub/epel/ 
> > s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/epel/
> > +"${CMD3[@]}" /srv/pub/epel/ "s3://$S3_MIRROR/pub/epel/"
> >  
> >  # Sync Fedora
> > -#echo $CMD /srv/pub/fedora/ 
> > s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/fedora/
> > +#echo $CMD /srv/pub/fedora/ s3://$S3_MIRROR/pub/fedora/
> >  echo "Starting Fedora sync at $(date)" >> /var/log/s3-mirror/timestamps
> > -$CMD1 /srv/pub/fedora/ 
> > s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/fedora/
> > -$CMD2 /srv/pub/fedora/ 
> > s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/fedora/
> > +"${CMD1[@]}" /srv/pub/fedora/ "s3://$S3_MIRROR/pub/fedora/"
> > +"${CMD2[@]}" /srv/pub/fedora/ "s3://$S3_MIRROR/pub/fedora/"
> >  echo "Ending Fedora sync at $(date)" >> /var/log/s3-mirror/timestamps
> >  
> >  for file in $(echo /srv/pub/fedora/linux/updates/*/*/*/repodata/repomd.xml 
> > | sed 's#/srv##g'); do
> > -  aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU 
> > --paths "$file"
> > +  aws cloudfront create-invalidation --distribution-id "$DIST_ID" --paths 
> > "$file"
> >  done
> > -$CMD2 --delete /srv/pub/fedora/ 
> > s3://s3-mirror-us-west-1-02.fedoraproject.org/pub/fedora/
> > +"${CMD3[@]}" /srv/pub/fedora/ s3://$S3_MIRROR/pub/fedora/
> > -- 
> > 2.25.1
> > ___
> > infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> > To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
> > Fedora Code of Conduct: 
> > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives: 
> > https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
> > 
> 
> 
> 
> ___
> infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
> Fedora Code of Conduct: 
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Adrian

-- 
Adrian Reber http://lisas.de/~adrian/
"It's hard to believe that something which is neither seen nor felt can
do so much harm."
"That's true.  But an idea can't be seen or felt.  And that's what kept
the Troglytes in the mines all these centuries.  A mistaken idea."
-- Vanna and Kirk, "The Cloud Minders", stardate 5819.0
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

MirrorManager upgraded to 0.13

[Adrian Reber]

I have upgraded MirrorManager in staging to 0.12, saw that the
webinterface broke, released 0.13, upgraded staging to 0.13 and after
seeing that everything still worked I also upgraded the production
instance.

It contains a couple of small changes which were already active (EPEL 8
modular repository support) in production, but also a security fix which
easily allowed viewing the report_mirror password of all other sites.

The biggest change is that I disabled report_mirror for public mirrors.
Private mirrors still need to run report_mirror, but for public mirrors
the information submitted via report_mirror is ignored. The main reason
for dropping report_mirror for public mirrors is that we regularly have
seen problems of mirrors being disabled by the crawler because they were
out of date and then being re-enabled by report_mirror. The problem was
that report_mirror only submits a list of up to date directories without
the actual content of the directories (to reduce I/O on the mirrors).

Another addition is that MirrorManager can now filter out paths which
should be ignored when detecting a version. MirrorManager regularly
detected new versions in pub/alt like 25757 or 25748 which were IoT test
trees or risc-v trees. So pub/alt is now filtered out when detecting new
versions.

Please let me know if anything breaks.

Adrian
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Freeze Break Request: s3-mirror adjustments

[Adrian Reber]

Looks correct from my point of view, for both scripts we are now doing:

1. sync everything besides repodata
2. sync everything including (or only) repodata
3. invalidate repodata
4. delete files

The only problem I can see now is that the files on the master are
changing between those steps. Not sure how often the master is updated,
but probably not more often than once or twice a day, right?

In theory this should help with the problems we have been seeing in
COPR.

+1

Adrian

On Fri, Mar 06, 2020 at 02:04:21PM -0800, Kevin Fenzi wrote:
> Final version? You be the judge. :) 
> 
> kevin
> --

> From bd3c100e3a5fdd453ebbd4b88cc3bba365d260c3 Mon Sep 17 00:00:00 2001
> From: Kevin Fenzi 
> Date: Thu, 5 Mar 2020 20:46:53 +
> Subject: [PATCH] s3-mirror: Split things into 2 sync runs, one without
>  repodata and delete, the other with both.
> 
> In order to make sure the s3 mirror always is consistent, split out the 
> commands
> to make it sync without repodata and delete, then do another run with those, 
> then finally
> invalidate all the repodata/* files.
> 
> Some of the cron jobs are adjusted to allow the repodata invalidation.
> 
> Signed-off-by: Kevin Fenzi 
> ---
>  roles/s3-mirror/files/s3-sync-path.sh |  42 +
>  roles/s3-mirror/files/s3.sh   | 114 
> --
>  roles/s3-mirror/tasks/main.yml|   4 +-
>  3 files changed, 140 insertions(+), 20 deletions(-)
> 
> diff --git a/roles/s3-mirror/files/s3-sync-path.sh 
> b/roles/s3-mirror/files/s3-sync-path.sh
> index e6ac994..79b4d63 100644
> --- a/roles/s3-mirror/files/s3-sync-path.sh
> +++ b/roles/s3-mirror/files/s3-sync-path.sh
> @@ -9,7 +9,30 @@ if [[ "$1" == "" ]] || [[ $1 != /pub* ]] || [[ $1 != */ ]]; 
> then
>exit 1
>  fi
>  
> -CMD="aws s3 sync   \
> +# first run do not delete anything or copy the repodata.
> +CMD1="aws s3 sync   \
> +  --exclude */repodata/* \
> +  --exclude *.snapshot/*  \
> +  --exclude *source/* \
> +  --exclude *SRPMS/*  \
> +  --exclude *debug/*  \
> +  --exclude *beta/*   \
> +  --exclude *ppc/*\
> +  --exclude *ppc64/*  \
> +  --exclude *repoview/*   \
> +  --exclude *Fedora/* \
> +  --exclude *EFI/*\
> +  --exclude *core/*   \
> +  --exclude *extras/* \
> +  --exclude *LiveOS/* \
> +  --exclude *development/rawhide/* \
> +  --no-follow-symlinks\
> +  --only-show-errors  \
> +  "
> +  #--dryrun \
> +
> +# second we delete old content and also copy the repodata
> +CMD2="aws s3 sync   \
>--delete \
>--exclude *.snapshot/*  \
>--exclude *source/* \
> @@ -32,19 +55,12 @@ CMD="aws s3 sync   \
>  
>  #echo "$CMD /srv$1 s3://s3-mirror-us-west-1-02.fedoraproject.org$1"
>  echo "Starting $1 sync at $(date)" >> /var/log/s3-mirror/timestamps
> -$CMD /srv$1 s3://s3-mirror-us-west-1-02.fedoraproject.org$1
> -echo "Ending $1 sync at $(date)" >> /var/log/s3-mirror/timestamps
> -
> +$CMD1 /srv$1 s3://s3-mirror-us-west-1-02.fedoraproject.org$1
> +$CMD1 /srv$1/repodata/ 
> s3://s3-mirror-us-west-1-02.fedoraproject.org$1/repodata/
>  # Always do the invalidations because they are quick and prevent issues
>  # depending on which path is synced.
> -for file in $(echo /srv/pub/epel/6/*/repodata/repomd.xml | sed 's#/srv##g'); 
> do
> -  aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU 
> --paths "$file" > /dev/null
> -done
> -
> -for file in $(echo /srv/pub/epel/7/*/repodata/repomd.xml | sed 's#/srv##g'); 
> do
> -  aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU 
> --paths "$file" > /dev/null
> -done
> -
> -for file in $(echo /srv/pub/fedora/linux/updates/*/*/*/repodata/repomd.xml | 
> sed 's#/srv##g'); do
> +for file in $(echo $1/repodata/* ); do
>aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU 
> --paths "$file" > /dev/null
>  done
> +$CMD2 /srv$1 s3://s3-mirror-us-west-1-02.fedoraproject.org$1
> +echo "Ending $1 sync at $(date)" >> /var/log/s3-mirror/timestamps
> diff --git a/roles/s3-mirror/files/s3.sh b/roles/s3-mirror/files/s3.sh
> index 55c1940..c70defb 100644
> --- a/roles/s3-mirror/files/s3.sh
> +++ b/roles/s3-mirror/files/s3.sh
> @@ -3,8 +3,10 @@
>  # LGPL
>  # Author: Rick Elrod 
>  
> -CMD="aws s3 sync   \
> -  --delete \
> +# first run this command that syncs, but does not delete.
> +# It also excludes repodata. 
> +CMD1="aws s3 sync   \
> +  --exclude */repodata/*   \
>--exclude */.snapshot/*  \
>--exclude */source/* \
>--exclude */SRPMS/*  \
> @@ -38,6 +40,9 @@ CMD="aws s3 sync   \
>--exclude */releases/24/*\
>--exclude 

Re: Freeze Break Request: s3-mirror adjustments

[Adrian Reber]

On Thu, Mar 05, 2020 at 02:39:36PM -0800, Kevin Fenzi wrote:
> -CMD="aws s3 sync   \
> +# first run do not delete anything or copy the repodata.
> +CMD1="aws s3 sync   \
> +  --exclude */repodata/* \
> +  --exclude *.snapshot/*  \
> +  --exclude *source/* \
> +  --exclude *SRPMS/*  \
> +  --exclude *debug/*  \
> +  --exclude *beta/*   \
> +  --exclude *ppc/*\
> +  --exclude *ppc64/*  \
> +  --exclude *repoview/*   \
> +  --exclude *Fedora/* \
> +  --exclude *EFI/*\
> +  --exclude *core/*   \
> +  --exclude *extras/* \
> +  --exclude *LiveOS/* \
> +  --exclude *development/rawhide/* \
> +  --no-follow-symlinks\
> +  --only-show-errors  \
> +  "
> +  #--dryrun \
> +
> +# second we delete old content and also copy the repodata
> +CMD2="aws s3 sync   \
>--delete \
>--exclude *.snapshot/*  \
>--exclude *source/* \
> @@ -32,19 +55,12 @@ CMD="aws s3 sync   \
>  
>  #echo "$CMD /srv$1 s3://s3-mirror-us-west-1-02.fedoraproject.org$1"
>  echo "Starting $1 sync at $(date)" >> /var/log/s3-mirror/timestamps
> -$CMD /srv$1 s3://s3-mirror-us-west-1-02.fedoraproject.org$1
> -echo "Ending $1 sync at $(date)" >> /var/log/s3-mirror/timestamps
> -
> +$CMD1 /srv$1 s3://s3-mirror-us-west-1-02.fedoraproject.org$1
> +$CMD1 /srv$1/repodata/ 
> s3://s3-mirror-us-west-1-02.fedoraproject.org$1/repodata/
>  # Always do the invalidations because they are quick and prevent issues
>  # depending on which path is synced.
> -for file in $(echo /srv/pub/epel/6/*/repodata/repomd.xml | sed 's#/srv##g'); 
> do
> -  aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU 
> --paths "$file" > /dev/null
> -done
> -
> -for file in $(echo /srv/pub/epel/7/*/repodata/repomd.xml | sed 's#/srv##g'); 
> do
> -  aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU 
> --paths "$file" > /dev/null
> -done
> -
> -for file in $(echo /srv/pub/fedora/linux/updates/*/*/*/repodata/repomd.xml | 
> sed 's#/srv##g'); do
> +for file in $(echo $1/repodata/* ); do
>aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU 
> --paths "$file" > /dev/null
>  done
> +$CMD2 /srv$1 s3://s3-mirror-us-west-1-02.fedoraproject.org$1
> +echo "Ending $1 sync at $(date)" >> /var/log/s3-mirror/timestamps
> diff --git a/roles/s3-mirror/files/s3.sh b/roles/s3-mirror/files/s3.sh
> index 55c1940..97abc59 100644
> --- a/roles/s3-mirror/files/s3.sh
> +++ b/roles/s3-mirror/files/s3.sh
> @@ -3,8 +3,10 @@
>  # LGPL
>  # Author: Rick Elrod 
>  
> -CMD="aws s3 sync   \
> -  --delete \
> +# first run this command that syncs, but does not delete.
> +# It also excludes repodata. 
> +CMD1="aws s3 sync   \
> +  --exclude */repodata/*   \
>--exclude */.snapshot/*  \
>--exclude */source/* \
>--exclude */SRPMS/*  \
> @@ -38,6 +40,9 @@ CMD="aws s3 sync   \
>--exclude */releases/24/*\
>--exclude */releases/25/*\
>--exclude */releases/26/*\
> +  --exclude */releases/27/*\
> +  --exclude */releases/28/*\
> +  --exclude */releases/29/*\
>--exclude */updates/8/*  \
>--exclude */updates/9/*  \
>--exclude */updates/10/* \
> @@ -57,6 +62,9 @@ CMD="aws s3 sync   \
>--exclude */updates/24/* \
>--exclude */updates/25/* \
>--exclude */updates/26/* \
> +  --exclude */updates/27/* \
> +  --exclude */updates/28/* \
> +  --exclude */updates/29/* \
>--exclude */updates/testing/8/*  \
>--exclude */updates/testing/9/*  \
>--exclude */updates/testing/10/* \
> @@ -76,6 +84,95 @@ CMD="aws s3 sync   \
>--exclude */updates/testing/24/* \
>--exclude */updates/testing/25/* \
>--exclude */updates/testing/26/* \
> +  --exclude */updates/testing/27/* \
> +  --exclude */updates/testing/28/* \
> +  --exclude */updates/testing/29/* \
> +  --no-follow-symlinks \
> +  "
> +  #--dryrun \
> +
> +# Next we run this command which also includes repodata.
> +CMD2="aws s3 sync   \
> +  --exclude */.snapshot/*  \
> +  --exclude */source/* \
> +  --exclude */SRPMS/*  \
> +  --exclude */debug/*  \
> +  --exclude */beta/*   \
> +  --exclude */ppc/*\
> +  --exclude */ppc64/*  \
> +  --exclude */repoview/*   \
> +  --exclude */Fedora/* \
> +  --exclude */EFI/*\
> +  --exclude */core/*   \
> +  --exclude */extras/* \
> +  --exclude */LiveOS/* \
> +  

Re: Freeze Break Request: s3-mirror adjustments

[Adrian Reber]

The only thing which is not clear in the documentation is when the
delete is happening. With rsync there is an option '--delete-after' to
make sure first all files are transferred and deletion happens
afterwards. If 'aws s3 sync' does the deletion during or before all
files are transferred you might again have a small time window where
repodata has not been updated but files have already been deleted.

To be on the safe side the repodata probably needs to be transferred
before files are being deleted.

Deletion should probably only happen after doing the invalidate to be
sure that no files are being referenced by the caches which are
currently being deleted.

Adrian

On Thu, Mar 05, 2020 at 01:05:20PM -0800, Kevin Fenzi wrote:
> First, I noticed we are running the full sync twice right now, at the
> same time: 
> 
> [root@mm-backend01 cron.d][PROD]# cat /etc/cron.d/s3.sh 
> #Ansible: s3sync
> 0 0,11 * * * s3-mirror /usr/local/bin/lock-wrapper s3sync 
> /usr/local/bin/s3.sh 2>&1 | /usr/local/bin/nag-once s3.sh 1d 2>
> &1
> #Ansible: s3sync-main
> 0 0 * * * s3-mirror /usr/local/bin/lock-wrapper s3sync-main 
> /usr/local/bin/s3.sh 2>&1 | /usr/local/bin/nag-once s3.sh 1d 
> 2>&1
> 
> Second, the attached patch changes the sync scripts to: 
> 
> * do one sync with no --delete and excluding repodata
> * do another one with --delete and including repodata
> * invalidate the repodata
> 
> I adjusted the cron jobs to handle the repodata invalidate (I think). 
> 
> TODO: only sync when things have changed. 
> 
> +1s?
> 
> kevin
> --

> From 72a86a9f4d80344b96f8752ead2cc2877de292ef Mon Sep 17 00:00:00 2001
> From: Kevin Fenzi 
> Date: Thu, 5 Mar 2020 20:46:53 +
> Subject: [PATCH] s3-mirror: Split things into 2 sync runs, one without
>  repodata and delete, the other with both.
> 
> In order to make sure the s3 mirror always is consistent, split out the 
> commands
> to make it sync without repodata and delete, then do another run with those, 
> then finally
> invalidate all the repodata/* files.
> 
> Some of the cron jobs are adjusted to allow the repodata invalidation.
> 
> Signed-off-by: Kevin Fenzi 
> ---
>  roles/s3-mirror/files/s3-sync-path.sh |  38 
>  roles/s3-mirror/files/s3.sh   | 110 
> +-
>  roles/s3-mirror/tasks/main.yml|   4 +-
>  3 files changed, 136 insertions(+), 16 deletions(-)
> 
> diff --git a/roles/s3-mirror/files/s3-sync-path.sh 
> b/roles/s3-mirror/files/s3-sync-path.sh
> index e6ac994..40a0f90 100644
> --- a/roles/s3-mirror/files/s3-sync-path.sh
> +++ b/roles/s3-mirror/files/s3-sync-path.sh
> @@ -9,7 +9,30 @@ if [[ "$1" == "" ]] || [[ $1 != /pub* ]] || [[ $1 != */ ]]; 
> then
>exit 1
>  fi
>  
> -CMD="aws s3 sync   \
> +# first run do not delete anything or copy the repodata.
> +CMD1="aws s3 sync   \
> +  --exclude */repodata/* \
> +  --exclude *.snapshot/*  \
> +  --exclude *source/* \
> +  --exclude *SRPMS/*  \
> +  --exclude *debug/*  \
> +  --exclude *beta/*   \
> +  --exclude *ppc/*\
> +  --exclude *ppc64/*  \
> +  --exclude *repoview/*   \
> +  --exclude *Fedora/* \
> +  --exclude *EFI/*\
> +  --exclude *core/*   \
> +  --exclude *extras/* \
> +  --exclude *LiveOS/* \
> +  --exclude *development/rawhide/* \
> +  --no-follow-symlinks\
> +  --only-show-errors  \
> +  "
> +  #--dryrun \
> +
> +# second we delete old content and also copy the repodata
> +CMD2="aws s3 sync   \
>--delete \
>--exclude *.snapshot/*  \
>--exclude *source/* \
> @@ -32,19 +55,12 @@ CMD="aws s3 sync   \
>  
>  #echo "$CMD /srv$1 s3://s3-mirror-us-west-1-02.fedoraproject.org$1"
>  echo "Starting $1 sync at $(date)" >> /var/log/s3-mirror/timestamps
> -$CMD /srv$1 s3://s3-mirror-us-west-1-02.fedoraproject.org$1
> +$CMD1 /srv$1 s3://s3-mirror-us-west-1-02.fedoraproject.org$1
> +$CMD2 /srv$1 s3://s3-mirror-us-west-1-02.fedoraproject.org$1
>  echo "Ending $1 sync at $(date)" >> /var/log/s3-mirror/timestamps
>  
>  # Always do the invalidations because they are quick and prevent issues
>  # depending on which path is synced.
> -for file in $(echo /srv/pub/epel/6/*/repodata/repomd.xml | sed 's#/srv##g'); 
> do
> -  aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU 
> --paths "$file" > /dev/null
> -done
> -
> -for file in $(echo /srv/pub/epel/7/*/repodata/repomd.xml | sed 's#/srv##g'); 
> do
> -  aws cloudfront create-invalidation --distribution-id E2KJMDC0QAJDMU 
> --paths "$file" > /dev/null
> -done
> -
> -for file in $(echo /srv/pub/fedora/linux/updates/*/*/*/repodata/repomd.xml | 
> sed 's#/srv##g'); do
> +for file in $(echo $1/repodata/* ); do
>aws cloudfront create-invalidation 

Re: Issue with Fedora GeoIP service

[Adrian Reber]

On Mon, Dec 09, 2019 at 12:19:17PM -0800, Kevin Fenzi wrote:
> On Sat, Dec 07, 2019 at 09:39:46AM -0500, Kevin Sandy wrote:
> > I have an updated copy working with geoip2, with the exception of area code 
> > data (which isn’t included in the new database). But if I’m understanding 
> > it’s purpose correctly, telephone area code isn’t needed. I need to do some 
> > testing with IPs around the world to ensure that the data is sane outside 
> > the US; once I’ve had a chance to do that I’ll submit a PR on GitHub.
> 
> Wow. Awesome work!
> 
> Thanks for diving in and getting this going. 
> 
> In another email thread someone, Adrian said he had some code to do
> this, but if you already have it working, perhaps he could review your
> code/setup and we could then push it into ansible?

I had a look at the PR (https://github.com/fedora-infra/geoip-city-wsgi/pull/1)
and it works and the output is almost the same. Maybe one of the known
data consumers can have a short look if those subtle changes in the
output are not a problem.

Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: New mirrorlist server implementation

[Adrian Reber]

Thanks a lot. I will change the container to be based on this instead of
the binary I have built locally. Perfect!

Adrian

On Tue, Nov 19, 2019 at 07:54:34AM +0100, Igor Gnatenko wrote:
> And I finally got this done. The update for F31 is here:
> https://bodhi.fedoraproject.org/updates/FEDORA-2019-48a3fc7eb5
> 
> On Tue, Oct 8, 2019 at 8:53 AM Igor Gnatenko
>  wrote:
> >
> > I'll try to package new Rust implementation in Fedora :)
> >
> > And thanks for working on this!
> >
> > On Tue, Oct 8, 2019 at 8:52 AM Adrian Reber  wrote:
> > >
> > >
> > > Fedora's complete MirrorManager setup is still running on Python2. The
> > > code has been ported to Python3 probably over two years ago but we have
> > > not switched yet. One of the reasons is that the backend is running on
> > > RHEL7 which means we are not in a hurry to deploy the Python3 version.
> > >
> > > The mirrorlist server which is answering the actual dnf/yum queries for
> > > a mirrorlist/metalink is, however, running in a Fedora 29 container.
> > > This container also still uses Python2 and it actually cannot use the
> > > Python3 version.
> > >
> > > One of MirrorManager's design points is that the mirrorlist servers,
> > > which are answering around 27 000 000 requests per day, are not directly
> > > accessing the database. The backend creates a snapshot of the relevant
> > > data (113MB) and the mirrorlist servers are using this snapshot to
> > > answer client requests.
> > >
> > > This data exchange is based on Python's pickle format and that does not
> > > seem to work with Python3 if it is generated using Python2.
> > >
> > > Having used protobuf before, I added code to also export the data for the
> > > mirrorlist servers based on protobuf.
> > >
> > > The good news with protobuf is, that the resulting file is only 66MB
> > > instead of 113MB. The bad news is, that loading it from Python requires
> > > 3.5 times the amount of memory during runtime (3.5GB instead of 1GB).
> > >
> > > In addition to the data exchange problems between backend and
> > > mirrorlist servers the architecture of the mirrorlist server does not
> > > really make sense today. 12 years ago it made a lot of sense as it could
> > > be easily integrated into httpd and it could be easily reloaded without
> > > stopping the service. Today the mirrorlist server and httpd is all part
> > > of a container which is then behind haproxy. So there is a lot of
> > > infrastructure in the container which is not really useful.
> > >
> > > To get rid of the pickle format and to have a simpler architecture I
> > > reimplemented the mirrorlist-server in Rust. This was brought up some
> > > time ago on a ticket and with the protobuf problems I was seeing in
> > > Python it made sense to try it out.
> > >
> > > My code currently can be found at 
> > > https://github.com/adrianreber/mirrorlist-server
> > > and so far the results from the new mirrorlist server are the same as
> > > from the Python based mirrorlist server.
> > >
> > > It requires less than 700MB instead of the 1GB in Python with production
> > > based data and seems really fast.
> > >
> > > I have set up a test instance with the mirror data from Sunday at:
> > >
> > > https://lisas.de/metalink?repo=updates-testing-f31=x86_64
> > > https://lisas.de/mirrorlist?repo=updates-testing-f31=x86_64
> > >
> > > The instance is based on the container I pushed to quay.io:
> > >
> > >  $ podman run quay.io/adrianreber/mirrorlist-server:latest -h
> > >
> > > With this change the mirrorlist server would also finally switch to
> > > geoip2. The currently running mirrorlist server still uses the legacy
> > > geoip database.
> > >
> > > After the Fedora 31 freeze I would like to introduce this new mirrorlist
> > > server implementation on the proxies. I already verified that I can run
> > > this mirrorlist container rootless. This new container can be a drop-in
> > > replacement for the current container and no infrastructure around it
> > > needs to be changed.
> > >
> > > The main changes to get it into production is to change 
> > > mirrorlist1.service
> > > and mirrorlist2.service to include a line "User=mirrormanager" and
> > > replace the current container name with new container.
> > >
> > >

Re: Mirrorlist server bug report - behaviour change requested

[Adrian Reber]

On Mon, Nov 11, 2019 at 02:15:03PM +0100, Adrian Reber wrote:
> I was contacted by someone working on openSUSE's Open Build System
> (OBS). They are using download.fedoraproject.org and some layout change
> concerning noarch breaks their workflow. The problem is, that the
> redirect service of download.fedoraproject.org based on the mirrorlist
> servers returns a 200 and 'error: invalid path'. See
> 
> https://github.com/adrianreber/mirrorlist-server/issues/1
> 
> So this problem is unrelated to the new mirrorlist implementation. The
> old code did the same. Error message and a 200 return code.
> 
> I think changing the return code to 404 is the correct thing to do, but
> I wanted to check if anybody sees any problems with this behaviour
> change of the mirrorlist server.

The new mirrorlist-server which returns a 404 for non-existing
directories in its database has been released and deployed in staging.
So far everything still seems to work correctly. I will make the
necessary ansible changes tomorrow.

Adrian
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Mirrorlist server bug report - behaviour change requested

[Adrian Reber]

I was contacted by someone working on openSUSE's Open Build System
(OBS). They are using download.fedoraproject.org and some layout change
concerning noarch breaks their workflow. The problem is, that the
redirect service of download.fedoraproject.org based on the mirrorlist
servers returns a 200 and 'error: invalid path'. See

https://github.com/adrianreber/mirrorlist-server/issues/1

So this problem is unrelated to the new mirrorlist implementation. The
old code did the same. Error message and a 200 return code.

I think changing the return code to 404 is the correct thing to do, but
I wanted to check if anybody sees any problems with this behaviour
change of the mirrorlist server.

Adrian
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: [FBR 0/1] Enable new mirrorlist server on proxy14

[Adrian Reber]

proxy14 is now running the new rust based mirrorlist server using
rootless containers (running as the mirrormanager user).

So far both containers are happily answering requests and it seems to
work.

Manual testing showed correct results and now we are waiting if anyone
complains about it.

Problems could be wrong results, so people could report being redirected
to the wrong mirrors. Another problem could be related to missing
replies or timeouts, so maybe something like no replies at all. Maybe
during restarts/reloads of the used data.

So far we have not heard any bug reports. Let's hope it keeps on
working.

Memory consumption seems to be less (around 500MB less) and hardly any
CPU required. So far this seems to be an improvement.

The plan is now to keep it running like this until the freeze is over
and see how it handles the actual release traffic. Then decide if we
switch everything to the new code-base.

Adrian

On Wed, Oct 16, 2019 at 10:50:08AM +0200, Adrian Reber wrote:
> This is a freeze break request to enable the new mirrorlist server on
> proxy14 as discussed on the mailing list.
> 
> I hope my conditionals are correct for the Ansible and Jinja2 files.
> 
> If this freeze break request gets accepted someone needs to run the
> playbook against proxy14.
> 
> Before running the playbook, proxy14 should be removed from DNS to make
> sure, that the old mirrorlist containers are correctly stopped and
> deleted and that the new mirrorlist containers are correctly running.
> 
> 
> Adrian Reber (1):
>   Enable new mirrorlist server on proxy14
> 
>  roles/mirrormanager/backend/files/backend.cron| 8 +---
>  .../backend/templates/sync_pkl_to_mirrorlists.sh  | 2 +-
>  roles/mirrormanager/mirrorlist_proxy/tasks/main.yml   | 2 +-
>  .../mirrorlist_proxy/templates/mirrorlist.service.j2  | 4 ++--
>  4 files changed, 9 insertions(+), 7 deletions(-)
> 
> -- 
> 2.23.0
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: [FBR 1/1] Enable new mirrorlist server on proxy14

[Adrian Reber]

On Wed, Oct 16, 2019 at 03:37:17PM -0400, Randy Barlow wrote:
> On Wed, 2019-10-16 at 10:50 +0200, Adrian Reber wrote:
> > {% if env == "staging" or inventory_hostname ==
> > "proxy14.fedoraproject.org" %}
> 
> Rather than inlining host names like this, why not use host vars to
> define a boolean variable like "enable_cool_new_rust_mirrorlist". Then
> you can make that true in staging, true on proxy 14, and false
> everywhere else and make this if statement simpler and more "permanent"
> (i.e., you don't to touch this or the other two places with similar
> adjustments later when you want to add the other proxies.
> 
> I'm guilty of doing the above in my Ansible too, but recently I've been
> thinking that it would be better to use the host and group vars
> instead. And honestly this is pretty similar to what a lot of our
> playbook does, so I am not opposed.
> 
> Anyways, I'm +1 to the change as-is, but would prefer a change that
> uses vars over this one, if you are so inclined.

Thanks for that recommendation. Can you point me to a place in ansible
where you are using this? For this FBR I would rather not change it and
go with the current solution. The goal is to completely remove the
conditionals because it either works (everything uses the new approach)
or it does not work at all (keep on using the current approach). If we
would actually go the way to slowly switch more proxies one after
another I would change the conditionals to how you described it. For now
I would rather try it the way I currently have it.

Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: [FBR 0/1] Enable new mirrorlist server on proxy14

[Adrian Reber]

On Wed, Oct 16, 2019 at 10:21:46AM -0400, Randy Barlow wrote:
> On Wed, 2019-10-16 at 10:50 +0200, Adrian Reber wrote:
> > This is a freeze break request to enable the new mirrorlist server on
> > proxy14 as discussed on the mailing list.
> 
> Would it be feasible to wait until the freeze is over, or is this
> important to do before then?

Of course, that was the original plan. During tests in staging the idea
was brought up to try it on only one production proxies to make sure it
also works in production under load.

Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: [FBR 1/1] Enable new mirrorlist server on proxy14

[Adrian Reber]

On Wed, Oct 16, 2019 at 10:02:09AM -0700, Kevin Fenzi wrote:
> On Wed, Oct 16, 2019 at 10:50:09AM +0200, Adrian Reber wrote:
> > After a successful test of the new mirrorlist server code in staging,
> > this is the next step. Try to make it work in production on one of the
> > proxies. As discussed on the mailing list we are trying the new setup
> > with proxy14.
> > 
> > All the conditionals have been updated to be 'staging or proxy14' and
> > the mirrorlist data generation has been enhanced to also write out the
> > new protobuf based format on the backend system.
> > 
> > At the same time this moves the pickle/protobuf generation from :55 to
> > :30 as this generation takes much longer than it used to be and the
> > mirrorlist server restarted at :15 was never using the new data during
> > the last few months.
> 
> Bummer. Is the protobuf faster than the pickle? Or are they about the
> same...

The database queries are the problem. During DB query time it is running
with 40% CPU. Once it has all the data from the CPU it runs with 100%.
Reading all the trouble you currently have with koji I am sure you are
thrilled to hear that the DB is also slow for this. The generation of
pickle or protobuf makes not difference as far as I can tell (at least
compared to the DB query time).

> +1 here. 

Thanks.

Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

[FBR 1/1] Enable new mirrorlist server on proxy14

[Adrian Reber]

After a successful test of the new mirrorlist server code in staging,
this is the next step. Try to make it work in production on one of the
proxies. As discussed on the mailing list we are trying the new setup
with proxy14.

All the conditionals have been updated to be 'staging or proxy14' and
the mirrorlist data generation has been enhanced to also write out the
new protobuf based format on the backend system.

At the same time this moves the pickle/protobuf generation from :55 to
:30 as this generation takes much longer than it used to be and the
mirrorlist server restarted at :15 was never using the new data during
the last few months.

Signed-off-by: Adrian Reber 
---
 roles/mirrormanager/backend/files/backend.cron| 8 +---
 .../backend/templates/sync_pkl_to_mirrorlists.sh  | 2 +-
 roles/mirrormanager/mirrorlist_proxy/tasks/main.yml   | 2 +-
 .../mirrorlist_proxy/templates/mirrorlist.service.j2  | 4 ++--
 4 files changed, 9 insertions(+), 7 deletions(-)

diff --git a/roles/mirrormanager/backend/files/backend.cron 
b/roles/mirrormanager/backend/files/backend.cron
index e973ba24c..46869b994 100644
--- a/roles/mirrormanager/backend/files/backend.cron
+++ b/roles/mirrormanager/backend/files/backend.cron
@@ -1,8 +1,10 @@
 MAILTO=root
 
-# Refresh the mirrorlist cache at the top of the hour and sync it to the
-# mirrorlist servers
-55 * * * * mirrormanager /usr/bin/mm2_update-mirrorlist-server && 
/usr/local/bin/sync_pkl_to_mirrorlists.sh
+# Refresh the mirrorlist cache once every hour.
+# This can take up to 35 minutes. The mirrorlist servers
+# are restarted :15 after the full hour. Starting at :30
+# should give us enough time.
+30 * * * * mirrormanager /usr/bin/mm2_update-mirrorlist-server -p && 
/usr/local/bin/sync_pkl_to_mirrorlists.sh
 
 # update master directory
 # logs sent to /var/log/mirrormanager/umdl.log by default
diff --git a/roles/mirrormanager/backend/templates/sync_pkl_to_mirrorlists.sh 
b/roles/mirrormanager/backend/templates/sync_pkl_to_mirrorlists.sh
index c05a62b30..8428695d2 100644
--- a/roles/mirrormanager/backend/templates/sync_pkl_to_mirrorlists.sh
+++ b/roles/mirrormanager/backend/templates/sync_pkl_to_mirrorlists.sh
@@ -5,5 +5,5 @@
 MIRRORLIST_PROXY="{% for host in groups['mirrorlist_proxies'] %} {{ host }} {% 
endfor %}"
 
 for s in ${MIRRORLIST_PROXY}; do
-   rsync -az --delete-delay --delay-updates --delete 
/var/lib/mirrormanager/{*pkl,*txt} ${s}:/srv/mirrorlist/data/mirrorlist1/
+   rsync -az --delete-delay --delay-updates --delete 
/var/lib/mirrormanager/{*pkl,*txt,*proto} ${s}:/srv/mirrorlist/data/mirrorlist1/
 done
diff --git a/roles/mirrormanager/mirrorlist_proxy/tasks/main.yml 
b/roles/mirrormanager/mirrorlist_proxy/tasks/main.yml
index 669554d0e..63596ee4a 100644
--- a/roles/mirrormanager/mirrorlist_proxy/tasks/main.yml
+++ b/roles/mirrormanager/mirrorlist_proxy/tasks/main.yml
@@ -59,7 +59,7 @@
   - /var/log/mirrormanager
   tags:
   - mirrorlist_proxy
-  when: env == 'staging'
+  when: env == 'staging' or inventory_hostname.startswith('proxy14')
 
 - name: set logrotate_read_inside_containers so logrotate works
   seboolean: name=logrotate_read_inside_containers state=yes persistent=yes
diff --git 
a/roles/mirrormanager/mirrorlist_proxy/templates/mirrorlist.service.j2 
b/roles/mirrormanager/mirrorlist_proxy/templates/mirrorlist.service.j2
index b0ce4ded5..2d26807c8 100644
--- a/roles/mirrormanager/mirrorlist_proxy/templates/mirrorlist.service.j2
+++ b/roles/mirrormanager/mirrorlist_proxy/templates/mirrorlist.service.j2
@@ -2,12 +2,12 @@
 Description=Mirrorlist Container {{ item }}
 
 [Service]
-{% if env == "staging" %}
+{% if env == "staging" or inventory_hostname == "proxy14.fedoraproject.org" %}
 User=mirrormanager
 {% endif %}
 ExecStartPre=-/usr/bin/podman stop %n
 ExecStartPre=-/usr/bin/podman rm %n --force
-{% if env == "staging" %}
+{% if env == "staging" or inventory_hostname == "proxy14.fedoraproject.org" %}
 ExecStart=/usr/bin/podman run \
 --rm=true \
 --net=host --userns=keep-id \
-- 
2.23.0
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

[FBR 0/1] Enable new mirrorlist server on proxy14

[Adrian Reber]

This is a freeze break request to enable the new mirrorlist server on
proxy14 as discussed on the mailing list.

I hope my conditionals are correct for the Ansible and Jinja2 files.

If this freeze break request gets accepted someone needs to run the
playbook against proxy14.

Before running the playbook, proxy14 should be removed from DNS to make
sure, that the old mirrorlist containers are correctly stopped and
deleted and that the new mirrorlist containers are correctly running.


Adrian Reber (1):
  Enable new mirrorlist server on proxy14

 roles/mirrormanager/backend/files/backend.cron| 8 +---
 .../backend/templates/sync_pkl_to_mirrorlists.sh  | 2 +-
 roles/mirrormanager/mirrorlist_proxy/tasks/main.yml   | 2 +-
 .../mirrorlist_proxy/templates/mirrorlist.service.j2  | 4 ++--
 4 files changed, 9 insertions(+), 7 deletions(-)

-- 
2.23.0
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: New mirrorlist server implementation

[Adrian Reber]

On Mon, Oct 14, 2019 at 07:42:30AM -0700, Kevin Fenzi wrote:
> On Mon, Oct 14, 2019 at 08:53:26AM -0400, Stephen John Smoogen wrote:
> > On Mon, 14 Oct 2019 at 03:39, Adrian Reber  wrote:
> > 
> > I would say proxy14. It should have ipv6 and is not a OMG we broke
> > koji/everything else like proxy01,10,110,101 can be.
> 
> +1 

I will prepare the ansible changes and post them here as a freeze break
request.

Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: New mirrorlist server implementation

[Adrian Reber]

On Tue, Oct 08, 2019 at 08:38:18AM -0700, Kevin Fenzi wrote:
> On Tue, Oct 08, 2019 at 08:42:13AM +0200, Adrian Reber wrote:
> > After the Fedora 31 freeze I would like to introduce this new mirrorlist
> > server implementation on the proxies. I already verified that I can run
> > this mirrorlist container rootless. This new container can be a drop-in
> > replacement for the current container and no infrastructure around it
> > needs to be changed.
> > 
> > The main changes to get it into production is to change mirrorlist1.service
> > and mirrorlist2.service to include a line "User=mirrormanager" and
> > replace the current container name with new container.
> 
> Awesome. 
> 
> How about we get this deployed in stg soonish so we can test it out
> more. 

Thanks to smooge's help we were able to switch staging to the new
mirrorlist. All changes in ansible are staging only.

So you should get almost the same result (modulo randomness) from:

https://mirrors.fedoraproject.org/mirrorlist?repo=epel-7=x86_64
https://mirrors.stg.fedoraproject.org/mirrorlist?repo=epel-7=x86_64

My plan was to wait until after the freeze to switch prod to the new
setup, but there was also the proposal to switch one production proxy
server to the new setup earlier to see if it also works under real load.

To test the new setup on one production proxy sounds like a good idea to
me, especially if we can try it before the actual Fedora 31 release.

If it does not work, we can easily revert to the old setup. If it works,
however, I am not sure yet what this would mean. Running only one proxy
with the new code and everything else with the current code does not
seem like a good idea. So if the test on one production proxy is
successful it would mean to switch everything to the new setup during
the freeze?? Maybe also not the best idea.

I like the idea of trying it in prod, but I am unsure what to do with
result of that try.

Any further comments about trying this during the freeze?

Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: New mirrorlist server implementation

[Adrian Reber]

On Tue, Oct 08, 2019 at 08:38:18AM -0700, Kevin Fenzi wrote:
> On Tue, Oct 08, 2019 at 08:42:13AM +0200, Adrian Reber wrote:
> ...snip...
> > 
> > After the Fedora 31 freeze I would like to introduce this new mirrorlist
> > server implementation on the proxies. I already verified that I can run
> > this mirrorlist container rootless. This new container can be a drop-in
> > replacement for the current container and no infrastructure around it
> > needs to be changed.
> > 
> > The main changes to get it into production is to change mirrorlist1.service
> > and mirrorlist2.service to include a line "User=mirrormanager" and
> > replace the current container name with new container.
> 
> Awesome. 
> 
> How about we get this deployed in stg soonish so we can test it out
> more. 

I was not aware, that the mirrorlist containers are also running in stg,
but they are, good.

Assuming we want to run the containers rootless as the mirrormanager
user I would need the necessary entries in /etc/subuid and /etc/subgid.
Grepping through the ansible repository this does not seem to be used
yet. If someone can setup subuid and subgid I can do everything else.

Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: FBR: clean up disk space on mirrors

[Adrian Reber]

On Tue, Mar 26, 2019 at 12:59:16PM -0400, Stephen John Smoogen wrote:
> On Tue, 26 Mar 2019 at 11:48, Adrian Reber  wrote:
> >
> > On Mon, Mar 25, 2019 at 04:28:09PM -0400, Stephen John Smoogen wrote:
> > > On Mon, 25 Mar 2019 at 16:01, Mohan Boddu  wrote:
> > > >
> > > > On Mon, Mar 25, 2019 at 8:37 AM Stephen John Smoogen  
> > > > wrote:
> > > >>
> > > >> 1. Remove F29beta from mirrors so mirrors can keep up
> > > >> /pub/fedora/linux/releases/test/29_Beta/
> > > >
> > > > +1
> > > >>
> > > >>
> > > >> 2. Move F27 to archives
> > > >> a. Have mirrormanager point to /pub/archives/fedora/linux/27
> > > >> b. Remove /pub/fedora/linux/27
> > > >
> > > > +1
> > >
> > > OK will look at tomorrow morning
> >
> > Let me know if you need any help on the mirrormanager side.
> 
> If you could make sure there aren't any links for 29_Beta and that the
> 27 is pointing to archives that would be great.

MirrorManager points to archive for primary and secondary architectures.
From my point of view 27 can be removed from the non-archive location.

Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: FBR: clean up disk space on mirrors

[Adrian Reber]

On Mon, Mar 25, 2019 at 04:28:09PM -0400, Stephen John Smoogen wrote:
> On Mon, 25 Mar 2019 at 16:01, Mohan Boddu  wrote:
> >
> > On Mon, Mar 25, 2019 at 8:37 AM Stephen John Smoogen  
> > wrote:
> >>
> >> 1. Remove F29beta from mirrors so mirrors can keep up
> >> /pub/fedora/linux/releases/test/29_Beta/
> >
> > +1
> >>
> >>
> >> 2. Move F27 to archives
> >> a. Have mirrormanager point to /pub/archives/fedora/linux/27
> >> b. Remove /pub/fedora/linux/27
> >
> > +1
> 
> OK will look at tomorrow morning

Let me know if you need any help on the mirrormanager side.

Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: mirrorlist server with Python3

[Adrian Reber]

On Sun, Feb 17, 2019 at 04:07:46PM +0100, Adrian Reber wrote:
> The latest release of MirrorManager2 is still in updates-testing. I did
> not want to push this version to updates-released because this is the
> first Python3 based released.
> 
> Could someone test the version from updates-testing? Update the
> mirrorlist container and test it on one of the proxies?

I tested it locally and it does not work. For someone knowing Python
better than me it is probably obvious, but the mirrorlist server using
Python 3 cannot read the pickle from the MirrorManager backend running
Python 2.

To be able to use a Python 2 based backend and Python 3 based mirrorlist
I added an additional export format using Protobuf.

The good news about this is that the file containing all the information
for the mirrorlist server using protbuf is smaller than the pickle file.
The protbuf is about 2/3 of the pickle file.

The bad thing about this change is that the memory required by the
mirrorlist server is 5 times higher using the protbuf file than using
the pickle file. It is amazing how reading a 32MB file results in a
process requiring 2.5GB of memory.

From the memory consumption point of view this change seems like a bad
idea. Being independent of the pickle format seems to be a good idea as
it would be possible to have Python2 and Python3 parts in our
MirrorManager deployment. It would also open up the possibility to
rewrite the mirrorlist server in something else which does not require
to understand Python's pickle format.

This is the pull request with all the changes:
https://github.com/fedora-infra/mirrormanager2/pull/266

Any recommendations if this is useful or how to convince Python to use
less memory.

Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

mirrorlist server with Python3

[Adrian Reber]

The latest release of MirrorManager2 is still in updates-testing. I did
not want to push this version to updates-released because this is the
first Python3 based released.

Could someone test the version from updates-testing? Update the
mirrorlist container and test it on one of the proxies?

Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

[release] MirrorManager2: 0.9.0

[Adrian Reber]

A new release of MirrorManager2 is available: 0.9.0

- crawler: Correctly calculate the remaining time
  https://github.com/fedora-infra/mirrormanager2/pull/244
- repomap: more modular repository detection logic
  https://github.com/fedora-infra/mirrormanager2/pull/243
- crawler: correctly handle keep-alive for HTTPS
  https://github.com/fedora-infra/mirrormanager2/pull/245
- crawler: only update directories of the current category
  https://github.com/fedora-infra/mirrormanager2/pull/250
- python3 compatibility
  https://github.com/fedora-infra/mirrormanager2/pull/185
- rpmmd: switch from yum.repoMDObject pyrpmmd
  https://github.com/fedora-infra/mirrormanager2/pull/254
- Migrate to new geoip API
  https://github.com/fedora-infra/mirrormanager2/pull/253
- Use InputRequired() instead of Required()
  https://github.com/fedora-infra/mirrormanager2/pull/256
- Enable MirrorManager2 to be built using Python 3 for Fedora
  https://github.com/fedora-infra/mirrormanager2/pull/260
- Fix tests with python3
  https://github.com/fedora-infra/mirrormanager2/pull/261
- Toggle private
  https://github.com/fedora-infra/mirrormanager2/pull/257


This release includes a few smaller fixes which are already running on
Fedora's production systems and it also includes the switch to GeoIP2.

Especially the switch to GeoIP2 means that it requires additional
changes to Fedora's MirrorManager2 instance to make sure the new GeoIP2
databases are available on the relevant systems.

I will look into updating Fedora's staging MirrorManager2 instance in
the next days.

Thanks to Zbigniew Jędrzejewski-Szmek and Neal Gompa for porting
MirrorManager2 to Python3. From now on the Fedora packages will be using
Python3. The EPEL7 packages which are used in Fedora's MirrorManager2
instance will still be Python2.

Tests are all green with Python3, but as far as I know there is no
MirrorManager2 instance running on Python3... yet.

Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: MirrorManager changes - report_mirror

[Adrian Reber]

On Fri, Sep 28, 2018 at 11:42:02AM -0500, Jason L Tibbitts III wrote:
> >>>>> "AR" == Adrian Reber  writes:
> 
> AR> I would like to ignore report_mirror results for non-private mirrors
> AR> in the future.
> 
> I don't really see why this would have to be done globally unless we're
> just going to abandon the concept of reporting for public mirrors.  Do
> you have an issue with quick-fedora-mirror checkins as well or just ones
> from report_mirror?  Can I have quick-fedora-mirror provide additional
> data so that you can distinguish it from report_mirror?

The problem is that people are reporting different things than they
have. Sometimes they are reporting from different machines than the ones
we scan. Sometimes it is some kind of cluster which is out of sync.

This has nothing to do with report_mirror or quick-fedora-mirror.

> And if checkins are ignored, then will mirrormanager have to wait until
> the next crawl before it knows that my mirrors have new content (even
> though that happens within a few minutes after the content is available)?

Which is not really a problem for most cases, as we are re-directing to
older mirrors anyway. That works as most of the times we are referencing
up to three repomd.xml files in the metalink.

> AR> For private mirrors report_mirror is the only way to
> AR> know which content it has, for public mirrors it is not really
> AR> necessary as the mirrors are crawled anyway and report_mirror does
> AR> not contain enough information anyway right now.
> 
> Well, what information do you need?

Everything the crawler has. I really like the idea behind report_mirror
but is does not work with what we have right now. The crawler is anyway
the instance which decides about the actual state of a mirror and right
now report_mirror is the main problem I am seeing with broken mirrors.

Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

MirrorManager changes - report_mirror

[Adrian Reber]

During the last months most mirror problems have been related to
report_mirror reporting that mirrors are up to date and the crawler then
marking the mirror as not up to date. Which resulted in mirrors being
added and removed from the mirrorlist every few hours.

As the crawler normally sees what the user sees, means that
report_mirror often reports the wrong mirror status.

I would like to ignore report_mirror results for non-private mirrors in
the future. For private mirrors report_mirror is the only way to know
which content it has, for public mirrors it is not really necessary as
the mirrors are crawled anyway and report_mirror does not contain enough
information anyway right now.

Any objections to ignore report_mirror reports for public mirrors
after the F29 release.

Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

MirrorManager 29 redirects removed

[Adrian Reber]

Now that MirrorManager detected the newly created Fedora 29
repositories, I removed the repository redirects from 29 to rawhide and
I added new redirects from 30 to rawhide.

Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/VFLX7HAUGIXFDJ4V4SAALKMPADYPFBPW/

Re: FBR: Change to category based crawling

[Adrian Reber]

On Thu, Apr 19, 2018 at 08:54:21AM +0200, Adrian Reber wrote:
> As discussed previously I would like to change the crawler to crawl each
> category separately. The goal is to reduce the load on the database by
> distributing the crawling better over the whole day and to reduce the
> chance of mirrors being disabled because of the high database load.
> 
> This should also remove the need for mirror administrators to create
> multiple hosts in MirrorManager to work around the 4 hours timeout per
> host.

On the mirror-admin mailing list there have been reports that mirrors
are no longer part of the mirrorlist/metalink.

With the switch to category based crawling a bug in the crawler was
uncovered:

https://github.com/fedora-infra/mirrormanager2/issues/249

It is fixed and the crawler should now mark directories correctly during
crawl.

Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org

Re: FBR: Change to category based crawling

[Adrian Reber]

On Thu, Apr 19, 2018 at 08:54:21AM +0200, Adrian Reber wrote:
> As discussed previously I would like to change the crawler to crawl each
> category separately. The goal is to reduce the load on the database by
> distributing the crawling better over the whole day and to reduce the
> chance of mirrors being disabled because of the high database load.
> 
> This should also remove the need for mirror administrators to create
> multiple hosts in MirrorManager to work around the 4 hours timeout per
> host.
> 
> Attached is my patch. Please +1. This affects mm-crawler01 and
> mm-crawler02.

Thanks for the '+1's. This change is active since yesterday and so far
it seems to work. If we still see hosts timing out, especially when
crawling the archive hosts, we can increase the timeout for archive
crawling to 6 hours. Another option to decrease the number of wrongly
auto-deactivated mirrors is to increase CRAWLER_AUTO_DISABLE from 4 to 6
or 8 crawls.

I will look at those changes after the freeze.

Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org

FBR: Change to category based crawling

[Adrian Reber]

As discussed previously I would like to change the crawler to crawl each
category separately. The goal is to reduce the load on the database by
distributing the crawling better over the whole day and to reduce the
chance of mirrors being disabled because of the high database load.

This should also remove the need for mirror administrators to create
multiple hosts in MirrorManager to work around the 4 hours timeout per
host.

Attached is my patch. Please +1. This affects mm-crawler01 and
mm-crawler02.

Adrian
From b10d5ffa7e48e934da3350186eaf8dd4fb0cebf3 Mon Sep 17 00:00:00 2001
From: Adrian Reber <adr...@lisas.de>
Date: Tue, 17 Apr 2018 19:42:16 +0200
Subject: [PATCH] mirror crawler: crawl each category separately

This is the first try to split up the mirror crawling by category. One
of the goals is to better distribute the load on the database. If this
actually works the effects of this change have to be monitored.

Another result could be that mirrors do not get auto-deactivated that
fast. Previously there was a crawl timeout of 4 hours for all categories
together. Now it is 4 hours per category.
---
 .../mirrormanager/crawler/files/crawler.cron  | 27 ---
 1 file changed, 24 insertions(+), 3 deletions(-)

diff --git a/roles/mirrormanager/crawler/files/crawler.cron 
b/roles/mirrormanager/crawler/files/crawler.cron
index 33f3967f3..24d141574 100644
--- a/roles/mirrormanager/crawler/files/crawler.cron
+++ b/roles/mirrormanager/crawler/files/crawler.cron
@@ -1,4 +1,4 @@
-# run the crawler twice a day
+# run the crawler for each MirrorManager category
 # logs sent to /var/log/mirrormanager/crawler.log and crawl/* by default
 #
 # [ "`hostname -s`" == "mm-crawler02" ] && sleep 6h is used to start the crawl
@@ -10,5 +10,26 @@
 # gracefully shutdown if it gets the signal SIGALRM(14).  After the signal we
 # wait for 5 minutes to give the crawler a chance to shutdown. After that the
 # crawler is killed.  To make sure we only end the cron started crawler we look
-# for the following process "/usr/bin/python /usr/bin/mm2_crawler --threads 
25".
-0 */12 * * * mirrormanager [ "`hostname -s`" == "mm-crawler02" ] && sleep 6h; 
pkill -14 -f "^/usr/bin/python2 -s /usr/bin/mm2_crawler --threads 20"; sleep 
5m; pkill -9 -f "^/usr/bin/python2 -s /usr/bin/mm2_crawler --threads 20"; 
/usr/bin/mm2_crawler --threads 20 --timeout-minutes 240 
`/usr/local/bin/run_crawler.sh 2` > /dev/null 2>&1
+# for the following process "/usr/bin/python /usr/bin/mm2_crawler 
--category=25".
+
+# The number of threads is based on the possible number of existing mirrors. 
More
+# threads for categories with more mirrors.
+
+# The goal is to distribute the crawling of all categories over the whole day.
+
+# The timeout is 4 hours, but for each category.
+
+# Category: 'Fedora Linux'; twice a day, 20 threads
+0 */12 * * * mirrormanager [ "`hostname -s`" == "mm-crawler02" ] && sleep 6h; 
pkill -14 -f "^/usr/bin/python2 -s /usr/bin/mm2_crawler --category=Fedora 
Linux"; sleep 5m; pkill -9 -f "^/usr/bin/python2 -s /usr/bin/mm2_crawler 
--category=Fedora Linux"; /usr/bin/mm2_crawler --category="Fedora Linux" 
--threads 20 --timeout-minutes 240 `/usr/local/bin/run_crawler.sh 2` > 
/dev/null 2>&1
+
+# Category: 'Fedora Secondary Arches'; twice a day, 10 threads
+0 3,9 * * * mirrormanager [ "`hostname -s`" == "mm-crawler02" ] && sleep 1h; 
pkill -14 -f "^/usr/bin/python2 -s /usr/bin/mm2_crawler --category=Fedora 
Secondary Arches"; sleep 5m; pkill -9 -f "^/usr/bin/python2 -s 
/usr/bin/mm2_crawler --category=Fedora Secondary Arches"; /usr/bin/mm2_crawler 
--category="Fedora Secondary Arches" --threads 10 --timeout-minutes 240 
`/usr/local/bin/run_crawler.sh 2` > /dev/null 2>&1
+
+# Category: 'Fedora EPEL'; four times a day, 20 threads
+45 */6 * * * mirrormanager [ "`hostname -s`" == "mm-crawler02" ] && sleep 1h; 
pkill -14 -f "^/usr/bin/python2 -s /usr/bin/mm2_crawler --category=Fedora 
EPEL"; sleep 5m; pkill -9 -f "^/usr/bin/python2 -s /usr/bin/mm2_crawler 
--category=Fedora EPEL"; /usr/bin/mm2_crawler --category="Fedora EPEL" 
--threads 20 --timeout-minutes 240 `/usr/local/bin/run_crawler.sh 2` > 
/dev/null 2>&1
+
+# Category: 'Fedora Archive'; once a day, 10 threads
+0 2 * * * mirrormanager [ "`hostname -s`" == "mm-crawler02" ] && sleep 6h; 
pkill -14 -f "^/usr/bin/python2 -s /usr/bin/mm2_crawler --category=Fedora 
Archive"; sleep 5m; pkill -9 -f "^/usr/bin/python2 -s /usr/bin/mm2_crawler 
--category=Fedora Archive"; /usr/bin/mm2_crawler --category="Fedora Archive" 
--threads 10 --timeout-minutes 240 `/usr/local/bin/run_crawler.sh 2` > 
/dev/

Re: Planned MirrorManager changes

[Adrian Reber]

On Sat, Apr 14, 2018 at 04:28:37PM -0700, Kevin Fenzi wrote:
> > I would like to change the setup of our mirror crawler and just wanted
> > to mention my planned changes here before working on them.
> > 
> > Currently we have two VMs which are crawling our mirrors. Each of the
> > machine is responsible for one half of the active mirrors. The crawl
> > starts every 12 hours on the first crawler and 6 hours later on the
> > second crawler. So every 6 hours one crawler is accessing the database.
> > 
> > Currently most of the crawling time is not spent crawling but updating
> > the database about which host has which directory up to date. With a
> > timeout of 4 hours per host we are hitting that timeout on some hosts
> > regularly and most of the time the database access is the problem.
> > 
> > What I would like to change is to crawl each category (Fedora Linux,
> > Fedora Other, Fedora EPEL, Fedora Secondary Arches, Fedora Archive)
> > separately and at different times and intervals.
> > 
> > We would not hit the timeout as often as now as only the information for
> > a single category has to be updated. We could scan 'Fedora Archive' only
> > once per day or every second day. We can scan 'Fedora EPEL' much more
> > often as it is usually really fast and get better data about the
> > available mirrors.
> > 
> > My goal would be to distribute the scanning in such a way to decrease
> > the load on the database and to decrease the cases of mirror
> > auto-deactivation due to slow database accesses. 
> > 
> > Let me know if you think that these planned changes are the wrong
> > direction of if you have other ideas how to improve the mirror crawling.
> 
> Sounds like all great ideas to me. ;)

Thanks.

> I wonder if we could also find some way to note which mirrors have
> iso/image files, and could communicate this to the
> download.fedoraproject.org redirect to only redirect people to mirrors
> that have that specific file if they are pointing to an iso/qcow2, etc.

This is one of the cases where MirrorManager, in theory, should almost
handle it correctly. The important part of this sentence is 'in theory'.
MirrorManager should know about the 3 most recent files in a directory
and if we are crawling via rsync we even download the complete listing
for a mirror. So besides the theory it would help to see a wrong
redirect live to understand why it is happening.

Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org

Re: Planned MirrorManager changes

[Adrian Reber]

On Sat, Apr 14, 2018 at 12:37:24AM +, Stephen John Smoogen wrote:
> On Fri, Apr 13, 2018 at 11:14 AM Adrian Reber <adr...@lisas.de> wrote:
> 
> > I would like to change the setup of our mirror crawler and just wanted
> > to mention my planned changes here before working on them.
> >
> > Currently we have two VMs which are crawling our mirrors. Each of the
> > machine is responsible for one half of the active mirrors. The crawl
> > starts every 12 hours on the first crawler and 6 hours later on the
> > second crawler. So every 6 hours one crawler is accessing the database.
> >
> > Currently most of the crawling time is not spent crawling but updating
> > the database about which host has which directory up to date. With a
> > timeout of 4 hours per host we are hitting that timeout on some hosts
> > regularly and most of the time the database access is the problem.
> >
> > What I would like to change is to crawl each category (Fedora Linux,
> > Fedora Other, Fedora EPEL, Fedora Secondary Arches, Fedora Archive)
> > separately and at different times and intervals.
> >
> > We would not hit the timeout as often as now as only the information for
> > a single category has to be updated. We could scan 'Fedora Archive' only
> > once per day or every second day. We can scan 'Fedora EPEL' much more
> > often as it is usually really fast and get better data about the
> > available mirrors.
> >
> > My goal would be to distribute the scanning in such a way to decrease
> > the load on the database and to decrease the cases of mirror
> > auto-deactivation due to slow database accesses.
> >
> > Let me know if you think that these planned changes are the wrong
> > direction of if you have other ideas how to improve the mirror crawling.
> 
> These look like a good way to deal with the fact that we have a lot of data
> and files and mirrors nd users get confused about how up to date they are.
> Would more VM’s help spread this out also?

From my point of view the main problem is the load MirrorManager creates
on the database. Currently I do not think that more VMs would help the
crawling. Someone once mentioned a dedicated database VM for
MirrorManager. That is something which could make a difference, but
first I would like to see if crawling per category can improve the
situation.

Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org

Planned MirrorManager changes

[Adrian Reber]

I would like to change the setup of our mirror crawler and just wanted
to mention my planned changes here before working on them.

Currently we have two VMs which are crawling our mirrors. Each of the
machine is responsible for one half of the active mirrors. The crawl
starts every 12 hours on the first crawler and 6 hours later on the
second crawler. So every 6 hours one crawler is accessing the database.

Currently most of the crawling time is not spent crawling but updating
the database about which host has which directory up to date. With a
timeout of 4 hours per host we are hitting that timeout on some hosts
regularly and most of the time the database access is the problem.

What I would like to change is to crawl each category (Fedora Linux,
Fedora Other, Fedora EPEL, Fedora Secondary Arches, Fedora Archive)
separately and at different times and intervals.

We would not hit the timeout as often as now as only the information for
a single category has to be updated. We could scan 'Fedora Archive' only
once per day or every second day. We can scan 'Fedora EPEL' much more
often as it is usually really fast and get better data about the
available mirrors.

My goal would be to distribute the scanning in such a way to decrease
the load on the database and to decrease the cases of mirror
auto-deactivation due to slow database accesses. 

Let me know if you think that these planned changes are the wrong
direction of if you have other ideas how to improve the mirror crawling.

Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org

Re: FBR: add exclude to mirror crawler

[Adrian Reber]

Pushed and live on both MM crawlers.

Adrian

On Thu, Mar 08, 2018 at 04:37:41PM +0100, Adrian Reber wrote:
> The rsync based crawler has two excludes hardcoded:
> 
>  cmd = "rsync --temp-dir=/tmp -r --exclude=.snapshot --exclude='*.~tmp~'"
> 
> I am just in discussions with a new mirror and the rsync based crawler
> fails there with:
> 
>  rsync: opendir "/lost+found" (in epel) failed: Permission denied (13)
> 
> To avoid this in this case and also on other mirrors I would like to
> apply following patch:
> 
> diff --git a/roles/mirrormanager/frontend2/templates/mirrormanager2.cfg 
> b/roles/mirrormanager/frontend2/templates/mirrormanager2.cfg
> index 2bdd60273..69c930b23 100644
> --- a/roles/mirrormanager/frontend2/templates/mirrormanager2.cfg
> +++ b/roles/mirrormanager/frontend2/templates/mirrormanager2.cfg
> @@ -161,7 +161,7 @@ CHECK_SESSION_IP = True
>  
>  # Specify additional rsync parameters for the crawler
>  # # --timeout 14400: abort rsync crawl after 4 hours
> -CRAWLER_RSYNC_PARAMETERS = '--no-motd --timeout 14400'
> +CRAWLER_RSYNC_PARAMETERS = '--no-motd --timeout 14400 --exclude=lost+found'
>  
>  ###
>  # Configuration options used by the crons
> 
> 
> This excludes 'lost+found' from being scanned on all hosts. Can I get two +1
> to update the crawler's configuration?
> 
>   Adrian



> ___
> infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org


Adrian

-- 
Adrian Reber <adr...@lisas.de>http://lisas.de/~adrian/
Change in Earth's rotational speed


signature.asc
Description: PGP signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org

FBR: add exclude to mirror crawler

[Adrian Reber]

The rsync based crawler has two excludes hardcoded:

 cmd = "rsync --temp-dir=/tmp -r --exclude=.snapshot --exclude='*.~tmp~'"

I am just in discussions with a new mirror and the rsync based crawler
fails there with:

 rsync: opendir "/lost+found" (in epel) failed: Permission denied (13)

To avoid this in this case and also on other mirrors I would like to
apply following patch:

diff --git a/roles/mirrormanager/frontend2/templates/mirrormanager2.cfg 
b/roles/mirrormanager/frontend2/templates/mirrormanager2.cfg
index 2bdd60273..69c930b23 100644
--- a/roles/mirrormanager/frontend2/templates/mirrormanager2.cfg
+++ b/roles/mirrormanager/frontend2/templates/mirrormanager2.cfg
@@ -161,7 +161,7 @@ CHECK_SESSION_IP = True
 
 # Specify additional rsync parameters for the crawler
 # # --timeout 14400: abort rsync crawl after 4 hours
-CRAWLER_RSYNC_PARAMETERS = '--no-motd --timeout 14400'
+CRAWLER_RSYNC_PARAMETERS = '--no-motd --timeout 14400 --exclude=lost+found'
 
 ###
 # Configuration options used by the crons


This excludes 'lost+found' from being scanned on all hosts. Can I get two +1
to update the crawler's configuration?

Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org

[release] MirrorManager2: 0.8.4

[Adrian Reber]

A new release of MirrorManager2 is available: 0.8.4

- Correctly handle Fedora 28 modular layout
  https://github.com/fedora-infra/mirrormanager2/pull/242
- Use "site", "host" and "mirror" consistently
  https://github.com/fedora-infra/mirrormanager2/pull/241
- crawler: support https only hosts
  https://github.com/fedora-infra/mirrormanager2/pull/240
- Make mm2_get_internet2_netblocks work again
  https://github.com/fedora-infra/mirrormanager2/pull/234
- crawler: use timeout also on rsync crawls
  https://github.com/fedora-infra/mirrormanager2/pull/229
- Fix existing test cases and re-enable tests on commits
- Enable tests in the %%check section
- publiclist: hide disabled arches and products
  https://github.com/fedora-infra/mirrormanager2/pull/223


Main reason for this release is to support the new layout of the
'Modular' tree.

Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org

[PATCH] Added script to check age of mirrorlist pkl

[Adrian Reber]

The following script can be used for nagios to contact the mirrolist
servers and query the age of the pickle currently loaded.

This output of the script looks like this:

 $./check_mirrorlist_pkl_age.py 02 3000 5000
 WARN: mirrorlist data on proxy02 older than 3000s (3894s)

 $ ./check_mirrorlist_pkl_age.py 14 5000 7000
 OK: up-to-date mirrorlist data on proxy14 (3921s)

 $ ./check_mirrorlist_pkl_age.py 08 2000 3000
 CRIT: mirrorlist data on proxy08 older than 3000s (3938s)

Signed-off-by: Adrian Reber <adr...@lisas.de>
---
 .../nagios/plugins/check_mirrorlist_pkl_age.py | 96 ++
 1 file changed, 96 insertions(+)
 create mode 100755 
roles/nagios_server/files/nagios/plugins/check_mirrorlist_pkl_age.py

diff --git 
a/roles/nagios_server/files/nagios/plugins/check_mirrorlist_pkl_age.py 
b/roles/nagios_server/files/nagios/plugins/check_mirrorlist_pkl_age.py
new file mode 100755
index 0..2aa5bb2ca
--- /dev/null
+++ b/roles/nagios_server/files/nagios/plugins/check_mirrorlist_pkl_age.py
@@ -0,0 +1,96 @@
+#!/usr/bin/python3
+#
+# Script to check the age of the data used by the mirrorlist servers.
+#
+# Fedora's mirrorlist server are using a python pkl which has the creation
+# date embedded. Querying the mirrorlist interface with '' returns
+# that timestamp:
+#
+#  '# database creation time: '
+#
+# This script connects to the specified proxy and reads out that value
+# and compares it with a warning and critical threshold.
+#
+# Tested with python2 and python3
+#
+# Requires python[2,3]-requests
+#
+# Usage:
+#  check_mirrorlist_pkl_age.py   
+#  check_mirrorlist_pkl_age.py 12 3600 7200
+#
+# Author: Adrian Reber <adr...@lisas.de>
+
+import requests
+import sys
+import time
+import datetime
+
+if len(sys.argv) != 4:
+print("Usage:")
+print(" %s needs 3 parameters\n" % sys.argv[0])
+print(" %s   " % sys.argv[0])
+print(" %s 12 3600 7200" % sys.argv[0])
+sys.exit(3)
+
+proxy = sys.argv[1]
+warn = int(sys.argv[2])
+crit = int(sys.argv[3])
+
+check_url = 'http://proxy%s.fedoraproject.org/' % proxy
+check_url += 'mirrorlist?repo=fedora-rawhide=x86_64'
+
+headers = {'Host': 'mirrors.fedoraproject.org'}
+
+try:
+r = requests.get(check_url, headers=headers)
+except:
+print('CRIT: getting data from proxy%s failed' % proxy)
+sys.exit(2)
+
+
+if r.status_code != 200:
+print('CRIT: unexpected response (not 200) from proxy%s' % proxy)
+sys.exit(2)
+
+for line in r.iter_lines():
+if b'database creation time' not in line:
+continue
+
+ts = 0
+now = 0
+try:
+time_from_proxy = line.decode().split(': ')[1:][0]
+ts = datetime.datetime.strptime(
+time_from_proxy,
+'%Y-%m-%d %H:%M:%S.%f'
+)
+ts = int(time.mktime(ts.timetuple()))
+
+now = datetime.datetime.utcnow()
+now = int(time.mktime(now.timetuple()))
+
+except:
+print('CRIT: failure parsing result from proxy%s' % proxy)
+sys.exit(2)
+
+if (len == 0) or (now == 0):
+print('CRIT: failure parsing result from proxy%s' % proxy)
+sys.exit(2)
+
+age = int(now - ts)
+
+if age > crit:
+print(
+'CRIT: mirrorlist data on proxy%s older than %ds (%ds)' %
+(proxy, crit, age))
+sys.exit(2)
+
+if age > warn:
+print(
+'WARN: mirrorlist data on proxy%s older than %ds (%ds)' %
+(proxy, warn, age))
+sys.exit(1)
+
+print('OK: up-to-date mirrorlist data on proxy%s (%ds)' % (proxy, age))
+sys.exit(0)
-- 
2.14.3
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org

Script to check mirrorlist age

[Adrian Reber]

The following patch would add a script to check for the age of the
mirrorlist servers. If somebody could add the necessary nagios
configuration to let this run against the proxies I would add this
script to the repository.

The script makes it easy to verify that the mirrorlist containers have
up-to-date data:

 $ ./check_mirrorlist_pkl_age.py 02 3000 5000
 WARN: mirrorlist data on proxy02 older than 3000s (3894s)

Is this a check that would make sense for the current Nagios setup? We
probably need to figure out the right values for warning and critical.

Adrian
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org

Re: FBR: Updating MirrorManager to 0.8.3

[Adrian Reber]

On Wed, Sep 27, 2017 at 08:20:51AM +0200, Adrian Reber wrote:
> The new MirrorManager2 0.8.3 release correctly detects and creates
> repositories for the newly created 'modular' tree:
> 
>  /srv/pub/fedora/linux/modular/
> 
> I have installed the new release in staging and it works correctly. The
> changes between MirrorManager2 0.8.1 which is installed in production
> and the newly released 0.8.3 is just the 'modular' repository detection
> and some additional repository creation/detection code which I also
> already used in production successfully.
> 
> The mirrormanager2-0.8.3 is in the epel-testing repository.
> 
> As I am not aware how important it is to get this installed on the
> production systems during beta freeze I am hoping Dennis can tell us if
> this is necessary or not.
> 
> The changes are minimal and only related to repository detection and
> creation. Testing in staging was successful and I do not think it will
> break anything.
> 
> Should we install the 0.8.3 release during beta freeze or should we
> wait?

Patrick tagged the builds into the infrastructure repository and so I
was able to update mm-backend01.prod. Only this system.

I had to reset the ctimes in the database:

 => update directory set ctime=0 where name like '%modular%';

so that umdl picks up the repositories and now the modular repositories
are active:

$ curl -s 
'https://mirrors.fedoraproject.org/mirrorlist?repo=modular-bikeshed-server=x86_64'
 | wc -l
27

Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org

FBR: Updating MirrorManager to 0.8.3

[Adrian Reber]

The new MirrorManager2 0.8.3 release correctly detects and creates
repositories for the newly created 'modular' tree:

 /srv/pub/fedora/linux/modular/

I have installed the new release in staging and it works correctly. The
changes between MirrorManager2 0.8.1 which is installed in production
and the newly released 0.8.3 is just the 'modular' repository detection
and some additional repository creation/detection code which I also
already used in production successfully.

The mirrormanager2-0.8.3 is in the epel-testing repository.

As I am not aware how important it is to get this installed on the
production systems during beta freeze I am hoping Dennis can tell us if
this is necessary or not.

The changes are minimal and only related to repository detection and
creation. Testing in staging was successful and I do not think it will
break anything.

Should we install the 0.8.3 release during beta freeze or should we
wait?

Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org

[release] MirrorManager2: 0.8.3

[Adrian Reber]

On Tue, Sep 26, 2017 at 08:18:21AM +0200, Adrian Reber wrote:
> A new release of MirrorManager2 is available: 0.8.2
> 
> - detect and setup mirrorlist/metalinks for modular Fedora
>   https://github.com/fedora-infra/mirrormanager2/pull/220
> - umdl: only create repositories for 'Everything'
>   https://github.com/fedora-infra/mirrormanager2/pull/219
> - Correctly detect repositories
>   https://github.com/fedora-infra/mirrormanager2/pull/218
> 
> The main reason for the release are the changes to detect the
> modular release directory tree correctly.

After trying the 0.8.2 release in staging a few small changes were
necessary to correctly detect the 'modular' repositories. These changes
are in the 0.8.3 release:

- umdl: fix 'modular' repository detection
  https://github.com/fedora-infra/mirrormanager2/pull/221

Now that 0.8.3 is running in staging the following repositories have
been created/found:

# repo=modular-bikeshed-server=aarch64
# repo=modular-bikeshed-server=armhfp
# repo=modular-bikeshed-server=i386
# repo=modular-bikeshed-server=ppc64
# repo=modular-bikeshed-server=ppc64le
# repo=modular-bikeshed-server=s390x
# repo=modular-bikeshed-server=x86_64
# repo=modular-bikeshed-server-debug=aarch64
# repo=modular-bikeshed-server-debug=armhfp
# repo=modular-bikeshed-server-debug=i386
# repo=modular-bikeshed-server-debug=ppc64
# repo=modular-bikeshed-server-debug=ppc64le
# repo=modular-bikeshed-server-debug=s390x
# repo=modular-bikeshed-server-debug=x86_64
# repo=modular-bikeshed-server-source=source


Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org

[release] MirrorManager2: 0.8.2

[Adrian Reber]

A new release of MirrorManager2 is available: 0.8.2

- detect and setup mirrorlist/metalinks for modular Fedora
  https://github.com/fedora-infra/mirrormanager2/pull/220
- umdl: only create repositories for 'Everything'
  https://github.com/fedora-infra/mirrormanager2/pull/219
- Correctly detect repositories
  https://github.com/fedora-infra/mirrormanager2/pull/218

The main reason for the release are the changes to detect the
modular release directory tree correctly.

Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org

Re: [release] MirrorManager2: 0.8

[Adrian Reber]

On Fri, Jun 16, 2017 at 03:26:42PM +0200, Adrian Reber wrote:
> A new release of MirrorManager2 is available: 0.8
> 
> - Specify rel="noopener noreferrer" to link including target='_blank'
> - Improve the runserver script
> - Make the propagation script more robust
> - crawler: also crawl https-only mirrors
>   https://github.com/fedora-infra/mirrormanager2/issues/183
> - mm2_move-devel-to-release: adapt to latest repository layout
>   https://github.com/fedora-infra/mirrormanager2/issues/195
> - Private URLs are now restricted to admins
>   https://github.com/fedora-infra/mirrormanager2/issues/149
> - mirrorlist: at least 5 mirrors should be returned for
>   country/continent
>   https://github.com/fedora-infra/mirrormanager2/issues/194
> - Remove 'Master rsync server Access Control List IPs' section
>   https://github.com/fedora-infra/mirrormanager2/issues/145
> - mirrorlist: add pkl generation time to pkl
>   https://github.com/fedora-infra/mirrormanager2/issues/184
> - restrict non-admin users to certain netblock sizes
>   https://github.com/fedora-infra/mirrormanager2/issues/71
> - Change all references from fedorahosted.org to use the github area
> - umdl: add fullfiletimelist-* based master scanning
>   https://github.com/fedora-infra/mirrormanager2/issues/206
> 
> I am planing to update staging today and production once I was able to
> make sure nothing breaks.

This version is now running in staging and production.

The biggest change is that the update-master-directory-list master
mirror scanning now relies on fullfiletimelist-*. This means almost no
stat() during scan and it is much faster. See:

https://github.com/fedora-infra/mirrormanager2/pull/207/commits/4ef8c1991f7d58808db6c541731162c107776186

I was not able to update the mirrorlist containers and
mm-frontend-checkin01 as I do not have access to update those systems.
Would be great if someone could install the updates also on those
systems.

All my tests were successful but if anything behaves strange please let
me know.

Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org

[release] MirrorManager2: 0.8

[Adrian Reber]

A new release of MirrorManager2 is available: 0.8

- Specify rel="noopener noreferrer" to link including target='_blank'
- Improve the runserver script
- Make the propagation script more robust
- crawler: also crawl https-only mirrors
  https://github.com/fedora-infra/mirrormanager2/issues/183
- mm2_move-devel-to-release: adapt to latest repository layout
  https://github.com/fedora-infra/mirrormanager2/issues/195
- Private URLs are now restricted to admins
  https://github.com/fedora-infra/mirrormanager2/issues/149
- mirrorlist: at least 5 mirrors should be returned for
  country/continent
  https://github.com/fedora-infra/mirrormanager2/issues/194
- Remove 'Master rsync server Access Control List IPs' section
  https://github.com/fedora-infra/mirrormanager2/issues/145
- mirrorlist: add pkl generation time to pkl
  https://github.com/fedora-infra/mirrormanager2/issues/184
- restrict non-admin users to certain netblock sizes
  https://github.com/fedora-infra/mirrormanager2/issues/71
- Change all references from fedorahosted.org to use the github area
- umdl: add fullfiletimelist-* based master scanning
  https://github.com/fedora-infra/mirrormanager2/issues/206

I am planing to update staging today and production once I was able to
make sure nothing breaks.

Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org

Re: fedora/linux/development/25

[Adrian Reber]

On Tue, Jan 17, 2017 at 09:20:41AM -0600, Dennis Gilmore wrote:
> El lun, 16-01-2017 a las 09:53 +0100, Adrian Reber escribió:
> > For some time now mirrormanager points to fedora/linux/releases/25
> > for all
> > the categories 'Fedora Linux' and 'Fedora Secondary Arches'.
> > 
> > The files at fedora/linux/development/25 could now be deleted. At
> > least
> > from MirrorManager's point of view.
> > 
> 
> Te content is managed by release engineering, filing an issue with them
> or sending an email to thier list would be more appropriate going
> forward.

Good point: https://pagure.io/releng/issue/6595

Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org

fedora/linux/development/25

[Adrian Reber]

For some time now mirrormanager points to fedora/linux/releases/25 for all
the categories 'Fedora Linux' and 'Fedora Secondary Arches'.

The files at fedora/linux/development/25 could now be deleted. At least
from MirrorManager's point of view.

Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org

Re: mirror lists plan

[Adrian Reber]

On Tue, Sep 13, 2016 at 12:05:27PM -0600, Kevin Fenzi wrote:
> Greetings. 
> 
> We currently have 3 lists associated with mirrors. 2 of them are still
> located at redhat.com and we want to move them over to our
> infrastructure. 
> 
> mirror-adm...@lists.fedoraproject.org 

It actually is mirror-ad...@lists.fedoraproject.org (without the 's'
after admin). Which is probably irrelevant for this discussion.

> 
> This list is open to all (but only allows posting to subscribers, I
> moderate it and adrianr also does). This is usually the list we have
> people with questions or issues post to to get help mirroring. 
> 
> mirror-list-annou...@redhat.com

Without announce, just mirror-l...@redhat.com

> This list was for announcing new releases and sizes for mirrors so they
> would know to sync up and how large our next release was and to expect
> more traffic. It was setup back in the Fedora Core days so it was
> restricted to only approved mirror admins (because it would get notice
> of releases before they were public). 
> 
> mirror-lis...@redhat.com
> 
> This was another Fedora Core days list thats restricted to list admins
> to discuss space issues or mirroring problems or whatever. 
> 
> So, I propose we do this: 
> 
> * Make a new mirror-announce list. 
> * Mass invite mirror-list-annou...@redhat.com people to it. 
> * Make it open to all, but moderated
> * Mass invite mirror-lis...@redhaat.com subscribers to mirror-admins
>   list. 
> * Mail redhat.com lists and close them to new posts. 

So both lists will be open to all and moderated? Or will one list be
only open for members posts? Or does 'Make it open to all, but
moderated' mean anybody can subscribe but every post (also of members)
will be moderated?

> Then, moving forward we send announcements about upcoming releases to
> mirror-announce and important changes (like archiving a release or a
> new mirroring tool or whatever). 
> 
> Thoughts? Other ideas?

Sounds good. It is not clear yet which list has which mail acceptance
policy but as it can be changed anytime this might also be something
which can be adapted to our needs.

Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

Re: Proposal to mirror Docker images

[Adrian Reber]

On Thu, Sep 01, 2016 at 10:49:15AM -0400, Colin Walters wrote:
> On Tue, Aug 16, 2016, at 11:33 AM, Randy Barlow wrote:
>   
>   
> > In summary, the proposal is to write a patch for the docker client that 
> > 
> > will give it the capability to accept metalink responses upon docker
> > 
> > pull operations. We would also need to add support for Docker images to 
> > 
> > mirror list and mirror manager. Additionally, we will need a small tool 
> > 
> > to pull the content to be mirrored out of a docker registry and write   
> > 
> > them to disk in a format that can be mirrored, as well as some Ansible  
> > 
> > code to run the tool when there is new content to be mirrored. 
> 
> Related to this, I think it'd be useful to target public IaaS (AWS, GCE, etc.)
> for inside-infra mirrors.  Basically we want Fedora images to hit a S3 bucket 
> in
> the region or equivalent by default for content.  This is how Amazon 
> configures
> Amazon Linux.
> 
> It seems like the kind of thing that we could ask Red Hat for sponsorship.

This already happens for EPEL:

$ curl -s 
"https://mirrors.fedoraproject.org/mirrorlist?repo=epel-7=x86_64=52.3.178.0;
 | head -2
# repo = epel-7 arch = x86_64 Using preferred netblock Using Internet2 country 
= US country = CA 
http://s3-mirror-us-east-1.fedoraproject.org/pub/epel/7/x86_64/

We are currently syncing EPEL to the following S3 regions:

s3-mirror-us-east-1 s3-mirror-us-west-1 s3-mirror-us-west-2 s3-mirror-eu-west-1 
s3-mirror-ap-northeast-1

So, it sounds doable for an additional target (like docker images).

Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

Re: Mirrormanager doesn't provide F25 x86_64 debuginfo mirror for updates repository

[Adrian Reber]

On Wed, Aug 31, 2016 at 09:31:43PM -, Christian Stadelmann wrote:
> I'm trying to report a bug in a package (gnome-settings-daemon) and it would 
> be useful to have debuginfo for this package. When running `dnf 
> debuginfo-install gnome-settings-daemon`, I'm getting an error that the 
> repomd.xml was not found in metalink. Therefore I manually downloaded the 
> metalink file from 
> https://mirrors.fedoraproject.org/metalink?repo=updates-released-debug-f25=x86_64
>  (URL from /etc/yum.repos.d/fedora-updates.repo file) and had a look at the 
> file. 
> 
> The file itself just contains this content:
> 
> http://www.metalinker.org/; type="dynamic" 
> pubdate="Wed, 31 Aug 2016 21:05:11 GMT" generator="mirrormanager" 
> xmlns:mm0="http://fedorahosted.org/mirrormanager;>
> 
> 
> 
> but the file does not contain a line like
> 
> # repo=updates-released-debug-f25=x86_64
> 
> but it contains everything else:
> different architectures:
> # repo=updates-released-debug-f25=aarch64
> # repo=updates-released-debug-f25=ppc64
> # repo=updates-released-debug-f25=ppc64le
> # repo=updates-released-debug-f25=s390x
> 
> different repos:
> …
> # repo=fedora-debug-25=x86_64
> …
> # repo=updates-released-f25=x86_64
> …
> # repo=updates-released-source-f25=source
> …
> # repo=updates-testing-debug-f25=x86_64
> …
> 
> the repo I'd need is just missing.

I can only comment from the MirrorManager side. The repository is
missing indeed. But it is missing because it does not exist on the
master mirror server. The repository exists for the alternate
architectures but not for for x86_64, i386 and armhfp. I guess this is
something rel-eng can fix.

Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

[release] MirrorManager2: 0.7.3

[Adrian Reber]

A new release of MirrorManager2 is available: 0.7.3

* Thu Jun 23 2016 Adrian Reber <adr...@lisas.de> - 0.7.3-1
- Update to 0.7.3
- Allow submission of checkin information via json (Patrick Uiterwijk)
  https://github.com/fedora-infra/mirrormanager2/issues/170
- Add logging to checkin code (Patrick Uiterwijk)
- mm2_crawler: Add missing field to stats dict
  https://github.com/fedora-infra/mirrormanager2/issues/176
- mirrolist: fix =1
  https://github.com/fedora-infra/mirrormanager2/issues/178

This is currently running in stg for those who want to test it.
We will update the production systems later.

Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

Re: [ansible] Disable crawler and automatic pkl push before the release

[Adrian Reber]

I git two +1's on IRC to quickly push this out. The goal was to better
control what is crawled and when the result is published on the
mirrorlist servers. I will revert this after the release to return to
normal operation.

Adrian

On Tue, Jun 21, 2016 at 11:57:36AM +, Adrian Reber wrote:
> This is the public repository, do not commit sensitive
> or confidential information here.
> X-Git-Refname: refs/heads/master
> X-Git-Oldrev: 12d3665003022025baa60c9980f6a48ae6c6bdb9
> X-Git-Newrev: 8e83d2cc8f827e6124ac698aef37ec123e9f3eb5
> 
> commit 8e83d2cc8f827e6124ac698aef37ec123e9f3eb5
> Author: Adrian Reber <adr...@lisas.de>
> Date:   Tue Jun 21 11:53:39 2016 +
> 
> Disable crawler and automatic pkl push before the release
> 
> To better control what is available on the mirrorlist servers the
> cronjob are disabled.
> 
> Signed-off-by: Adrian Reber <adr...@lisas.de>
> 
>  roles/mirrormanager/backend/files/backend.cron | 2 +-
>  roles/mirrormanager/crawler/files/crawler.cron | 2 +-
>  2 files changed, 2 insertions(+), 2 deletions(-)
> ---
> diff --git a/roles/mirrormanager/backend/files/backend.cron 
> b/roles/mirrormanager/backend/files/backend.cron
> index b63d3fb..bcf4cde 100644
> --- a/roles/mirrormanager/backend/files/backend.cron
> +++ b/roles/mirrormanager/backend/files/backend.cron
> @@ -2,7 +2,7 @@ MAILTO=root
>  
>  # Refresh the mirrorlist cache at the top of the hour and sync it to the
>  # mirrorlist servers
> -55 * * * * mirrormanager /usr/bin/mm2_update-mirrorlist-server && 
> /usr/local/bin/sync_pkl_to_mirrorlists.sh
> +#55 * * * * mirrormanager /usr/bin/mm2_update-mirrorlist-server && 
> /usr/local/bin/sync_pkl_to_mirrorlists.sh
>  
>  # update master directory
>  # logs sent to /var/log/mirrormanager/umdl.log by default
> diff --git a/roles/mirrormanager/crawler/files/crawler.cron 
> b/roles/mirrormanager/crawler/files/crawler.cron
> index 2b68c76..1060aa1 100644
> --- a/roles/mirrormanager/crawler/files/crawler.cron
> +++ b/roles/mirrormanager/crawler/files/crawler.cron
> @@ -4,4 +4,4 @@
>  # [ "`hostname -s`" == "mm-crawler02" ] && sleep 2h is used to start the 
> crawl
>  # later on the second crawler to reduce the number of parallel accesses to
>  # the database
> -0 */12 * * * mirrormanager [ "`hostname -s`" == "mm-crawler02" ] && sleep 
> 2h; /usr/bin/mm2_crawler --timeout-minutes 180 --threads 20 
> `/usr/local/bin/run_crawler.sh 2` > /dev/null 2>&1
> +#0 */12 * * * mirrormanager [ "`hostname -s`" == "mm-crawler02" ] && sleep 
> 2h; /usr/bin/mm2_crawler --timeout-minutes 180 --threads 20 
> `/usr/local/bin/run_crawler.sh 2` > /dev/null 2>&1


signature.asc
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

Re: Failed to synchronize cache for repo 'fedora', disabling.

[Adrian Reber]

On Fri, May 27, 2016 at 11:12:39AM -0600, Kevin Fenzi wrote:
> On Fri, 27 May 2016 10:35:43 +0200
> Adrian Reber <adr...@lisas.de> wrote:
> 
> > I wrote details about this in the above mentioned issue. I am not
> > convinced it is a MirrorManager bug. It actually exactly did what it
> > is supposed to do. The problem was that it had a newer checksum and
> > date of a repomd.xml file which does not exist on disk anymore. It
> > would be interesting to know if this newer repomd.xml file was at
> > some point actually available on the filesystem. Which seems hard to
> > find out.
> 
> Weird. I cannot think of a reason why it would have updated then
> un-updated. 

Yes. No idea. It's just strange that metalink checksum timestamp was
newer than the file and the checksum was different.

> > Ah, I could have a look at the netapp snapshots. But I don't seem
> > them. Are there netapps snapshots enable for
> > 
> > ntap-phx2-c01-fedora01-nfs.storage.phx2.redhat.com:/fedora_ftp/fedora.redhat.com/pub
> > 
> > Any way to look at older versions of that directory?
> 
> There are snapshots, but currently there's 7 of them one every 4 hours,
> so thats only 28 hours ago. This happened longer than that right?

No. It would have been interesting to see the state at:

$ date -ud @1464076094
Tue 24 May 07:48:14 UTC 2016

That was the timestamp in the metalink.

According to the current propagation result:

https://admin.fedoraproject.org/mirrormanager/propgation

Something changed between 2016-05-24-10-28-14 and 2016-05-24-14-28-14

Ah, the propagation logs might be a good place to check.

The following is the timestamp and the value of the sha256sum of
repomd.xml of
/srv/pub/fedora/linux/development/rawhide/Everything/x86_64/os/repodata
in the database. Shortened to only include the timestamps when it
changes:

2016-05-23-10-28-20 --- 
4b7495d42f9661ee9f1dcab1f235bba883d619cc5185b09e191c19429aa61e9e
2016-05-24-14-28-15 --- 
6c00530839492c598ba2daa41f068f9f673d39063645debb69e97a5a06dadc1a
2016-05-24-18-27-36 --- 
4414edff46c7a2d9754a5b711d23227e7840031871f0049d5a546062592964f7
2016-05-26-08-28-38 --- 
6c00530839492c598ba2daa41f068f9f673d39063645debb69e97a5a06dadc1a
2016-05-26-16-27-41 --- 
4414edff46c7a2d9754a5b711d23227e7840031871f0049d5a546062592964f7

last propagation check in the log:

2016-05-27-16-27-28 --- 
4414edff46c7a2d9754a5b711d23227e7840031871f0049d5a546062592964f7

So it seems to change sometimes back and forth. Which is strange.

Patrick, would did you do to fix the metalink errors on 2016-05-26? Did you wait
for a new compose or did you touch the directory and repomd.xml file so that
umdl picks up the 'changes'.

Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

Re: Failed to synchronize cache for repo 'fedora', disabling.

[Adrian Reber]

On Thu, May 26, 2016 at 09:14:55AM -0600, Kevin Fenzi wrote:
> On Thu, 26 May 2016 06:11:57 -0400 (EDT)
> Jun Aruga  wrote:
> 
> > Hi,
> > 
> > I got an error in mock building in my environment for fedora-rawhide
> > from today. Yesterday I could do mock building. My colleague also got
> > same error for another package's mock building today.
> > 
> > Is anyone working in the system now?
> 
> This seems to be a mirrormanager bug.
> 
> ( https://github.com/fedora-infra/mirrormanager2/issues/96 )
> 
> Basically it's supposed to each time rawhide updates add the new
> repomd.xml in and keep the previous 2 around for slow updating mirrors.
> However, we haven't had any rawhide composes that worked in the last 3
> days, so it rolled off the last known good repomd.xml leaving
> nothing. ;( 

I wrote details about this in the above mentioned issue. I am not
convinced it is a MirrorManager bug. It actually exactly did what it is
supposed to do. The problem was that it had a newer checksum and date of
a repomd.xml file which does not exist on disk anymore. It would be
interesting to know if this newer repomd.xml file was at some point
actually available on the filesystem. Which seems hard to find out.

Ah, I could have a look at the netapp snapshots. But I don't seem them.
Are there netapps snapshots enable for

ntap-phx2-c01-fedora01-nfs.storage.phx2.redhat.com:/fedora_ftp/fedora.redhat.com/pub

Any way to look at older versions of that directory?

Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

Re: RFC: Fedora Scale-Out Docker Registry Proposal

[Adrian Reber]

On Tue, May 10, 2016 at 08:52:00AM -0400, Stephen John Smoogen wrote:
> On 10 May 2016 at 03:02, Adrian Reber <adr...@lisas.de> wrote:
> > On Mon, May 09, 2016 at 01:49:26PM -0600, Kevin Fenzi wrote:
> >> On Fri, 6 May 2016 17:30:18 -0500
> >> Adam Miller <maxamill...@fedoraproject.org> wrote:
> >>
> >> ...snip...
> >>
> >> >
> >> > Proposal:
> >> >
> >> > Pulp[1] + Crane[2] + MirrorManager[3] + Docker Distribution[4]
> >>
> >> Are all of these packaged up? For EPEL?
> >> (aside mirrormanager).
> >>
> >> ...snip...
> >>
> >> > Workflow:
> >> > OSBS will perform Builds, as these builds complete they will be
> >> > pushed to the docker-distribution (v2) registry, these will be
> >> > considered "candidate images". Pulp will sync and publish the
> >> > candidate repository.
> >> >
> >> > Testing will occur using the "candidate images" (details of how
> >> > we want to handle that are outside the scope of this proposal).
> >>
> >> So, at this point the 'candidate image' is just in pulp?
> >> Or it's been published to a directory and mirrored out?
> >> I'm guessing it would be published and mirrored so people could test it?
> >>
> >> ...snip...
> >>
> >> > Mirror Manager will Pulp distribute to the mirrors the image
> >> > layers and their metadata.
> >>
> >> This should work for mirrors to just rsync the directories right?
> >
> > From a MirrorManager point of view it would be important, that this
> > does not result in thousands of new files. The atomic directory contains
> > over 70 files which massively increases sync time for the mirrors
> > and crawl times from our side as well as an increase in database size.
> > Depending on the number of files this might be a candidate for a new
> > rsync module.
> >
> 
> I was going to bring up that I think that atomic is a good candidate
> for its own sync module. The amount of lookups having to be done on
> the atomic directory has really increased the amount of IOPS on the
> data where only certain groups are interested in it.

Agreed. I just did not want to go into more details of the atomic tree
in the this thread. Important for me was to mention to not further
increase the number of files in fedora/linux/.

Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

Re: RFC: Fedora Scale-Out Docker Registry Proposal

[Adrian Reber]

On Mon, May 09, 2016 at 01:49:26PM -0600, Kevin Fenzi wrote:
> On Fri, 6 May 2016 17:30:18 -0500
> Adam Miller  wrote:
> 
> ...snip...
> 
> > 
> > Proposal:
> > 
> > Pulp[1] + Crane[2] + MirrorManager[3] + Docker Distribution[4]
> 
> Are all of these packaged up? For EPEL?
> (aside mirrormanager).
> 
> ...snip...
> 
> > Workflow:
> > OSBS will perform Builds, as these builds complete they will be
> > pushed to the docker-distribution (v2) registry, these will be
> > considered "candidate images". Pulp will sync and publish the
> > candidate repository.
> > 
> > Testing will occur using the "candidate images" (details of how
> > we want to handle that are outside the scope of this proposal).
> 
> So, at this point the 'candidate image' is just in pulp? 
> Or it's been published to a directory and mirrored out?
> I'm guessing it would be published and mirrored so people could test it?
> 
> ...snip...
> 
> > Mirror Manager will Pulp distribute to the mirrors the image
> > layers and their metadata.
> 
> This should work for mirrors to just rsync the directories right?

From a MirrorManager point of view it would be important, that this
does not result in thousands of new files. The atomic directory contains
over 70 files which massively increases sync time for the mirrors
and crawl times from our side as well as an increase in database size.
Depending on the number of files this might be a candidate for a new
rsync module.

Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

Re: [ansible] Remove mirror01.prgmr.com, it's now the same as mirror.prgmr.com

[Adrian Reber]

On Tue, Apr 19, 2016 at 10:55:12AM -0600, Kevin Fenzi wrote:
> On Fri, 15 Apr 2016 18:46:02 +0200
> Adrian Reber <adr...@lisas.de> wrote:
> 
> > These diffs of the rsync ACLs are not very useful. For me it is almost
> > impossible to see if and what has changed. It also seems we have to
> > maintain the ACL in 4 or 5 different files. Can the rsync ACL not be
> > handled in a more ansible way? I don't know much about ansible but
> > wouldn't it be possible to maintain the ACL one time with something
> > like this:
> > 
> > - name rsync acl
> >   template: some template with a loop statement over items
> >   with_items:
> >   - ip1
> >   - ip2
> >   - host1
> >   - host2
> > 
> > That would make the diffs readable any maybe we could maintain the
> > rsync ACL in only one place. Before trying to implement it I wanted
> > to see if there are some better ideas/ways to implement this
> > 'correctly'.
> 
> Yeah, I am all for this change. We should be able to use ansible
> variables and a template and make it much more readable. 
> Additionally we can then easily see who added a line when. 
> 
> If someone wants to come up with a patch and test it out, please do and
> we can look at it as a freeze break. 

I can try to come up with a patch. Which location would be the most
suited to hold the ACL content. Should this be in yml file or rather
under group_vars or host_vars (or some other place)?

Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

Re: [ansible] Remove mirror01.prgmr.com, it's now the same as mirror.prgmr.com

[Adrian Reber]

On Fri, Apr 15, 2016 at 02:01:34PM +, Nick Bebout wrote:
> diff --git a/roles/rsyncd/files/rsyncd.conf.download-ibiblio 
> b/roles/rsyncd/files/rsyncd.conf.download-ibiblio
> index 15375aa..478bf48 100644
> --- a/roles/rsyncd/files/rsyncd.conf.download-ibiblio
> +++ b/roles/rsyncd/files/rsyncd.conf.download-ibiblio
> @@ -68,7 +68,7 @@ refuse options = checksum
> list = no
> uid = 263
> gid = 263
> -   hosts allow = jobbot1.ibiblio.org 200.17.202.1/28 zeus1.kernel.org 
> zeus2.kernel.org zeus3.kernel.org zeus4.kernel.org 149.20.20.132 
> 204.152.191.36 199.6.1.170 130.239.17.3 sinclair.wpi.edu 
> bonaparte.hrz.tu-chemnitz.de josephine.hrz.tu-chemnitz.de 
> mirror.speedpartner.de rsyncer.ftp.heanet.ie archive.linux.duke.edu 
> lists.us.dell.com auslistsprd01.us.dell.com auslistsdr01.us.dell.com 
> 198.129.224.34 mirror.hiwaay.net sagres.c3sl.ufpr.br mail.fedoraunity.org 
> scrye.com odysseus.fi.muni.cz odysseus.linux.cz rhlx01.hs-esslingen.de 
> ftp.nrc.ca zaphod.gtlib.gatech.edu 128.171.104.148 129.21.171.98 
> torrent01.fedoraproject.org torrent02.fedoraproject.org sunsite.mff.cuni.cz 
> sunsite.ms.mff.cuni.cz ultra.linux.cz ftp.cz.kernel.org 202.158.214.12 
> speculum.rbc.ru 71.19.151.18 152.19.134.145 152.19.134.195 mirrors.mit.edu 
> solar-one.mit.edu 10.64.10.11 mirrors.xmission.com 182.255.111.7 
> 2001:388:1:4066:225:90ff:fec7:777e mirror.prgmr.com mirror01.prgmr.com 
> tiz-korg-mirror.kernel.org sfo-korg-mi
>  rror.kernel.org 129.7.128.189 129.7.128.190 129.101.198.59 frisal.switch.ch 
> 208.96.144.70 208.96.144.16
> +   hosts allow = jobbot1.ibiblio.org 200.17.202.1/28 zeus1.kernel.org 
> zeus2.kernel.org zeus3.kernel.org zeus4.kernel.org 149.20.20.132 
> 204.152.191.36 199.6.1.170 130.239.17.3 sinclair.wpi.edu 
> bonaparte.hrz.tu-chemnitz.de josephine.hrz.tu-chemnitz.de 
> mirror.speedpartner.de rsyncer.ftp.heanet.ie archive.linux.duke.edu 
> lists.us.dell.com auslistsprd01.us.dell.com auslistsdr01.us.dell.com 
> 198.129.224.34 mirror.hiwaay.net sagres.c3sl.ufpr.br mail.fedoraunity.org 
> scrye.com odysseus.fi.muni.cz odysseus.linux.cz rhlx01.hs-esslingen.de 
> ftp.nrc.ca zaphod.gtlib.gatech.edu 128.171.104.148 129.21.171.98 
> torrent01.fedoraproject.org torrent02.fedoraproject.org sunsite.mff.cuni.cz 
> sunsite.ms.mff.cuni.cz ultra.linux.cz ftp.cz.kernel.org 202.158.214.12 
> speculum.rbc.ru 71.19.151.18 152.19.134.145 152.19.134.195 mirrors.mit.edu 
> solar-one.mit.edu 10.64.10.11 mirrors.xmission.com 182.255.111.7 
> 2001:388:1:4066:225:90ff:fec7:777e mirror.prgmr.com 
> tiz-korg-mirror.kernel.org sfo-korg-mirror.kernel.org 129
>  .7.128.189 129.7.128.190 129.101.198.59 frisal.switch.ch 208.96.144.70 
> 208.96.144.16

These diffs of the rsync ACLs are not very useful. For me it is almost
impossible to see if and what has changed. It also seems we have to
maintain the ACL in 4 or 5 different files. Can the rsync ACL not be
handled in a more ansible way? I don't know much about ansible but
wouldn't it be possible to maintain the ACL one time with something like
this:

- name rsync acl
  template: some template with a loop statement over items
  with_items:
  - ip1
  - ip2
  - host1
  - host2

That would make the diffs readable any maybe we could maintain the rsync
ACL in only one place. Before trying to implement it I wanted to see if
there are some better ideas/ways to implement this 'correctly'.

Adrian
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

Re: Can not download repomd.xml for rawhide-source repo.

[Adrian Reber]

On Wed, Mar 30, 2016 at 02:52:02AM -0400, Jun Aruga wrote:
> > > https://mirrors.fedoraproject.org/metalink?repo=rawhide-source=x86_64,
> > > and downloaded "metalink" text file by my Firefox. Though several
> > > link sites were "not found 404" for the repomd.xml. I could find one
> > > available link site, that is the link of [3].
> > 
> > So, like I mentioned above... rawhide hasnt been composing until late
> > yesterday. So, possibly all mirrors got marked out of date... I see
> > some synced now. Is this working for you now?
> 
> Unfortunately, for this case [2], it is still not working now.
> 
> $ dnf repoquery --enablerepo=rawhide-source --arch src --whatrequires 
> 'rubygem(acts_as_list)'
> Fedora 23 - x86_64 - Updates 56 
> MB/s |  20 MB 00:00
> Error: Failed to synchronize cache for repo 'rawhide-source' from 
> 'https://mirrors.fedoraproject.org/metalink?repo=rawhide-source=x86_64': 
> Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors 
> were tried

Looking at the metalink it only lists a few mirrors which have an empty
source directory at the wrong location:

  development/rawhide/source/SRPMS/repodata

I am not 100% sure were data have been moved but I think it is now at:

  development/rawhide/Everything/source/tree/repodata

Somebody did some database changes for other parts of that move and
probably the rawhide-source repository still points at the old/wrong
location in MirrorManager.

Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

Re: a new MirrorManager-related tool for stripping old metadata and pushing critical updates in a defined time

[Adrian Reber]

On Mon, Feb 08, 2016 at 07:43:14AM -0500, Kamil Paral wrote:
> > On Fri, Jan 08, 2016 at 02:40:52PM -0500, Patrick Uiterwijk wrote:
> > > > 3. The tool will go through all metalinks for f23-updates (all primary
> > > > architectures), and drop each  section which has
> > > >  older than the provided date (let's say midnight UTC).
> > > 
> > > Please note that there are no metalink files: these files are generated on
> > > the fly from a cache
> > > by the mirrorlist servers.
> > > I have a patch for this that I'll submit upstream to add the feature
> > > itself, and will discuss
> > > with releng how they want to fire this off.
> > 
> > The new script is now available on mm-backend01: mm2_emergency-expire-repo
> > 
> > I think I have seen that Patrick also created a playbook to let the
> > script run in the correct environment and configuration.
> > 
> > We have successfully tested the script in the staging environment and
> > after it runs only the newest repomd.xml file is listed in the metalink.
> > 
> > As far as I understand MirrorManager this script only changes the number
> > of alternate repomd.xml files in the metalink. The number of mirrors
> > returned does not change. Depending on the last run of the master mirror
> > crawler (umdl), the state of the crawler checking the mirrors and the
> > mirrors which are running report_mirror this may lead to situation where
> > mirrors are offered to clients which might not yet have the newest
> > files.
> 
> In other words, some of those repos included in the metalink might not 
> correspond to the included repomd.xml hash, right?

Correct.

> Is that a problem? I believe DNF should just skip those repos and find the 
> first one which matches the provided metadata hash.

It shouldn't be a problem. It is a situation which can happen anytime.
Just wanted to mention it once more.

Adrian


pgpRyHfGz79k1.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

Re: a new MirrorManager-related tool for stripping old metadata and pushing critical updates in a defined time

[Adrian Reber]

On Fri, Jan 08, 2016 at 02:40:52PM -0500, Patrick Uiterwijk wrote:
> > 3. The tool will go through all metalinks for f23-updates (all primary
> > architectures), and drop each  section which has
> >  older than the provided date (let's say midnight UTC).
> 
> Please note that there are no metalink files: these files are generated on 
> the fly from a cache
> by the mirrorlist servers.
> I have a patch for this that I'll submit upstream to add the feature itself, 
> and will discuss
> with releng how they want to fire this off.

The new script is now available on mm-backend01: mm2_emergency-expire-repo

I think I have seen that Patrick also created a playbook to let the
script run in the correct environment and configuration.

We have successfully tested the script in the staging environment and
after it runs only the newest repomd.xml file is listed in the metalink.

As far as I understand MirrorManager this script only changes the number
of alternate repomd.xml files in the metalink. The number of mirrors
returned does not change. Depending on the last run of the master mirror
crawler (umdl), the state of the crawler checking the mirrors and the
mirrors which are running report_mirror this may lead to situation where
mirrors are offered to clients which might not yet have the newest
files.

Adrian


pgprqb0L5EUX9.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

Re: isn't mirrormanager smarter than this?

[Adrian Reber]

On Thu, Jan 28, 2016 at 08:41:19AM -0500, Matthew Miller wrote:
> I'm getting 404 errors from several places to which mirrormanager wants
> to send me to get
> https://download.fedoraproject.org/pub/alt/atomic/stable/Cloud-Images/x86_64/Images/Fedora-Cloud-Atomic-23-20160127.x86_64.qcow2
> (Specifically, mirrors.rit.edu and mirrors.kernel.org.) Presumbably,
> these just haven't synced yet. But doesn't mirrormanager get feedback
> about when mirrors are fresh?

I can try to give a few reasons (excuses) why you were redirected to a
wrong file.

* The file does not exist anymore on the master mirror. Not being 100%
  sure how the file names are constructed but it seems to include a date
  and the file on the master mirror (and a few other mirrors includes a
  '.2'). So it seems like the file was recreated. Using the new file
  name I am redirected to a mirror which has the file.

* /pub/alt (also known as the MM category 'Fedora Other') on the master
  mirror is only scanned once per day (12:15 UTC). As the scanning
  takes a very long time and as it creates a high I/O load the less
  important MM categories like 'Fedora Other' and 'Fedora Archive' are
  only scanned once a day.

* mirrors.rit.edu and mirrors.kernel.org are both running report_mirror
  to update the status of their mirror after the sync. We trust mirrors
  running report_mirror. We only get the information that they claim
  the transmitted directories are up to date. The actual content is not
  checked when using report_mirror. In the current case both mirrors
  have been crawled by since the last report_mirror, but at the time of
  the crawl the file 'Fedora-Cloud-Atomic-23-20160127.2.x86_64.qcow2'
  (with the '.2') was not yet part of the database as it has a time of
  'Jan 27 17:29' which is after the previous master mirror scan. By now
  the new file should be part of the database and soon the crawler
  should detect it also on the mirrors.


So there are many reasons why it did fail but the main reason probably
is, that due to large amount of files it is difficult to have real-time
view of what is on the mirrors and the master mirror. Finding the right
balance between massive amounts of I/O on the master and mirrors and
freshness of the MM database is the hard part.

Adrian


pgp4WY_UV3kPa.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

host_category_url_private

[Adrian Reber]

This is mainly a question for Matt.

Currently it seems there is a problem with newly added private mirrors
that they never seem to appear in the mirrorlist. I am tracking this for
the last few weeks, but I have not found the cause for this yet. As this
problem does not yet seem very widely spread I did not spent too much
time on it. Recently dl.phx2.fedoraproject.org has been added as a
private mirror and it does not appear in the mirrorlist.

I have now seen that a few URLs have the private flag set in
host_category_url and I am wondering what it exactly means. I see that
these URLs are not added to the mirrorlist servers as opposed to mirror
sites/hosts which have the private flag set. I also do not see in MM2
where this flag can actually be changed and that's why I am wondering if
we somehow set this flag by accident in certain situations and that
maybe it had some different meaning in MM1.

That's why I am hoping that maybe Matt remembers the idea behind this
flag which enables marking URls as private. In MM2 it seems to be
unused, besides that once it is enabled it cannot be disabled and
URLs using it are therefore for ever blocked from the mirrorlist server.

Adrian
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

Re: host_category_url_private

[Adrian Reber]

On Mon, Dec 14, 2015 at 01:48:33PM +0100, Adrian Reber wrote:
> This is mainly a question for Matt.
> 
> Currently it seems there is a problem with newly added private mirrors
> that they never seem to appear in the mirrorlist. I am tracking this for
> the last few weeks, but I have not found the cause for this yet. As this
> problem does not yet seem very widely spread I did not spent too much
> time on it. Recently dl.phx2.fedoraproject.org has been added as a
> private mirror and it does not appear in the mirrorlist.
> 
> I have now seen that a few URLs have the private flag set in
> host_category_url and I am wondering what it exactly means. I see that
> these URLs are not added to the mirrorlist servers as opposed to mirror
> sites/hosts which have the private flag set. I also do not see in MM2
> where this flag can actually be changed and that's why I am wondering if
> we somehow set this flag by accident in certain situations and that
> maybe it had some different meaning in MM1.
> 
> That's why I am hoping that maybe Matt remembers the idea behind this
> flag which enables marking URls as private. In MM2 it seems to be
> unused, besides that once it is enabled it cannot be disabled and
> URLs using it are therefore for ever blocked from the mirrorlist server.

Looking at the source code of MirrorManager (1) made it clearer (I
think). According to the descriptions on the MirrorManager1 pages those
URLs are not used for public or private mirrors but are used as URLs
between mirrors for better/easier syncing. It seems the description did
not survive the port to MM2 so that people started selecting private at
the URL level as it was not clear what it means.

Currently it can only be removed by either deleting the URLs completely
or by doing the change directly in the database. I modified the
dl.phx2.fedoraproject.org URLs to see if this actually helps.

Adrian
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org

Re: Freeze Break Request: Increase number of children for mirrorlist-server

[Adrian Reber]

On Thu, Oct 15, 2015 at 09:50:03AM -0600, Kevin Fenzi wrote:
> +1 here, but could we also increase the processes on the wsgi to 60 at
> the same time?

Like this:

diff --git a/inventory/group_vars/mirrorlist2 b/inventory/group_vars/mirrorlist2
index 42138a3..f0d5655 100644
--- a/inventory/group_vars/mirrorlist2
+++ b/inventory/group_vars/mirrorlist2
@@ -23,7 +23,7 @@ fas_client_groups: sysadmin-noc,fi-apprentice
 nrpe_procs_warn: 500
 nrpe_procs_crit: 600
 # By default run 45 wsgi procs
-mirrorlist_procs: 45
+mirrorlist_procs: 60
 
 # Set this to get the vpn postfix setup
 postfix_group: vpn


Adrian


pgptLNhpDvrPX.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/infrastructure@lists.fedoraproject.org

Freeze Break Request: Increase number of children for mirrorlist-server

[Adrian Reber]

As there are more reports of timeouts connected to the mirrorlist
servers I had a closer look at the code. It seems we are hitting the
default value of max_children = 40. This freeze break request tries to
increase this to 80 via a hotfix. See commit message below for further
details.

Adrian


commit 2b8e284e687d097c2cad6cd24dc3db3aa23f837d
Author: Adrian Reber <adr...@lisas.de>
Date:   Thu Oct 15 14:03:02 2015 +

Increase the number of possible child processes

The mirrorlist-server is the process which has the mirrorlist data
loaded and which is accessed by the public facing
mirrorlist_client.wsgi. The mirrorlist-server uses the
ForkingUnixStreamServer which has a default of max_children = 40.
(https://hg.python.org/cpython/file/2.7/Lib/SocketServer.py#l516)

Looking at the code of ForkingUnixStreamServer it says at

https://hg.python.org/cpython/file/2.7/Lib/SocketServer.py#l523

  # If we're above the max number of children, wait and reap them until
  # we go back below threshold. Note that we use waitpid(-1) below to be
  # able to collect children in size() syscalls instead
  # of size(): the downside is that this might reap children
  # which we didn't spawn, which is why we only resort to this when we're
  # above max_children.

As we are running the wsgi with processes=45 this sounds like it can
lead to situation where it might just hang.

This increases max_children to 80 and maybe processes could be upped to
60 or 70.

Signed-off-by: Adrian Reber <adr...@lisas.de>

diff --git a/files/hotfix/mirrorlist/mirrorlist_server.py 
b/files/hotfix/mirrorlist/mirrorlist_server.py
index 2d98fa0..2b81912 100644
--- a/files/hotfix/mirrorlist/mirrorlist_server.py
+++ b/files/hotfix/mirrorlist/mirrorlist_server.py
@@ -938,6 +938,7 @@ def sigterm_handler(signum, frame):
 
 class ForkingUnixStreamServer(ForkingMixIn, UnixStreamServer):
 request_queue_size = 300
+max_children = 80
 def finish_request(self, request, client_address):
 signal.signal(signal.SIGHUP, signal.SIG_IGN)
 BaseServer.finish_request(self, request, client_address)
diff --git a/roles/mirrormanager/mirrorlist2/tasks/main.yml 
b/roles/mirrormanager/mirrorlist2/tasks/main.yml
index 786432f..2e5f5ee 100644
--- a/roles/mirrormanager/mirrorlist2/tasks/main.yml
+++ b/roles/mirrormanager/mirrorlist2/tasks/main.yml
@@ -73,3 +73,12 @@
 #  tags:
 #  - mirrorlist2
 #  - selinux
+
+
+- name: HOTFIX - increase the number of possible child processes
+  copy: src="{{ files }}/hotfix/mirrorlist/mirrorlist_server.py" 
dest=/usr/share/mirrormanager2/mirrorlist_server.py
+owner=root group=root mode=0755
+  tags:
+  - hotfix
+  notify:
+  - restart mirrorlist-server

commit d04588274a6e161a36d8dbd9c91ea0fe78de017e
Author: Adrian Reber <adr...@lisas.de>
Date:   Thu Oct 15 13:54:53 2015 +

Original mirrorlist_server.py which needs to be hotfixed
    
    Signed-off-by: Adrian Reber <adr...@lisas.de>

diff --git a/files/hotfix/mirrorlist/mirrorlist_server.py 
b/files/hotfix/mirrorlist/mirrorlist_server.py
new file mode 100644
index 000..2d98fa0
--- /dev/null
+++ b/files/hotfix/mirrorlist/mirrorlist_server.py
@@ -0,0 +1,1136 @@
+#!/usr/bin/python
+#
+# Copyright (c) 2007-2013 Dell, Inc.
+#  by Matt Domsch <matt_dom...@dell.com>
+# Licensed under the MIT/X11 license
+
+# standard library modules in alphabetical order
+from collections import defaultdict
+import datetime
+import getopt
+import logging
+import logging.handlers
+import os
+import random
+import cPickle as pickle
+import select
+import signal
+import socket
+from SocketServer import (StreamRequestHandler, ForkingMixIn,
+  UnixStreamServer, BaseServer)
+import sys
+from string import zfill, atoi
+import time
+import traceback
+
+try:
+import threading
+except ImportError:
+import dummy_threading as threading
+
+# not-so-standard library modules that this program needs
+from IPy import IP
+import GeoIP
+import radix
+from weighted_shuffle import weighted_shuffle
+
+# can be overridden on the command line
+pidfile = '/var/run/mirrormanager/mirrorlist_server.pid'
+socketfile = '/var/run/mirrormanager/mirrorlist_server.sock'
+cachefile = '/var/lib/mirrormanager/mirrorlist_cache.pkl'
+internet2_netblocks_file = '/var/lib/mirrormanager/i2_netblocks.txt'
+global_netblocks_file = '/var/lib/mirrormanager/global_netblocks.txt'
+logfile = None
+debug = False
+must_die = False
+# at a point in time when we're no longer serving content for versions
+# that don't use yum prioritymethod=fallback
+# (e.g. after Fedora 7 is past end-of-life)
+# then we can set this value to True
+# this only affects results requested using path=...
+# for dirs which aren't repositories (such as iso/)
+# because we don't know the Version associated with that dir here.
+default_ordered_mirrorlist = Fals

Re: Freeze Break Request: Increase number of children for mirrorlist-server

[Adrian Reber]

On Thu, Oct 15, 2015 at 06:33:22PM +0200, Pierre-Yves Chibon wrote:
> On Thu, Oct 15, 2015 at 06:08:15PM +0200, Adrian Reber wrote:
> > On Thu, Oct 15, 2015 at 09:50:03AM -0600, Kevin Fenzi wrote:
> > > +1 here, but could we also increase the processes on the wsgi to 60 at
> > > the same time?
> > 
> > Like this:
> > 
> > diff --git a/inventory/group_vars/mirrorlist2 
> > b/inventory/group_vars/mirrorlist2
> > index 42138a3..f0d5655 100644
> > --- a/inventory/group_vars/mirrorlist2
> > +++ b/inventory/group_vars/mirrorlist2
> > @@ -23,7 +23,7 @@ fas_client_groups: sysadmin-noc,fi-apprentice
> >  nrpe_procs_warn: 500
> >  nrpe_procs_crit: 600
> >  # By default run 45 wsgi procs
> > -mirrorlist_procs: 45
> > +mirrorlist_procs: 60
> >  
> >  # Set this to get the vpn postfix setup
> >  postfix_group: vpn
> > 
> 
> +1 for me on both, but let's just make sure we have folks around when we push 
> it
> in case of a fire-burst :)

Thanks. Pushed. Somebody needs to run the playbook for the mirrorlist
servers. Maybe first in staging to detect early if it will make the
mirrorlist servers burn.

Adrian


pgpB8_wlWbPRw.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/infrastructure@lists.fedoraproject.org

Re: Master and mirror crawling

[Adrian Reber]

On Sat, Sep 12, 2015 at 12:52:55PM -0600, Kevin Fenzi wrote:
> > On Fri, Sep 11, 2015 at 04:56:41PM +0200, Adrian Reber wrote:
> > [...]
> > > So my main question is if we should insert a delay between umdl and
> > > the crawl of the mirrors? This would require a fedmsg emitted at
> > > the end of an umdl run and something on the crawler which waits
> > > some time before starting the crawls.
> > 
> > Thinking more about it, it actually does not make much sense to base
> > the mirror crawls on fedmsg. The mirrors are updated at (from our
> > point of view) random times. So with category based crawling we have
> > the possibility to increase the crawl frequency for Fedora Linux and
> > Fedora EPEL and decrease it for Fedora Archive. Which should
> > hopefully give MirrorManager a better view of the status of the
> > mirrors.
> 
> Well, mirrors that are using your script to trigger syncs after a
> fedmsg would be syncing right after that as well, but might depend on
> how long it takes them to sync. 

Yes, my mirror syncs from ::fedora-buffet0/ and that takes a few hours.

Adrian


pgpfoxWIvJ7AD.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/postorius/infrastructure@lists.fedoraproject.org

Freeze break request: s3sync

[Adrian Reber]

This is a try to fix the s3sync on mm-backend01. I have tested the
script and it successfully syncs the content to S3. I have, however, not
tested if the ansible rules work. Please have a look at following patch
and if the ansible rules are correct.

In addition to this patch it is necessary to install the new
mirrormanager2 release (0.4.2) in prod. It has now changes except an
additional subpackage mirrormanager2-client which provides report_mirror
which is run after the sync to S3.


Date: Thu, 30 Jul 2015 13:47:48 +
Subject: [PATCH] Fix s3sync and start it via cron.

This changes the s3sync setup at multiple places:

 * The s3sync script was using wrong paths which are now corrected
 * After the sync report_mirror was started which is now installed
 * The necessary report_mirror files are installed
 * The s3sync script is now running from cron instead as a service
 * Before touching any files fedmsg is queried to see if new files
   are available for syncing
 * logrotate has been adapted to the new setup
---
 roles/s3-mirror/files/s3-mirror.init |   77 
 roles/s3-mirror/files/s3-mirror.logrotate|3 -
 roles/s3-mirror/files/s3sync |  100 +-
 roles/s3-mirror/tasks/main.yml   |   19 -
 roles/s3-mirror/templates/report_mirror.conf |   60 +++
 5 files changed, 126 insertions(+), 133 deletions(-)
 delete mode 100644 roles/s3-mirror/files/s3-mirror.init
 create mode 100644 roles/s3-mirror/templates/report_mirror.conf

diff --git a/roles/s3-mirror/files/s3-mirror.init 
b/roles/s3-mirror/files/s3-mirror.init
deleted file mode 100644
index 59dbb39..000
--- a/roles/s3-mirror/files/s3-mirror.init
+++ /dev/null
@@ -1,77 +0,0 @@
-#!/bin/sh
-#
-# s3-mirror - sync content to S3
-#
-# chkconfig:   - 99 99
-# description: sync content to S3
-
-# http://fedoraproject.org/wiki/FCNewInit/Initscripts
-### BEGIN INIT INFO
-# Provides: 
-# Required-Start: $network $named $remote_fs
-# Required-Stop: 
-# Should-Start: 
-# Should-Stop: 
-# Default-Start: 
-# Default-Stop: 
-# Short-Description: 
-# Description: 
-### END INIT INFO
-
-# Source function library.
-. /etc/rc.d/init.d/functions
-
-exec=/usr/local/bin/s3sync
-prog=${exec##*/}
-
-lockfile=/var/lock/subsys/$prog
-pidfile=/var/run/s3-mirror/pid
-
-start() {
-echo -n $Starting $prog: 
-mkdir -p /var/run/s3-mirror
-chown -R s3-mirror:s3-mirror /var/run/s3-mirror
-daemon --user s3-mirror --pidfile $pidfile $exec 
-retval=$?
-echo
-[ $retval -eq 0 ]  touch $lockfile
-return $retval
-}
-
-stop() {
-echo -n $Stopping $prog: 
-killproc -p $pidfile
-retval=$?
-echo
-[ $retval -eq 0 ]  rm -f $lockfile
-return $retval
-}
-
-restart() {
-stop
-start
-}
-
-case $1 in
-start|stop|restart)
-$1
-;;
-force-reload)
-restart
-;;
-status)
-status $prog
-;;
-try-restart|condrestart)
-if status $prog /dev/null ; then
-restart
-fi
-   ;;
-reload)
-action $Service ${0##*/} does not support the reload action:  
/bin/false
-exit 3
-;;
-*)
-echo $Usage: $0 {start|stop|status|restart|try-restart|force-reload}
-exit 2
-esac
diff --git a/roles/s3-mirror/files/s3-mirror.logrotate 
b/roles/s3-mirror/files/s3-mirror.logrotate
index 12e66aa..7549028 100644
--- a/roles/s3-mirror/files/s3-mirror.logrotate
+++ b/roles/s3-mirror/files/s3-mirror.logrotate
@@ -9,7 +9,4 @@
 compressext .bz2
 dateext
 copytruncate
-postrotate
-/bin/kill -HUP `cat /var/run/s3-mirror/pid 2/dev/null` 2/dev/null || 
true
-endscript
 }
diff --git a/roles/s3-mirror/files/s3sync b/roles/s3-mirror/files/s3sync
index 6d0c552..f8d0413 100755
--- a/roles/s3-mirror/files/s3sync
+++ b/roles/s3-mirror/files/s3sync
@@ -1,16 +1,16 @@
 #!/bin/bash
 
-pidfile=/var/run/s3-mirror/pid
 logfile=/var/log/s3-mirror/s3sync.log
+tree=/srv/pub/
+s3cmd=/usr/bin/s3cmd
+curdate=`date +%s`
 
-trap sighup_handler HUP
-trap rm -f ${pidfile} QUIT EXIT INT TERM
-
-function sighup_handler()
-{
-exec 1${logfile} 21
-}
+if [ ! -d ${tree} ]; then
+echo Specified root of the directory tree (${tree}) does not exist. 
Exiting. | tee ${logfile}
+exit 1
+fi
 
+exec 1${logfile} 21
 
 function newer()
 {
@@ -20,16 +20,6 @@ function newer()
 return 1
 }
 
-function repeat()
-{
-while :; do
-   $1
-/bin/sleep $((${RANDOM} % 300))
-done
-}
-
-s3cmd=/usr/bin/s3cmd
-
 S3CMD_ARGS=sync \
   --verbose \
   --preserve \
@@ -44,41 +34,51 @@ S3CMD_ARGS=sync \
 
 content=epel
 targets=s3-mirror-us-east-1 s3-mirror-us-west-1 s3-mirror-us-west-2 
s3-mirror-eu-west-1 s3-mirror-ap-northeast-1
+report=0
 
-function parallel_sync_full()
-{
-report=0
-for c in ${content}; do
-   if $(newer /pub/${c}/fullfilelist /var/lib/s3-mirror/${c}-fullfilelist) 
; then
-   echo 

Re: Freeze break request: decrease number of crawlers to 29

[Adrian Reber]

On Wed, Jul 29, 2015 at 01:48:57PM -0600, Kevin Fenzi wrote:
 On Wed, 29 Jul 2015 21:24:02 +0200
 Adrian Reber adr...@lisas.de wrote:
 
  Can I get two +1's for:
 
 +1. 

Thanks Kevin and Pierre. Pushed.

 Also, if we want, I could also give them another 32GB memory... 

Either that or we can setup a crawler in Europe to crawl European
mirrors from there. The crawler has the necessary functionality and we
could try continent based crawls. I was aiming for after alpha freeze.

Adrian


pgp4XbSgiZtsw.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

MM related changes

[Adrian Reber]

I have a few MM related ansible changes which I would like to get
reviewed before committing them.

 * The crawler logs are now on mm-crawler01 - relative simple change

 * Increase cache time for mirrorlist - This tries to increase the cache
   time from 2 minutes to 6 hours for the mirrorlist. I have tested it
   on a local varnish installation and it seems to work.

 * Use the new api interface for Nagios checks - This changes two nagios
   checks to use the api interface which performs a much simpler
   database query.

Adrian

commit 477f2ac02e30be805eacffda47c7c926f25bc892
Author: Adrian Reber adr...@lisas.de
Date:   Fri Jun 12 08:20:24 2015 +

The crawler logs are now on mm-crawler01

diff --git 
a/roles/httpd/reverseproxy/templates/reversepassproxy.mirrormanager.conf 
b/roles/httpd/reverseproxy/templates/reversepassproxy.mirrormanager.conf
index 44987a1..8ea3700 100644
--- a/roles/httpd/reverseproxy/templates/reversepassproxy.mirrormanager.conf
+++ b/roles/httpd/reverseproxy/templates/reversepassproxy.mirrormanager.conf
@@ -4,8 +4,8 @@ SetEnv force-proxy-request-1.0 1
 SetEnv proxy-nokeepalive 1
 /Location
 
-ProxyPass {{localpath}}/crawler http://bapp02/mirrormanager/crawler
-ProxyPassReverse {{localpath}}/crawler http://bapp02/mirrormanager/crawler
+ProxyPass {{localpath}}/crawler http://mm-crawler01/mirrormanager/crawler
+ProxyPassReverse {{localpath}}/crawler 
http://mm-crawler01/mirrormanager/crawler
 
 ProxyPass {{localpath}} {{proxyurl}}{{remotepath}}
 ProxyPassReverse {{localpath}} {{proxyurl}}{{remotepath}}

commit 8e09f5a37620c52265aa83305c99173f9e982d76
Author: Adrian Reber adr...@lisas.de
Date:   Fri Jun 12 08:13:17 2015 +

Increase cache time for mirrorlist

The mirrorlist which can be viewed in the browser used to be generated
once or twice per day with MM1. This is listed is now cached in varnish
but only for two minutes. This patch increases the cache time to 6 hours.

diff --git a/roles/varnish/files/proxy.vcl b/roles/varnish/files/proxy.vcl
index 0fd5aa6..6d2f340 100644
--- a/roles/varnish/files/proxy.vcl
+++ b/roles/varnish/files/proxy.vcl
@@ -303,5 +303,6 @@ sub vcl_recv {
 sub vcl_backend_response {
 if (bereq.url ~ ^/mirrormanager/mirrors) {
 unset beresp.http.set-cookie;
+set beresp.ttl = 6h;
 }
 }

commit 498f16861fbde5190b6d4fbab731185fb01c7662
Author: Adrian Reber adr...@lisas.de
Date:   Fri Jun 12 07:28:38 2015 +

MM: Use the new api interface for Nagios checks

The newly introduced api interface provides a way to query the MM
frontend for availability which only performs a minimal database query.
Using this interface should reduce the load on the frontend and the
database.

diff --git a/roles/nagios_server/files/nagios-external/services/websites.cfg 
b/roles/nagios_server/files/nagios-external/services/websites.cfg
index 55ad34a..0787e53 100644
--- a/roles/nagios_server/files/nagios-external/services/websites.cfg
+++ b/roles/nagios_server/files/nagios-external/services/websites.cfg
@@ -145,7 +145,7 @@ define service {
 define service {
   host_name 209.132.181.16-phx2, 85.236.55.6-internetx, 
proxy03.fedoraproject.org, 152.19.134.142-ibiblio, proxy06.fedoraproject.org, 
213.175.193.206-bodhost, 67.203.2.67-coloamerica, 66.135.62.187-serverbeach
   service_description   mirrors.fedoraproject.org - publiclist
-  check_command 
check_website_ssl!admin.fedoraproject.org!/mirrormanager/!Fedora Public Active 
Mirrors
+  check_command 
check_website_ssl!admin.fedoraproject.org!/mirrormanager/api/mirroradmins/?name=dl.fedoraproject.org!admins
   use   websitetemplate
 }
 
diff --git a/roles/nagios_server/files/nagios/services/websites.cfg 
b/roles/nagios_server/files/nagios/services/websites.cfg
index 1beda33..101c115 100644
--- a/roles/nagios_server/files/nagios/services/websites.cfg
+++ b/roles/nagios_server/files/nagios/services/websites.cfg
@@ -46,7 +46,7 @@ define service {
 define service {
   host_name mm-frontend01,mm-frontend01.stg
   service_description   mm-publiclist-internal
-  check_command check_website!localhost!/mirrormanager/
+  check_command 
check_website!localhost!/mirrormanager/api/mirroradmins/?name=dl.fedoraproject.org
   use   internalwebsitetemplate



pgpf0bdN3HKTt.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: MM related changes

[Adrian Reber]

On Fri, Jun 12, 2015 at 11:13:28AM +0200, Pierre-Yves Chibon wrote:
   ProxyPass {{localpath}} {{proxyurl}}{{remotepath}}
   ProxyPassReverse {{localpath}} {{proxyurl}}{{remotepath}}
  
  commit 8e09f5a37620c52265aa83305c99173f9e982d76
  Author: Adrian Reber adr...@lisas.de
  Date:   Fri Jun 12 08:13:17 2015 +
  
  Increase cache time for mirrorlist
  
  The mirrorlist which can be viewed in the browser used to be generated
  once or twice per day with MM1. This is listed is now cached in varnish
  but only for two minutes. This patch increases the cache time to 6 
  hours.
  
  diff --git a/roles/varnish/files/proxy.vcl b/roles/varnish/files/proxy.vcl
  index 0fd5aa6..6d2f340 100644
  --- a/roles/varnish/files/proxy.vcl
  +++ b/roles/varnish/files/proxy.vcl
  @@ -303,5 +303,6 @@ sub vcl_recv {
   sub vcl_backend_response {
   if (bereq.url ~ ^/mirrormanager/mirrors) {
   unset beresp.http.set-cookie;
  +set beresp.ttl = 6h;
   }
   }
 
 
 Seems ok, I was kind of expecting that there would already be a value here,
 but I do not know enoug of varnish to make a full +1, so I'll +0.5 there :)

Looking at the documentation it says, that the default is 120 seconds or
whatever the backend specifies via Expires/Cache-Control. At least
that's what I understood.

Adrian


pgpM2jb7Bo7aD.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: MM related changes

[Adrian Reber]

On Fri, Jun 12, 2015 at 09:47:34AM -0600, Kevin Fenzi wrote:
 These changes look ok to me. ;) 

Thanks, for the reviews. I will commit them and somebody needs to run
the corresponding playbooks.

 Related however, what is our plan for crawlers currently?
 We are only going to use mm-crawler01? 

No, both crawlers are used. There is script in the crawler command-line
which retrieves the crawler with the highest ID and divides it by 2, so that
every crawler gets half of the mirrors to crawl.

/usr/bin/mm2_crawler --timeout-minutes 180 --threads 35 
`/usr/local/bin/run_crawler.sh 2`

 Should we nuke 02?

No, as we are using it.

 Should we give 01 more memory?

No, we need to crawl better and not throw more resources at it.

 I still have seen them hit swap alerts in nagios, which we should avoid
 if at all possible. 

Okay, currently we have 35 parallel crawlers running. We can decrease it to
32. The problem is that it is hard to predict which mirrors are crawled
at the same time and thus it is hard to predict the required memory.

The current setup, however, is not optimal. We have two crawlers with
64GB of memory in total which is only used for maybe 10 hours per day
and the rest of the day the memory is completely idling and wasted.

So we could crawl all mirrors on one crawler. First the first half and
then the second half. The reason this is not yet implemented is that we
do not have a way to gracefully shutdown the crawler if it takes to
long. Sometimes the crawler is hanging on a single mirror for hours and
it is not clear why. Even with all the timeout checks all over the place
it just hangs.

I am currently working on some code to gracefully shutdown the crawler
if it takes too long. With this and with canary and repodata mode we can
use only one crawler and make sure its resources are in use most of the
time. If this feature to gracefully shutdown the crawler is finished I
would like to join one of the next infrastructure meetings to discuss in
a larger group what the best combination of full, repodata and canary
crawls would be.

Adrian


pgpACovTArTFs.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Re: [release] MirrorManager2: 0.2.0

[Adrian Reber]

On Fri, Jun 05, 2015 at 05:09:06PM +0200, Pierre-Yves Chibon wrote:
 Good morning everyone,
 
 Here is another release of the day, MirrorManager2: 0.2.0
 
 And here is the changelog:
 * Fri Jun 05 2015 Pierre-Yves Chibon pin...@pingoured.fr - 0.2.0-1
 - Update to 0.2.0
 - Include the background header file in MM2 itself (Adrian Reber)
 - Support always update hosts which are unreachable in the crawler (Adrian
   Reber)
 - Adjust the spec file to the systemd packaging guidelines for Fedora
 - Multiple improvements to the crawler, including a start of a canary mode
   (Adrian Reber)
 - Offer possibility to sort by product, bringing back MM1 behavior (Adrian
   Reber)
 - Couple of UI fixes about who is allowed to access what
 - Fix peer ASNs (in the same spirit, who can access)
 - Create noauthed master for mirror publiclist so that it can be cached in
   memcachd (Patrick Uiterwijk)

I think this should read varnish and not memcachd. I think it still uses
the 'default' cache age of 120 seconds. The publiclist used to be
created once or twice a day so that we could easily keep the cache much
longer than the current 120 seconds. I think 12 hours would be no
problem, as this list is 'only' used as an overview and independent of
the results of the actual yum/dny mirrorlist/metalink.

 - Fix the report_mirror to correctly catch the xmlrpclib.ProtocolError
 - Add a new utility script to upgrade repo from -alpha or -beta to release
 - Adjust the logrotate configuration to fix the permission denied error
 - Create 2 API endpoints, one for zodbot's .mirroradmin and one for nagios
 
 Note: this is currently only in *staging*, we did not feel like pushing this 
 to
 prod on a Friday afternoon :)
 
 We'll likely push on Monday though, assuming nothing burns down over the
 week-end.

I tested the new canary and repodata crawl mode and I was able to crawl
all mirrors in a much short time:

$ time mm2_crawler --repodata -c ~/mirrormanager2.cfg --threads 75 --timeout 30
real36m38.639s
user27m20.689s
sys 3m13.829s

$ time mm2_crawler --canary -c ~/mirrormanager2.cfg --threads 75 --timeout 30
real6m33.624s
user0m16.061s
sys 0m2.279s

Canary mode only checks if at least one base URL of all categories on
the mirror works. If no URL works the whole mirror is marked as not
being update. This also means that the mirror will not be re-enabled
until a full crawl or successful report_mirror run. This behaviour that
canary mode disables the mirror until a full crawl or report_mirror run
could be changed but would require database changes and therefore we
decided to support this form of canary mode as a first step.

Repodata mode crawls all repodata directories and only marks those
directories as being up to date or not. A mirror will not be completely
disabled in repodata mode in contrast to canary mode.

As canary mode seems to finish in under 10 minutes we can now think
about using full crawls and canary or repodata mode in combination to
get better overview about the availability and the status of the
mirrors.

Adrian


pgpbVkitTwRWR.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

[adr...@fedoraproject.org: [ansible] Re-enable /mirrormanager without slash at the end]

[Adrian Reber]

This change still needs a playbook to be run. If this change is correct
it would be great if someone could run the playbook. Thanks.

Adrian

- Forwarded message from Adrian Reber adr...@fedoraproject.org -

Date: Sun, 31 May 2015 09:30:09 + (UTC)
From: Adrian Reber adr...@fedoraproject.org
To: sysadmin-memb...@fedoraproject.org
Subject: [ansible] Re-enable /mirrormanager without slash at the end

This is the public repository, do not commit sensitive
or confidential information here.
X-Git-Refname: refs/heads/master
X-Git-Oldrev: af9462caaff1cb4736d6ffaec93338c2f7b8
X-Git-Newrev: 9a90bb869ff7982dc6c05c804d83f7444249fc52

commit 9a90bb869ff7982dc6c05c804d83f7444249fc52
Author: Adrian Reber adr...@lisas.de
Date:   Sun May 31 09:27:54 2015 +

Re-enable /mirrormanager without slash at the end

The mirrormanager application and the publiclist re-write used to work
without a slash at the end. Re-enable /mirrormanager without a slash at
the end of the URL.

 roles/varnish/files/proxy.vcl |2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
---
diff --git a/roles/varnish/files/proxy.vcl b/roles/varnish/files/proxy.vcl
index ed8333b..788b717 100644
--- a/roles/varnish/files/proxy.vcl
+++ b/roles/varnish/files/proxy.vcl
@@ -181,7 +181,7 @@ sub vcl_recv {
 set req.url = regsub(req.url, \?.*, );
 }
 }
-if (req.url ~ ^/mirrormanager/) {
+if (req.url ~ ^/mirrormanager) {
 set req.backend_hint = mirrormanager;
 if (req.url ~ ^/mirrormanager/static/) {
 unset req.http.cookie;

- End forwarded message -
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

[piotr.mal...@grupaonet.pl: New public mirror Fedora PL.]

[Adrian Reber]

There is a fix missing, I think for varnish, to make the mirrormanager
flask application work without a '/' at the end. Would be good if this
could be added. I changed the MirrorManager instructions in the wiki to
point to the correct URL to avoid troubles like this.

Adrian

- Forwarded message from Maluty Piotr piotr.mal...@grupaonet.pl -

Date: Mon, 25 May 2015 18:08:38 +0200
From: Maluty Piotr piotr.mal...@grupaonet.pl
To: 'mirror-ad...@fedoraproject.org' mirror-ad...@fedoraproject.org
Subject: New public mirror Fedora PL.

Hi,

I run mirror Fedora in Poland.
HTTP:  http://mirror.onet.pl/pub/mirrors/fedora/linux/
FTP: ftp://mirror.onet.pl/pub/mirrors/fedora/linux/
RSYNC:  rsync://mirror.onet.pl/pub/mirrors/fedora/
Sync schedule: 4 times/day
Bandwidth: 10Gb/s
Location: Poland
Sponsor: Grupa Onet.pl S.A.
Sponsor URL: http://www.onet.pl/
IP : 213.180.139.200, 2a02:c10:2170:4::bebe
Email contact: piotr.mal...@grupaonet.plmailto:piotr.mal...@grupaonet.pl.

follow instructions and get an error.

Not Found
The requested URL /mirrormanager was not found on this server.
Apache/2.2.15 (Red Hat) Server at localhost Port 6081


Pozdrawiam
Piotr Maluty
Administrator Systemow
Grupa Onet.pl SA
ul. Bobrzynskiego 12 E
30-348 Krakow
tel.: +48 600 206 065


___
Mirror-admin mailing list
mirror-ad...@fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/mirror-admin


- End forwarded message -


pgpnY0brOh_7G.pgp
Description: PGP signature
___
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure