Thanks a lot. I will change the container to be based on this instead of
the binary I have built locally. Perfect!
Adrian
On Tue, Nov 19, 2019 at 07:54:34AM +0100, Igor Gnatenko wrote:
> And I finally got this done. The update for F31 is here:
> https://bodhi.fedoraproject.org/updates/FEDORA-2019-48a3fc7eb5
>
> On Tue, Oct 8, 2019 at 8:53 AM Igor Gnatenko
> <ignatenkobr...@fedoraproject.org> wrote:
> >
> > I'll try to package new Rust implementation in Fedora :)
> >
> > And thanks for working on this!
> >
> > On Tue, Oct 8, 2019 at 8:52 AM Adrian Reber <adr...@lisas.de> wrote:
> > >
> > >
> > > Fedora's complete MirrorManager setup is still running on Python2. The
> > > code has been ported to Python3 probably over two years ago but we have
> > > not switched yet. One of the reasons is that the backend is running on
> > > RHEL7 which means we are not in a hurry to deploy the Python3 version.
> > >
> > > The mirrorlist server which is answering the actual dnf/yum queries for
> > > a mirrorlist/metalink is, however, running in a Fedora 29 container.
> > > This container also still uses Python2 and it actually cannot use the
> > > Python3 version.
> > >
> > > One of MirrorManager's design points is that the mirrorlist servers,
> > > which are answering around 27 000 000 requests per day, are not directly
> > > accessing the database. The backend creates a snapshot of the relevant
> > > data (113MB) and the mirrorlist servers are using this snapshot to
> > > answer client requests.
> > >
> > > This data exchange is based on Python's pickle format and that does not
> > > seem to work with Python3 if it is generated using Python2.
> > >
> > > Having used protobuf before, I added code to also export the data for the
> > > mirrorlist servers based on protobuf.
> > >
> > > The good news with protobuf is, that the resulting file is only 66MB
> > > instead of 113MB. The bad news is, that loading it from Python requires
> > > 3.5 times the amount of memory during runtime (3.5GB instead of 1GB).
> > >
> > > In addition to the data exchange problems between backend and
> > > mirrorlist servers the architecture of the mirrorlist server does not
> > > really make sense today. 12 years ago it made a lot of sense as it could
> > > be easily integrated into httpd and it could be easily reloaded without
> > > stopping the service. Today the mirrorlist server and httpd is all part
> > > of a container which is then behind haproxy. So there is a lot of
> > > infrastructure in the container which is not really useful.
> > >
> > > To get rid of the pickle format and to have a simpler architecture I
> > > reimplemented the mirrorlist-server in Rust. This was brought up some
> > > time ago on a ticket and with the protobuf problems I was seeing in
> > > Python it made sense to try it out.
> > >
> > > My code currently can be found at
> > > https://github.com/adrianreber/mirrorlist-server
> > > and so far the results from the new mirrorlist server are the same as
> > > from the Python based mirrorlist server.
> > >
> > > It requires less than 700MB instead of the 1GB in Python with production
> > > based data and seems really fast.
> > >
> > > I have set up a test instance with the mirror data from Sunday at:
> > >
> > > https://lisas.de/metalink?repo=updates-testing-f31&arch=x86_64
> > > https://lisas.de/mirrorlist?repo=updates-testing-f31&arch=x86_64
> > >
> > > The instance is based on the container I pushed to quay.io:
> > >
> > > $ podman run quay.io/adrianreber/mirrorlist-server:latest -h
> > >
> > > With this change the mirrorlist server would also finally switch to
> > > geoip2. The currently running mirrorlist server still uses the legacy
> > > geoip database.
> > >
> > > After the Fedora 31 freeze I would like to introduce this new mirrorlist
> > > server implementation on the proxies. I already verified that I can run
> > > this mirrorlist container rootless. This new container can be a drop-in
> > > replacement for the current container and no infrastructure around it
> > > needs to be changed.
> > >
> > > The main changes to get it into production is to change
> > > mirrorlist1.service
> > > and mirrorlist2.service to include a line "User=mirrormanager" and
> > > replace the current container name with new container.
> > >
> > > Adrian
> > > _______________________________________________
> > > infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> > > To unsubscribe send an email to
> > > infrastructure-le...@lists.fedoraproject.org
> > > Fedora Code of Conduct:
> > > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > List Archives:
> > > https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
> _______________________________________________
> infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
_______________________________________________
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
