Another Rust MirrorManager experiment

2020-06-01 Thread Adrian Reber
Our MirrorManager setup exports the current state of all mirrors every
hour at :30 to a protobuf based file which is then used by the
mirrorlist servers to answer the requests from yum and dnf.

The Python script requires up to 10GB of memory and takes between 35 and
50 minutes. The script does a lot of SQL queries and also some really
big SQL queries joining up to 6 large MirrorManager tables.

I have rewritten this Python script in Rust and now it only needs around
1 minute instead of 35 to 50 minutes and only 600MB instead of 10GB.

I think the biggest difference is that I am almost not doing any joins
in my SQL request. I download all the tables once and then I do a lot of
loops over the downloaded tables and this seems to be massively faster.

As the mirrorlist-server in Rust has proven to be extremely stable over
the last months we have been using it I would also like to replace the
mirrorlist protbuf input generation with my new Rust based code.

I am planing to try out the new protobuf file in staging in the next
days and would then try to get my new protobuf generation program into
Fedora. Once it is packaged I would discuss here how and if we want to
deploy in Fedora's infrastructure.

Having the possibility to generate the mirrorlist input data in about a
minute would significantly reduce the load on the database server and
enable us to react much faster if broken protobuf data has been synced
to the mirrorlist servers on the proxies.

Adrian


signature.asc
Description: PGP signature
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org


Re: The future of loopabull

2020-06-01 Thread Pierre-Yves Chibon
On Fri, May 29, 2020 at 03:18:58PM +0200, Clement Verna wrote:
>On Thu, 28 May 2020 at 14:27, Pierre-Yves Chibon <[1]pin...@pingoured.fr>
>wrote:
> 
>  Good Morning Everyone,
> 
>  I know this question has already been raised a few times, but I think we
>  should
>  raise it once more: what do we see as future for loopabull?
> 
>  It is currently triggered on 4 topics (3 from prod and 1 from stg) to do
>  basically
>  three actions:
>  - Flag commit successfully built in koji, in other words it adds these
>  flags
>    to dist-git:
>   
>  
> [2]https://src.fedoraproject.org/rpms/mingw-filesystem/c/717f2a929bd25b62a0427e8e5c3792a0939dbfce
>  - Flag when the Fedora CI start testing a PR
>  - Flag when the Fedora CI finished testing a PR (and thus reports
>  Pass/Fail)
> 
>  Upstream released yesterday a 0.0.7 release which brings supports for
>  fedora-messaging (contributed by your servitor).
>  Looking at the code, it should be python3 compatible, but it doesn't say
>  specifically in the setup.py and I honestly don't remember if I've
>  tested that
>  or not.
>  The package has been orphaned in Fedora for over 10 months and has thus
>  been
>  retired.
> 
>  I've had a chat with upstream yesterday and they are still interested in
>  the
>  project but more as a pet project and their time is just like the rest
>  of us,
>  limited for pet projects these days.
>  That being said the code base is really quite small and involves
>  technologies
>  we're already using in other places (python-pika, celery, rabbitmq,
>  ansible...)
>  so there isn't really anything new there.
> 
>  One of its limitation currently is with secrets, how to pass/specify
>  them.
>  This is something we could circumvent via ansible-vault or so, but it
>  needs a
>  little investigation.
> 
>  I basically see three ways forward with this:
>  - We continue with loopabull and we need to check its python3 support,
>  how to
>    deal with secrets, if we can get it to run in openshift & so on.
>  - We look for something else, similar. The requirements being:
>    - Run a task when seeing a message in our message bus
>    - Handle secrets
>    - Scalable up/down
>    - Runnable in openshift is a bonus
>    - Preferably in a language we can debug (python++, potentially rust)
>  - We write something that fits our needs and requirements
> 
>There is a PR[0] in fedora-messaging to add a 'run' callback that would
>let you execute any command, I think that might be a nice solution and I
>think it would meet most of the requirements.
> 
>[0] - [3]https://github.com/fedora-infra/fedora-messaging/pull/163

Isn't that the equivalent of having us write a custom fedora-messaging consumer
for each task we want to automate?
In a way I like this, it's simple(r), really straight forward, constraint and
self-contained. There is no risk that a previous task prevents a following one
to be executed.
On the other hand it also means that if we move to, say fed-messaging, in the
future, we will have to convert more consumers.

If that's a trade off we're willing to live with, then I'm ok with it as well.


Pierre
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org