> > <for what it is worth> > I work in telecom for meteorology, and we ended up with a general method > for file copying (catchphrase: rsync on steroids*.) ( *every catchphrase is > a distortion, no dis to rsync, but in certain cases we do work much faster, > it just communicates the idea.) Sarracenia ( > https://github.com/MetPX/Sarracenia) is a GPL2 app (Python and C > implementations) that use mozilla public license rabbitmq broker, as well > as openssh and/or any web server to do fastish file synching, and/or > processing/orchestration. The app is just json messages with file metadata > sent through the broker. Then you daisy chain brokers through clients. No > centralization (every entity installs their own broker), No federated > identity required (authentication is to each broker, but they can pass > files/messages to each other.) > A firstish thing to do with it would be to sync the debian mirrors in > real-time rather than periodically. Each mirror has a broker, they get > advertisements (AMQP messages containing JSON file metadata) download the > corresponding file, and re-advertise (publish on the local broker with the > local file URL) for downstream clients. You can then make a mesh of > mirrors, where, if each mirror is subscribed to at least two others, then > it can withstand the failure of any node. If you add more connections, you > increase redundancy. > Once you have that sort of anchor tenant for an AMQP message bus, people > might want to use it to provide other forms of automation, but way quicker > and in some ways much simpler than SMTP. but yeah... SMTP is a lot more > well-known/common. RabbitMQ is the industry dominant open solution for AMQP > brokers. sounds like marketing bs, but if you look around it is what the > vast majority are using, and there are thousands upon thousands of > deployments. It's a much more viable starting point, for stability, and a > lot less assembly required to get something going. Sarracenia makes it a > bit easier again, but messages are kind of alien and different, so it takes > a while to get used to them. > </for what it is worth>
Peter, I like the solution and for the mirrors it sounds great but I have a few nitpicks: - the file syncing part is makes a perfect sense for the debian mirrors but in general case you might only want to send a message and skip the file syncing part - I am currently, personally more intrigued by even more standard technologies than RabbitMQ and I believe that a good solution might lie there What I particularly like about Sarracenia is that it is decentralized because each host has its own broker - that I think is cool and I would like to potentially do something similar... clime On Wed, 25 Mar 2020 at 01:07, clime <cli...@gmail.com> wrote: > On Wed, 25 Mar 2020 at 01:00, clime <cli...@gmail.com> wrote: > > > > On Tue, 24 Mar 2020 at 22:45, Nicolas Dandrimont <ol...@debian.org> > wrote: > > > > > > On Tue, Mar 24, 2020, at 21:51, clime wrote: > > > > On Tue, 24 Mar 2020 at 20:40, Nicolas Dandrimont <ol...@debian.org> > wrote: > > > > > > > > > > Hi! > > > > > > > > > > On Sun, Mar 22, 2020, at 13:06, clime wrote: > > > > > > Hello! > > > > > > > > > > > > Ad. https://lists.debian.org/debian-devel/2016/07/msg00377.html > - > > > > > > fedmsg usage in Debian. > > > > > > > > > > > > There is a note: "it seems that people actually like parsing > emails" > > > > > > > > > > This was just a way to say that fedmsg never got much of a user > base in the services that run on Debian infra, and that even the new > services introduced at the time kept parsing emails. > > > > > > > > Hello Nicolas! > > > > > > > > Do you remember some such service and how it used email parsing > specifically? > > > > > > I believe that tracker.debian.org was introduced around that time. > > > > > > At the point it was created, tracker.d.o was mostly consuming emails > from packages.debian.org to update its data. These days tracker.d.o has > replaced packages.d.o as "email router", in that it receives all the mails > from services (e.g. the BTS, the archive maintenance software, buildds, > salsa webhooks, ...) and forwards them to the public. > > > > > > > I am still a bit unclear how email parsing is used in Debian > > > > infrastructure, don't get me wrong, I find it elegant > > > > > > Ha. I find that it's a big mess. > > > > > > Here's the set of headers of a message I received today from > tracker.d.o, which are supposed to make parsing these emails better: > > > > > > X-PTS-Approved: yes > > > X-Distro-Tracker-Package: facter > > > X-Distro-Tracker-Keyword: derivatives > > > X-Remote-Delivered-To: dispa...@tracker.debian.org > > > X-Loop: dispa...@tracker.debian.org > > > X-Distro-Tracker-Keyword: derivatives > > > X-Distro-Tracker-Package: facter > > > List-Id: <facter.tracker.debian.org> > > > X-Debian: tracker.debian.org > > > X-Debian-Package: facter > > > X-PTS-Package: facter > > > X-PTS-Keyword: derivatives > > > Precedence: list > > > List-Unsubscribe: <mailto:cont...@tracker.debian.org > ?body=unsubscribe%20facter> > > > > > > I'll leave you to judge whether this makes sense or not. > > > > > > (and it turns out that the actual useful payload was just plaintext > with no real chance of automated parsing) > > > > > > > but from what I have found (e.g. reportbug), in the beginning there > is an > > > > email being sent by some human which will then trigger some automatic > > > > action (e.g. putting the bug into db). So it's like you could do all > > > > your work simply by sending emails (some of them machine-parsable). > > > > > > > > So do you have the opposite? I do some clicking action somewhere and > > > > it will send an email to a certain mailing list to inform human > > > > beings? Or let's not just clicking but e.g. `git push` (something > that > > > > you can still do from command line). > > > > > > > > Do you have: I do some clicking action somewhere and it will send an > > > > email to a certain mailing list where the email is afterward parsed > by > > > > another service which will do an action (e.g. launch a build) based > on > > > > it? > > > > > > Both of these are somewhat true. > > > > > > Some examples of email-based behaviors: > > > - Our bug tracking system is fully controlled by email. > > > - Closing a bug in reaction to an upload is done by an email from the > archive maintenance system (dak) to the bug tracking system. > > > - Salsa has a webhook service that react to UI clicks (e.g. "clicking > the merge button") by sending an email to the BTS (e.g. to tag bugs as > pending), or to tracker.d.o (for new commit notifications). > > > - Some of our IRC bots are triggered by procmail rules. > > > - At some point mentors.debian.net depended on a NNTP gateway to the > debian-devel-changes mailing list to trigger removal of superseded packages > (...) > > > - etc. etc. > > > > > > I'm still not sure where your trail of questions is going? fedmsg in > Debian has been dead for years at this point, and there still doesn't seem > to be much interest to implement anything beyond email parsing in some of > our core systems. > > > > Cool, so basically what I am thinking about is to create a free > > software from what you are describing. I.e. create reusable tooling > > out of the Debian messaging system. Something that a new linux > > distribution can easily start using to connect their services. > > > > I didn't know Debian infra works like this but I find it very > > elegant/efficient and I would like the solution you have to be > > reusable by others. > > > > So basically the tooling should contain: > > - unified email message format > > - library that is able to translate a message to a language data > > structure (e.g. dictionary in python) > > - email receiver that would be listening for emails coming from the > > bus and emitting events based on that (this could be part of the > > library so you would be able to attach a callback for an incoming > > message or just do blocking waits) > > - email publisher - something that can send a new message into the > > bus, i.e. to a preconfigured mail server (a "broker" or "hub") > > - mail server that would have an http API to manage topic > > subscriptions (i.e. add/delete me from a given topic) - it would > > receive a message from a publisher for a given topic, found out who is > > subscribed to it, and duplicated the email message for each consumer > > and send it to them > > > > For the mail server I am thinking about https://www.courier-mta.org/ > > and using https://www.courier-mta.org/maildropgdbm.html for > > subscription management. > > > > Basically, this I thought could be a new "email backend" in fedmsg > > instead of zeromq one... > > > > I am not very familiar with email technology but I like the idea because: > > - if you do an email setup for people, you are going to already be > > technically skilled to do it for services or vice versa > > - one of communicating agents may be a human being that is watching > > what's going on in system by having dedicated inbox folders for each > > type of event (topic) - no amqp/zeromq/mqtt -> email translation is > > needed here - everything is just email (except for irc messages > > emitted based on those) > > - i think this can be optimized to work very reliably inside one > > infrastructure (e.g. debian.org) but at the same time it is easy for > > an outside listener to join in with his/her own service and start > > doing some stuff based on Debian events (if the subscription hub is > > public) > > - it uses the most standard and compatible protocol possible (SMTP) so > > shouldn't be an opinionated technology - theoretical message > > throughput will be limited because of that (i suspect SMTP is not > > extremely fast) but it should be still sufficient to handle all the > > distribution events > > I forgot one large advantage - it is compatible with your way of > operating services by sending emails to them, it is just about making > the interface standardized across applications... > > > > > I am still exploring ideas to do a federated message bus so this is one > of them > > Please, take this as a wild brainstorming, maybe I should have given > > this more time to settle in my head but on the other hand, I won't > > mind being pwned too much here > > clime > > > > > > > > Bye, > > > -- > > > Nicolas Dandrimont >
