[Mailman-Developers] Re: Introduction, FOSDEM, scaling down, latency, OpenPGP support
Justus Winter writes: > But currently Mailman3 does fork+exec, so it doesn't get to share > the parent's pages. I experimented with fork-and-dont-exec [0], > but the results were underwhelming, because reference counting can > cause pages to diverge. Surprisingly, gc.freeze didn't seem to > help much, so there may have been issues beyond the reference > counts. > [...] > I think Python just doesn't support sharing code across processes > well. Seems likely. I know that Emacsen have always advised running just one process for this reason (also because users usually want all their recent hacks available in all buffers, but memory hogging is a big reason). ___ Mailman-Developers mailing list -- mailman-developers@python.org To unsubscribe send an email to mailman-developers-le...@python.org https://mail.python.org/mailman3/lists/mailman-developers.python.org/ Mailman FAQ: https://wiki.list.org/x/AgA3 Security Policy: https://wiki.list.org/x/QIA9
[Mailman-Developers] Re: Introduction, FOSDEM, scaling down, latency, OpenPGP support
Hi :) "Stephen J. Turnbull" writes: > Hi Justus! > > > Besides cleanups and bugfixes, there are three things I'd like to > > do: > > > > - Improve Mailman to better scale down to small installations > > Not sure what you can really do about that without rearchitecting. > The full suite of daemons is something like 13, including 3 WSGI > processes, the master daemon for mailman and about 7 or 8 runners. > > But I'm pretty sure people have run Mailman 3 on a Raspberry Pi. How > constrained an environment are you aiming for? I had problems on my shared hoster that provided 1 gigabyte of RAM per user (I'm not a 100% on how they measure that). I first noticed the problem because every now and then the OOM killer would kill a Mailman runner process, and because of a bug in the master process [0] it wasn't restarted, resulting in stalled mail processing with no indication, quite frustrating. 0: https://gitlab.com/mailman/mailman/-/merge_requests/1094 And, while I fixed the reliability issue, seeing my small installations (I'd be surprised if we see more than 1 message per day on average) consume so much memory was frustrating [1]. 1: https://gitlab.com/mailman/mailman/-/issues/1050 > > - Improve latency of messages > > What latency are you observing? My last project was getting about > 100,000 incoming per day across 20K lists, two incoming runners, 8 > outgoing, 1 each for the other Mailman runners. Never saw more than > about 5 seconds dwell in the Mailman system, except when the Mailman > to outgoing Postfix SMTP connection started glitching. We fixed that > by reconfiguring the Mailman host (in Dallas) to use an MX in the same > datacenter instead of one in Boston. (!!) And the normal case with a > process where I'd do "ls queue/*" evey 5s was completely empty queues. > Stuff just didn't stay around long enough for ls to see it. > > I see no reason to suppose you can do much better than that, but > again, tell me what you're seeing. I'm not experienced in dealing > with Mailman at scale, and that host was quite beefy. Still I have a > strong feeling that latency is mostly a communication with MTA issue, > not in Mailman 3 itself. The latency may be currently small, in absolute terms, but this comes at a considerable cost: the runners are polling their queues in loops. My installations that hardly see any traffic at all are all doing: do I have work, no, sleep 1, do I have work, no, sleep 1... I can see that this will amortize in big installations, but for small ones this is quite sad. And even for big installations, or if we say that efficiency is not important, if a mail goes through the hands of three queue runners, the worst-case latency is three seconds in an otherwise idle installation! We can definitively improve upon that. The key insight here is that emails in queues don't appear out of thin air, another runner is putting them there. If each runner that goes to sleep does so by waiting on a condition variable associated with its queue, and every runner that deposits a mail into the queue signals the sleeping runners, that latency goes away while at the same time improving efficiency by no longer having to poll the queue every second. > > - Implement OpenPGP support > > What does that mean? OpenPGP can be used to provide confidentiality and integrity for email. What exactly that means in the setting of mailing lists varies by threat model and policy. My prototype [2] simply records associations between addresses and OpenPGP certificates by consuming Autocrypt headers [3] and when sending an outgoing mail opportunistically encrypting it if a certificate is known. Details and future work in [2]. 2: https://gitlab.com/mailman/mailman/-/merge_requests/1166 3: https://autocrypt.org > > Here are the things I did so far: > > > > - I have Mailman running with runners in threads instead of > > processes, but that is in a proof-of-concept stage at this > > point and needs some cleaning up > > I guess this is supposed to address the resource consumption (memory > footprint?) issue? Yes. > After working with Mailman 3 and Postfix, I've become fond of the HUPD > (HUPD of Uncontrolled Proliferation of Daemons) model of application > design, at least for email. I feel *much* more comfortable messing > with individual daemons this way, knowing that I can't affect the > others. I'm not going to object to providing the threaded version if > people want it, but I would object to wholesale conversion to that > model without a lot of production experience based on it. My prototype let's you chose, for every kind of runner, whether to use the process or thread model, so it is actually a continuum between the current model, and using threads for all runners (with the exception of the REST runner, because gunicorn doesn't like to be run in the non-main thread). I don't quite buy (or maybe I'm not understanding the whole picture) into the argument that having
[Mailman-Developers] Re: Introduction, FOSDEM, scaling down, latency, OpenPGP support
"Stephen J. Turnbull" writes: > Дилян Палаузов writes: > > > Do you know by accident, if the runner processes use identical > > memory, and by > > calling https://man7.org/linux/man-pages/man2/madvise.2.html the > > kernel can somehow detect these identical memory and then use a > > single instance for all identical regions of memory? > > The memory size is a side effect of the way fork works (formally, it > creates a new process by duplicating the old one, what "really" > happens is up to the CPU's MMU and the kernel's memory management). I > believe that as long as copy-on-write is enabled it only actually > copies pages with changes. But currently Mailman3 does fork+exec, so it doesn't get to share the parent's pages. I experimented with fork-and-dont-exec [0], but the results were underwhelming, because reference counting can cause pages to diverge. Surprisingly, gc.freeze didn't seem to help much, so there may have been issues beyond the reference counts. 0: https://gitlab.com/mailman/mailman/-/merge_requests/1093 I think Python just doesn't support sharing code across processes well. Best, Justus ___ Mailman-Developers mailing list -- mailman-developers@python.org To unsubscribe send an email to mailman-developers-le...@python.org https://mail.python.org/mailman3/lists/mailman-developers.python.org/ Mailman FAQ: https://wiki.list.org/x/AgA3 Security Policy: https://wiki.list.org/x/QIA9
[Mailman-Developers] Re: Introduction, FOSDEM, scaling down, latency, OpenPGP support
Дилян Палаузов writes: > Do you know by accident, if the runner processes use identical > memory, and by > calling https://man7.org/linux/man-pages/man2/madvise.2.html the > kernel can somehow detect these identical memory and then use a > single instance for all identical regions of memory? The memory size is a side effect of the way fork works (formally, it creates a new process by duplicating the old one, what "really" happens is up to the CPU's MMU and the kernel's memory management). I believe that as long as copy-on-write is enabled it only actually copies pages with changes. Use ps or top to check to see how much of the virtually memory is actually in RAM, and how much is shared. I'm pretty sure real memory usage was considered in the original design to use processes for each runner rather than threads. Steve ___ Mailman-Developers mailing list -- mailman-developers@python.org To unsubscribe send an email to mailman-developers-le...@python.org https://mail.python.org/mailman3/lists/mailman-developers.python.org/ Mailman FAQ: https://wiki.list.org/x/AgA3 Security Policy: https://wiki.list.org/x/QIA9
[Mailman-Developers] Re: Introduction, FOSDEM, scaling down, latency, OpenPGP support
Hello Justus, I find it very good that you are trying to reduce the memory consumption of mailman 3. I cannot help in doing this. I hope you find some way to reduce the memory. Do you know by accident, if the runner processes use identical memory, and by calling https://man7.org/linux/man-pages/man2/madvise.2.html the kernel can somehow detect these identical memory and then use a single instance for all identical regions of memory? Greetings Дилян -Original Message- From: Justus Winter To: mailman-developers@python.org Subject: [Mailman-Developers] Introduction, FOSDEM, scaling down, latency, OpenPGP support Date: 01/21/2024 02:08:34 PM Hello everyone :) I want to introduce myself, I'm Justus, I have worked on GnuPG in the past and am now working on Sequoia-PGP, and I'm running two Mailman installations in a resource constrained shared hosting environment. I'd like to contribute a little to Mailman, and I'd like to better understand how the Mailman project is doing nowadays. I have gotten a bugfix merged in the past, but I now have what I think is a fairly uncontroversial cleanup merge request that has neither been merged nor has it gotten comments. - https://gitlab.com/mailman/mailman/-/merge_requests/1158 As FOSDEM is around the corner, are any of you going to be there and are up for a chat? Besides cleanups and bugfixes, there are three things I'd like to do: - Improve Mailman to better scale down to small installations - Improve latency of messages - Implement OpenPGP support Here are the things I did so far: - https://gitlab.com/mailman/mailman/-/merge_requests/1094 - https://gitlab.com/mailman/mailman/-/issues/1050 - https://gitlab.com/mailman/mailman/-/merge_requests/1166 - I have Mailman running with runners in threads instead of processes, but that is in a proof-of-concept stage at this point and needs some cleaning up (I understand that Mailman is a GNU project that wants copyright assignments, and I have done that in the past for other GNU projects, and would be happy to do that for Mailman, but at the same time I feel like putting up *any* barrier to contributing is unfortunate.) Best, Justus ___ Mailman-Developers mailing list -- mailman-developers@python.org To unsubscribe send an email to mailman-developers-le...@python.org https://mail.python.org/mailman3/lists/mailman-developers.python.org/ Mailman FAQ: https://wiki.list.org/x/AgA3 Security Policy: https://wiki.list.org/x/QIA9
