[pfx] Re: Active queue congestion
You can also configure a non-zero smtpd_client_message_rate_limit On 07.03.24 17:21, Colin McKinnon via Postfix-users wrote: H, not so sure about that. The docs do advise against this for legitimate traffic - and I've yet to see anything in the documentation that describes what happens when these rates are exceeded is it a 4xx? a 5xx? Is the IP just blocked? I have set this number on some servers to big enough (1000), just to see maximum number in anvil stats. It helps with setting limits later. And yes, there are better ways for this, e.g. using postfwd. you could use a policy service to impose rate limits per SASL login, or sender address I had not considered that as a means of load balancing across the available relays (delaying the message at the origin is very much a last resort). I will do some reading on this. Note that policy limits incoming mail, not outgoing. Just like smtpd_*_limit - these are to limit receiving mail from your clients, not sending it out. -- Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. Despite the cost of living, have you noticed how popular it remains? ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Active queue congestion
On Thu, Mar 07, 2024 at 01:11:09PM -0500, Wietse Venema via Postfix-users wrote: > > I am planning to look at increasing the size of the Active queue however I > > would need to resize to a minimum of 50x based on past events. > > That should be OK as long as your syustem has enough memory. A million messages on a typical modern server system with 100+ GB of RAM should not be a problem. If a typical message costs ~10k of RAM, that'd still be "just" 10GB, which should be fine. Somewhere north of that you'd start to run out of room. But it is of course best to not let a small set of users "hog" the queue to the tune of of multiple 100k messages. Hence rate limits. If the limits are sufficiently well explained to the users, they may/should be able to tune their submission rates to avoid hitting the limits. -- Viktor. ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Active queue congestion
Colin McKinnon via Postfix-users: > Thank you, Viktor. > > I am planning to look at increasing the size of the Active queue however I > would need to resize to a minimum of 50x based on past events. That should be OK as long as your syustem has enough memory. > > You can also configure a non-zero smtpd_client_message_rate_limit > > H, not so sure about that. The docs do advise against this for > legitimate traffic - and I've yet to see anything in the documentation that > describes what happens when these rates are exceeded is it a 4xx? a 5xx? Is > the IP just blocked? It does not interact well when the client is an MTA as it messes up scheduling and can result in huge delays. If the clients are human-operated MUAs then that would not apply. postfix does not reject mail that exceeds the limit (WTF?) but it returns a 4xx status. Wietse > I now know that the config settings do not do what I expected - which is > unfortunate as this would have been a very simple solution. > > > you could use a policy service to impose rate limits per SASL login, or > sender address > > I had not considered that as a means of load balancing across the available > relays (delaying the message at the origin is very much a last resort). I > will do some reading on this. > > C. > > On Thu, 7 Mar 2024 at 13:46, Viktor Dukhovni via Postfix-users < > postfix-users@postfix.org> wrote: > > > On Thu, Mar 07, 2024 at 12:26:06PM +, Colin McKinnon via Postfix-users > > wrote: > > > > > I look after a SAAS site where customers can send emails to their own > > > domains. At times some of our customers can initiate sending of large > > mail > > > volumes - which can swamp the active queue. > > > > Given sufficient memory, you can substantially raise the active queue > > size limit. Servers have a lot more RAM now than they did in 2001. > > The default of 20,000 could easily be raised by 10x to 20 on a > > server-class machine. > > > > If customers indeed send mail only to their own domain, the destination > > concurrency limits should ensure fairness, given sufficient space in the > > queue and sufficiently many delivery agent slots. > > > > Speaking of delivery agent slots, if you have enough network bandwidth, > > you can raise the smtp(8) delivery process limit in master.cf from 100 > > to 1000: > > > > smtp unix - - n - 1000smtp > > > > Not that this could require some system-dependent tuning of the open > > file hard limit in whatever code starts Postfix, if the limit is not > > already very generous (on a Fedora 39 system with 65GB RAM, "ulimit -Hn" > > reports ~1.8 million max open files). > > > > > >From [1]: > > > "The only way to reduce congestion is to either reduce the input rate or > > > increase the throughput. Increasing the throughput requires either > > > increasing the concurrency or reducing the latency of deliveries." > > > > I am suggesting increasing concurrency, and also increasing the queue > > depth to allow your customer to send larger bursts of mail without > > overflowing the queue size limit. You can also configure a non-zero > > > > smtpd_client_message_rate_limit > > > > if abuse of your resources is plausible even with the larger queue size. > > If that's too crude, you could use a policy service to impose rate > > limits per SASL login, or sender address, ... > > > > > I thought that reducing TRANSPORT_recipient_refill_limit and > > > TRANSPORT_recipient_limit would prevent messages sent to the customers > > > domain from dominating the active queue / prevent blocking of other > > > customers / backup messages in the incoming queue. > > > > These controls affect deliveries of single messages with many > > recipients, but have no effect on a flood of single-recipient messages. > > > > -- > > Viktor. > > ___ > > Postfix-users mailing list -- postfix-users@postfix.org > > To unsubscribe send an email to postfix-users-le...@postfix.org > > > > > -- > -BEGIN GEEK CODE BLOCK- > Version: 3.1 > GCM d s+:+ a+ C+++(---)$ UL+++ P+(--) L+++ E--- W+++ N++ w-- PS++(+++()) > t+ 5+ X R- tv-- b++ DI++ D e+++ h > --END GEEK CODE BLOCK-- > ___ > Postfix-users mailing list -- postfix-users@postfix.org > To unsubscribe send an email to postfix-users-le...@postfix.org ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Active queue congestion
Thank you, Viktor. I am planning to look at increasing the size of the Active queue however I would need to resize to a minimum of 50x based on past events. > You can also configure a non-zero smtpd_client_message_rate_limit H, not so sure about that. The docs do advise against this for legitimate traffic - and I've yet to see anything in the documentation that describes what happens when these rates are exceeded is it a 4xx? a 5xx? Is the IP just blocked? I now know that the config settings do not do what I expected - which is unfortunate as this would have been a very simple solution. > you could use a policy service to impose rate limits per SASL login, or sender address I had not considered that as a means of load balancing across the available relays (delaying the message at the origin is very much a last resort). I will do some reading on this. C. On Thu, 7 Mar 2024 at 13:46, Viktor Dukhovni via Postfix-users < postfix-users@postfix.org> wrote: > On Thu, Mar 07, 2024 at 12:26:06PM +, Colin McKinnon via Postfix-users > wrote: > > > I look after a SAAS site where customers can send emails to their own > > domains. At times some of our customers can initiate sending of large > mail > > volumes - which can swamp the active queue. > > Given sufficient memory, you can substantially raise the active queue > size limit. Servers have a lot more RAM now than they did in 2001. > The default of 20,000 could easily be raised by 10x to 20 on a > server-class machine. > > If customers indeed send mail only to their own domain, the destination > concurrency limits should ensure fairness, given sufficient space in the > queue and sufficiently many delivery agent slots. > > Speaking of delivery agent slots, if you have enough network bandwidth, > you can raise the smtp(8) delivery process limit in master.cf from 100 > to 1000: > > smtp unix - - n - 1000smtp > > Not that this could require some system-dependent tuning of the open > file hard limit in whatever code starts Postfix, if the limit is not > already very generous (on a Fedora 39 system with 65GB RAM, "ulimit -Hn" > reports ~1.8 million max open files). > > > >From [1]: > > "The only way to reduce congestion is to either reduce the input rate or > > increase the throughput. Increasing the throughput requires either > > increasing the concurrency or reducing the latency of deliveries." > > I am suggesting increasing concurrency, and also increasing the queue > depth to allow your customer to send larger bursts of mail without > overflowing the queue size limit. You can also configure a non-zero > > smtpd_client_message_rate_limit > > if abuse of your resources is plausible even with the larger queue size. > If that's too crude, you could use a policy service to impose rate > limits per SASL login, or sender address, ... > > > I thought that reducing TRANSPORT_recipient_refill_limit and > > TRANSPORT_recipient_limit would prevent messages sent to the customers > > domain from dominating the active queue / prevent blocking of other > > customers / backup messages in the incoming queue. > > These controls affect deliveries of single messages with many > recipients, but have no effect on a flood of single-recipient messages. > > -- > Viktor. > ___ > Postfix-users mailing list -- postfix-users@postfix.org > To unsubscribe send an email to postfix-users-le...@postfix.org > -- -BEGIN GEEK CODE BLOCK- Version: 3.1 GCM d s+:+ a+ C+++(---)$ UL+++ P+(--) L+++ E--- W+++ N++ w-- PS++(+++()) t+ 5+ X R- tv-- b++ DI++ D e+++ h --END GEEK CODE BLOCK-- ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
[pfx] Re: Active queue congestion
On Thu, Mar 07, 2024 at 12:26:06PM +, Colin McKinnon via Postfix-users wrote: > I look after a SAAS site where customers can send emails to their own > domains. At times some of our customers can initiate sending of large mail > volumes - which can swamp the active queue. Given sufficient memory, you can substantially raise the active queue size limit. Servers have a lot more RAM now than they did in 2001. The default of 20,000 could easily be raised by 10x to 20 on a server-class machine. If customers indeed send mail only to their own domain, the destination concurrency limits should ensure fairness, given sufficient space in the queue and sufficiently many delivery agent slots. Speaking of delivery agent slots, if you have enough network bandwidth, you can raise the smtp(8) delivery process limit in master.cf from 100 to 1000: smtp unix - - n - 1000smtp Not that this could require some system-dependent tuning of the open file hard limit in whatever code starts Postfix, if the limit is not already very generous (on a Fedora 39 system with 65GB RAM, "ulimit -Hn" reports ~1.8 million max open files). > >From [1]: > "The only way to reduce congestion is to either reduce the input rate or > increase the throughput. Increasing the throughput requires either > increasing the concurrency or reducing the latency of deliveries." I am suggesting increasing concurrency, and also increasing the queue depth to allow your customer to send larger bursts of mail without overflowing the queue size limit. You can also configure a non-zero smtpd_client_message_rate_limit if abuse of your resources is plausible even with the larger queue size. If that's too crude, you could use a policy service to impose rate limits per SASL login, or sender address, ... > I thought that reducing TRANSPORT_recipient_refill_limit and > TRANSPORT_recipient_limit would prevent messages sent to the customers > domain from dominating the active queue / prevent blocking of other > customers / backup messages in the incoming queue. These controls affect deliveries of single messages with many recipients, but have no effect on a flood of single-recipient messages. -- Viktor. ___ Postfix-users mailing list -- postfix-users@postfix.org To unsubscribe send an email to postfix-users-le...@postfix.org
