On Tue, May 14, 2013 at 04:00:27PM +0200, Patrik Rak wrote:
> You try to detect them, by initiating the request and measuring how
> long it takes. No matter what exactly the test is, that means
> allocating some resources for each such test (giving away). However,
> unless you are willing to tear down the connection if it doesn't
> complete on time (taking away), the resource is wasted until the
> test completes. And the bad guys will eventually take it all.
> Therefore, you have merely moved the bottleneck elsewhere - instead
> of competing for the delivery agents, they compete for the bad/good
> guy test resources.
> The fact that they are the same resourcein what you describe makes
> no difference.
Repeating the same non-fact without analytical backup does not make
it true. There is no infinite supply of "bad guys" to take it all.
There is a finite arrival rate of mail that takes a long time to
process (perhaps time out, perhaps be delivered), this arrival rate
creates a demand for equal or higher output throughput, otherwise
such mail *accumulates* (until the arrival rate drops or perhaps the
supply is exhausted).
Throughput = concurrency / latency. If we provide enough concurrency,
the throughput exceeds the arrival rate and there is no congestion.
Queueing systems are bimodal. When the demand is below the critical
level, the queue is empty. When it spikes above, the queue grows
(until demand levels off).
You must switch from gut reaction to detailed analysis or we are
wasting time.
I am assuming that the case that lead to this thread is accumulation
of repeatedly timed-out mail in the deferred queue. With a
sufficiently narrow slow output channel, and enough such mail, at
least half of it will be in the active queue at any given time,
once the time it takes for a message at the back of the queue to
reach the front is equal to the maximum backoff time or higher.
You can attempt to quench supply by suppressing deferred queue
scans, but this hurts innocent bystanders. The crude deferred
== slow heuristic is not by itself a good solution.
> Therefore, I say the guess is the best what we have got.
We must increase concurrency for such mail, perhaps in a separate
transport, but then risk "thundering hurd" when the guess is wrong,
hence bi-modal process limits.
> >There is no "giving away" to maintain throughput, high latency
> >tasks warrant higher concurrency, such concurrency is cheap since
> >the delivery agents spend most of their time just sitting there
> >waiting.
>
> You say it's cheap. I believe it is in your environment, as well as
> it is in mine and in many others. However, I believe that no one is
> willing to pay for more RAM every month for their cloud servers,
> just so they can dedicate it for dealing with mail that never gets
> delivered. That seems like total waste of money. Especially if there
> is another solution which works with fixed resources at hand.
Most people don't have this problem. The RAM required is rather
light. You have not thought carefully through the dynamics of the
queue or the impact on deferred queue processing, which is also
important, it is not all junk, or we would not bother deferring
mail at all.
> The problem with your approach is that whatever the bad guys want,
> you give it to them.
Nonsense. There are no "bad guys", just an arrival rate of mail
that takes longer to process. If someone wants to DoS your MTA,
it is far easier than trying to starve SMTP output with slow mail.
Also for most sites, I can't inject arbitary recipients into their
queue.
> That's meek. They want more, you give it to them.
There is no "them". This is not an analysis.
> Not anymore. What I suggest is the solution 1 and 3 from my previous
> mail, which both merely restrain how much of the available resources
> we are willing to give to the bad guys in the worst case. Note that
> there is no slow/fast way either, which would somehow affect either
> group. We just make sure that those who we think are bad never get
> everything.
But if this is an issue, the slow path *needs more* concurrency or
else the slow path is starved and it contains real mail to be
delivered.
Please come back with a model to which we can apply thought experiment
analyses (even simulations if we wanted to write code for that).
Without a model, there is no way to validate the design.
> Now if one group becomes really slow, like 30 or 60 times slower
> than the other one, it's effectively the case when it starts
> starving the other one.
This is too crude. The deferred queue emits mail at a fixed rate,
proportional to its population and dependent on the maximal backoff
time, the messages in the deferred queue that are just sitting
there do no harm. If this rate is below the throughput for slow
messages, we have no congestion problem.
We can:
- Raise the throughput (expanded concurrency).
- Reduce the rate (increase the maximal backoff time,
reduce the queue lifetime).
If you want to guarantee a reserved pool of processes for new mail,
we can do that too (most easily second instance and smtp_fallback_relay).
> If it is the slow group which becomes this
> slow, it gets 60:1 split, which with ~100 delivery agents available
> is obviously not enough to get new mail delivered fast enough.
Only if the throughput is not high enough to exceed the arrival rate.
A healthy MTA has a small deferred queue, and most deferred queue
scans finish quickly, thus at any given moment the deferred queue
is not a source of new mail (every 5 minutes a new scan injects
a small amount of previously deferred mail).
The flow from the deferred queue is a trickle, not a stream. It does
not much additional concurrency to provide an output rate equal to the
deferred queue input rate.
> Finally, what happens if the fast group becomes terribly slow
> instead, and the slow group not? I'd conclude that that doesn't have
> to bother us. It makes no sense to try to take resources away from
> new mail which suddenly became slow, just so we can try more new
> mail which can be just as slow. And taking resources away from slow
> new mail just so we can retry some deferred mail seems equally
> pointless. So I would say it's perfectly fine to wait until the
> situation gets back to normal, shall this ever happen.
There is no taking resources away. I run slow deliveries at high
concurrency (since the network can afford this), and fast deliveries
at lower concurrency (since that's enough, and we don't want thrashing).
> Now, does this make it any clearer what I have in mind?
No, because all this was already clear. You need a more complete
model of the queue dynamics.
Yes, separate output pools can help. But once the deferred queue
is creating congestion, starving it is not a good idea. Provide
more concurrency in a separate pool if you want some sort of
guaranteed capacity for new mail (which may itself include bursts
of slow messages that clog the queue).
Don't starve the deferred mail, it needs to processed too.
Grey-listed mail should not be delayed for hours or days.
Reduce the deferred queue output rate, by tuning parameters.
Implement DNS prefetch.
All or some of of these are reasonable, just starving the deferred
queue with reduced concurrency is not.
--
Viktor.
