ouvtam created JAMES-4192:
-----------------------------
Summary: ActiveMQ: delayed mails block the queue while dequeuing
Key: JAMES-4192
URL: https://issues.apache.org/jira/browse/JAMES-4192
Project: James Server
Issue Type: Bug
Components: Queue
Affects Versions: 3.7.0, 3.8.0, 3.7.1, 3.7.3, 3.7.4, 3.8.1, 3.7.5, 3.8.2,
3.8.3
Reporter: ouvtam
We operate James and observed issues when encountering a lot of temporary
failures on remote delivery. It seemed James could not dequeue the outgoing
queue.
*Bug*
Delayed mails were enqueued into the regular ActiveMQ queue with
{{{}NEXT_DELIVERY_TIME{}}}.
Because of ActiveMQ queue paging ({{{}maxPageSize defaults to 200{}}}), delayed
messages at the head of the queue could block ready messages behind them,
preventing non-delayed mails from being dequeued.
*Steps to reproduce*
* Configure RemoteDelivery with <maxRetries>1</maxRetries> and a high delay
time (e.g. <delayTime>60 minutes</delayTime>)
* Send at least 200 mails to a remote server that causes temporary exceptions
(e.g. 421 greylist). In our case we configured a <gateway> that simulated such
behaviour (i.e. always sending a temporary exception 421)
* Sending the 201 mail will never be dequeued (i.e. returns always null),
because the 200 prior mails are sitting in the queue (ActiveMQ's
maxPageSize=200). After 60 minutes of delay time the queue will processed
eventually.
*Fix*
Delayed mails are no longer kept in the regular JMS queue before they are due.
They are stored in the ActiveMQ Scheduler and only appear in the regular queue
once their delay expires.
*Result*
Non-delayed mails are no longer blocked by delayed mails.
*Implementation Details*
I will provide a PR that takes advantage of the ActiveMQ's Scheduler Support.
We already tested this in our local setup.
Mails with the delayed JMS property set (i.e.
org.apache.activemq.ScheduledMessage.AMQ_SCHEDULED_DELAY) will be intercepted
by ActiveMQ's Scheduler and not added to the queue (e.g. outgoing). Instead,
the scheduler stores the messages in the scheduler store. After the delay has
passed the scheduler sends the messages into the queue for the consumer.
*Alternative*
Another approach would be to increase maxPageSize (e.g. using Per Destination
Policies). This would increase the memory footprint of ActiveMQ, because it
would hold the amount of maxPageSize messages in memory. For delayed messages
this would not be very efficient, because with high delays they just sit in
memory. In addition, the problem could occur nonetheless (e.g. lot of
deliveries)
*Questions*
Using ** the scheduler introduces an important question:
* For instance, if 100 mails are delayed, the
ActiveMQCacheableMailQueue#getSize would return 0 if no mails are in the queue.
The reason is the delayed mails sit in the scheduler store. What is the
contract for ManageableMailQueue#getSize? Should it count the total mails with
the delayed messages? Or only the ones that have no delay? Since we make use of
WebAdmin this would be changing behaviour.
** Unfortunately, we cannot count the size of the scheduled messages using the
same approach as in normal queues (i.e. using Statistics Plugin).
*** Either we use a special JMS browse command to browse the messages in the
scheduler store while filtering the scheduled messages by queue (i.e. only
scheduled messages for the queue "outgoing").
*** Or we activate JMX on the Broker to access the ScheduledMessages directly
and iterate through all jobs, unmarshal the payload, and filter the scheduler
jobs by queue (i.e. only scheduled messages for the queue "outgoing").
*** Using WebAdmin it would be great to pass params such as
/mailQueues/outgoing?delayed
** An alternative would be to create two manageable queues by
ActiveMQMailQueueFactory#createCacheableMailQueue (e.g. "outgoing" and
"outgoing-delayed") that are manageable through "/mailQueues/outgoing" and
"/mailQueues/outgoing-delayed"
Happy for feedback, so I can create the PR in the next few days.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
