Hi all,
I'm an Apache JAMES committer and I'm "almost" new to ActiveMQ.
I'm starting analysis on how to replace our default spool with ActiveMQ
and I hope you can give me some hints :-)
It would be better to use ActiveMQ via JMS (more flexibility) but if
there is any better solution to our problems by using specific ActiveMQ
APIs then why not!!
Our scenario is an SMTP Server so we have something like this:
1) SMTP Server receives messages and put them to the spool. The spool
have to be persistent because once the message has been posted via SMTP
we cannot loose it. Most time the message will be consumed very fast, so
in past I looked at using Kaha directly for this, but maybe the 5.0 AMQ
Message Store already handle this one in a performant way?
2) Our current spooling have this architecture:
we have a single "spool" that contains messages with a "state". We read
a random message from the spool, look at its state and then start the
processing depending on the state itself at the end of the processing we
can alter the state and leave the message in the spool, or we can remove
it from the spool. In the processing we could even push more messages
into the spool (e.g: to split the message to 2 different paths). ATM the
re is no transaction management.
The processing from a state to another (or to delete) is a sequence of
micro-processings (named matchers/mailets in james), so the actual
status depends also on what matchers/mailets have been processed so far,
but we currently keep this in memory and never store this. So if
something goes wrong (given that we don't have transactions) we simply
start from the beginning of that "state processor" (I'd like to improve
this issue, too, with the new ActiveMQ based spool).
Some times the message is simply moved from one state to another a few
times and then it is removed from the spool because of 2 causes:
a) it has been moved to the "outgoing spool" (the spool for the messages
to be sent to other smtp servers)
b) it has been posted to an user inbox.
Other times the message is altered in its content.
So you see in James we currently have a single "message store" and we
can "lock on a message" (so no other thread will take it) "retrieve it",
"update and unlock it" (alter its state or state+content) or "remove
it". How would you manage this with ActiveMQ?
3) Outgoing spool:
The outgoing spool in JAMES is a spool like the main spool, with the
difference that a message delivery could fail and there is a retry
schedule. So we try to send a message, on failure we try again 10
minutes later, then 30 minutes later, then 2 hours later (it is
configurable) and so on. ATM we store the "next-attempt-date" and then
each "deliverer" simply take the message with the minor
next-attempt-date and if it is due for delivery it starts its work,
otherwise it will simply wait the needed time (one deliverer is noticed
when a *new* message enter this spool / They all "wait" on the spool and
the spool is noticed one at each store).
The most common case is:
a) the message we received at #1 entered the spool #2 and is processed
very fast and it ends in the outgoing spool #3 where it is delivered on
the first attempt. In this case it would be cool if the message was in
memory and simply written once for safety because the processing should
be fast and it would be slow to read it again from the disk.
b) we fail our first attempt, then it does not make sense to keep it in
memory because we know we won't need it in the next X minutes/hours.
Any suggestions on how to do this with ActiveMQ?
As a last point we have to take care of 2 different use-cases:
I) most traffic is done by fastmoving small messages but
II) many messages are 1-10MB in size, and a few message could be even
100MB or even more: how should we handle this messages in ActiveMQ given
that we can't take them in memory but we simply want to stream then in
and out from the server?
I understand this is a lot of questions, but I would really appreciate
any hint, even partial. I'm collecting ideas :-)
Stefano
PS: we are also evaluating using JCR for inboxes if you was wondering,
but this is another story, for another list ;-)
