Hi Mark,
answers inlined:
On 27 Nov 2006, at 06:59, mark angrish 2 wrote:
Hi,
I've had a search around but couldn't find anything that could help.
I currently have around 2 million messages that are persisted in my
queue.
The situation I'm in is that the queue must hold onto 1 months (say
200Gb)
worth of data before I can put consumers on to process the backlog. I
currently only have the default config file setup. My consumer is
set up to
use XA between activemq 4.0.2 and oracle 10g using atomikos
transaction
manager 3 integrated with spring 2.0, which allows me to use
message driven
pojos! I am using the marathon fix (if you search the bug fix area
you will
find the patch) for activemq to allow it to work with XA.
My questions:
1. My problem is that when i start active mq with the small 2
million record
scenario, it takes ages to start up. Is there any way to overcome
this poor
startup time?
There's new functionality that is in trunk at the moment, but is
due to be released in ActiveMQ 4.2 that solves this. In the current
release, all messages are loaded into the broker and a message
reference is then held in memory that has a pointer to the message in
the store. This can be optionally replaced by a paging mechanism that
pages messages into the broker when required.
However, there may be some work required to optimize some of the
queries for oracle - see http://www.nabble.com/Statements.java%2C-
etc.-Performance-tf2662372.html#a7425760
2. This is actually my big pain point at the moment. When activemq
does
eventually start up with the 2 million messages, I can't use hermes
to see
the messages (it seems to take hours and still no result), and i
can't see
any processing happening on the xa consumer side to the database.
When used
normally say 10's or 100's of messages, so a steady stream of
messages,
everything processs to the database just fine. I tried tinkering
with the
prefetch limit but that didn't seem to help either. Something that
could
say "right, let the consumer take x amount of messages off the
queue and
process them with XA" would be awesome here.
Hopefully the paging mechanism will help here too.
3. We have about 6 queues on 6 machines with 24 applications acting as
producers into them (so 4 producers going to each queue). We have
a SAN set
up so that each queue can write to its own storage area on the san
(i.e. 6
activemq data directories on the SAN). If i wanted to set up a
store and
forward kind of setup to 2 'central queues' where the consumers of
the queue
lie, do i just configure a network connector on the 6 queues to
point to the
central queue locations and get those central queues to read from
all 6 SAN
data directories? Or do i actually need 6 consumer queues? In
either case
is it ok for them to point to the SAN to read the message stores,
considering the producers are writing and the consumers are removing?
Any help would be massively appreciated!
There isn't a problem with either 2 central queues or 6 consumer
queues - but I'd like to understand a little more about your setup -
do you really need store and forward, or could you just have a
centralized hub around your SAN ?
cheers,
::mark
--
View this message in context: http://www.nabble.com/starting-a-
consumer-on-a-queue-with-lots-of-messages-tf2710121.html#a7555664
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
cheers,
rob
