Also, like in the other thread you recommended, do you think i should look at another storage mechanism other than derby (like a kaha store, or postgres?)
rajdavies wrote: > > Hi Mark, > answers inlined: > > On 27 Nov 2006, at 06:59, mark angrish 2 wrote: > >> >> Hi, >> >> I've had a search around but couldn't find anything that could help. >> >> I currently have around 2 million messages that are persisted in my >> queue. >> The situation I'm in is that the queue must hold onto 1 months (say >> 200Gb) >> worth of data before I can put consumers on to process the backlog. I >> currently only have the default config file setup. My consumer is >> set up to >> use XA between activemq 4.0.2 and oracle 10g using atomikos >> transaction >> manager 3 integrated with spring 2.0, which allows me to use >> message driven >> pojos! I am using the marathon fix (if you search the bug fix area >> you will >> find the patch) for activemq to allow it to work with XA. >> >> My questions: >> >> 1. My problem is that when i start active mq with the small 2 >> million record >> scenario, it takes ages to start up. Is there any way to overcome >> this poor >> startup time? > There's new functionality that is in trunk at the moment, but is > due to be released in ActiveMQ 4.2 that solves this. In the current > release, all messages are loaded into the broker and a message > reference is then held in memory that has a pointer to the message in > the store. This can be optionally replaced by a paging mechanism that > pages messages into the broker when required. > However, there may be some work required to optimize some of the > queries for oracle - see http://www.nabble.com/Statements.java%2C- > etc.-Performance-tf2662372.html#a7425760 >> >> 2. This is actually my big pain point at the moment. When activemq >> does >> eventually start up with the 2 million messages, I can't use hermes >> to see >> the messages (it seems to take hours and still no result), and i >> can't see >> any processing happening on the xa consumer side to the database. >> When used >> normally say 10's or 100's of messages, so a steady stream of >> messages, >> everything processs to the database just fine. I tried tinkering >> with the >> prefetch limit but that didn't seem to help either. Something that >> could >> say "right, let the consumer take x amount of messages off the >> queue and >> process them with XA" would be awesome here. > Hopefully the paging mechanism will help here too. > >> >> 3. We have about 6 queues on 6 machines with 24 applications acting as >> producers into them (so 4 producers going to each queue). We have >> a SAN set >> up so that each queue can write to its own storage area on the san >> (i.e. 6 >> activemq data directories on the SAN). If i wanted to set up a >> store and >> forward kind of setup to 2 'central queues' where the consumers of >> the queue >> lie, do i just configure a network connector on the 6 queues to >> point to the >> central queue locations and get those central queues to read from >> all 6 SAN >> data directories? Or do i actually need 6 consumer queues? In >> either case >> is it ok for them to point to the SAN to read the message stores, >> considering the producers are writing and the consumers are removing? >> >> Any help would be massively appreciated! >> > There isn't a problem with either 2 central queues or 6 consumer > queues - but I'd like to understand a little more about your setup - > do you really need store and forward, or could you just have a > centralized hub around your SAN ? > >> cheers, >> >> ::mark >> -- >> View this message in context: http://www.nabble.com/starting-a- >> consumer-on-a-queue-with-lots-of-messages-tf2710121.html#a7555664 >> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >> > > cheers, > > rob > > -- View this message in context: http://www.nabble.com/starting-a-consumer-on-a-queue-with-lots-of-messages-tf2710121.html#a7560162 Sent from the ActiveMQ - User mailing list archive at Nabble.com.
