Re: partially durable queue?
Sorry this took a while (I've not had much free time to look through the source code recently) and you may well have already found what you need in the meantime, but in case you haven't yet, I think you'd want to be looking in the activemq-kahadb-store project and particularly org.apache.activemq.store.kahadb.plist.PListImpl or PListStoreImpl. Based on what you've said, I think you'll be looking to have the PListImpl store messages in memory without writing to the PageFile until some threshold is reached, and then begin writing messages into the PageFile at that time. Since I've not done very much poking in the KahaDB code, it's possible that I'm off the mark here on where best to implement this, but it's where I'd start looking if I were you. Tim On Mon, Jan 29, 2018 at 3:53 AM, Noel Grandinwrote: > Thanks for all the answers. It sounds like I can't quite achieve what I > want out of the box. (Which is not a surprise) > > I'll have a look at the internal code and see how hard it will be to build > something custom that is basically a combination of a small non-persistent > in-memory queue, and a larger on-disk persistent queue. > > Any ideas where to start looking in the codebase? >
Re: partially durable queue?
It'll only throw away unconsumed messages. So a single event like a whole-house power failure won't lose many messages, because the consumer will probably be up to date and there won't be any messages on the broker. You're only at risk of data loss if there are multiple failures separated by a significant amount of time (the consumer dies, and then substantially later the broker dies due to something unrelated). It might be that the odds of that are low enough and the consequences reasonable enough that you can accept that risk, or maybe not. Also, keep in mind that if you're keeping data in memory until you fill memory, your messages are non-persistent and will be lost when the message broker cycles. That's true for any messaging product configured the way you've said you want it, so it's not unique to ActiveMQ 5.x. If losing most or all of your unconsumed messages is a problem, you need to be storing them on disk even if it wears out your cheap SSDs, or you need to stand up a central broker on something with an attached spinning disk. Tim On Jan 26, 2018 10:51 AM, "Noel Grandin"wrote: On Fri, 26 Jan 2018 at 19:22, Tim Bain wrote: > > 5.x can also do in-memory-till-full-then-spool-to-disk when using > non-persistent messages (which go to the memory store and then to the temp > store if the memory store is full). > That’s useful info thanks. But non persistent means that the broker will throw away the messages on a reboot? Which wouldn’t be good for me. I can lose a small window of data, but if the wan disconnects for a while and then the machine reboots I need to keep most of the data. >
Re: partially durable queue?
It will work with artemis. It will start to steam to disk when beyond configured limit. On Fri, Jan 26, 2018 at 2:12 PM Noel Grandinwrote: > On 26 January 2018 at 20:58, Clebert Suconic > wrote: > > > With artemis you can simply disable syncs on producer and you would get > > similar performance numbers. Usually people turn of persistence to gain > > > > Ah, it's not perf I'm after, think of this as a kind of IOT/sensor device, > I want to preserve my rather cheap SSD device from wearing out. > > I'm producing a number of measurements on 24/7 basis (10-50 messages / sec) > > In the common case I should be able to stream this out without ever hitting > disk, but for backup I need to spool messages if the consumer becomes > temporarily unreachable. > > This is an abnormal use-case, so I'm not surprised it's not readily > available. > > But I figured > (a) I might get lucky > (b) if not, it might be worth adding the feature to ActiveMQ myself to > avoid re-inventing all the other parts > -- Clebert Suconic
Re: partially durable queue?
On 26 January 2018 at 20:58, Clebert Suconicwrote: > With artemis you can simply disable syncs on producer and you would get > similar performance numbers. Usually people turn of persistence to gain > Ah, it's not perf I'm after, think of this as a kind of IOT/sensor device, I want to preserve my rather cheap SSD device from wearing out. I'm producing a number of measurements on 24/7 basis (10-50 messages / sec) In the common case I should be able to stream this out without ever hitting disk, but for backup I need to spool messages if the consumer becomes temporarily unreachable. This is an abnormal use-case, so I'm not surprised it's not readily available. But I figured (a) I might get lucky (b) if not, it might be worth adding the feature to ActiveMQ myself to avoid re-inventing all the other parts
Re: partially durable queue?
With artemis you can simply disable syncs on producer and you would get similar performance numbers. Usually people turn of persistence to gain some performance. With artemis you can just always persist and increase perf by batching multiple writes. SyncOnDurable=false on the connection factory. On Fri, Jan 26, 2018 at 12:51 PM Noel Grandinwrote: > On Fri, 26 Jan 2018 at 19:22, Tim Bain wrote: > > > > > 5.x can also do in-memory-till-full-then-spool-to-disk when using > > non-persistent messages (which go to the memory store and then to the > temp > > store if the memory store is full). > > > > That’s useful info thanks. But non persistent means that the broker will > throw away the messages on a reboot? Which wouldn’t be good for me. > I can lose a small window of data, but if the wan disconnects for a while > and then the machine reboots I need to keep most of the data. > > > > > > > -- Clebert Suconic
Re: partially durable queue?
On Fri, 26 Jan 2018 at 19:22, Tim Bainwrote: > > 5.x can also do in-memory-till-full-then-spool-to-disk when using > non-persistent messages (which go to the memory store and then to the temp > store if the memory store is full). > That’s useful info thanks. But non persistent means that the broker will throw away the messages on a reboot? Which wouldn’t be good for me. I can lose a small window of data, but if the wan disconnects for a while and then the machine reboots I need to keep most of the data. >
Re: partially durable queue?
This mailing list is for the two ActiveMQ forks: 5.x and Artemis (which is expected to become 6.x). RabbitMQ and all other message brokers are unrelated products. 5.x can also do in-memory-till-full-then-spool-to-disk when using non-persistent messages (which go to the memory store and then to the temp store if the memory store is full). Pro tip: spending two minutes Googling products you've never heard of is a much better use of everyone's time than just replying to say you don't know what they are. Tim On Jan 26, 2018 9:49 AM, "Noel Grandin"wrote: > On Fri, 26 Jan 2018 at 17:59, Justin Bertram wrote: > > > > > If you're asking about ActiveMQ, can you clarify if you're asking about > the > > 5.x or Artemis broker? > > > > > I’ve only just started looking at tech options, so I don’t even know what > those are. I assume different forks of active mq? >
Re: partially durable queue?
On Fri, 26 Jan 2018 at 17:59, Justin Bertramwrote: > > If you're asking about ActiveMQ, can you clarify if you're asking about the > 5.x or Artemis broker? > I’ve only just started looking at tech options, so I don’t even know what those are. I assume different forks of active mq?
Re: partially durable queue?
You can do that with Artemis and Paging.. I'm not sure about Rabbit :) On Fri, Jan 26, 2018 at 10:15 AM, Noel Grandinwrote: > > Hi > > I have a need for a queue that operates purely in-memory until the > consumer(s) can't keep up, and only then starts spooling to disk. > I'm ok with losing some messages in the event of a hardware or software > crash. > > Is there such a configuration in RabbitMQ? If not, could one be created with > a plugin? > > For context, the requirement is slightly weird in that I'm > (a) wanting to push some data back over a flakey WAN > (b) from an underpowered piece of hardware that doesn't like it's little SSD > drive being written too hard > :-) > > Thanks, Noel Grandin > -- Clebert Suconic
Re: partially durable queue?
Are you asking about RabbitMQ? If so, the ActiveMQ user list might not be the best place to get answers about RabbitMQ. If you're asking about ActiveMQ, can you clarify if you're asking about the 5.x or Artemis broker? Justin On Fri, Jan 26, 2018 at 9:15 AM, Noel Grandinwrote: > > Hi > > I have a need for a queue that operates purely in-memory until the > consumer(s) can't keep up, and only then starts spooling to disk. > I'm ok with losing some messages in the event of a hardware or software > crash. > > Is there such a configuration in RabbitMQ? If not, could one be created > with a plugin? > > For context, the requirement is slightly weird in that I'm > (a) wanting to push some data back over a flakey WAN > (b) from an underpowered piece of hardware that doesn't like it's little > SSD drive being written too hard > :-) > > Thanks, Noel Grandin > >