FYI.. I closed this bug as WONTFIX as I think this is the best strategy post feedback.
Thanks guys. On Mon, Feb 2, 2015 at 11:52 AM, Kevin Burton <bur...@spinn3r.com> wrote: > the problem with this strategy is that queues can come and go.. it’s a > scheduler system for a web crawler. so if a URL request comes in for an IP > we place it in that queue. And if, a month from now, we have to request > another resources from that IP, we place it in a queue for that IP. > > MOST of the time, we just have a few IPs because most web properties are > large. However, sometimes, we have a request once per hour to an IP. In > that situation we want to GC the queue 60 seconds later. > > Kevin > > On Mon, Feb 2, 2015 at 11:17 AM, Tim Bain <tb...@alumni.duke.edu> wrote: > >> It sounds like the advisory message you mentioned should let you work >> around the problem. >> >> But what it sounds like you really need is a way to know, for sure, that >> the producer is finished using a queue. If your producers write an EOF >> message when they're through, your consumer can then safely go away >> without >> having to worry about stragglers. Then your only remaining problem is how >> to know about producers that die unexpectedly without writing the EOF or >> that take a really long time to producer their next message. A heartbeat >> mechanism would solve those issues: every X amount of time, the producer >> produces a small no-op heartbeat message that tells the consumer it's >> still >> there; failure to receive one of those messages after N heartbeat >> intervals >> indicates that the producer is no longer alive. The only thing it doesn't >> protect you against is temporary network perturbations (since they would >> make it appear that the producer is gone when it's not really), but you >> might be able to live with that. >> >> I'm not sure that heartbeats are a better solution than processing the >> advisory messages as you've suggested, but I think at a minimum you should >> have an EOF that's always sent (during normal operations) when the >> producer >> closes, and to which your consumer reacts by disconnecting as well. Then >> the advisory messages can be used only for handling truly exceptional >> conditions. >> >> On Mon, Feb 2, 2015 at 11:29 AM, Kevin Burton <bur...@spinn3r.com> wrote: >> >> > > ActiveMQ provides no means to guarantee that a producer will fail to >> > > deliver >> > > a message when there are no consumers interested in the message. It >> is >> > > actually designed more for cases of consumers being away and returning >> > at a >> > > later time. >> > > >> > > >> > Perhaps. But I think the advisory queue message >> > >> > ActiveMQ.Advisory.NoConsumer.Queue >> > >> > … actually solves my problem. >> > >> > If a producer creates a queue, and no one is consuming, I’ll just >> create a >> > new consumer once I get the advisory message about the queue. >> > >> > This is actually how I do it *now* except I was listening to NEW queues… >> > >> > I’ve actually gotten a lot of use out of the advisory queue system. >> > >> > >> > > As you noted correctly, ActiveMQ automatically recreates destinations >> > every >> > > time they are needed, so even if they were GC'ed, they would just be >> > > automatically recreated when a producer posted a message. >> > > >> > >> > Yes. And in my case, I think that’s ok. >> > >> > I might actually write up a blog post about this design. It’s novel but >> AMQ >> > actually does seem to be working out well for us without the limitations >> > you describe due to advisory messages. >> > >> > Kevin >> > >> > -- >> > >> > Founder/CEO Spinn3r.com >> > Location: *San Francisco, CA* >> > blog: http://burtonator.wordpress.com >> > … or check out my Google+ profile >> > <https://plus.google.com/102718274791889610666/posts> >> > <http://spinn3r.com> >> > >> > > > > -- > > Founder/CEO Spinn3r.com > Location: *San Francisco, CA* > blog: http://burtonator.wordpress.com > … or check out my Google+ profile > <https://plus.google.com/102718274791889610666/posts> > <http://spinn3r.com> > > -- Founder/CEO Spinn3r.com Location: *San Francisco, CA* blog: http://burtonator.wordpress.com … or check out my Google+ profile <https://plus.google.com/102718274791889610666/posts> <http://spinn3r.com>
