> If you implement this in the broker, 1) you don't protect against a > duplicate message sent after the consumer has processed the first one, and
Sort of.. If you only want ONE inside the broker, you’ve accomplished that goal. There’s a race where message m0 could be sent to a consumer, and then another producer could create m0’ (m0 prime, isomorphic copy of m0 which would normally be a duplicate) and then immediately enqueued. I think this is reasonable. It would just mean that a message is processed twice in a short duration. After the consumer has processed that message, even if it resends it, then it will get de-duplicated. Of course there might be a possibility here of having messages that are resent to brokers that are continually being consumed and re-send back to the broker. So maybe if that’s a problem for some apps, keeping a consistent set of messages to catch duplicates would be more valuable. > 2) you tax the broker's memory, rather than putting the burden on the > consumer (and it would be a lighter burden for the consumer because the > consumer only has to worry about messages destined for it). > It depends what the semantics are. We need to make sure that a message is only processed once globally during an interval. So in this situation the consumer would need a set of all the IDs of all the messages which wouldn’t really scale. I guess I could split up the consumer space and have consumers use groups but that means now my consumers have to be smart which isn’t fun either. Another idea may be to have a high water marker and only messages above the marker can be processed. So if you want to re-load all the messages again, you increment the marker (or set it to something like the current time) and a consumer or broker sees a message with a marker lower than the correct marker, it’s immediately discarded on the assumption that a new message with the appropriate marker has been sent. Kevin -- Founder/CEO Spinn3r.com Location: *San Francisco, CA* blog: http://burtonator.wordpress.com … or check out my Google+ profile <https://plus.google.com/102718274791889610666/posts> <http://spinn3r.com>
