Thank you. Did you get a chance to test the behavior if you allowed messages to back up and then expire on the broker? That would be valuable information for the JIRA issue you submitted, so if you have the ability to do that test (since you already have a system that's known to do this), that would be helpful for the investigation..
Tim On Mon, Sep 24, 2018, 5:24 AM Simon Lundström <si...@su.se> wrote: > Thank you Tim! > > https://issues.apache.org/jira/browse/AMQ-7061 > > BR, > - Simon > > On Fri, 2018-09-21 at 14:55:14 +0200, Tim Bain wrote: > >Please submit this as a bug in JIRA, and provide the link here. > > > >My first guess would be that the message is dispatched to the consumer > >before it expires, but expires before the consumer pulls it from the > >dispatch queue (so this is consumer-side expiration rather than > broker-side > >expiration), and that the broker code isn't detecting that this particular > >Throwable counts as expiration. But I could be wrong about those guesses, > >and whoever investigates will have to see if it's that or something else. > > > >One thing you can do to help the investigation is to allow a backlog of > >messages to build up on the queue (e.g. by pointing the consumer at a > >different queue for a little while) and then see if all the built-up > >messages are properly discarded once the consumer reconnects. That would > >help test whether broker-side expiration is working. > > > >For your setup, I suspect that fixing your 60ms problem will reduce the > >frequency at which you see this to nearly zero, so I expect that this > won't > >affect you in any meaningful way, but it's still a bug that should be > >fixed. > > > >Thanks, > >Tim > > > >On Fri, Sep 21, 2018, 3:03 AM Simon Lundström <si...@su.se> wrote: > > > >> Hey all! > >> > >> We're running 5.15.3 and we're using > >> <https://github.com/ARGOeu/nagios-plugins-activemq/> for monitoring > >> (SSL'ed) OpenWire. Healthchecks fail and to not get our aliveness-check > >> queue filled with old messages we added TTL via > >> < > >> > https://github.com/ARGOeu/nagios-plugins-activemq/blob/3d6d43c/lib/OpenWireProbe/src/org/activemq/probes/OpenWireProbe.java#L124-L126 > >> >. > >> We got one DLQ'd OpenWire check message and just discarded it but when > >> we got the second one I started to investigate. > >> > >> The message which got put into the DLQ has the propery: > >> dlqDeliveryFailureCause java.lang.Throwable: Message Expired. > >> Expiration:1537468027378 > >> > >> Our configuration related to the queue aliveness-test in activemq.xml > >> is: > >> > >> [...] > >> <authorizationEntry > >> queue="aliveness-test.>" > >> read="monitoring" > >> write="monitoring" > >> admin="admins" > >> /> > >> [...] > >> <queue physicalName="aliveness-test"/> > >> [...] > >> <policyEntry queue="aliveness-test"> > >> <!-- > >> Tell the dead letter strategy not to process expired > messages > >> so that they will just be discarded instead of being sent > to > >> the DLQ > >> --> > >> <deadLetterStrategy> > >> <sharedDeadLetterStrategy processExpired="false" /> > >> </deadLetterStrategy> > >> </policyEntry> > >> [...] > >> > >> I just noticed a bug that our TTL is set to 60ms (instead of seconds) > >> but that's just how fast ActiveMQ is ; ) So I don't think that's > >> related. > >> > >> So what we expect is for those messages to just get deleted but instead > >> we they get DLQ'd into ActiveMQ.DLQ (since no DLQ is configured). > >> > >> Thanks and have a great weekend! > >> > >> BR, > >> - Simon > >> > >> ____________________________________ > >> > >> Simon Lundström > >> Section for Infrastructure > >> > >> IT Services > >> Stockholm University > >> SE-106 91 Stockholm, Sweden > >> > >> www.su.se/english/staff-info/it > >> https://www.su.se/english/about-this-website/privacy-policy > >> > > > >On Sep 21, 2018 3:03 AM, "Simon Lundström" <si...@su.se> wrote: > > > >Hey all! > > > >We're running 5.15.3 and we're using > ><https://github.com/ARGOeu/nagios-plugins-activemq/> for monitoring > >(SSL'ed) OpenWire. Healthchecks fail and to not get our aliveness-check > >queue filled with old messages we added TTL via > >< > > > https://github.com/ARGOeu/nagios-plugins-activemq/blob/3d6d43c/lib/OpenWireProbe/src/org/activemq/probes/OpenWireProbe.java#L124-L126 > >>. > >We got one DLQ'd OpenWire check message and just discarded it but when > >we got the second one I started to investigate. > > > >The message which got put into the DLQ has the propery: > >dlqDeliveryFailureCause java.lang.Throwable: Message Expired. > >Expiration:1537468027378 > > > >Our configuration related to the queue aliveness-test in activemq.xml > >is: > > > >[...] > > <authorizationEntry > > queue="aliveness-test.>" > > read="monitoring" > > write="monitoring" > > admin="admins" > > /> > >[...] > > <queue physicalName="aliveness-test"/> > >[...] > > <policyEntry queue="aliveness-test"> > > <!-- > > Tell the dead letter strategy not to process expired > messages > > so that they will just be discarded instead of being sent to > > the DLQ > > --> > > <deadLetterStrategy> > > <sharedDeadLetterStrategy processExpired="false" /> > > </deadLetterStrategy> > > </policyEntry> > >[...] > > > >I just noticed a bug that our TTL is set to 60ms (instead of seconds) > >but that's just how fast ActiveMQ is ; ) So I don't think that's > >related. > > > >So what we expect is for those messages to just get deleted but instead > >we they get DLQ'd into ActiveMQ.DLQ (since no DLQ is configured). > > > >Thanks and have a great weekend! > > > >BR, > >- Simon > > > >____________________________________ > > > >Simon Lundström > >Section for Infrastructure > > > >IT Services > >Stockholm University > >SE-106 91 Stockholm, Sweden > > > >www.su.se/english/staff-info/it > >https://www.su.se/english/about-this-website/privacy-policy >
