No problem. Hopefully whoever picks up the JIRA issue will be able to reproduce the problem.
Tim On Mon, Sep 24, 2018, 11:14 PM Simon Lundström <si...@su.se> wrote: > Hey Tim! > > Sorry, but I can't actually reproduce it. I tried sending persistent > messages with 60ms TTL from the ActiveMQ console and then starting the > producer/consumer monitoring check client. All messages are eaten and > done are DLQ'd = ( > > You're welcome for nothing ; ) > > BR, > - Simon > > On Mon, 2018-09-24 at 14:41:14 +0200, Tim Bain wrote: > >Thank you. > > > >Did you get a chance to test the behavior if you allowed messages to back > >up and then expire on the broker? That would be valuable information for > >the JIRA issue you submitted, so if you have the ability to do that test > >(since you already have a system that's known to do this), that would be > >helpful for the investigation.. > > > >Tim > > > >On Mon, Sep 24, 2018, 5:24 AM Simon Lundström <si...@su.se> wrote: > > > >> Thank you Tim! > >> > >> https://issues.apache.org/jira/browse/AMQ-7061 > >> > >> BR, > >> - Simon > >> > >> On Fri, 2018-09-21 at 14:55:14 +0200, Tim Bain wrote: > >> >Please submit this as a bug in JIRA, and provide the link here. > >> > > >> >My first guess would be that the message is dispatched to the consumer > >> >before it expires, but expires before the consumer pulls it from the > >> >dispatch queue (so this is consumer-side expiration rather than > >> broker-side > >> >expiration), and that the broker code isn't detecting that this > particular > >> >Throwable counts as expiration. But I could be wrong about those > guesses, > >> >and whoever investigates will have to see if it's that or something > else. > >> > > >> >One thing you can do to help the investigation is to allow a backlog of > >> >messages to build up on the queue (e.g. by pointing the consumer at a > >> >different queue for a little while) and then see if all the built-up > >> >messages are properly discarded once the consumer reconnects. That > would > >> >help test whether broker-side expiration is working. > >> > > >> >For your setup, I suspect that fixing your 60ms problem will reduce the > >> >frequency at which you see this to nearly zero, so I expect that this > >> won't > >> >affect you in any meaningful way, but it's still a bug that should be > >> >fixed. > >> > > >> >Thanks, > >> >Tim > >> > > >> >On Fri, Sep 21, 2018, 3:03 AM Simon Lundström <si...@su.se> wrote: > >> > > >> >> Hey all! > >> >> > >> >> We're running 5.15.3 and we're using > >> >> <https://github.com/ARGOeu/nagios-plugins-activemq/> for monitoring > >> >> (SSL'ed) OpenWire. Healthchecks fail and to not get our > aliveness-check > >> >> queue filled with old messages we added TTL via > >> >> < > >> >> > >> > https://github.com/ARGOeu/nagios-plugins-activemq/blob/3d6d43c/lib/OpenWireProbe/src/org/activemq/probes/OpenWireProbe.java#L124-L126 > >> >> >. > >> >> We got one DLQ'd OpenWire check message and just discarded it but > when > >> >> we got the second one I started to investigate. > >> >> > >> >> The message which got put into the DLQ has the propery: > >> >> dlqDeliveryFailureCause java.lang.Throwable: Message Expired. > >> >> Expiration:1537468027378 > >> >> > >> >> Our configuration related to the queue aliveness-test in activemq.xml > >> >> is: > >> >> > >> >> [...] > >> >> <authorizationEntry > >> >> queue="aliveness-test.>" > >> >> read="monitoring" > >> >> write="monitoring" > >> >> admin="admins" > >> >> /> > >> >> [...] > >> >> <queue physicalName="aliveness-test"/> > >> >> [...] > >> >> <policyEntry queue="aliveness-test"> > >> >> <!-- > >> >> Tell the dead letter strategy not to process expired > >> messages > >> >> so that they will just be discarded instead of being > sent > >> to > >> >> the DLQ > >> >> --> > >> >> <deadLetterStrategy> > >> >> <sharedDeadLetterStrategy processExpired="false" /> > >> >> </deadLetterStrategy> > >> >> </policyEntry> > >> >> [...] > >> >> > >> >> I just noticed a bug that our TTL is set to 60ms (instead of seconds) > >> >> but that's just how fast ActiveMQ is ; ) So I don't think that's > >> >> related. > >> >> > >> >> So what we expect is for those messages to just get deleted but > instead > >> >> we they get DLQ'd into ActiveMQ.DLQ (since no DLQ is configured). > >> >> > >> >> Thanks and have a great weekend! > >> >> > >> >> BR, > >> >> - Simon > >> >> > >> >> ____________________________________ > >> >> > >> >> Simon Lundström > >> >> Section for Infrastructure > >> >> > >> >> IT Services > >> >> Stockholm University > >> >> SE-106 91 Stockholm, Sweden > >> >> > >> >> www.su.se/english/staff-info/it > >> >> https://www.su.se/english/about-this-website/privacy-policy > >> >> > >> > > >> >On Sep 21, 2018 3:03 AM, "Simon Lundström" <si...@su.se> wrote: > >> > > >> >Hey all! > >> > > >> >We're running 5.15.3 and we're using > >> ><https://github.com/ARGOeu/nagios-plugins-activemq/> for monitoring > >> >(SSL'ed) OpenWire. Healthchecks fail and to not get our aliveness-check > >> >queue filled with old messages we added TTL via > >> >< > >> > > >> > https://github.com/ARGOeu/nagios-plugins-activemq/blob/3d6d43c/lib/OpenWireProbe/src/org/activemq/probes/OpenWireProbe.java#L124-L126 > >> >>. > >> >We got one DLQ'd OpenWire check message and just discarded it but when > >> >we got the second one I started to investigate. > >> > > >> >The message which got put into the DLQ has the propery: > >> >dlqDeliveryFailureCause java.lang.Throwable: Message Expired. > >> >Expiration:1537468027378 > >> > > >> >Our configuration related to the queue aliveness-test in activemq.xml > >> >is: > >> > > >> >[...] > >> > <authorizationEntry > >> > queue="aliveness-test.>" > >> > read="monitoring" > >> > write="monitoring" > >> > admin="admins" > >> > /> > >> >[...] > >> > <queue physicalName="aliveness-test"/> > >> >[...] > >> > <policyEntry queue="aliveness-test"> > >> > <!-- > >> > Tell the dead letter strategy not to process expired > >> messages > >> > so that they will just be discarded instead of being > sent to > >> > the DLQ > >> > --> > >> > <deadLetterStrategy> > >> > <sharedDeadLetterStrategy processExpired="false" /> > >> > </deadLetterStrategy> > >> > </policyEntry> > >> >[...] > >> > > >> >I just noticed a bug that our TTL is set to 60ms (instead of seconds) > >> >but that's just how fast ActiveMQ is ; ) So I don't think that's > >> >related. > >> > > >> >So what we expect is for those messages to just get deleted but instead > >> >we they get DLQ'd into ActiveMQ.DLQ (since no DLQ is configured). > >> > > >> >Thanks and have a great weekend! > >> > > >> >BR, > >> >- Simon > >> > > >> >____________________________________ > >> > > >> >Simon Lundström > >> >Section for Infrastructure > >> > > >> >IT Services > >> >Stockholm University > >> >SE-106 91 Stockholm, Sweden > >> > > >> >www.su.se/english/staff-info/it > >> >https://www.su.se/english/about-this-website/privacy-policy > >> >
