Hi Rob, Thanks so much a prompt response and create JIRA to track this.
We don't set a TTL on our messages though -- i.e., we use 0 (unlimited). We do send it in non-persistent mode; not sure if that matters or could trigger the same bug? For the fix, would it be possible to fix it in 0.32 as well as in 6.0.x ? CCing, Helen from our team. Thanks, Ramayan On Thu, Oct 6, 2016 at 1:21 PM, Rob Godfrey <[email protected]> wrote: > I've added a patch file which should apply to 0.32 on the following > JIRA: https://issues.apache.org/jira/browse/QPID-7451 > > Hope this helps, > Rob > > On 6 October 2016 at 22:04, Rob Godfrey <[email protected]> wrote: > > Hi Ramayan, > > > > this is exception indicates that a message has been deleted from the > > store, but the in-memory queue still references it. Unfortunately the > > exception doesn't really tell us anything about how the broker will > > have got to this state. > > > > Having looked at the code I have an idea about what may be happening - > > do your messages have a TTL set? I *think* that the AMQP 0-10 message > > path may be vulnerable to a race condition if a message expires at > > precisely the same time that the message is picked up to be sent to a > > consumer (essentially the message has to be available to consume and > > then three lines of code later it must have been deleted). The 0-9-1 > > codepath is not vulnerable to this as it caches the size of the > > message (as you can see in the stack trace, the consumer is checking > > the size of the message to make sure that the consumer has enough > > credit to receive it). > > > > We're unlikely to do a patch release for 0.32, but we will likely be > > putting out a new 6.0.x release soon, and soon after a 6.1 release. > > Would you be able to upgrade to one of these, or would you prefer me > > to send you a patch file that you could apply to the 0.32 source to > > test? > > > > -- Rob > > > > On 6 October 2016 at 21:27, Ramayan Tiwari <[email protected]> > wrote: > >> Hi, > >> > >> > >> We are ran into this StoreException in our production environment > multiple > >> times on different brokers, which caused broker shutdown. We are running > >> 0.32 Java broker with 0.16 client. I see that this was reported and > fixed > >> here: > >> https://issues.apache.org/jira/browse/QPID-4012 > >> > >> This is still happening, I don't have enough context to reproduce this > >> locally. Any help is appreciated! > >> > >> Thanks > >> Ramayan > >> > >> *Exception* > >> > >> Uncaught exception, shutting down. > >> org.apache.qpid.server.store.StoreException: Metadata not found for > message > >> with id 1762118451 > >> at > >> org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore. > getMessageMetaData(AbstractBDBMessageStore.java:343) > >> at > >> org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore$ > StoredBDBMessage.getMetaData(AbstractBDBMessageStore.java:1224) > >> at > >> org.apache.qpid.server.protocol.v0_10.MessageTransferMessage. > getMetaData(MessageTransferMessage.java:41) > >> at > >> org.apache.qpid.server.protocol.v0_10.MessageTransferMessage.getSize( > MessageTransferMessage.java:56) > >> at > >> org.apache.qpid.server.protocol.v0_10.ConsumerTarget_ > 0_10.allocateCredit(ConsumerTarget_0_10.java:494) > >> at > >> org.apache.qpid.server.queue.QueueConsumerImpl.wouldSuspend( > QueueConsumerImpl.java:278) > >> at > >> org.apache.qpid.server.queue.AbstractQueue.attemptDelivery( > AbstractQueue.java:2059) > >> at > >> org.apache.qpid.server.queue.AbstractQueue.flushConsumer( > AbstractQueue.java:1981) > >> at > >> org.apache.qpid.server.queue.AbstractQueue.flushConsumer( > AbstractQueue.java:1957) > >> at > >> org.apache.qpid.server.queue.QueueConsumerImpl.flush( > QueueConsumerImpl.java:318) > >> at > >> org.apache.qpid.server.protocol.v0_10.ConsumerTarget_ > 0_10.flush(ConsumerTarget_0_10.java:605) > >> at > >> org.apache.qpid.server.protocol.v0_10.ServerSessionDelegate. > messageFlush(ServerSessionDelegate.java:521) > >> at > >> org.apache.qpid.server.protocol.v0_10.ServerSessionDelegate. > messageFlush(ServerSessionDelegate.java:82) > >> at org.apache.qpid.transport.MessageFlush.dispatch( > MessageFlush.java:87) > >> at > >> org.apache.qpid.transport.SessionDelegate.command( > SessionDelegate.java:55) > >> at > >> org.apache.qpid.server.protocol.v0_10.ServerSessionDelegate.command( > ServerSessionDelegate.java:99) > >> at > >> org.apache.qpid.server.protocol.v0_10.ServerSessionDelegate.command( > ServerSessionDelegate.java:82) > >> at org.apache.qpid.transport.Method.delegate(Method.java:159) > >> at org.apache.qpid.transport.Session.received(Session.java:596) > >> at org.apache.qpid.transport.Connection.dispatch(Connection.java:452) > >> at > >> org.apache.qpid.transport.ConnectionDelegate.handle( > ConnectionDelegate.java:64) > >> at > >> org.apache.qpid.transport.ConnectionDelegate.handle( > ConnectionDelegate.java:40) > >> at > >> org.apache.qpid.transport.MethodDelegate.messageFlush( > MethodDelegate.java:143) > >> at org.apache.qpid.transport.MessageFlush.dispatch( > MessageFlush.java:87) > >> at > >> org.apache.qpid.transport.ConnectionDelegate.command( > ConnectionDelegate.java:54) > >> at > >> org.apache.qpid.transport.ConnectionDelegate.command( > ConnectionDelegate.java:40) > >> at org.apache.qpid.transport.Method.delegate(Method.java:159) > >> at org.apache.qpid.transport.Connection.received(Connection.java:405) > >> at > >> org.apache.qpid.server.protocol.v0_10.ServerConnection.access$001( > ServerConnection.java:64) > >> at > >> org.apache.qpid.server.protocol.v0_10.ServerConnection$1.run( > ServerConnection.java:316) > >> at > >> org.apache.qpid.server.protocol.v0_10.ServerConnection$1.run( > ServerConnection.java:312) > >> at java.security.AccessController.doPrivileged(Native Method) > >> at javax.security.auth.Subject.doAs(Subject.java:360) > >> at > >> org.apache.qpid.server.protocol.v0_10.ServerConnection.received( > ServerConnection.java:311) > >> at > >> org.apache.qpid.server.protocol.v0_10.ServerConnection.received( > ServerConnection.java:64) > >> at org.apache.qpid.transport.network.Assembler.emit(Assembler.java:97) > >> at org.apache.qpid.transport.network.Assembler.assemble( > Assembler.java:198) > >> at org.apache.qpid.transport.network.Assembler.frame( > Assembler.java:131) > >> at org.apache.qpid.transport.network.Frame.delegate(Frame.java:128) > >> at org.apache.qpid.transport.network.Assembler.received( > Assembler.java:102) > >> at org.apache.qpid.transport.network.Assembler.received( > Assembler.java:44) > >> at > >> org.apache.qpid.transport.network.InputHandler.next( > InputHandler.java:199) > >> at > >> org.apache.qpid.transport.network.InputHandler.received( > InputHandler.java:114) > >> at > >> org.apache.qpid.server.protocol.v0_10.ProtocolEngine_ > 0_10.received(ProtocolEngine_0_10.java:179) > >> at > >> org.apache.qpid.server.protocol.v0_10.ProtocolEngine_ > 0_10.received(ProtocolEngine_0_10.java:43) > >> at > >> org.apache.qpid.server.protocol.MultiVersionProtocolEngine.received( > MultiVersionProtocolEngine.java:153) > >> at > >> org.apache.qpid.server.protocol.MultiVersionProtocolEngine.received( > MultiVersionProtocolEngine.java:51) > >> at org.apache.qpid.transport.network.io.IoReceiver.run( > IoReceiver.java:161) > >> at java.lang.Thread.run(Thread.java:745) > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
