(Re-sent this as I wanted to change the subject) Hi,
Last week our ActiveMQ message broker lost a message from the queue. We have a load balanced system with two separate brokers sharing a data store. At some point during the transfer of the lease, the original lease holder started shutting down, which started a cascade of failures as the Transport Connection failed to deliver the message. 16:54:07.869 Starting Job Scheduler Store 16:54:07.869 Persistence Adapter successfully started 16:54:08.456 Apache ActiveMQ 5.18.3 (gbldnsrv4pw4564, ID:GBLDNSRV4PW4564-50268-1705135858444-0:3) is starting 16:54:10.807 gbldnsrv4pw4563, no longer able to keep the exclusive lock so giving up being a master 16:54:10.807 Apache ActiveMQ 5.18.3 (gbldnsrv4pw4563, ID:GBLDNSRV4PW4563-63611-1706742264783-0:2) is shutting down 16:54:10.807 Transport Connection to: tcp://10.18.136.56:51504 failed: Broker BrokerService[gbldnsrv4pw4563] is being stopped 16:54:10.807 socketQueue interrupted - stopping 16:54:10.807 Could not accept connection during shutdown : null (null) 16:54:10.823 Transport Connection to: tcp://10.18.136.38:57430 failed: Broker BrokerService[gbldnsrv4pw4563] is being stopped 16:54:10.823 Failed delivery for (MessageId: ID-GBLDNSRV4PW4563-1706742293708-1-145409 on ExchangeId: ID-GBLDNSRV4PW4563-1706742293708-1-145309). On delivery attempt: 0 caught: org.springframework.jms.UncategorizedJmsException: Uncategorized exception occurred during JMS processing; nested exception is javax.jms.JMSException: Peer (vm://localhost#35661) disposed. However, when the second broker came to process the failed messages, there was an Primary Key exception from the sqldb that the message was already stored in the database. 16:54:37.357 Error while closing connection: Violation of PRIMARY KEY constraint 'PK__ACTIVEMQ__3214EC27C81AADFA'. Cannot insert duplicate key in object 'dbo.ACTIVEMQ_MSGS'. The duplicate key value is (510684907). 16:54:37.374 Ignoring SQLException, java.io.IOException: Violation of PRIMARY KEY constraint 'PK__ACTIVEMQ__3214EC27C81AADFA'. Cannot insert duplicate key in object 'dbo.ACTIVEMQ_MSGS'. The duplicate key value is (510684907). 16:54:37.421 Ignoring SQLException, java.io.IOException: Violation of PRIMARY KEY constraint 'PK__ACTIVEMQ__3214EC27C81AADFA'. Cannot insert duplicate key in object 'dbo.ACTIVEMQ_MSGS'. The duplicate key value is (510684909). 16:54:37.421 Commit failed: Violation of PRIMARY KEY constraint 'PK__ACTIVEMQ__3214EC27C81AADFA'. Cannot insert duplicate key in object 'dbo.ACTIVEMQ_MSGS'. The duplicate key value is (510684909). 16:54:37.483 Store COMMIT FAILED: 16:54:40.824 Failed delivery for (MessageId: ID-GBLDNSRV4PW4563-1706742293708-1-145409 on ExchangeId: ID-GBLDNSRV4PW4563-1706742293708-1-145309). On delivery attempt: 1 caught: java.lang.IllegalStateException: SendProcessor has not been started: sendTo(activemq://queue: 16:54:40.824 Failed delivery for (MessageId: ID-GBLDNSRV4PW4563-1706742293708-1-145410 on ExchangeId: ID-GBLDNSRV4PW4563-1706742293708-1-145312). On delivery attempt: 1 caught: java.lang.IllegalStateException: SendProcessor has not been started: sendTo(activemq://queue:) Would anyone here know what happened here? Is this a bug? Many thanks, Jack
