Looking at the logs again, we noticed this warning on the broker as well: Store limit is 30720 mb (current store usage is 0 mb). The data directory: D:\Apps\apache-activemq-5.18.3\data only has 25672 mb of usable space. - resetting to maximum available disk space: 25672 mb
Could this be related? If there was not enough available disk space would It cause this exception to occur? -----Original Message----- From: Jack Fry (They/Them) <j...@scottlogic.com.INVALID> Sent: Monday, February 12, 2024 2:35 PM To: users@activemq.apache.org Subject: Message delivery failure and Primary Key violation on load balanced ActiveMQ Broker [You don't often get email from j...@scottlogic.com.invalid. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] (Re-sent this as I wanted to change the subject) Hi, Last week our ActiveMQ message broker lost a message from the queue. We have a load balanced system with two separate brokers sharing a data store. At some point during the transfer of the lease, the original lease holder started shutting down, which started a cascade of failures as the Transport Connection failed to deliver the message. 16:54:07.869 Starting Job Scheduler Store 16:54:07.869 Persistence Adapter successfully started 16:54:08.456 Apache ActiveMQ 5.18.3 (gbldnsrv4pw4564, ID:GBLDNSRV4PW4564-50268-1705135858444-0:3) is starting 16:54:10.807 gbldnsrv4pw4563, no longer able to keep the exclusive lock so giving up being a master 16:54:10.807 Apache ActiveMQ 5.18.3 (gbldnsrv4pw4563, ID:GBLDNSRV4PW4563-63611-1706742264783-0:2) is shutting down 16:54:10.807 Transport Connection to: tcp://10.18.136.56:51504 failed: Broker BrokerService[gbldnsrv4pw4563] is being stopped 16:54:10.807 socketQueue interrupted - stopping 16:54:10.807 Could not accept connection during shutdown : null (null) 16:54:10.823 Transport Connection to: tcp://10.18.136.38:57430 failed: Broker BrokerService[gbldnsrv4pw4563] is being stopped 16:54:10.823 Failed delivery for (MessageId: ID-GBLDNSRV4PW4563-1706742293708-1-145409 on ExchangeId: ID-GBLDNSRV4PW4563-1706742293708-1-145309). On delivery attempt: 0 caught: org.springframework.jms.UncategorizedJmsException: Uncategorized exception occurred during JMS processing; nested exception is javax.jms.JMSException: Peer (vm://localhost#35661) disposed. However, when the second broker came to process the failed messages, there was an Primary Key exception from the sqldb that the message was already stored in the database. 16:54:37.357 Error while closing connection: Violation of PRIMARY KEY constraint 'PK__ACTIVEMQ__3214EC27C81AADFA'. Cannot insert duplicate key in object 'dbo.ACTIVEMQ_MSGS'. The duplicate key value is (510684907). 16:54:37.374 Ignoring SQLException, java.io.IOException: Violation of PRIMARY KEY constraint 'PK__ACTIVEMQ__3214EC27C81AADFA'. Cannot insert duplicate key in object 'dbo.ACTIVEMQ_MSGS'. The duplicate key value is (510684907). 16:54:37.421 Ignoring SQLException, java.io.IOException: Violation of PRIMARY KEY constraint 'PK__ACTIVEMQ__3214EC27C81AADFA'. Cannot insert duplicate key in object 'dbo.ACTIVEMQ_MSGS'. The duplicate key value is (510684909). 16:54:37.421 Commit failed: Violation of PRIMARY KEY constraint 'PK__ACTIVEMQ__3214EC27C81AADFA'. Cannot insert duplicate key in object 'dbo.ACTIVEMQ_MSGS'. The duplicate key value is (510684909). 16:54:37.483 Store COMMIT FAILED: 16:54:40.824 Failed delivery for (MessageId: ID-GBLDNSRV4PW4563-1706742293708-1-145409 on ExchangeId: ID-GBLDNSRV4PW4563-1706742293708-1-145309). On delivery attempt: 1 caught: java.lang.IllegalStateException: SendProcessor has not been started: sendTo(activemq://queue: 16:54:40.824 Failed delivery for (MessageId: ID-GBLDNSRV4PW4563-1706742293708-1-145410 on ExchangeId: ID-GBLDNSRV4PW4563-1706742293708-1-145312). On delivery attempt: 1 caught: java.lang.IllegalStateException: SendProcessor has not been started: sendTo(activemq://queue:) Would anyone here know what happened here? Is this a bug? Many thanks, Jack
