I also mentioned that when an address with two queues was undeployed, there were also two log messages for undeploying each cluster queue. I tried a workaround with adding cluster queues to the configuration. During the configuration reload, each cluster queue has been undeployed and deployed back again. There may be a message loss between these two events (although this interval was relatively small, about 19 ms).
вт, 26 нояб. 2024 г. в 11:51, Alexander Milovidov <milovid...@gmail.com>: > Hi All, > > I also reproduced this case. > When the setting config-delete-addresses is set to FORCE on the particular > address match with wildcard (not matches with cluster address/queue names), > when undeploying this address, cluster queues are also deleted. > This does not happen if I set only config-delete-queues to FORCE. Only > config-delete-address has an effect. > > It is also strange because config-delete-addresses setting specifies which > addresses broker should delete, not queues. > > > > > пн, 25 нояб. 2024 г. в 14:51, Jean-Pascal Briquet <jpdigit...@gmail.com>: > >> Hi, >> >> I have found simple reproduction scenarios for cluster-connection failure >> and related message loss. >> All is documented in details under >> https://issues.apache.org/jira/browse/ARTEMIS-5086 >> >> In short, configuration must have FORCE on config-delete-addresses/ >> config-delete-queues. >> When the configuration is refreshed and if an address (with force >> deletion) >> is removed from configuration, then "Remote binding" queues are deleted at >> the same time as the "Local binding" queue. >> >> Regards, >> >> Jean-Pascal >> >> >> >> >> On Sat, Nov 23, 2024 at 3:01 PM Alexander Milovidov <milovid...@gmail.com >> > >> wrote: >> >> > Hi Justin, >> > >> > I also had this message in the logs. There were also two other queues >> with >> > addresses which had been removed from configuration and undeployed >> during >> > this configuration reload. >> > >> > The possible scenario to reproduce is: >> > - create a cluster with 2 active nodes; >> > - create two broker.xml files, one with only default addresses/queues, >> > other with approx. 3000 addresses with queues; >> > - the configuration file should include all types of objects (addresses, >> > anycast and multicast queues); >> > - cycle between these two configurations to deploy/undeploy addresses >> and >> > queues - copy configuration file 1 or 2 to the broker.xml; >> > - wait for the period needed to reload the configuration and repeat for >> > each file. >> > >> > I'm not sure it will be reproduced because products in the test lab >> usually >> > work as designed. >> > Also it is possible to continuously send/receive messages to some queues >> > during the entire test to make the test to be closer to the real. >> > >> > пт, 22 нояб. 2024 г. в 23:41, Justin Bertram <jbert...@apache.org>: >> > >> > > At this point I don't see how this could happen unless the relevant >> > > address-setting had FORCE for the configDeleteQueues or >> > > configDeleteAddresses. This code has been essentially the same for the >> > last >> > > 5 years or so. >> > > >> > > Do you have a way to reproduce this? >> > > >> > > Do you also see this logging? >> > > >> > > AMQ224076 Undeploying address... >> > > >> > > >> > > Justin >> > > >> > > On Thu, Nov 21, 2024 at 4:31 PM Alexander Milovidov < >> > milovid...@gmail.com> >> > > wrote: >> > > >> > > > Hi Justin, >> > > > >> > > > Thanks for your help. >> > > > I will define cluster addresses and queues in the configuration. >> > Usually >> > > we >> > > > don't clean Artemis data on production servers, and IDs of the nodes >> > > remain >> > > > the same. >> > > > >> > > > > Can you invoke the "getAddressSettingsAsJSON" management method on >> > the >> > > > ActiveMQServerControl passing in "$.artemis.internal.sf.#" as the >> match >> > > > and provide the output here? >> > > > >> > > > { >> > > > "addressFullMessagePolicy": "PAGE", >> > > > "maxSizeBytes": 524288000, >> > > > "pageSizeBytes": 10485760, >> > > > "maxDeliveryAttempts": 10, >> > > > "messageCounterHistoryDayLimit": 10, >> > > > "redeliveryDelay": 0, >> > > > "deadLetterAddress": "DLQ", >> > > > "minExpiryDelay": 1000, >> > > > "maxExpiryDelay": 2592000000, >> > > > "redistributionDelay": 1000, >> > > > "slowConsumerThresholdMeasurementUnit": "MESSAGES_PER_SECOND", >> > > > "autoCreateQueues": true, >> > > > "autoDeleteQueues": false, >> > > > "autoDeleteCreatedQueues": false, >> > > > "configDeleteQueues": "OFF", >> > > > "autoCreateAddresses": true, >> > > > "autoDeleteAddresses": false, >> > > > "configDeleteAddresses": "OFF", >> > > > "managementBrowsePageSize": 200, >> > > > "autoCreateDeadLetterResources": true, >> > > > "deadLetterQueuePrefix": "DLQ.", >> > > > "managementMessageAttributeSizeLimit": 10000 >> > > > } >> > > > >> > > > (formatted for better reading) >> > > > >> > > > чт, 21 нояб. 2024 г. в 20:01, Justin Bertram <jbert...@apache.org>: >> > > > >> > > > > > There are no address settings like autoDeleteQueues, >> > > > > autoDeleteCreatedQueues, configDeleteQueues etc. I wonder how >> could >> > > this >> > > > > happen. >> > > > > >> > > > > Can you invoke the "getAddressSettingsAsJSON" management method on >> > the >> > > > > ActiveMQServerControl [1] passing in "$.artemis.internal.sf.#" as >> the >> > > > match >> > > > > and provide the output here? >> > > > > >> > > > > >> > > > > Justin >> > > > > >> > > > > [1] >> > > > > >> > > > > >> > > > >> > > >> > >> https://activemq.apache.org/components/artemis/documentation/javadocs/javadoc-latest/org/apache/activemq/artemis/api/core/management/ActiveMQServerControl.html#getAddressSettingsAsJSON(java.lang.String) >> > > > > >> > > > > On Thu, Nov 21, 2024 at 3:41 AM Alexander Milovidov < >> > > > milovid...@gmail.com> >> > > > > wrote: >> > > > > >> > > > > > Hi All! >> > > > > > >> > > > > > We have Artemis cluster with two primary / backups, and it >> worked >> > > > > normally >> > > > > > before. Suddenly, the cluster queue was undeployed on one of the >> > > > cluster >> > > > > > nodes during reload of the broker configuration. There was a log >> > > > message >> > > > > > with event id AMQ224077 Undeploying queue >> > > > > > $.artemis.internal.sf.cluster-name.cluster-node-uuid. >> > > > > > >> > > > > > After this queue was undeployed, the messages which were routed >> to >> > > > other >> > > > > > cluster node were unrouted and discarded. >> > > > > > >> > > > > > There are no address settings like autoDeleteQueues, >> > > > > > autoDeleteCreatedQueues, configDeleteQueues etc. I wonder how >> could >> > > > this >> > > > > > happen. >> > > > > > The cluster queue was recreated after restart of the cluster >> > > connector. >> > > > > > >> > > > > > I don't know the root cause of the problem and we would like to >> > > prevent >> > > > > > this situation in the future because it leads to message loss. >> Is >> > it >> > > ok >> > > > > to >> > > > > > make cluster addresses and queues to be configuration-managed on >> > both >> > > > > > cluster nodes? >> > > > > > >> > > > > > ActiveMQ Artemis version is 2.37.0. >> > > > > > >> > > > > > -- >> > > > > > Regards, >> > > > > > Alexander >> > > > > > >> > > > > >> > > > >> > > >> > -- >> > Regards, >> > Alexander >> > >> >
