Is there anything on DLQ or Expiry? Any exceptions in the logs?
Things you need to understand in order to write a reproducer: How you're sending (protocol you're using? size, how often, how many producers, how you open the connection and how you commit (say if you are using XA). are you pooling connections? How you're consuming (same questions except to the size). How you're configuring clustering How many nodes are you using? Is your message considered a large message? (beyond the min-large-message-size limit) write an example of your load... track every message you send.... and check they are received.... We have a lot of tests doing that and none of them are failing.... it's either a misconfiguration or you are using some edge case that I would like to know about. If you produce such things, than your report for a missed message would be complete.. until then it's either a misconfiguration or I don't have how to investigate it. Another way I could help.. if you reproduce the issue with your data and retention folder enabled.. send me the data, telling me what message you're missing (with properties and track) and I try to identify what happened to the message. On Thu, Oct 20, 2022 at 8:27 AM Clebert Suconic <clebert.suco...@gmail.com> wrote: > > Regarding the reproducer. Write a test that mimics your payload and > configuration In an independent way. You could use the examples for that. > > If you reproduce your issue I way someone can actually see what’s happening > (even jf it’s misconfig) is the only way someone would be able to help you > remotely. > > On Thu, Oct 20, 2022 at 8:25 AM Clebert Suconic <clebert.suco...@gmail.com> > wrote: >> >> Can you reproduce it in your test at least ? >> >> Use journal retention and look for the message your lost on the print data. >> >> It could be misconfiguration in your cluster. There are no changes between >> 2.22 and 2.26 that could lead to that. >> >> Run your test with retention enabled and I can help you out figuring out >> what happened in your test. >> >> On Thu, Oct 20, 2022 at 7:57 AM Gašper Čefarin <gasper.cefa...@actual-it.si> >> wrote: >>> >>> I hope you don't mind if I reply to this thread. I'd also like to report >>> messages getting lost. >>> >>> I've had 2 occurrence of losing messages when using simple replication (1 >>> live and 1 backup server). >>> >>> >>> I was using artemis v2.22.0. >>> I was not able to replicate the issue, and I think it happened when I >>> rebooted the live server. >>> >>> The messages lost were stored persistently, in a durable queue, with no >>> consumers online. Not sure about producers. >>> >>> All I see in the logs are warnings like these two: >>> >>> - 2022-07-01 14:52:16,282 WARN [org.apache.activemq.artemis.core.server] >>> AMQ222092: Connection to the backup node failed, removing replication now: >>> ActiveMQRemoteDisconnectException[errorType=REMOTE_DISCONNECT message=null] >>> >>> - 2022-07-01 14:52:16,295 WARN [org.apache.activemq.artemis.core.client] >>> AMQ212037: Connection failure to 10.108.28.52/10.108.28.52:9000 has been >>> detected: AMQ219015: The connection was disconnected because of server >>> shutdown [code=DISCONNECTED] >>> >>> The only thing that comes to mind that could be the problem is changing the >>> port for cluster communication from default 61616 to 9000 (i've experienced >>> some problems unrelated to message loss when changing the port). >>> >>> Any advice on reproducing the issue or where to look for more data >>> appreciated. >>> >>> -----Original Message----- >>> From: Clebert Suconic <clebert.suco...@gmail.com> >>> Sent: Wednesday, October 19, 2022 4:58 PM >>> To: users@activemq.apache.org >>> Subject: Re: Messages getting lost on Artemis 2.25 >>> >>> >>> To sporočilo izvira izven naše organizacije. Bodite pozorni pri vsebini in >>> odpiranju povezav ali prilog. >>> >>> >>> >>> >>> Basically I'm telling you how to investigate it.. and if you find an issue >>> on the broker, we will need a way to reproduce it. >>> >>> I have no other report about a message loss situation... >>> >>> (we do have situations with page-counters going wrong while paging..which >>> I'm working now to fix it... but no message loss). >>> >>> On Wed, Oct 19, 2022 at 10:55 AM Clebert Suconic >>> <clebert.suco...@gmail.com> wrote: >>> > >>> > I am not aware of any issues that would lead to message loss... >>> > >>> > Garbage Collection itself has no effect on anything regarding paging or >>> > journal. >>> > >>> > >>> > Are you able to chase which message is lost on a test? >>> > >>> > >>> > you could use the retention feature, replay the message.. and you >>> > could also look on the ./artemis data print on what happened to the >>> > message. >>> > >>> > >>> > One other suggestion I could make is to use Federation instead of >>> > clustering. Perhaps message are stranded on the Store and forward >>> > queue? >>> > >>> > >>> > also.. you have consumers in all the nodes.. you should use clustering >>> > with OFF-WITH-REDISTRIBUTION, or use Federation. you should always >>> > favor the local consumers. >>> > >>> > On Wed, Oct 19, 2022 at 8:16 AM Walter de Boer <walterdeb...@dbso.nl> >>> > wrote: >>> > > >>> > > All, >>> > > >>> > > This week we lost 23.000 messages in a few days time on our >>> > > production Cluster running Artemis 2.26.0, see our settings below. >>> > > We've reverted back to Artemis 2.20.0 just in case >>> > > >>> > > A few observatoins: >>> > > >>> > > * In version 2.24.0, 2.25.0 and 2.26.0 running on ZGC we noticed >>> > > messages being produced to a queue without errors, that we didn't >>> > > find in that queue. At the same time we saw incorrect counters. We >>> > > did restart nodes to resolve, but on one occasion the error >>> > > continued for some time after that, and we never found the messages >>> > > again. Not even when exporting the journal files. The errors showed >>> > > after running a few days >>> > > * In version 2.20.0 running on G1GC and on ZGC we did not lose any >>> > > messages. We did experience memory issues resulting in (to) long >>> > > garbage collection times every other week, maybe due to lack of JVM >>> > > tuning on our side. We were running 2.20 on G1GC for serveral >>> > > months >>> > > >>> > > We're running a symetric Cluser of 3 live/backup pairs in Docker JRE >>> > > (temurin) containers on VMWare CentOS7 hosts. Each live node has >>> > > around >>> > > 1.000 producers & consumers continuously. >>> > > >>> > > I hope the Artemis community can advise us in this? >>> > > >>> > > Best Regards, >>> > > >>> > > Walter >>> > > >>> > > >>> > > Our setup: >>> > > >>> > > * >>> > > **docker-compose.yaml** >>> > > * >>> > > version: "3.8" >>> > > >>> > > services: >>> > > artemis: >>> > > container_name: 'artemis' >>> > > network_mode: "host" >>> > > image: "cdplatform/activemq-artemis:2.26.0" >>> > > restart: 'always' >>> > > hostname: cjiblx8408.ato.cjib.minjus.nl >>> > > volumes: >>> > > - "/data/artemis/data:/var/lib/artemis/data" >>> > > - "/data/artemis/plugins:/var/lib/artemis/lib" >>> > > - "/data/artemis/etc:/var/lib/artemis/etc" >>> > > - "/data/artemis/etc-override:/var/lib/artemis/etc-override" >>> > > - "/logging/artemis:/var/lib/artemis/log" >>> > > environment: >>> > > ARTEMIS_MIN_MEMORY: "14051615047" >>> > > ARTEMIS_MAX_MEMORY: "14051615047" >>> > > JAVA_XTRA_ARGS: "-XX:ActiveProcessorCount=4 -XX:+UseZGC >>> > > -XX:+UseDynamicNumberOfGCThreads -XX:+UseStringDeduplication " >>> > > BROKER_SETTINGS_FILE: "broker-settings.xml" >>> > > ENABLE_JMX: "true" >>> > > JMX_PORT: "3333" >>> > > ENABLE_JMX_EXPORTER: "true" >>> > > JMX_RMI_PORT: "1098" >>> > > mem_swappiness: 0 >>> > > memswap_limit: 20073735782 >>> > > deploy: >>> > > resources: >>> > > limits: >>> > > memory: "20073735782" >>> > > reservations: >>> > > memory: "20073735782" >>> > > >>> > > *Command line options:* >>> > > >>> > > /opt/java/openjdk/bin/java >>> > > >>> > > -javaagent:/opt/jmx-exporter/jmx_prometheus_javaagent.jar=9404:/opt/jmx-exporter/etc/jmx-exporter-config.yaml >>> > > -Xmx17564518809 >>> > > -Xms17564518809 >>> > > -Dcom.sun.management.jmxremote.authenticate=true >>> > > >>> > > -Dcom.sun.management.jmxremote.password.file=/var/lib/artemis/etc/jmxremote.password >>> > > >>> > > -Dcom.sun.management.jmxremote.access.file=/var/lib/artemis/etc/jmxremote.access >>> > > -Dcom.sun.management.jmxremote.port=3333 >>> > > -Dcom.sun.management.jmxremote.rmi.port=1098 >>> > > -Dcom.sun.management.jmxremote.ssl=false >>> > > -Djava.net.preferIPv4Addresses=true >>> > > -Djava.net.preferIPv4Stack=true >>> > > -XX:ActiveProcessorCount=4 >>> > > -XX:+UseZGC >>> > > -XX:+UseDynamicNumberOfGCThreads >>> > > -XX:+UseStringDeduplication >>> > > -Dhawtio.realm=activemq >>> > > -Dhawtio.offline=true >>> > > -Dhawtio.role=gs-auth-Artemis_Admin,gs-auth-Artemis_User >>> > > >>> > > -DPrincipalClasses=org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal >>> > > >>> > > -Djolokia.policyLocation=file:/var/lib/artemis/etc/jolokia-access.xml >>> > > -Dcom.sun.management.jmxremote.ssl=false >>> > > -Xbootclasspath/a:/var/lib/artemis/lib/javax.json-1.1.4.jar >>> > > -Dhawtio.role=gs-auth-Artemis_Admin,gs-auth-Artemis_User >>> > > >>> > > -Xbootclasspath/a:/opt/apache-artemis/lib/jboss-logmanager-2.1.18.Final.jar:/opt/apache-artemis/lib/wildfly-common-1.5.2.Final.jar:/opt/apache-artemis/lib/javax.json-1.1.4.jar >>> > > -Djava.security.auth.login.config=/var/lib/artemis/etc/login.config >>> > > -classpath /opt/apache-artemis/lib/artemis-boot.jar >>> > > -Dartemis.home=/opt/apache-artemis >>> > > -Dartemis.instance=/var/lib/artemis >>> > > -Djava.library.path=/opt/apache-artemis/bin/lib/linux-x86_64 >>> > > -Djava.io.tmpdir=/var/lib/artemis/tmp >>> > > -Ddata.dir=/var/lib/artemis/data >>> > > -Dartemis.instance.etc=/var/lib/artemis/etc >>> > > -Djava.util.logging.manager=org.jboss.logmanager.LogManager >>> > > >>> > > -Dlogging.configuration=file:/var/lib/artemis/etc//logging.properties >>> > > -Dartemis.default.sensitive.string.codec.key= >>> > > org.apache.activemq.artemis.boot.Artemis >>> > > run >>> > > >>> > > *broker-settings.xml**:* >>> > > >>> > > <core xmlns="urn:activemq:core"> >>> > > <global-max-size>2810323009</global-max-size> >>> > > <name>xxxxxxx.xxxxxx.xx</name> >>> > > <graceful-shutdown-enabled >>> > > xmlns="urn:activemq:core">true</graceful-shutdown-enabled> >>> > > <graceful-shutdown-timeout >>> > > xmlns="urn:activemq:core">10000</graceful-shutdown-timeout> >>> > > <management-address >>> > > xmlns="urn:activemq:core">activemq.management</management-address> >>> > > <persistence-enabled >>> > > xmlns="urn:activemq:core">true</persistence-enabled> >>> > > <id-cache-size xmlns="urn:activemq:core">20000</id-cache-size> >>> > > <persist-id-cache >>> > > xmlns="urn:activemq:core">true</persist-id-cache> >>> > > <paging-directory >>> > > xmlns="urn:activemq:core">data/paging</paging-directory> >>> > > <bindings-directory >>> > > xmlns="urn:activemq:core">data/bindings</bindings-directory> >>> > > <large-messages-directory >>> > > >>> > > xmlns="urn:activemq:core">data/large-messages</large-messages-directory> >>> > > <journal-directory >>> > > xmlns="urn:activemq:core">data/journal</journal-directory> >>> > > <journal-type xmlns="urn:activemq:core">ASYNCIO</journal-type> >>> > > <journal-datasync >>> > > xmlns="urn:activemq:core">true</journal-datasync> >>> > > <journal-min-files >>> > > xmlns="urn:activemq:core">2</journal-min-files> >>> > > <journal-pool-files >>> > > xmlns="urn:activemq:core">10</journal-pool-files> >>> > > <journal-device-block-size >>> > > xmlns="urn:activemq:core">4096</journal-device-block-size> >>> > > <journal-file-size >>> > > xmlns="urn:activemq:core">10MB</journal-file-size> >>> > > <journal-buffer-size >>> > > xmlns="urn:activemq:core">490KB</journal-buffer-size> >>> > > <journal-compact-min-files >>> > > xmlns="urn:activemq:core">10</journal-compact-min-files> >>> > > <journal-compact-percentage >>> > > xmlns="urn:activemq:core">30</journal-compact-percentage> >>> > > <journal-lock-acquisition-timeout >>> > > xmlns="urn:activemq:core">-1</journal-lock-acquisition-timeout> >>> > > <journal-file-open-timeout >>> > > xmlns="urn:activemq:core">5</journal-file-open-timeout> >>> > > <journal-sync-non-transactional >>> > > xmlns="urn:activemq:core">true</journal-sync-non-transactional> >>> > > <journal-sync-transactional >>> > > xmlns="urn:activemq:core">true</journal-sync-transactional> >>> > > <disk-scan-period >>> > > xmlns="urn:activemq:core">5000</disk-scan-period> >>> > > <max-disk-usage xmlns="urn:activemq:core">90</max-disk-usage> >>> > > <critical-analyzer >>> > > xmlns="urn:activemq:core">true</critical-analyzer> >>> > > <critical-analyzer-timeout >>> > > xmlns="urn:activemq:core">120000</critical-analyzer-timeout> >>> > > <critical-analyzer-check-period >>> > > xmlns="urn:activemq:core">60000</critical-analyzer-check-period> >>> > > <critical-analyzer-policy >>> > > xmlns="urn:activemq:core">LOG</critical-analyzer-policy> >>> > > <page-sync-timeout >>> > > xmlns="urn:activemq:core">548000</page-sync-timeout> >>> > > <acceptors xmlns="urn:activemq:core"> >>> > > <acceptor >>> > > >>> > > name="artemis">tcp://0.0.0.0:61616?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;amqpMinLargeMessageSize=102400;connectionsAllowed=1536;directDeliver=false;useEpoll=true;amqpCredits=1000;amqpLowCredits=300;amqpDuplicateDetection=true;protocols=CORE,AMQP,STOMP,HORNETQ,OPENWIRE;</acceptor> >>> > > </acceptors> >>> > > <connectors xmlns="urn:activemq:core"> >>> > > <connector >>> > > name="artemis">tcp://xxxxxxx.xxxxxx.xx:61616</connector> >>> > > </connectors> >>> > > <cluster-user xmlns="urn:activemq:core">artemis</cluster-user> >>> > > <cluster-password >>> > > xmlns="urn:activemq:core">xxxxxxxx</cluster-password> >>> > > <broadcast-groups xmlns="urn:activemq:core"> >>> > > <broadcast-group name="bg-group1"> >>> > > <group-address>231.7.7.10</group-address> >>> > > <group-port>9876</group-port> >>> > > <broadcast-period>5000</broadcast-period> >>> > > <connector-ref>artemis</connector-ref> >>> > > </broadcast-group> >>> > > </broadcast-groups> >>> > > <discovery-groups xmlns="urn:activemq:core"> >>> > > <discovery-group name="dg-group1"> >>> > > <group-address>231.7.7.10</group-address> >>> > > <group-port>9876</group-port> >>> > > <refresh-timeout>10000</refresh-timeout> >>> > > </discovery-group> >>> > > </discovery-groups> >>> > > <cluster-connections xmlns="urn:activemq:core"> >>> > > <cluster-connection name="artemis-ato"> >>> > > <connector-ref>artemis</connector-ref> >>> > > <retry-interval>2000</retry-interval> >>> > > <initial-connect-attempts>1000</initial-connect-attempts> >>> > > <reconnect-attempts>1000</reconnect-attempts> >>> > > <message-load-balancing>ON_DEMAND</message-load-balancing> >>> > > <max-hops>1</max-hops> >>> > > <discovery-group-ref discovery-group-name="dg-group1"/> >>> > > </cluster-connection> >>> > > </cluster-connections> >>> > > <ha-policy xmlns="urn:activemq:core"> >>> > > <replication> >>> > > <master> >>> > > <check-for-live-server>true</check-for-live-server> >>> > > <vote-on-replication-failure>true</vote-on-replication-failure> >>> > > <group-name>ato-hapair-1</group-name> >>> > > </master> >>> > > </replication> >>> > > </ha-policy> >>> > > <metrics xmlns="urn:activemq:core"> >>> > > <jvm-memory>true</jvm-memory> >>> > > <jvm-gc>true</jvm-gc> >>> > > <jvm-threads>true</jvm-threads> >>> > > <plugin >>> > > >>> > > class-name="org.apache.activemq.artemis.core.server.metrics.plugins.ArtemisPrometheusMetricsPlugin"/> >>> > > </metrics> >>> > > <security-settings xmlns="urn:activemq:core"> >>> > > <security-setting match="activemq.management"> >>> > > <permission type="manage" roles="amq,service"/> >>> > > </security-setting> >>> > > <security-setting match="#"> >>> > > <permission type="manage" roles="amq,service"/> >>> > > <permission type="send" roles="amq,service,b2bi"/> >>> > > <permission type="consume" roles="amq,service,b2bi"/> >>> > > <permission type="browse" roles="amq,service"/> >>> > > <permission type="createAddress" roles="amq,service"/> >>> > > <permission type="deleteAddress" roles="amq,service"/> >>> > > <permission type="createDurableQueue" roles="amq,service"/> >>> > > <permission type="deleteDurableQueue" roles="amq,service"/> >>> > > <permission type="createNonDurableQueue" >>> > > roles="amq,service"/> >>> > > <permission type="deleteNonDurableQueue" >>> > > roles="amq,service"/> >>> > > </security-setting> >>> > > <role-mapping from="gs-auth-Artemis_Admin" to="amq"/> >>> > > <role-mapping from="gs-auth-Artemis_User" to="service"/> >>> > > </security-settings> >>> > > <address-settings xmlns="urn:activemq:core"> >>> > > <address-setting match="activemq.management#"> >>> > > <dead-letter-address>DLQ</dead-letter-address> >>> > > <expiry-address>ExpiryQueue</expiry-address> >>> > > <redelivery-delay>0</redelivery-delay> >>> > > >>> > > <message-counter-history-day-limit>10</message-counter-history-day-limit> >>> > > <max-size-bytes>-1</max-size-bytes> >>> > > <max-size-messages>-1</max-size-messages> >>> > > <address-full-policy>PAGE</address-full-policy> >>> > > <auto-create-queues>true</auto-create-queues> >>> > > <auto-create-addresses>true</auto-create-addresses> >>> > > <auto-create-jms-queues>true</auto-create-jms-queues> >>> > > <auto-create-jms-topics>true</auto-create-jms-topics> >>> > > </address-setting> >>> > > <address-setting match="#"> >>> > > <dead-letter-address>DLQ</dead-letter-address> >>> > > <expiry-address>ExpiryQueue</expiry-address> >>> > > <redelivery-delay>0</redelivery-delay> >>> > > >>> > > <message-counter-history-day-limit>10</message-counter-history-day-limit> >>> > > <max-size-bytes>-1</max-size-bytes> >>> > > <max-size-messages>-1</max-size-messages> >>> > > <address-full-policy>PAGE</address-full-policy> >>> > > <auto-create-queues>true</auto-create-queues> >>> > > <auto-create-addresses>true</auto-create-addresses> >>> > > <auto-create-jms-queues>true</auto-create-jms-queues> >>> > > <auto-create-jms-topics>true</auto-create-jms-topics> >>> > > </address-setting> >>> > > <address-setting match="jms.#"> >>> > > <dead-letter-address>DLQ</dead-letter-address> >>> > > <expiry-address>ExpiryQueue</expiry-address> >>> > > <max-delivery-attempts>5</max-delivery-attempts> >>> > > <redelivery-delay>500</redelivery-delay> >>> > > <redelivery-delay-multiplier>1.5</redelivery-delay-multiplier> >>> > > >>> > > <redelivery-collision-avoidance-factor>0.5</redelivery-collision-avoidance-factor> >>> > > <redistribution-delay>30000</redistribution-delay> >>> > > <send-to-dla-on-no-route>true</send-to-dla-on-no-route> >>> > > <max-size-bytes>-1</max-size-bytes> >>> > > <max-size-messages>-1</max-size-messages> >>> > > <address-full-policy>PAGE</address-full-policy> >>> > > >>> > > <message-counter-history-day-limit>10</message-counter-history-day-limit> >>> > > <auto-create-queues>false</auto-create-queues> >>> > > <auto-delete-queues>false</auto-delete-queues> >>> > > <auto-delete-created-queues>false</auto-delete-created-queues> >>> > > <auto-delete-queues-delay>30000</auto-delete-queues-delay> >>> > > <config-delete-queues>OFF</config-delete-queues> >>> > > <auto-create-addresses>false</auto-create-addresses> >>> > > <auto-delete-addresses>false</auto-delete-addresses> >>> > > <auto-delete-addresses-delay>30000</auto-delete-addresses-delay> >>> > > <config-delete-addresses>OFF</config-delete-addresses> >>> > > </address-setting> >>> > > <address-setting match="activemq.notifications"> >>> > > <max-size-bytes>-1</max-size-bytes> >>> > > <max-size-messages>-1</max-size-messages> >>> > > <address-full-policy>PAGE</address-full-policy> >>> > > </address-setting> >>> > > <address-setting match="jms.queue.#"> >>> > > <default-address-routing-type>ANYCAST</default-address-routing-type> >>> > > <default-queue-routing-type>ANYCAST</default-queue-routing-type> >>> > > </address-setting> >>> > > <address-setting match="jms.topic.#"> >>> > > >>> > > <default-address-routing-type>MULTICAST</default-address-routing-type> >>> > > <default-queue-routing-type>MULTICAST</default-queue-routing-type> >>> > > </address-setting> >>> > > </address-settings> >>> > > <addresses xmlns="urn:activemq:core"> >>> > > <address name="DLQ"> >>> > > <anycast> >>> > > <queue name="DLQ"/> >>> > > </anycast> >>> > > </address> >>> > > <address name="ExpiryQueue"> >>> > > <anycast> >>> > > <queue name="ExpiryQueue"/> >>> > > </anycast> >>> > > </address> >>> > > </addresses> >>> > > <broker-plugins xmlns="urn:activemq:core"> >>> > > <broker-plugin >>> > > >>> > > class-name="org.apache.activemq.artemis.core.server.plugin.impl.LoggingActiveMQServerPlugin"> >>> > > <property key="LOG_ALL_EVENTS" value="false"/> >>> > > <property key="LOG_CONNECTION_EVENTS" value="false"/> >>> > > <property key="LOG_SESSION_EVENTS" value="false"/> >>> > > <property key="LOG_CONSUMER_EVENTS" value="false"/> >>> > > <property key="LOG_DELIVERING_EVENTS" value="false"/> >>> > > <property key="LOG_SENDING_EVENTS" value="false"/> >>> > > <property key="LOG_INTERNAL_EVENTS" value="false"/> >>> > > </broker-plugin> >>> > > </broker-plugins> >>> > > </core> >>> > > >>> > > >>> > > >>> > >>> > >>> > -- >>> > Clebert Suconic >>> >>> >>> >>> -- >>> Clebert Suconic >>> >>> NOTICE - NOT TO BE REMOVED. >>> This e-mail and any attachments are confidential and may contain legally >>> privileged information and/or copyright material of Actual I.T. or third >>> parties. If you are not an authorised recipient of this e-mail, please >>> contact Actual I.T. immediately by return email or by telephone or >>> facsimile on the above numbers. >>> You should not read, print, re-transmit, store or act in reliance on this >>> email or any attachments and you should destroy all copies of them. >> >> -- >> Clebert Suconic > > -- > Clebert Suconic -- Clebert Suconic
