[
https://issues.apache.org/jira/browse/ARTEMIS-3588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17454098#comment-17454098
]
Lakatos Gyula edited comment on ARTEMIS-3588 at 12/6/21, 3:54 PM:
------------------------------------------------------------------
{quote}
Since "large" messages can be arbitrarily large it's likely they could
overwhelm the memory capacity of the broker and/or client so they are dealt
with in chunks.
{quote}
I had memory problems before on the client-side, because I load the large
messages in the memory, then parse them (they are PDF files). However, recently
I added more memory to the server and the OOM errors went away. Instead, I get
these AMQ219060 errors rarely, once around every couple of days.
One of the problems is that it is a RuntimeException so quite hard to catch
(without catching almost everything else).
{quote}What happens to the client at this point? Keep in mind that failover is
not designed to be 100% transparent in every use-case.
{quote}
The client is blocking on a
{{org.apache.activemq.artemis.api.core.client.ClientConsumer#receive()}} call,
then the exception is fired and propagated to the application, without any
failover catching it, etc. So eventually I'll get a {{RuntimeException}} on a
{{{}ClientConsumer.receive(){}}}. I should be able to catch the exception,
close the client then create a new one, but I think this should be handled by
the failover?
Also, it can be an intermittent networking issue. The network is not
enterprise-grade but some reliable homelab gear.
I'll look into the log files on the broker side too. Unfortunately, I can't
access the servers this week (need physical access).
was (Author: laxika):
{quote}
Since "large" messages can be arbitrarily large it's likely they could
overwhelm the memory capacity of the broker and/or client so they are dealt
with in chunks.
{quote}
I had memory problems before, because I load the large messages in the memory,
then parse them (they are PDF files). However, recently I added more memory to
the server and the OOM errors went away. Instead, I get these AMQ219060 errors
rarely, once around every couple of days.
One of the problems is that it is a RuntimeException so quite hard to catch
(without catching almost everything else).
{quote}What happens to the client at this point? Keep in mind that failover is
not designed to be 100% transparent in every use-case.
{quote}
The client is blocking on a
{{org.apache.activemq.artemis.api.core.client.ClientConsumer#receive()}} call,
then the exception is fired and propagated to the application, without any
failover catching it, etc. So eventually I'll get a {{RuntimeException}} on a
{{{}ClientConsumer.receive(){}}}. I should be able to catch the exception,
close the client then create a new one, but I think this should be handled by
the failover?
Also, it can be an intermittent networking issue. The network is not
enterprise-grade but some reliable homelab gear.
> AMQ219060 - Large Message Transmission interrupted error stops the
> application even when reconnect is set to infinite
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: ARTEMIS-3588
> URL: https://issues.apache.org/jira/browse/ARTEMIS-3588
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Affects Versions: 2.18.0
> Environment: Artemis 2.18.0
> Ubuntu 20.10
> Both the server and the client running on local site addresses (ie
> 192.168.0.*).
> Reporter: Lakatos Gyula
> Priority: Major
>
> I get the following error occasionally in my application:
> {code:java}
> 2021-11-20 00:27:32.337 INFO 20184 --- [oundedElastic-1]
> c.g.l.v.service.VaultDocumentManager : Archiving document with id:
> 6ae3ed96-ecaa-43e2-84a5-5b8371e93765.
> 2021-11-20 00:27:33.014 ERROR 20184 --- [oundedElastic-1]
> reactor.core.publisher.Operators : Operator called default
> onErrorDropped
> reactor.core.Exceptions$ErrorCallbackNotImplemented:
> java.lang.RuntimeException: AMQ219060: Large Message Transmission interrupted
> on consumer shutdown.
> Caused by: java.lang.RuntimeException: AMQ219060: Large Message Transmission
> interrupted on consumer shutdown.
> at
> org.apache.activemq.artemis.core.client.impl.ClientLargeMessageImpl.getBodyBuffer(ClientLargeMessageImpl.java:93)
> ~[artemis-core-client-2.18.0.jar!/:2.18.0]
> at
> com.github.loa.queue.artemis.service.consumer.deserializer.DocumentArchivingMessageDeserializer.deserialize(DocumentArchivingMessageDeserializer.java:17)
> ~[loa-queue-artemis-service-1.4.0.jar!/:na]
> at
> com.github.loa.queue.artemis.service.consumer.deserializer.DocumentArchivingMessageDeserializer.deserialize(DocumentArchivingMessageDeserializer.java:11)
> ~[loa-queue-artemis-service-1.4.0.jar!/:na]
> at
> com.github.loa.queue.artemis.service.ArtemisQueueManipulator.lambda$readMessage$3(ArtemisQueueManipulator.java:147)
> ~[loa-queue-artemis-service-1.4.0.jar!/:na]
> at
> com.github.loa.queue.artemis.service.consumer.ClientConsumerExecutor.invokeConsumer(ClientConsumerExecutor.java:26)
> ~[loa-queue-artemis-service-1.4.0.jar!/:na]
> at
> com.github.loa.queue.artemis.service.ArtemisQueueManipulator.readMessage(ArtemisQueueManipulator.java:139)
> ~[loa-queue-artemis-service-1.4.0.jar!/:na]
> at
> com.github.loa.vault.service.listener.VaultQueueConsumer.accept(VaultQueueConsumer.java:22)
> ~[classes!/:1.4.0]
> at
> com.github.loa.vault.service.listener.VaultQueueConsumer.accept(VaultQueueConsumer.java:13)
> ~[classes!/:1.4.0]
> at
> reactor.core.publisher.FluxGenerate.lambda$new$1(FluxGenerate.java:57)
> ~[reactor-core-3.4.10.jar!/:3.4.10]
> at
> reactor.core.publisher.FluxGenerate$GenerateSubscription.slowPath(FluxGenerate.java:265)
> ~[reactor-core-3.4.10.jar!/:3.4.10]
> at
> reactor.core.publisher.FluxGenerate$GenerateSubscription.request(FluxGenerate.java:207)
> ~[reactor-core-3.4.10.jar!/:3.4.10]
> at
> reactor.core.publisher.FluxPublishOn$PublishOnSubscriber.runAsync(FluxPublishOn.java:447)
> ~[reactor-core-3.4.10.jar!/:3.4.10]
> at
> reactor.core.publisher.FluxPublishOn$PublishOnSubscriber.run(FluxPublishOn.java:527)
> ~[reactor-core-3.4.10.jar!/:3.4.10]
> at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84)
> ~[reactor-core-3.4.10.jar!/:3.4.10]
> at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37)
> ~[reactor-core-3.4.10.jar!/:3.4.10]
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> ~[na:na]
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
> ~[na:na]
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> ~[na:na]
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> ~[na:na]
> at java.base/java.lang.Thread.run(Thread.java:833) ~[na:na]
> Caused by:
> org.apache.activemq.artemis.api.core.ActiveMQLargeMessageInterruptedException:
> AMQ219060: Large Message Transmission interrupted on consumer shutdown.
> at
> org.apache.activemq.artemis.core.client.impl.LargeMessageControllerImpl.checkException(LargeMessageControllerImpl.java:334)
> ~[artemis-core-client-2.18.0.jar!/:2.18.0]
> at
> org.apache.activemq.artemis.core.client.impl.LargeMessageControllerImpl.waitCompletion(LargeMessageControllerImpl.java:315)
> ~[artemis-core-client-2.18.0.jar!/:2.18.0]
> at
> org.apache.activemq.artemis.core.client.impl.LargeMessageControllerImpl.saveBuffer(LargeMessageControllerImpl.java:275)
> ~[artemis-core-client-2.18.0.jar!/:2.18.0]
> at
> org.apache.activemq.artemis.core.client.impl.ClientLargeMessageImpl.checkBuffer(ClientLargeMessageImpl.java:159)
> ~[artemis-core-client-2.18.0.jar!/:2.18.0]
> at
> org.apache.activemq.artemis.core.client.impl.ClientLargeMessageImpl.getBodyBuffer(ClientLargeMessageImpl.java:91)
> ~[artemis-core-client-2.18.0.jar!/:2.18.0]
> ... 19 common frames omitted
> Caused by:
> org.apache.activemq.artemis.api.core.ActiveMQLargeMessageInterruptedException:
> AMQ219060: Large Message Transmission interrupted on consumer shutdown.
> at
> org.apache.activemq.artemis.core.client.impl.LargeMessageControllerImpl.cancel(LargeMessageControllerImpl.java:207)
> ~[artemis-core-client-2.18.0.jar!/:2.18.0]
> at
> org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.resetLargeMessageController(ClientConsumerImpl.java:755)
> ~[artemis-core-client-2.18.0.jar!/:2.18.0]
> at
> org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.clearAtFailover(ClientConsumerImpl.java:528)
> ~[artemis-core-client-2.18.0.jar!/:2.18.0]
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.handleFailover(ClientSessionImpl.java:1412)
> ~[artemis-core-client-2.18.0.jar!/:2.18.0]
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.reconnectSessions(ClientSessionFactoryImpl.java:840)
> ~[artemis-core-client-2.18.0.jar!/:2.18.0]
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:664)
> ~[artemis-core-client-2.18.0.jar!/:2.18.0]
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:542)
> ~[artemis-core-client-2.18.0.jar!/:2.18.0]
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:535)
> ~[artemis-core-client-2.18.0.jar!/:2.18.0]
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.access$100(ClientSessionFactoryImpl.java:75)
> ~[artemis-core-client-2.18.0.jar!/:2.18.0]
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:398)
> ~[artemis-core-client-2.18.0.jar!/:2.18.0]
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42)
> ~[artemis-commons-2.18.0.jar!/:2.18.0]
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
> ~[artemis-commons-2.18.0.jar!/:2.18.0]
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65)
> ~[artemis-commons-2.18.0.jar!/:2.18.0]
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> ~[na:na]
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> ~[na:na]
> at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> ~[artemis-commons-2.18.0.jar!/:2.18.0] {code}
> I have a hard time figuring out why this happens. The internet doesn't have
> any results when I search for it in Google.
> Also, I have
> {{org.apache.activemq.artemis.api.core.client.ServerLocator#setReconnectAttempts}}
> set to {{-1}} and I would expect Artemis to reconnect to the server in the
> case of a client error.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
