[
https://issues.apache.org/jira/browse/ARTEMIS-4217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17705313#comment-17705313
]
daves commented on ARTEMIS-4217:
--------------------------------
Hi [Justin
Bertram|https://issues.apache.org/jira/secure/ViewProfile.jspa?name=jbertram]
The log is all the traces I got from finding the problem to reproducing* it. I
thought it's better to have too much log available than not enough.
The use-case is this: Client A sends a AMQP message to client B via Artemis.
Both clients use AMQP as protocol. The message is a proprietary structure
serialized with protobuf. The payload contains 1-5 PNG images (therefore the
large size). Sending the message works, all send attempts are successful, no
errors. Consuming messages also does not produce any client side errors, BUT
some messages are never "pushed" to the client. For all "lost" messages, I can
find the stack trace (ticket description) in the Artemis log.
Sadly, the problem occurs only on a test environment with real applications.
Also, it only happens a few times a day. I tried to make a small consumer and
producer example, but I could not reproduce the problem. I used a recorded
message from the test environment as test data for my client, although I
believe the problem has "nothing" to do with the payload. The real application
sends the same data, just with a different client id, to multiple clients
(100-2000) and typically only 1 or 2 messages do not arrive at the client. The
problem does not always occur with the same client.
If you want to try it for yourself, I've added my sample client as an
attachment. You can change the connection string in the appsettings.json file.
> AMQ111005: Failed to convert message. Sending it to Dead Letter Address.
> ------------------------------------------------------------------------
>
> Key: ARTEMIS-4217
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4217
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: AMQP
> Affects Versions: 2.28.0
> Environment: Windows Server 2022 21H2
> openjdk 19.0.1 2022-10-18
> OpenJDK Runtime Environment (build 19.0.1+10-21)
> OpenJDK 64-Bit Server VM (build 19.0.1+10-21, mixed mode, sharing)
> Reporter: daves
> Priority: Major
> Attachments: ArtemisConvertError.zip, artemis-service.wrapper.log,
> artemis.log, artemis.log.2023-03-23-1.log.gz,
> artemis.log.2023-03-23-10.log.gz, artemis.log.2023-03-23-100.log.gz,
> artemis.log.2023-03-23-11.log.gz, artemis.log.2023-03-23-12.log.gz,
> artemis.log.2023-03-23-13.log.gz, artemis.log.2023-03-23-14.log.gz,
> artemis.log.2023-03-23-15.log.gz, artemis.log.2023-03-23-16.log.gz,
> artemis.log.2023-03-23-17.log.gz, artemis.log.2023-03-23-18.log.gz,
> artemis.log.2023-03-23-19.log.gz, artemis.log.2023-03-23-2.log.gz,
> artemis.log.2023-03-23-20.log.gz, artemis.log.2023-03-23-21.log.gz,
> artemis.log.2023-03-23-22.log.gz, artemis.log.2023-03-23-23.log.gz,
> artemis.log.2023-03-23-24.log.gz, artemis.log.2023-03-23-25.log.gz,
> artemis.log.2023-03-23-26.log.gz, artemis.log.2023-03-23-27.log.gz,
> artemis.log.2023-03-23-28.log.gz, artemis.log.2023-03-23-29.log.gz,
> artemis.log.2023-03-23-3.log.gz, artemis.log.2023-03-23-30.log.gz,
> artemis.log.2023-03-23-31.log.gz, artemis.log.2023-03-23-32.log.gz,
> artemis.log.2023-03-23-33.log.gz, artemis.log.2023-03-23-34.log.gz,
> artemis.log.2023-03-23-35.log.gz, artemis.log.2023-03-23-36.log.gz,
> artemis.log.2023-03-23-37.log.gz, artemis.log.2023-03-23-38.log.gz,
> artemis.log.2023-03-23-39.log.gz, artemis.log.2023-03-23-4.log.gz,
> artemis.log.2023-03-23-40.log.gz, artemis.log.2023-03-23-41.log.gz,
> artemis.log.2023-03-23-42.log.gz, artemis.log.2023-03-23-43.log.gz,
> artemis.log.2023-03-23-44.log.gz, artemis.log.2023-03-23-45.log.gz,
> artemis.log.2023-03-23-46.log.gz, artemis.log.2023-03-23-47.log.gz,
> artemis.log.2023-03-23-48.log.gz, artemis.log.2023-03-23-49.log.gz,
> artemis.log.2023-03-23-5.log.gz, artemis.log.2023-03-23-50.log.gz,
> artemis.log.2023-03-23-51.log.gz, artemis.log.2023-03-23-52.log.gz,
> artemis.log.2023-03-23-53.log.gz, artemis.log.2023-03-23-54.log.gz,
> artemis.log.2023-03-23-55.log.gz, artemis.log.2023-03-23-56.log.gz,
> artemis.log.2023-03-23-57.log.gz, artemis.log.2023-03-23-58.log.gz,
> artemis.log.2023-03-23-59.log.gz, artemis.log.2023-03-23-6.log.gz,
> artemis.log.2023-03-23-60.log.gz, artemis.log.2023-03-23-61.log.gz,
> artemis.log.2023-03-23-62.log.gz, artemis.log.2023-03-23-63.log.gz,
> artemis.log.2023-03-23-64.log.gz, artemis.log.2023-03-23-65.log.gz,
> artemis.log.2023-03-23-66.log.gz, artemis.log.2023-03-23-67.log.gz,
> artemis.log.2023-03-23-68.log.gz, artemis.log.2023-03-23-69.log.gz,
> artemis.log.2023-03-23-7.log.gz, artemis.log.2023-03-23-70.log.gz,
> artemis.log.2023-03-23-71.log.gz, artemis.log.2023-03-23-72.log.gz,
> artemis.log.2023-03-23-73.log.gz, artemis.log.2023-03-23-74.log.gz,
> artemis.log.2023-03-23-75.log.gz, artemis.log.2023-03-23-76.log.gz,
> artemis.log.2023-03-23-77.log.gz, artemis.log.2023-03-23-78.log.gz,
> artemis.log.2023-03-23-79.log.gz, artemis.log.2023-03-23-8.log.gz,
> artemis.log.2023-03-23-80.log.gz, artemis.log.2023-03-23-81.log.gz,
> artemis.log.2023-03-23-82.log.gz, artemis.log.2023-03-23-83.log.gz,
> artemis.log.2023-03-23-84.log.gz, artemis.log.2023-03-23-85.log.gz,
> artemis.log.2023-03-23-86.log.gz, artemis.log.2023-03-23-87.log.gz,
> artemis.log.2023-03-23-88.log.gz, artemis.log.2023-03-23-89.log.gz,
> artemis.log.2023-03-23-9.log.gz, artemis.log.2023-03-23-90.log.gz,
> artemis.log.2023-03-23-91.log.gz, artemis.log.2023-03-23-92.log.gz,
> artemis.log.2023-03-23-93.log.gz, artemis.log.2023-03-23-94.log.gz,
> artemis.log.2023-03-23-95.log.gz, artemis.log.2023-03-23-96.log.gz,
> artemis.log.2023-03-23-97.log.gz, artemis.log.2023-03-23-98.log.gz,
> artemis.log.2023-03-23-99.log.gz
>
>
> Some of the AMQP messages sent by my client never arrive at the consumer. In
> the Artemis log I found the following exception:
> {noformat}
> 2023-03-23 18:06:58,084 WARN
> [org.apache.activemq.artemis.protocol.amqp.logger] AMQ111005: Failed to
> convert message. Sending it to Dead Letter Address.
> org.apache.activemq.artemis.protocol.amqp.converter.coreWrapper.ConversionException:
> java.nio.channels.ClosedChannelException
> at
> org.apache.activemq.artemis.protocol.amqp.converter.CoreAmqpConverter.fromCore(CoreAmqpConverter.java:318)
> ~[artemis-amqp-protocol-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.protocol.amqp.converter.CoreAmqpConverter.checkAMQP(CoreAmqpConverter.java:79)
> ~[artemis-amqp-protocol-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.protocol.amqp.proton.ProtonServerSenderContext.executeDelivery(ProtonServerSenderContext.java:561)
> ~[artemis-amqp-protocol-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.core.server.impl.MessageReferenceImpl.run(MessageReferenceImpl.java:131)
> ~[artemis-server-2.28.0.jar:2.28.0]
> at
> io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
> ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> at
> io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
> ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
> ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
> ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
> ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> at
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> ~[artemis-commons-2.28.0.jar:?] Caused by: java.lang.RuntimeException:
> java.nio.channels.ClosedChannelException
> at
> org.apache.activemq.artemis.core.persistence.impl.journal.LargeBody.getBodyBufferSize(LargeBody.java:293)
> ~[artemis-server-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl.getBodyBufferSize(LargeServerMessageImpl.java:263)
> ~[artemis-server-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.protocol.amqp.converter.coreWrapper.CoreBytesMessageWrapper.getBodyLength(CoreBytesMessageWrapper.java:98)
> ~[artemis-amqp-protocol-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.protocol.amqp.converter.coreWrapper.CoreBytesMessageWrapper.getBinaryFromMessageBody(CoreBytesMessageWrapper.java:68)
> ~[artemis-amqp-protocol-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.protocol.amqp.converter.coreWrapper.CoreBytesMessageWrapper.createAMQPSection(CoreBytesMessageWrapper.java:78)
> ~[artemis-amqp-protocol-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.protocol.amqp.converter.CoreAmqpConverter.fromCore(CoreAmqpConverter.java:106)
> ~[artemis-amqp-protocol-2.28.0.jar:2.28.0]
> ... 10 more
> Caused by: java.nio.channels.ClosedChannelException
> at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:165)
> ~[?:?]
> at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:416) ~[?:?]
> at
> org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.size(NIOSequentialFile.java:339)
> ~[artemis-journal-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.core.persistence.impl.journal.LargeBody.getBodyBufferSize(LargeBody.java:285)
> ~[artemis-server-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl.getBodyBufferSize(LargeServerMessageImpl.java:263)
> ~[artemis-server-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.protocol.amqp.converter.coreWrapper.CoreBytesMessageWrapper.getBodyLength(CoreBytesMessageWrapper.java:98)
> ~[artemis-amqp-protocol-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.protocol.amqp.converter.coreWrapper.CoreBytesMessageWrapper.getBinaryFromMessageBody(CoreBytesMessageWrapper.java:68)
> ~[artemis-amqp-protocol-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.protocol.amqp.converter.coreWrapper.CoreBytesMessageWrapper.createAMQPSection(CoreBytesMessageWrapper.java:78)
> ~[artemis-amqp-protocol-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.protocol.amqp.converter.CoreAmqpConverter.fromCore(CoreAmqpConverter.java:106)
> ~[artemis-amqp-protocol-2.28.0.jar:2.28.0]
> ... 10 more{noformat}
> I tried to reproduce the error with trace log level enabled, but the problem
> never occurs with trace enabled (same message payload). Maybe it is a timing
> issue which not occurs when Artemis processes messages much slower due to
> additional tracing...?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
