[jira] [Work logged] (ARTEMIS-4641) Allow AMQP federation to recover from missing or removed resources
[ https://issues.apache.org/jira/browse/ARTEMIS-4641?focusedWorklogId=904472=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-904472 ] ASF GitHub Bot logged work on ARTEMIS-4641: --- Author: ASF GitHub Bot Created on: 09/Feb/24 21:49 Start Date: 09/Feb/24 21:49 Worklog Time Spent: 10m Work Description: tabish121 commented on PR #4817: URL: https://github.com/apache/activemq-artemis/pull/4817#issuecomment-1936649789 I've run the full test suite with these changes. Issue Time Tracking --- Worklog Id: (was: 904472) Time Spent: 20m (was: 10m) > Allow AMQP federation to recover from missing or removed resources > -- > > Key: ARTEMIS-4641 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4641 > Project: ActiveMQ Artemis > Issue Type: Improvement > Components: AMQP >Affects Versions: 2.32.0 >Reporter: Timothy A. Bish >Assignee: Timothy A. Bish >Priority: Major > Fix For: 2.33.0 > > Time Spent: 20m > Remaining Estimate: 0h > > When an AMQP federation instance attempts to federate an address or queue it > can fail if the remote address or queue is not present or cannot be created > based on broker policy. A federation link can also closed if the federated > resource is removed from the remote broker by management etc. In those cases > the remote broker should note the resources that were targets of federation > and send alerts to the source federation broker to notify it that these > resources become available for federation and the source should attempt again > to create federation links if demand still exists. This allows an AMQP > federation instance to heal itself based on updates from the remote. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4641) Allow AMQP federation to recover from missing or removed resources
[ https://issues.apache.org/jira/browse/ARTEMIS-4641?focusedWorklogId=904471=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-904471 ] ASF GitHub Bot logged work on ARTEMIS-4641: --- Author: ASF GitHub Bot Created on: 09/Feb/24 21:48 Start Date: 09/Feb/24 21:48 Worklog Time Spent: 10m Work Description: tabish121 opened a new pull request, #4817: URL: https://github.com/apache/activemq-artemis/pull/4817 When an AMQP federation instance attempts to federate an address or queue it can fail if the remote address or queue is not present or cannot be created based on broker policy. A federation link can also closed if the federated resource is removed from the remote broker by management etc. In those cases the remote broker should note the resources that were targets of federation and send alerts to the source federation broker to notify it that these resources become available for federation and the source should attempt again to create federation links if demand still exists. This allows an AMQP federation instance to heal itself based on updates from the remote. Issue Time Tracking --- Worklog Id: (was: 904471) Remaining Estimate: 0h Time Spent: 10m > Allow AMQP federation to recover from missing or removed resources > -- > > Key: ARTEMIS-4641 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4641 > Project: ActiveMQ Artemis > Issue Type: Improvement > Components: AMQP >Affects Versions: 2.32.0 >Reporter: Timothy A. Bish >Assignee: Timothy A. Bish >Priority: Major > Fix For: 2.33.0 > > Time Spent: 10m > Remaining Estimate: 0h > > When an AMQP federation instance attempts to federate an address or queue it > can fail if the remote address or queue is not present or cannot be created > based on broker policy. A federation link can also closed if the federated > resource is removed from the remote broker by management etc. In those cases > the remote broker should note the resources that were targets of federation > and send alerts to the source federation broker to notify it that these > resources become available for federation and the source should attempt again > to create federation links if demand still exists. This allows an AMQP > federation instance to heal itself based on updates from the remote. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (ARTEMIS-4641) Allow AMQP federation to recover from missing or removed resources
Timothy A. Bish created ARTEMIS-4641: Summary: Allow AMQP federation to recover from missing or removed resources Key: ARTEMIS-4641 URL: https://issues.apache.org/jira/browse/ARTEMIS-4641 Project: ActiveMQ Artemis Issue Type: Improvement Components: AMQP Affects Versions: 2.32.0 Reporter: Timothy A. Bish Assignee: Timothy A. Bish Fix For: 2.33.0 When an AMQP federation instance attempts to federate an address or queue it can fail if the remote address or queue is not present or cannot be created based on broker policy. A federation link can also closed if the federated resource is removed from the remote broker by management etc. In those cases the remote broker should note the resources that were targets of federation and send alerts to the source federation broker to notify it that these resources become available for federation and the source should attempt again to create federation links if demand still exists. This allows an AMQP federation instance to heal itself based on updates from the remote. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (ARTEMIS-4217) AMQ111005: Failed to convert message. Sending it to Dead Letter Address.
[
https://issues.apache.org/jira/browse/ARTEMIS-4217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17816222#comment-17816222
]
daves commented on ARTEMIS-4217:
Thank you [~jbertram] and [~tabish] for finding the cause and helping me.
Changing amqpMinLargeMessageSize or journal-buffer-size works. I cannot
reproduce the problem with correct settings.
> AMQ111005: Failed to convert message. Sending it to Dead Letter Address.
>
>
> Key: ARTEMIS-4217
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4217
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: AMQP
>Affects Versions: 2.28.0
> Environment: Windows Server 2022 21H2
> openjdk 19.0.1 2022-10-18
> OpenJDK Runtime Environment (build 19.0.1+10-21)
> OpenJDK 64-Bit Server VM (build 19.0.1+10-21, mixed mode, sharing)
>Reporter: daves
>Priority: Major
> Attachments: ArtemisConvertError.zip, ArtemisReproMemory.zip,
> image-2023-06-16-15-59-25-689.png, image-2023-06-16-15-59-25-721.png
>
>
> Some of the AMQP messages sent by my client never arrive at the consumer. In
> the Artemis log I found the following exception:
> {noformat}
> 2023-03-23 18:06:58,084 WARN
> [org.apache.activemq.artemis.protocol.amqp.logger] AMQ111005: Failed to
> convert message. Sending it to Dead Letter Address.
> org.apache.activemq.artemis.protocol.amqp.converter.coreWrapper.ConversionException:
> java.nio.channels.ClosedChannelException
> at
> org.apache.activemq.artemis.protocol.amqp.converter.CoreAmqpConverter.fromCore(CoreAmqpConverter.java:318)
> ~[artemis-amqp-protocol-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.protocol.amqp.converter.CoreAmqpConverter.checkAMQP(CoreAmqpConverter.java:79)
> ~[artemis-amqp-protocol-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.protocol.amqp.proton.ProtonServerSenderContext.executeDelivery(ProtonServerSenderContext.java:561)
> ~[artemis-amqp-protocol-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.core.server.impl.MessageReferenceImpl.run(MessageReferenceImpl.java:131)
> ~[artemis-server-2.28.0.jar:2.28.0]
> at
> io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
> ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> at
> io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
> ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
> ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
> ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
> ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> at
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> ~[artemis-commons-2.28.0.jar:?] Caused by: java.lang.RuntimeException:
> java.nio.channels.ClosedChannelException
> at
> org.apache.activemq.artemis.core.persistence.impl.journal.LargeBody.getBodyBufferSize(LargeBody.java:293)
> ~[artemis-server-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl.getBodyBufferSize(LargeServerMessageImpl.java:263)
> ~[artemis-server-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.protocol.amqp.converter.coreWrapper.CoreBytesMessageWrapper.getBodyLength(CoreBytesMessageWrapper.java:98)
> ~[artemis-amqp-protocol-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.protocol.amqp.converter.coreWrapper.CoreBytesMessageWrapper.getBinaryFromMessageBody(CoreBytesMessageWrapper.java:68)
> ~[artemis-amqp-protocol-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.protocol.amqp.converter.coreWrapper.CoreBytesMessageWrapper.createAMQPSection(CoreBytesMessageWrapper.java:78)
> ~[artemis-amqp-protocol-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.protocol.amqp.converter.CoreAmqpConverter.fromCore(CoreAmqpConverter.java:106)
> ~[artemis-amqp-protocol-2.28.0.jar:2.28.0]
> ... 10 more
> Caused by: java.nio.channels.ClosedChannelException
> at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:165)
> ~[?:?]
> at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:416) ~[?:?]
> at
> org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.size(NIOSequentialFile.java:339)
> ~[artemis-journal-2.28.0.jar:2.28.0]
[jira] [Comment Edited] (ARTEMIS-4217) AMQ111005: Failed to convert message. Sending it to Dead Letter Address.
[
https://issues.apache.org/jira/browse/ARTEMIS-4217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17816212#comment-17816212
]
Justin Bertram edited comment on ARTEMIS-4217 at 2/9/24 8:45 PM:
-
I was able to reproduce the problem with a relatively simple test in the
test-suite based on the information that [~tabish] provided in his previous
comment. The conversion from AMQP to core is happening because you're sending a
message whose size is less than {{amqpMinLargeMessageSize}} but larger than the
{{journal-buffer-size}}. Therefore, since the broker can't fit the message into
the journal buffer it converts it into a _core_ large message and stores it
directly to disk. Then the broker converts it back to AMQP to send it to your
consumer. This is when/where the exceptions are happening.
Ideally the broker wouldn't covert the message to a _core_ large message but
would instead keep it as an AMQP large message. I don't yet have code that will
do that. However, I believe you can avoid the issue by adjusting your
configuration. Currently you're using an {{amqpMinLargeMessageSize}} of
{{500}} (i.e. 5MB) and the default {{journal-buffer-size}} of {{501760}}
(i.e. 490KiB). You almost certainly don't want to be storing messages which are
close to 5MB in memory so I recommend you lower your
{{amqpMinLargeMessageSize}} to something a bit more reasonable. The default is
{{102400}} (i.e. 100KiB) so maybe double it to be {{204800}}. If you find
that's still too low for your liking then feel free to increase it and the
{{journal-buffer-size}} if necessary as well. However, keep in mind that the
larger the {{amqpMinLargeMessageSize}} and {{journal-buffer-size}} is the more
heap the broker will consume.
Let me know if that resolves the issue for you.
was (Author: jbertram):
I was able to reproduce the problem with a relatively simple test in the
test-suite based on the information that [~tabish] provided in his previous
comment. The conversion from AMQP to core is happening because you're sending a
message whose size is less than {{amqpMinLargeMessageSize}} but larger than the
{{journal-buffer-size}}. Therefore, since the broker can't fit the message into
the journal buffer it convert it into a _core_ large message and stores it
directly to disk. Then the broker converts it back to AMQP to send it to your
consumer. This is when/where the exceptions are happening.
Ideally the broker wouldn't covert the message to a _core_ large message but
would instead keep it as an AMQP large message. I don't yet have code that will
do that. However, I believe you can avoid the issue by adjusting your
configuration. Currently you're using an {{amqpMinLargeMessageSize}} of
{{500}} (i.e. 5MB) and the default {{journal-buffer-size}} of {{501760}}
(i.e. 490KiB). You almost certainly don't want to be storing messages which are
close to 5MB in memory so I recommend you lower your
{{amqpMinLargeMessageSize}} to something a bit more reasonable. The default is
{{102400}} (i.e. 100KiB) so maybe double it to be {{204800}}. If you find
that's still too low for your liking then feel free to increase it and the
{{journal-buffer-size}} if necessary as well. However, keep in mind that the
larger the {{amqpMinLargeMessageSize}} and {{journal-buffer-size}} is the more
heap the broker will consume.
Let me know if that resolves the issue for you.
> AMQ111005: Failed to convert message. Sending it to Dead Letter Address.
>
>
> Key: ARTEMIS-4217
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4217
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: AMQP
>Affects Versions: 2.28.0
> Environment: Windows Server 2022 21H2
> openjdk 19.0.1 2022-10-18
> OpenJDK Runtime Environment (build 19.0.1+10-21)
> OpenJDK 64-Bit Server VM (build 19.0.1+10-21, mixed mode, sharing)
>Reporter: daves
>Priority: Major
> Attachments: ArtemisConvertError.zip, ArtemisReproMemory.zip,
> image-2023-06-16-15-59-25-689.png, image-2023-06-16-15-59-25-721.png
>
>
> Some of the AMQP messages sent by my client never arrive at the consumer. In
> the Artemis log I found the following exception:
> {noformat}
> 2023-03-23 18:06:58,084 WARN
> [org.apache.activemq.artemis.protocol.amqp.logger] AMQ111005: Failed to
> convert message. Sending it to Dead Letter Address.
> org.apache.activemq.artemis.protocol.amqp.converter.coreWrapper.ConversionException:
> java.nio.channels.ClosedChannelException
> at
> org.apache.activemq.artemis.protocol.amqp.converter.CoreAmqpConverter.fromCore(CoreAmqpConverter.java:318)
> ~[artemis-amqp-protocol-2.28.0.jar:2.28.0]
> at
>
[jira] [Commented] (ARTEMIS-4217) AMQ111005: Failed to convert message. Sending it to Dead Letter Address.
[
https://issues.apache.org/jira/browse/ARTEMIS-4217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17816212#comment-17816212
]
Justin Bertram commented on ARTEMIS-4217:
-
I was able to reproduce the problem with a relatively simple test in the
test-suite based on the information that [~tabish] provided in his previous
comment. The conversion from AMQP to core is happening because you're sending a
message whose size is less than {{amqpMinLargeMessageSize}} but larger than the
{{journal-buffer-size}}. Therefore, since the broker can't fit the message into
the journal buffer it convert it into a _core_ large message and stores it
directly to disk. Then the broker converts it back to AMQP to send it to your
consumer. This is when/where the exceptions are happening.
Ideally the broker wouldn't covert the message to a _core_ large message but
would instead keep it as an AMQP large message. I don't yet have code that will
do that. However, I believe you can avoid the issue by adjusting your
configuration. Currently you're using an {{amqpMinLargeMessageSize}} of
{{500}} (i.e. 5MB) and the default {{journal-buffer-size}} of {{501760}}
(i.e. 490KiB). You almost certainly don't want to be storing messages which are
close to 5MB in memory so I recommend you lower your
{{amqpMinLargeMessageSize}} to something a bit more reasonable. The default is
{{102400}} (i.e. 100KiB) so maybe double it to be {{204800}}. If you find
that's still too low for your liking then feel free to increase it and the
{{journal-buffer-size}} if necessary as well. However, keep in mind that the
larger the {{amqpMinLargeMessageSize}} and {{journal-buffer-size}} is the more
heap the broker will consume.
Let me know if that resolves the issue for you.
> AMQ111005: Failed to convert message. Sending it to Dead Letter Address.
>
>
> Key: ARTEMIS-4217
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4217
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: AMQP
>Affects Versions: 2.28.0
> Environment: Windows Server 2022 21H2
> openjdk 19.0.1 2022-10-18
> OpenJDK Runtime Environment (build 19.0.1+10-21)
> OpenJDK 64-Bit Server VM (build 19.0.1+10-21, mixed mode, sharing)
>Reporter: daves
>Priority: Major
> Attachments: ArtemisConvertError.zip, ArtemisReproMemory.zip,
> image-2023-06-16-15-59-25-689.png, image-2023-06-16-15-59-25-721.png
>
>
> Some of the AMQP messages sent by my client never arrive at the consumer. In
> the Artemis log I found the following exception:
> {noformat}
> 2023-03-23 18:06:58,084 WARN
> [org.apache.activemq.artemis.protocol.amqp.logger] AMQ111005: Failed to
> convert message. Sending it to Dead Letter Address.
> org.apache.activemq.artemis.protocol.amqp.converter.coreWrapper.ConversionException:
> java.nio.channels.ClosedChannelException
> at
> org.apache.activemq.artemis.protocol.amqp.converter.CoreAmqpConverter.fromCore(CoreAmqpConverter.java:318)
> ~[artemis-amqp-protocol-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.protocol.amqp.converter.CoreAmqpConverter.checkAMQP(CoreAmqpConverter.java:79)
> ~[artemis-amqp-protocol-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.protocol.amqp.proton.ProtonServerSenderContext.executeDelivery(ProtonServerSenderContext.java:561)
> ~[artemis-amqp-protocol-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.core.server.impl.MessageReferenceImpl.run(MessageReferenceImpl.java:131)
> ~[artemis-server-2.28.0.jar:2.28.0]
> at
> io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
> ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> at
> io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
> ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
> ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
> ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
> ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> at
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> ~[artemis-commons-2.28.0.jar:?] Caused by: java.lang.RuntimeException:
> java.nio.channels.ClosedChannelException
> at
>
[jira] [Comment Edited] (ARTEMIS-4217) AMQ111005: Failed to convert message. Sending it to Dead Letter Address.
[
https://issues.apache.org/jira/browse/ARTEMIS-4217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17816194#comment-17816194
]
daves edited comment on ARTEMIS-4217 at 2/9/24 7:32 PM:
Hi [~jbertram]
I planned to upload the file together with a comment, but I got interrupted
while writing the comment, sorry.
I did some more testing and found a way to reproduce the problem rellably.
For the problem to occure the system running the broker must be in a low memory
state. I've added a small project to the issue to reproduce the problem.
Setup:
Broker: The problem exists since version 2.28 I used 2.31.2 for this test.
2.27.1 is the last version in which the problem never occurs.
Broker config:
{code:xml}
http://www.w3.org/2001/XMLSchema-instance;
xmlns:xi="http://www.w3.org/2001/XInclude; xsi:schemaLocation="urn:activemq
/schema/artemis-configuration.xsd">
http://www.w3.org/2001/XMLSchema-instance;
xsi:schemaLocation="urn:activemq:core ">
Argos
true
NIO
C:/Data/Artemis/Argos/data/paging
C:/Data/Artemis/Argos/data/bindings
C:/Data/Artemis/Argos/data/journal
C:/Data/Artemis/Argos/data/large-messages
true
2
10
4096
10M
1404000
1
5000
99
true
12
6
HALT
1196000
tcp://0.0.0.0:5672?tcpSendBufferSize=2000;tcpReceiveBufferSize=2000;minLargeMessageSize=500;amqpMinLargeMessageSize=500;consumerWindowSize=10485760;directDeliver=false;protocols=AMQP;useEpoll=true;amqpCredits=1000;amqpMinCredits=300
0
-1
10
PAGE
true
true
true
true
1.5
5000
6
10
10485760
104857600
2
PAGE
10
true
true
true
true
false
false
false
false
{code}
Repro tool:
Extract ArtemisReproMemory.zip and run the project.
It is a .net application and should run on all major platforms, but I tested it
only on Windows (11) . To build and run it, you need the .net8.0 SDK from here:
https://dotnet.microsoft.com/en-us/download/visual-studio-sdks
Run this command to build the project (in the root of the extracted directory):
{code:bash}
dotnet build -c Release
{code}
Start the built application like this:
{code:bash}
cd .\ArtemisReproMemory\bin\Release\net8.0\
dotnet .\ArtemisReproMemory.dll -a amqp://admin:password@localhost:5672 -b 10
{code}
Use the -a switch to specify a broker URL (AMQP)
Use -b to allocate the specified number of bytes to simulate a high memory
state. (Only makes sense if the broker is running on the same machine)
The repro tool will send large messages to the specified broker using the AMQP
protocol and simultaneously consume the messages. The actions of the repro tool
trigger the problem (
Failed to convert message. Sending it to Dead Letter Address.) on all machines
I tested with.
In some cases, it took considerably longer for the problem to occur, if the
broker was started before the system memory was all "in use". So it can help to
start the broker after the repro tool is up and running (using the -b flag).
Artemis will, using this setup, generate the following log entries and loose
messages:
{code:java}
2024-02-09 10:30:17,320 WARN [org.apache.activemq.artemis.core.server]
AMQ222150: Sending message
Reference[8590002935]:NON-RELIABLE:LargeServerMessage[messageID=8590002935,durable=false,userID=d5f605da-c72d-11ee-891c-a059505d02d7,priority=4,
timestamp=0,expiration=0, durable=false, address=test-q,
properties=TypedProperties[NATIVE_MESSAGE_ID=ID:AMQP_NO_PREFIX:test-producer
#18, JMS_AMQP_ORIGINAL_ENCODING=6, _AMQ_LARGE_SIZE=1048576]]@964273977 to Dead
Letter Address, but there is no Dead Letter Address configured for queue test-q
so dropping it
2024-02-09 10:30:29,752 WARN
[org.apache.activemq.artemis.protocol.amqp.logger] AMQ111005: Failed to convert
message. Sending it to Dead Letter Address.
org.apache.activemq.artemis.protocol.amqp.converter.coreWrapper.ConversionException:
java.nio.channels.AsynchronousCloseException
at
org.apache.activemq.artemis.protocol.amqp.converter.CoreAmqpConverter.fromCore(CoreAmqpConverter.java:318)
~[artemis-amqp-protocol-2.31.2.jar:2.31.2]
at
org.apache.activemq.artemis.protocol.amqp.converter.CoreAmqpConverter.checkAMQP(CoreAmqpConverter.java:79)
~[artemis-amqp-protocol-2.31.2.jar:2.31.2]
at
org.apache.activemq.artemis.protocol.amqp.proton.ProtonServerSenderContext.executeDelivery(ProtonServerSenderContext.java:572)
~[artemis-amqp-protocol-2.31.2.jar:2.31.2]
at
[jira] [Comment Edited] (ARTEMIS-4217) AMQ111005: Failed to convert message. Sending it to Dead Letter Address.
[
https://issues.apache.org/jira/browse/ARTEMIS-4217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17816194#comment-17816194
]
daves edited comment on ARTEMIS-4217 at 2/9/24 7:30 PM:
Hi [~jbertram]
I planned to upload the file together with a comment, but I got interrupted
while writing the comment, sorry.
I did some more testing and found a way to reproduce the problem rellably.
For the problem to occure the system running the broker must be in a low memory
state. I've added a small project to the issue to reproduce the problem.
Setup:
Broker: The problem exists since version 2.28 I used 2.31.2 for this test.
2.27.1 is the last version in which the problem never occurs.
Broker config:
{code:xml}
http://www.w3.org/2001/XMLSchema-instance;
xmlns:xi="http://www.w3.org/2001/XInclude; xsi:schemaLocation="urn:activemq
/schema/artemis-configuration.xsd">
http://www.w3.org/2001/XMLSchema-instance;
xsi:schemaLocation="urn:activemq:core ">
Argos
true
NIO
C:/Data/Artemis/Argos/data/paging
C:/Data/Artemis/Argos/data/bindings
C:/Data/Artemis/Argos/data/journal
C:/Data/Artemis/Argos/data/large-messages
true
2
10
4096
10M
1404000
1
5000
99
true
12
6
HALT
1196000
tcp://0.0.0.0:5672?tcpSendBufferSize=2000;tcpReceiveBufferSize=2000;minLargeMessageSize=500;amqpMinLargeMessageSize=500;consumerWindowSize=10485760;directDeliver=false;protocols=AMQP;useEpoll=true;amqpCredits=1000;amqpMinCredits=300
0
-1
10
PAGE
true
true
true
true
1.5
5000
6
10
10485760
104857600
2
PAGE
10
true
true
true
true
false
false
false
false
{code}
Repro tool:
Extract ArtemisReproMemory.zip and run the project.
It is a .net application and should run on all major platforms, but I tested it
only on Windows (11) . To build and run it, you need the .net8.0 SDK from here:
https://dotnet.microsoft.com/en-us/download/visual-studio-sdks
Run this command to build the project (in the root of the extracted directory):
{code:cmd}
dotnet build -c Release
{code}
Start the built application like this:
{code:cmd}
cd .\ArtemisReproMemory\bin\Release\net8.0\
dotnet .\ArtemisReproMemory.dll -a amqp://admin:password@localhost:5672 -b 10
{code}
Use the -a switch to specify a broker URL (AMQP)
Use -b to allocate the specified number of bytes to simulate a high memory
state. (Only makes sense if the broker is running on the same machine)
The repro tool will send large messages to the specified broker using the AMQP
protocol and simultaneously consume the messages. The actions of the repro tool
trigger the problem (
Failed to convert message. Sending it to Dead Letter Address.) on all machines
I tested with.
In some cases, it took considerably longer for the problem to occur, if the
broker was started before the system memory was all "in use". So it can help to
start the broker after the repro tool is up and running (using the -b flag).
Artemis will, using this setup, generate the following log entries and loose
messages:
{code:java}
2024-02-09 10:30:17,320 WARN [org.apache.activemq.artemis.core.server]
AMQ222150: Sending message
Reference[8590002935]:NON-RELIABLE:LargeServerMessage[messageID=8590002935,durable=false,userID=d5f605da-c72d-11ee-891c-a059505d02d7,priority=4,
timestamp=0,expiration=0, durable=false, address=test-q,
properties=TypedProperties[NATIVE_MESSAGE_ID=ID:AMQP_NO_PREFIX:test-producer
#18, JMS_AMQP_ORIGINAL_ENCODING=6, _AMQ_LARGE_SIZE=1048576]]@964273977 to Dead
Letter Address, but there is no Dead Letter Address configured for queue test-q
so dropping it
2024-02-09 10:30:29,752 WARN
[org.apache.activemq.artemis.protocol.amqp.logger] AMQ111005: Failed to convert
message. Sending it to Dead Letter Address.
org.apache.activemq.artemis.protocol.amqp.converter.coreWrapper.ConversionException:
java.nio.channels.AsynchronousCloseException
at
org.apache.activemq.artemis.protocol.amqp.converter.CoreAmqpConverter.fromCore(CoreAmqpConverter.java:318)
~[artemis-amqp-protocol-2.31.2.jar:2.31.2]
at
org.apache.activemq.artemis.protocol.amqp.converter.CoreAmqpConverter.checkAMQP(CoreAmqpConverter.java:79)
~[artemis-amqp-protocol-2.31.2.jar:2.31.2]
at
org.apache.activemq.artemis.protocol.amqp.proton.ProtonServerSenderContext.executeDelivery(ProtonServerSenderContext.java:572)
~[artemis-amqp-protocol-2.31.2.jar:2.31.2]
at
[jira] [Commented] (ARTEMIS-4217) AMQ111005: Failed to convert message. Sending it to Dead Letter Address.
[
https://issues.apache.org/jira/browse/ARTEMIS-4217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17816194#comment-17816194
]
daves commented on ARTEMIS-4217:
Hi [~jbertram]
I planned to upload the file together with a comment, but I got interrupted
while writing the comment, sorry.
I did some more testing and found a way to reproduce the problem rellably.
For the problem to occure the system running the broker must be in a low memory
state. I've added a small project to the issue to reproduce the problem.
Setup:
Broker: The problem exists since version 2.28 I used 2.31.2 for this test.
2.27.1 is the last version in which the problem never occurs.
Broker config:
{code:xml}
http://www.w3.org/2001/XMLSchema-instance;
xmlns:xi="http://www.w3.org/2001/XInclude; xsi:schemaLocation="urn:activemq
/schema/artemis-configuration.xsd">
http://www.w3.org/2001/XMLSchema-instance;
xsi:schemaLocation="urn:activemq:core ">
Argos
true
NIO
C:/Logisoft/Data/Artemis/Argos/data/paging
C:/Logisoft/Data/Artemis/Argos/data/bindings
C:/Logisoft/Data/Artemis/Argos/data/journal
C:/Logisoft/Data/Artemis/Argos/data/large-messages
true
2
10
4096
10M
1404000
1
5000
99
true
12
6
HALT
1196000
tcp://0.0.0.0:5672?tcpSendBufferSize=2000;tcpReceiveBufferSize=2000;minLargeMessageSize=500;amqpMinLargeMessageSize=500;consumerWindowSize=10485760;directDeliver=false;protocols=AMQP;useEpoll=true;amqpCredits=1000;amqpMinCredits=300
0
-1
10
PAGE
true
true
true
true
1.5
5000
6
10
10485760
104857600
2
PAGE
10
true
true
true
true
false
false
false
false
{code}
Repro tool:
Extract ArtemisReproMemory.zip and run the project.
It is a .net application and should run on all major platforms, but I tested it
only on Windows (11) . To build and run it, you need the .net8.0 SDK from here:
https://dotnet.microsoft.com/en-us/download/visual-studio-sdks
Run this command to build the project (in the root of the extracted directory):
{code:cmd}
dotnet build -c Release
{code}
Start the built application like this:
{code:cmd}
cd .\ArtemisReproMemory\bin\Release\net8.0\
dotnet .\ArtemisReproMemory.dll -a amqp://admin:password@localhost:5672 -b 10
{code}
Use the -a switch to specify a broker URL (AMQP)
Use -b to allocate the specified number of bytes to simulate a high memory
state. (Only makes sense if the broker is running on the same machine)
The repro tool will send large messages to the specified broker using the AMQP
protocol and simultaneously consume the messages. The actions of the repro tool
trigger the problem (
Failed to convert message. Sending it to Dead Letter Address.) on all machines
I tested with.
In some cases, it took considerably longer for the problem to occur, if the
broker was started before the system memory was all "in use". So it can help to
start the broker after the repro tool is up and running (using the -b flag).
Artemis will, using this setup, generate the following log entries and loose
messages:
{code:java}
2024-02-09 10:30:17,320 WARN [org.apache.activemq.artemis.core.server]
AMQ222150: Sending message
Reference[8590002935]:NON-RELIABLE:LargeServerMessage[messageID=8590002935,durable=false,userID=d5f605da-c72d-11ee-891c-a059505d02d7,priority=4,
timestamp=0,expiration=0, durable=false, address=test-q,
properties=TypedProperties[NATIVE_MESSAGE_ID=ID:AMQP_NO_PREFIX:test-producer
#18, JMS_AMQP_ORIGINAL_ENCODING=6, _AMQ_LARGE_SIZE=1048576]]@964273977 to Dead
Letter Address, but there is no Dead Letter Address configured for queue test-q
so dropping it
2024-02-09 10:30:29,752 WARN
[org.apache.activemq.artemis.protocol.amqp.logger] AMQ111005: Failed to convert
message. Sending it to Dead Letter Address.
org.apache.activemq.artemis.protocol.amqp.converter.coreWrapper.ConversionException:
java.nio.channels.AsynchronousCloseException
at
org.apache.activemq.artemis.protocol.amqp.converter.CoreAmqpConverter.fromCore(CoreAmqpConverter.java:318)
~[artemis-amqp-protocol-2.31.2.jar:2.31.2]
at
org.apache.activemq.artemis.protocol.amqp.converter.CoreAmqpConverter.checkAMQP(CoreAmqpConverter.java:79)
~[artemis-amqp-protocol-2.31.2.jar:2.31.2]
at
org.apache.activemq.artemis.protocol.amqp.proton.ProtonServerSenderContext.executeDelivery(ProtonServerSenderContext.java:572)
~[artemis-amqp-protocol-2.31.2.jar:2.31.2]
at
[jira] [Created] (AMQ-9433) Add configurable advanced timing metrics to Destination Policy for queues and topics
Matt Pavlovich created AMQ-9433: --- Summary: Add configurable advanced timing metrics to Destination Policy for queues and topics Key: AMQ-9433 URL: https://issues.apache.org/jira/browse/AMQ-9433 Project: ActiveMQ Issue Type: Improvement Reporter: Matt Pavlovich Tracking timestamps for every message in and out could incur performance impact with all the System.. get millis or nanos time stamps. This feature enhancement would add a configurable flag 'enableTimingStatistics' to destination policy and then only track those statistics when the flag is enabled. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4586) Auto reload web binding SSL stores on change
[ https://issues.apache.org/jira/browse/ARTEMIS-4586?focusedWorklogId=904439=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-904439 ] ASF GitHub Bot logged work on ARTEMIS-4586: --- Author: ASF GitHub Bot Created on: 09/Feb/24 17:21 Start Date: 09/Feb/24 17:21 Worklog Time Spent: 10m Work Description: gemmellr commented on PR #4760: URL: https://github.com/apache/activemq-artemis/pull/4760#issuecomment-1936304096 Yep update looks good, merged. Issue Time Tracking --- Worklog Id: (was: 904439) Time Spent: 1h (was: 50m) > Auto reload web binding SSL stores on change > > > Key: ARTEMIS-4586 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4586 > Project: ActiveMQ Artemis > Issue Type: Improvement >Reporter: Domenico Francesco Bruscino >Assignee: Domenico Francesco Bruscino >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > Add the web binding attribute `sslAutoReload` to enable auto reload of the > SSL stores related to a binding when they are updated. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (ARTEMIS-4586) Auto reload web binding SSL stores on change
[ https://issues.apache.org/jira/browse/ARTEMIS-4586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17816157#comment-17816157 ] ASF subversion and git services commented on ARTEMIS-4586: -- Commit 29781bd5da6f9892a6fb80c2da5dd7eb3c0e4ebd in activemq-artemis's branch refs/heads/main from Domenico Francesco Bruscino [ https://gitbox.apache.org/repos/asf?p=activemq-artemis.git;h=29781bd5da ] ARTEMIS-4586 Auto reload web binding SSL stores on change > Auto reload web binding SSL stores on change > > > Key: ARTEMIS-4586 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4586 > Project: ActiveMQ Artemis > Issue Type: Improvement >Reporter: Domenico Francesco Bruscino >Assignee: Domenico Francesco Bruscino >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > > Add the web binding attribute `sslAutoReload` to enable auto reload of the > SSL stores related to a binding when they are updated. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4586) Auto reload web binding SSL stores on change
[ https://issues.apache.org/jira/browse/ARTEMIS-4586?focusedWorklogId=904438=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-904438 ] ASF GitHub Bot logged work on ARTEMIS-4586: --- Author: ASF GitHub Bot Created on: 09/Feb/24 17:20 Start Date: 09/Feb/24 17:20 Worklog Time Spent: 10m Work Description: gemmellr merged PR #4760: URL: https://github.com/apache/activemq-artemis/pull/4760 Issue Time Tracking --- Worklog Id: (was: 904438) Time Spent: 50m (was: 40m) > Auto reload web binding SSL stores on change > > > Key: ARTEMIS-4586 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4586 > Project: ActiveMQ Artemis > Issue Type: Improvement >Reporter: Domenico Francesco Bruscino >Assignee: Domenico Francesco Bruscino >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > > Add the web binding attribute `sslAutoReload` to enable auto reload of the > SSL stores related to a binding when they are updated. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4586) Auto reload web binding SSL stores on change
[ https://issues.apache.org/jira/browse/ARTEMIS-4586?focusedWorklogId=904436=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-904436 ] ASF GitHub Bot logged work on ARTEMIS-4586: --- Author: ASF GitHub Bot Created on: 09/Feb/24 17:13 Start Date: 09/Feb/24 17:13 Worklog Time Spent: 10m Work Description: gemmellr commented on PR #4760: URL: https://github.com/apache/activemq-artemis/pull/4760#issuecomment-1936294026 didnt get notified it had been updated, so havent looked yet Issue Time Tracking --- Worklog Id: (was: 904436) Time Spent: 40m (was: 0.5h) > Auto reload web binding SSL stores on change > > > Key: ARTEMIS-4586 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4586 > Project: ActiveMQ Artemis > Issue Type: Improvement >Reporter: Domenico Francesco Bruscino >Assignee: Domenico Francesco Bruscino >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > Add the web binding attribute `sslAutoReload` to enable auto reload of the > SSL stores related to a binding when they are updated. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4586) Auto reload web binding SSL stores on change
[ https://issues.apache.org/jira/browse/ARTEMIS-4586?focusedWorklogId=904428=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-904428 ] ASF GitHub Bot logged work on ARTEMIS-4586: --- Author: ASF GitHub Bot Created on: 09/Feb/24 16:37 Start Date: 09/Feb/24 16:37 Worklog Time Spent: 10m Work Description: jbertram commented on PR #4760: URL: https://github.com/apache/activemq-artemis/pull/4760#issuecomment-1936240743 @gemmellr, this OK to merge? Issue Time Tracking --- Worklog Id: (was: 904428) Time Spent: 0.5h (was: 20m) > Auto reload web binding SSL stores on change > > > Key: ARTEMIS-4586 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4586 > Project: ActiveMQ Artemis > Issue Type: Improvement >Reporter: Domenico Francesco Bruscino >Assignee: Domenico Francesco Bruscino >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > Add the web binding attribute `sslAutoReload` to enable auto reload of the > SSL stores related to a binding when they are updated. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (ARTEMIS-4217) AMQ111005: Failed to convert message. Sending it to Dead Letter Address.
[
https://issues.apache.org/jira/browse/ARTEMIS-4217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17816114#comment-17816114
]
Justin Bertram commented on ARTEMIS-4217:
-
[~daves], I see you attached [^ArtemisReproMemory.zip]. Can you provide any
details about what this is for?
> AMQ111005: Failed to convert message. Sending it to Dead Letter Address.
>
>
> Key: ARTEMIS-4217
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4217
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: AMQP
>Affects Versions: 2.28.0
> Environment: Windows Server 2022 21H2
> openjdk 19.0.1 2022-10-18
> OpenJDK Runtime Environment (build 19.0.1+10-21)
> OpenJDK 64-Bit Server VM (build 19.0.1+10-21, mixed mode, sharing)
>Reporter: daves
>Priority: Major
> Attachments: ArtemisConvertError.zip, ArtemisReproMemory.zip,
> image-2023-06-16-15-59-25-689.png, image-2023-06-16-15-59-25-721.png
>
>
> Some of the AMQP messages sent by my client never arrive at the consumer. In
> the Artemis log I found the following exception:
> {noformat}
> 2023-03-23 18:06:58,084 WARN
> [org.apache.activemq.artemis.protocol.amqp.logger] AMQ111005: Failed to
> convert message. Sending it to Dead Letter Address.
> org.apache.activemq.artemis.protocol.amqp.converter.coreWrapper.ConversionException:
> java.nio.channels.ClosedChannelException
> at
> org.apache.activemq.artemis.protocol.amqp.converter.CoreAmqpConverter.fromCore(CoreAmqpConverter.java:318)
> ~[artemis-amqp-protocol-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.protocol.amqp.converter.CoreAmqpConverter.checkAMQP(CoreAmqpConverter.java:79)
> ~[artemis-amqp-protocol-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.protocol.amqp.proton.ProtonServerSenderContext.executeDelivery(ProtonServerSenderContext.java:561)
> ~[artemis-amqp-protocol-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.core.server.impl.MessageReferenceImpl.run(MessageReferenceImpl.java:131)
> ~[artemis-server-2.28.0.jar:2.28.0]
> at
> io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
> ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> at
> io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
> ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
> ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
> ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
> ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> at
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> ~[artemis-commons-2.28.0.jar:?] Caused by: java.lang.RuntimeException:
> java.nio.channels.ClosedChannelException
> at
> org.apache.activemq.artemis.core.persistence.impl.journal.LargeBody.getBodyBufferSize(LargeBody.java:293)
> ~[artemis-server-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl.getBodyBufferSize(LargeServerMessageImpl.java:263)
> ~[artemis-server-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.protocol.amqp.converter.coreWrapper.CoreBytesMessageWrapper.getBodyLength(CoreBytesMessageWrapper.java:98)
> ~[artemis-amqp-protocol-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.protocol.amqp.converter.coreWrapper.CoreBytesMessageWrapper.getBinaryFromMessageBody(CoreBytesMessageWrapper.java:68)
> ~[artemis-amqp-protocol-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.protocol.amqp.converter.coreWrapper.CoreBytesMessageWrapper.createAMQPSection(CoreBytesMessageWrapper.java:78)
> ~[artemis-amqp-protocol-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.protocol.amqp.converter.CoreAmqpConverter.fromCore(CoreAmqpConverter.java:106)
> ~[artemis-amqp-protocol-2.28.0.jar:2.28.0]
> ... 10 more
> Caused by: java.nio.channels.ClosedChannelException
> at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:165)
> ~[?:?]
> at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:416) ~[?:?]
> at
> org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.size(NIOSequentialFile.java:339)
> ~[artemis-journal-2.28.0.jar:2.28.0]
> at
>
[jira] [Work logged] (ARTEMIS-4639) Artemis Cluster and NullpointerException in "sending topology"
[
https://issues.apache.org/jira/browse/ARTEMIS-4639?focusedWorklogId=904374=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-904374
]
ASF GitHub Bot logged work on ARTEMIS-4639:
---
Author: ASF GitHub Bot
Created on: 09/Feb/24 10:55
Start Date: 09/Feb/24 10:55
Worklog Time Spent: 10m
Work Description: gemmellr merged PR #4814:
URL: https://github.com/apache/activemq-artemis/pull/4814
Issue Time Tracking
---
Worklog Id: (was: 904374)
Time Spent: 50m (was: 40m)
> Artemis Cluster and NullpointerException in "sending topology"
> --
>
> Key: ARTEMIS-4639
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4639
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: Broker, Clustering
>Affects Versions: 2.32.0
>Reporter: Jean-Pascal Briquet
>Assignee: Justin Bertram
>Priority: Major
> Fix For: 2.33.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> A {{NullPointerException}} may appear while the "sending topology" operation
> is being executed.
> *Configuration:*
> Artemis clustering enabled with static connectors using the option
> {{{}allow-direct-connections-only="true"{}}}.
> *Broker logs:*
> {noformat}
> 2024-02-06 22:06:14,259 WARN [org.apache.activemq.artemis.core.client]
> AMQ212030: error sending topology
> java.lang.NullPointerException: Cannot invoke
> "org.apache.activemq.artemis.api.core.TransportConfiguration.newTransportConfig(String)"
> because the return value of
> "org.apache.activemq.artemis.api.core.client.TopologyMember.getLive()" is null
> at
> org.apache.activemq.artemis.core.server.cluster.impl.ClusterConnectionImpl.nodeUP(ClusterConnectionImpl.java:755)
> ~[artemis-server-2.30.0.jar:2.30.0]
> at
> org.apache.activemq.artemis.core.client.impl.Topology$1.run(Topology.java:284)
> ~[artemis-core-client-2.30.0.jar:2.30.0]
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
> ~[artemis-commons-2.30.0.jar:?]
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
> ~[artemis-commons-2.30.0.jar:?]
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
> ~[artemis-commons-2.30.0.jar:?]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> [?:?]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> [?:?]
> at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> [artemis-commons-2.30.0.jar:?]
> 2024-02-06 22:06:14,260 WARN [org.apache.activemq.artemis.core.client]
> AMQ212030: error sending topology
> java.lang.NullPointerException: Cannot invoke
> "org.apache.activemq.artemis.api.core.TransportConfiguration.getParams()"
> because "liveConnector" is null
> at
> org.apache.activemq.artemis.core.client.impl.TopologyMemberImpl.toURI(TopologyMemberImpl.java:133)
> ~[artemis-core-client-2.30.0.jar:2.30.0]
> at
> org.apache.activemq.artemis.core.protocol.openwire.OpenWireProtocolManager.generateMembersURI(OpenWireProtocolManager.java:536)
> ~[artemis-openwire-protocol-2.30.0.jar:2.30.0]
> at
> org.apache.activemq.artemis.core.protocol.openwire.OpenWireProtocolManager.newConnectionControl(OpenWireProtocolManager.java:521)
> ~[artemis-openwire-protocol-2.30.0.jar:2.30.0]
> at
> org.apache.activemq.artemis.core.protocol.openwire.OpenWireProtocolManager.updateClientClusterInfo(OpenWireProtocolManager.java:288)
> ~[artemis-openwire-protocol-2.30.0.jar:2.30.0]
> at
> org.apache.activemq.artemis.core.protocol.openwire.OpenWireProtocolManager.nodeUP(OpenWireProtocolManager.java:236)
> ~[artemis-openwire-protocol-2.30.0.jar:2.30.0]
> at
> org.apache.activemq.artemis.core.client.impl.Topology$1.run(Topology.java:284)
> ~[artemis-core-client-2.30.0.jar:2.30.0]
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
> ~[artemis-commons-2.30.0.jar:?]
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
> ~[artemis-commons-2.30.0.jar:?]
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
> ~[artemis-commons-2.30.0.jar:?]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> [?:?]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> [?:?]
> at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
>
[jira] [Commented] (ARTEMIS-4639) Artemis Cluster and NullpointerException in "sending topology"
[
https://issues.apache.org/jira/browse/ARTEMIS-4639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17816014#comment-17816014
]
ASF subversion and git services commented on ARTEMIS-4639:
--
Commit 20840cfdf1de78bf6a53aa9a98510cb52456b81a in activemq-artemis's branch
refs/heads/main from Justin Bertram
[ https://gitbox.apache.org/repos/asf?p=activemq-artemis.git;h=20840cfdf1 ]
ARTEMIS-4639 NPEs when TopologyMember's primary is null
> Artemis Cluster and NullpointerException in "sending topology"
> --
>
> Key: ARTEMIS-4639
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4639
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: Broker, Clustering
>Affects Versions: 2.32.0
>Reporter: Jean-Pascal Briquet
>Assignee: Justin Bertram
>Priority: Major
> Fix For: 2.33.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> A {{NullPointerException}} may appear while the "sending topology" operation
> is being executed.
> *Configuration:*
> Artemis clustering enabled with static connectors using the option
> {{{}allow-direct-connections-only="true"{}}}.
> *Broker logs:*
> {noformat}
> 2024-02-06 22:06:14,259 WARN [org.apache.activemq.artemis.core.client]
> AMQ212030: error sending topology
> java.lang.NullPointerException: Cannot invoke
> "org.apache.activemq.artemis.api.core.TransportConfiguration.newTransportConfig(String)"
> because the return value of
> "org.apache.activemq.artemis.api.core.client.TopologyMember.getLive()" is null
> at
> org.apache.activemq.artemis.core.server.cluster.impl.ClusterConnectionImpl.nodeUP(ClusterConnectionImpl.java:755)
> ~[artemis-server-2.30.0.jar:2.30.0]
> at
> org.apache.activemq.artemis.core.client.impl.Topology$1.run(Topology.java:284)
> ~[artemis-core-client-2.30.0.jar:2.30.0]
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
> ~[artemis-commons-2.30.0.jar:?]
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
> ~[artemis-commons-2.30.0.jar:?]
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
> ~[artemis-commons-2.30.0.jar:?]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> [?:?]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> [?:?]
> at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> [artemis-commons-2.30.0.jar:?]
> 2024-02-06 22:06:14,260 WARN [org.apache.activemq.artemis.core.client]
> AMQ212030: error sending topology
> java.lang.NullPointerException: Cannot invoke
> "org.apache.activemq.artemis.api.core.TransportConfiguration.getParams()"
> because "liveConnector" is null
> at
> org.apache.activemq.artemis.core.client.impl.TopologyMemberImpl.toURI(TopologyMemberImpl.java:133)
> ~[artemis-core-client-2.30.0.jar:2.30.0]
> at
> org.apache.activemq.artemis.core.protocol.openwire.OpenWireProtocolManager.generateMembersURI(OpenWireProtocolManager.java:536)
> ~[artemis-openwire-protocol-2.30.0.jar:2.30.0]
> at
> org.apache.activemq.artemis.core.protocol.openwire.OpenWireProtocolManager.newConnectionControl(OpenWireProtocolManager.java:521)
> ~[artemis-openwire-protocol-2.30.0.jar:2.30.0]
> at
> org.apache.activemq.artemis.core.protocol.openwire.OpenWireProtocolManager.updateClientClusterInfo(OpenWireProtocolManager.java:288)
> ~[artemis-openwire-protocol-2.30.0.jar:2.30.0]
> at
> org.apache.activemq.artemis.core.protocol.openwire.OpenWireProtocolManager.nodeUP(OpenWireProtocolManager.java:236)
> ~[artemis-openwire-protocol-2.30.0.jar:2.30.0]
> at
> org.apache.activemq.artemis.core.client.impl.Topology$1.run(Topology.java:284)
> ~[artemis-core-client-2.30.0.jar:2.30.0]
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
> ~[artemis-commons-2.30.0.jar:?]
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
> ~[artemis-commons-2.30.0.jar:?]
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
> ~[artemis-commons-2.30.0.jar:?]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> [?:?]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> [?:?]
> at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> [artemis-commons-2.30.0.jar:?]{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (ARTEMIS-4639) Artemis Cluster and NullpointerException in "sending topology"
[
https://issues.apache.org/jira/browse/ARTEMIS-4639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robbie Gemmell resolved ARTEMIS-4639.
-
Fix Version/s: 2.33.0
Resolution: Fixed
> Artemis Cluster and NullpointerException in "sending topology"
> --
>
> Key: ARTEMIS-4639
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4639
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: Broker, Clustering
>Affects Versions: 2.32.0
>Reporter: Jean-Pascal Briquet
>Assignee: Justin Bertram
>Priority: Major
> Fix For: 2.33.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> A {{NullPointerException}} may appear while the "sending topology" operation
> is being executed.
> *Configuration:*
> Artemis clustering enabled with static connectors using the option
> {{{}allow-direct-connections-only="true"{}}}.
> *Broker logs:*
> {noformat}
> 2024-02-06 22:06:14,259 WARN [org.apache.activemq.artemis.core.client]
> AMQ212030: error sending topology
> java.lang.NullPointerException: Cannot invoke
> "org.apache.activemq.artemis.api.core.TransportConfiguration.newTransportConfig(String)"
> because the return value of
> "org.apache.activemq.artemis.api.core.client.TopologyMember.getLive()" is null
> at
> org.apache.activemq.artemis.core.server.cluster.impl.ClusterConnectionImpl.nodeUP(ClusterConnectionImpl.java:755)
> ~[artemis-server-2.30.0.jar:2.30.0]
> at
> org.apache.activemq.artemis.core.client.impl.Topology$1.run(Topology.java:284)
> ~[artemis-core-client-2.30.0.jar:2.30.0]
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
> ~[artemis-commons-2.30.0.jar:?]
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
> ~[artemis-commons-2.30.0.jar:?]
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
> ~[artemis-commons-2.30.0.jar:?]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> [?:?]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> [?:?]
> at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> [artemis-commons-2.30.0.jar:?]
> 2024-02-06 22:06:14,260 WARN [org.apache.activemq.artemis.core.client]
> AMQ212030: error sending topology
> java.lang.NullPointerException: Cannot invoke
> "org.apache.activemq.artemis.api.core.TransportConfiguration.getParams()"
> because "liveConnector" is null
> at
> org.apache.activemq.artemis.core.client.impl.TopologyMemberImpl.toURI(TopologyMemberImpl.java:133)
> ~[artemis-core-client-2.30.0.jar:2.30.0]
> at
> org.apache.activemq.artemis.core.protocol.openwire.OpenWireProtocolManager.generateMembersURI(OpenWireProtocolManager.java:536)
> ~[artemis-openwire-protocol-2.30.0.jar:2.30.0]
> at
> org.apache.activemq.artemis.core.protocol.openwire.OpenWireProtocolManager.newConnectionControl(OpenWireProtocolManager.java:521)
> ~[artemis-openwire-protocol-2.30.0.jar:2.30.0]
> at
> org.apache.activemq.artemis.core.protocol.openwire.OpenWireProtocolManager.updateClientClusterInfo(OpenWireProtocolManager.java:288)
> ~[artemis-openwire-protocol-2.30.0.jar:2.30.0]
> at
> org.apache.activemq.artemis.core.protocol.openwire.OpenWireProtocolManager.nodeUP(OpenWireProtocolManager.java:236)
> ~[artemis-openwire-protocol-2.30.0.jar:2.30.0]
> at
> org.apache.activemq.artemis.core.client.impl.Topology$1.run(Topology.java:284)
> ~[artemis-core-client-2.30.0.jar:2.30.0]
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
> ~[artemis-commons-2.30.0.jar:?]
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
> ~[artemis-commons-2.30.0.jar:?]
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
> ~[artemis-commons-2.30.0.jar:?]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> [?:?]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> [?:?]
> at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> [artemis-commons-2.30.0.jar:?]{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (ARTEMIS-4217) AMQ111005: Failed to convert message. Sending it to Dead Letter Address.
[
https://issues.apache.org/jira/browse/ARTEMIS-4217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
daves updated ARTEMIS-4217:
---
Attachment: ArtemisReproMemory.zip
> AMQ111005: Failed to convert message. Sending it to Dead Letter Address.
>
>
> Key: ARTEMIS-4217
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4217
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: AMQP
>Affects Versions: 2.28.0
> Environment: Windows Server 2022 21H2
> openjdk 19.0.1 2022-10-18
> OpenJDK Runtime Environment (build 19.0.1+10-21)
> OpenJDK 64-Bit Server VM (build 19.0.1+10-21, mixed mode, sharing)
>Reporter: daves
>Priority: Major
> Attachments: ArtemisConvertError.zip, ArtemisReproMemory.zip,
> image-2023-06-16-15-59-25-689.png, image-2023-06-16-15-59-25-721.png
>
>
> Some of the AMQP messages sent by my client never arrive at the consumer. In
> the Artemis log I found the following exception:
> {noformat}
> 2023-03-23 18:06:58,084 WARN
> [org.apache.activemq.artemis.protocol.amqp.logger] AMQ111005: Failed to
> convert message. Sending it to Dead Letter Address.
> org.apache.activemq.artemis.protocol.amqp.converter.coreWrapper.ConversionException:
> java.nio.channels.ClosedChannelException
> at
> org.apache.activemq.artemis.protocol.amqp.converter.CoreAmqpConverter.fromCore(CoreAmqpConverter.java:318)
> ~[artemis-amqp-protocol-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.protocol.amqp.converter.CoreAmqpConverter.checkAMQP(CoreAmqpConverter.java:79)
> ~[artemis-amqp-protocol-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.protocol.amqp.proton.ProtonServerSenderContext.executeDelivery(ProtonServerSenderContext.java:561)
> ~[artemis-amqp-protocol-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.core.server.impl.MessageReferenceImpl.run(MessageReferenceImpl.java:131)
> ~[artemis-server-2.28.0.jar:2.28.0]
> at
> io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
> ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> at
> io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
> ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
> ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
> ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
> ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> at
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> ~[artemis-commons-2.28.0.jar:?] Caused by: java.lang.RuntimeException:
> java.nio.channels.ClosedChannelException
> at
> org.apache.activemq.artemis.core.persistence.impl.journal.LargeBody.getBodyBufferSize(LargeBody.java:293)
> ~[artemis-server-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl.getBodyBufferSize(LargeServerMessageImpl.java:263)
> ~[artemis-server-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.protocol.amqp.converter.coreWrapper.CoreBytesMessageWrapper.getBodyLength(CoreBytesMessageWrapper.java:98)
> ~[artemis-amqp-protocol-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.protocol.amqp.converter.coreWrapper.CoreBytesMessageWrapper.getBinaryFromMessageBody(CoreBytesMessageWrapper.java:68)
> ~[artemis-amqp-protocol-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.protocol.amqp.converter.coreWrapper.CoreBytesMessageWrapper.createAMQPSection(CoreBytesMessageWrapper.java:78)
> ~[artemis-amqp-protocol-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.protocol.amqp.converter.CoreAmqpConverter.fromCore(CoreAmqpConverter.java:106)
> ~[artemis-amqp-protocol-2.28.0.jar:2.28.0]
> ... 10 more
> Caused by: java.nio.channels.ClosedChannelException
> at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:165)
> ~[?:?]
> at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:416) ~[?:?]
> at
> org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.size(NIOSequentialFile.java:339)
> ~[artemis-journal-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.core.persistence.impl.journal.LargeBody.getBodyBufferSize(LargeBody.java:285)
> ~[artemis-server-2.28.0.jar:2.28.0]
> at
>
