Trying out master results in weird issues:
Broker A logs a very large amount of this:
10:06:37,292 ERROR [org.apache.activemq.artemis.core.client] AMQ214031: Failed
to decode buffer, disconnect immediately.: java.lang.IllegalStateException:
java.lang.IllegalArgumentException: AMQ119032: Invalid type: 1
at
org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:378)
[artemis-core-client-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at
org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1144)
[artemis-core-client-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at
org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:68)
[artemis-core-client-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
[netty-all-4.1.22.Final.jar:4.1.22.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
[netty-all-4.1.22.Final.jar:4.1.22.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
[netty-all-4.1.22.Final.jar:4.1.22.Final]
at
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
[netty-all-4.1.22.Final.jar:4.1.22.Final]
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)
[netty-all-4.1.22.Final.jar:4.1.22.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
[netty-all-4.1.22.Final.jar:4.1.22.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
[netty-all-4.1.22.Final.jar:4.1.22.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
[netty-all-4.1.22.Final.jar:4.1.22.Final]
at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1414)
[netty-all-4.1.22.Final.jar:4.1.22.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
[netty-all-4.1.22.Final.jar:4.1.22.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
[netty-all-4.1.22.Final.jar:4.1.22.Final]
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:945)
[netty-all-4.1.22.Final.jar:4.1.22.Final]
at
io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:806)
[netty-all-4.1.22.Final.jar:4.1.22.Final]
at
io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:404)
[netty-all-4.1.22.Final.jar:4.1.22.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:304)
[netty-all-4.1.22.Final.jar:4.1.22.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
[netty-all-4.1.22.Final.jar:4.1.22.Final]
at
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
[artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
Caused by: java.lang.IllegalArgumentException: AMQ119032: Invalid type: 1
at
org.apache.activemq.artemis.core.protocol.core.impl.PacketDecoder.decode(PacketDecoder.java:455)
[artemis-core-client-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at
org.apache.activemq.artemis.core.protocol.ClientPacketDecoder.decode(ClientPacketDecoder.java:67)
[artemis-core-client-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at
org.apache.activemq.artemis.core.protocol.ServerPacketDecoder.slowPathDecode(ServerPacketDecoder.java:253)
[artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at
org.apache.activemq.artemis.core.protocol.ServerPacketDecoder.decode(ServerPacketDecoder.java:133)
[artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at
org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:365)
[artemis-core-client-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
... 19 more
Broker B logs this once (note, no other process listens to addresses defined in
its acceptors:
10:15:17,703 ERROR [org.apache.activemq.artemis.core.server] AMQ224000: Failure
in initialisation: io.netty.channel.unix.Errors$NativeIoException: bind(..)
failed: Address already in use
at io.netty.channel.unix.Errors.newIOException(Errors.java:122)
[netty-all-4.1.22.Final.jar:4.1.22.Final]
at io.netty.channel.unix.Socket.bind(Socket.java:287)
[netty-all-4.1.22.Final.jar:4.1.22.Final]
at
io.netty.channel.epoll.AbstractEpollChannel.doBind(AbstractEpollChannel.java:687)
[netty-all-4.1.22.Final.jar:4.1.22.Final]
at
io.netty.channel.epoll.EpollServerSocketChannel.doBind(EpollServerSocketChannel.java:70)
[netty-all-4.1.22.Final.jar:4.1.22.Final]
at
io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:558)
[netty-all-4.1.22.Final.jar:4.1.22.Final]
at
io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1338)
[netty-all-4.1.22.Final.jar:4.1.22.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:501)
[netty-all-4.1.22.Final.jar:4.1.22.Final]
at
io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:486)
[netty-all-4.1.22.Final.jar:4.1.22.Final]
at
io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:999)
[netty-all-4.1.22.Final.jar:4.1.22.Final]
at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:254)
[netty-all-4.1.22.Final.jar:4.1.22.Final]
at
io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:366)
[netty-all-4.1.22.Final.jar:4.1.22.Final]
at
io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
[netty-all-4.1.22.Final.jar:4.1.22.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
[netty-all-4.1.22.Final.jar:4.1.22.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:309)
[netty-all-4.1.22.Final.jar:4.1.22.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
[netty-all-4.1.22.Final.jar:4.1.22.Final]
at
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
[artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
All clients connecting to brokers log a significant amount of:
08:21:01.992 [Thread-2 (ActiveMQ-client-netty-threads-1829548907)] ERROR
o.a.activemq.artemis.core.client - AMQ214013: Failed to decode packet
java.lang.IllegalArgumentException: AMQ119032: Invalid type: 1
at
org.apache.activemq.artemis.core.protocol.core.impl.PacketDecoder.decode(PacketDecoder.java:424)
at
org.apache.activemq.artemis.core.protocol.ClientPacketDecoder.decode(ClientPacketDecoder.java:60)
at
org.apache.activemq.artemis.core.protocol.ClientPacketDecoder.decode(ClientPacketDecoder.java:39)
at
org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:349)
at
org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1143)
at
org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:68)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:372)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:358)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:350)
at
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:372)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:358)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:350)
at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:372)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:358)
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:129)
at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:610)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:551)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:465)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:437)
at
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873)
at java.lang.Thread.run(Thread.java:748)
And netstat on broker b server lists massive amounts of tcp connections from
local port 61616 to broker a host with seemingly random ports in TIME_WAIT
status. It seems like something fundamental is broken causing all tcp packets
to fail somehow?
Best regards,
- Ilkka
-----Original Message-----
From: Clebert Suconic [mailto:clebert.suco...@gmail.com]
Sent: 8. maaliskuuta 2018 2:08
To: users@activemq.apache.org
Subject: Re: Artemis 2.4.0 - Issues with memory leaks and JMS message
redistribution
Can you try master?
On Wed, Mar 7, 2018 at 2:11 AM, Ilkka Virolainen <ilkka.virolai...@bitwise.fi>
wrote:
> My setup is the same as described in issue #1 of the first message in this
> thread: NMS clients listening to JMS topics for messages sent by Artemis
> 1.5.4 clients, both connecting randomly to either of the two brokers in the
> cluster. Load balancing works at first but eventually this error occurs and
> the cluster consumer seems to be removed.
>
> I will try to produce a test case reproducing this.
>
> Best regards,
> - Ilkka
>
> -----Original Message-----
> From: Clebert Suconic [mailto:clebert.suco...@gmail.com]
> Sent: 7. maaliskuuta 2018 4:23
> To: users@activemq.apache.org
> Subject: Re: Artemis 2.4.0 - Issues with memory leaks and JMS message
> redistribution
>
> How do you run into this ?
>
> Are you running amqp messages?
>
> I was going to release 2.5 tomorrow but I wanted to find out first.
>
>
> I am in USA and about to sleep. If you are in APAC please include as much
> info as you can for me to replicate this so I can make progress tomorrow.
>
>
> Thanks.
>
> On Tue, Mar 6, 2018 at 8:40 AM Ilkka Virolainen
> <ilkka.virolai...@bitwise.fi>
> wrote:
>
>> While the memory leak is now fixed in master (thanks Justin!),
>> unfortunately more issues regarding the load balancing have occurred.
>> The messages are redistributed at first but after a few minutes the
>> broker
>> (2.5.0 current snapshot) logs the following error and load balancing
>> of topic messages stop. I found an issue that lists similar errors
>> but it is marked as closed:
>>
>>
>> https://issues.apache.org/jira/browse/ARTEMIS-1345?jql=project%20%3D%
>> 2 0ARTEMIS%20AND%20text%20~%20NoSuchElementException
>>
>> Does this error seem familiar?
>>
>> 15:06:08,000 WARN [org.apache.activemq.artemis.core.server] AMQ222151:
>> removing consumer which did not handle a message,
>> consumer=ClusterConnectionBridge@6a50dbc1
>> [name=$.artemis.internal.sf.my-cluster.73a86a21-2066-11e8-a8bc-0021f6
>> e
>> 7cd2d,
>> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.73a86a21-2066-1
>> 1
>> e8-a8bc-0021f6e7cd2d,
>> postOffice=PostOfficeImpl
>> [server=ActiveMQServerImpl::serverUUID=ae2a7067-2066-11e8-ba18-005056
>> b
>> c41e7],
>> temp=false]@46ca2a06 targetConnector=ServerLocatorImpl
>> (identity=(Cluster-connection-bridge::ClusterConnectionBridge@6a50dbc
>> 1
>> [name=$.artemis.internal.sf.my-cluster.73a86a21-2066-11e8-a8bc-0021f6
>> e
>> 7cd2d,
>> queue=QueueImpl[name=$.artemis.internal.sf.my-cluster.73a86a21-2066-1
>> 1
>> e8-a8bc-0021f6e7cd2d,
>> postOffice=PostOfficeImpl
>> [server=ActiveMQServerImpl::serverUUID=ae2a7067-2066-11e8-ba18-005056
>> b
>> c41e7],
>> temp=false]@46ca2a06 targetConnector=ServerLocatorImpl
>> [initialConnectors=[TransportConfiguration(name=netty-connector,
>> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyCon
>> n
>> ectorFactory)
>> ?port=61616&host=broker-a-host&activemq-passwordcodec=****],
>> discoveryGroupConfiguration=null]]::ClusterConnectionImpl@2143139988[
>> n odeUUID=ae2a7067-2066-11e8-ba18-005056bc41e7,
>> connector=TransportConfiguration(name=netty-connector,
>> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyCon
>> n
>> ectorFactory)
>> ?port=61616&host=broker-b-host&activemq-passwordcodec=****, address=,
>> server=ActiveMQServerImpl::serverUUID=ae2a7067-2066-11e8-ba18-005056b
>> c
>> 41e7]))
>> [initialConnectors=[TransportConfiguration(name=netty-connector,
>> factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyCon
>> n
>> ectorFactory)
>> ?port=61616&host=broker-a-host&activemq-passwordcodec=****],
>> discoveryGroupConfiguration=null]],
>> message=Reference[19136928]:NON-RELIABLE:CoreMessage[messageID=191369
>> 2 8,durable=false,userID=null,priority=0,
>> timestamp=0,expiration=0, durable=false,
>> address=ActiveMQ.Advisory.TempQueue,size=1089,properties=TypedPropert
>> i
>> es[__HDR_BROKER_IN_TIME=1520341567999,_AMQ_ROUTING_TYPE=0,__HDR_GROUP
>> _
>> SEQUENCE=0,__HDR_COMMAND_ID=0,__HDR_DATASTRUCTURE=[0000
>> 0062 0800 0000 0000 0178 0100 2449 443A 616C 3232 2D35 3833 3735 2D36 ...
>> 3233 2D38 3031 372D 6330 3564 3235 3339 3365 3364 0100 0000 0000 0000
>> 0000),_AMQ_DUPL_ID=ID:livisovt49l-28730-1520248604578-1:2:0:0:1212,__
>> H
>> DR_MESSAGE_ID=[0000 004D 6E00 017B 0100 2649 443A 6C69 7669 736F 7674
>> 3439 6C2D 3238 3733 ...
>> 00 0000 0000 0000 0000 0000 0000 0000 0000 0000 0004 BC00 0000 0000
>> 0000
>> 00),__HDR_DROPPABLE=false,__HDR_ARRIVAL=0,__HDR_PRODUCER_ID=[0000
>> 003A
>> 7B01
>> 0026 4944 3A6C 6976 6973 6F76 7434 396C 2D32 3837 3330 2D31 ...
>> 3032
>> 3438
>> 3630 3435 3738 2D31 3A32 0000 0000 0000 0000 0000 0000 0000
>> 0000),JMSType=Advisory,_AMQ_ROUTE_TO$.artemis.internal.sf.my-cluster.
>> 7
>> 3a86a21-2066-11e8-a8bc-0021f6e7cd2d=[0000
>> 0000 009B AA9C 0000 0000 009B
>> AA9E),bytesAsLongs(10201756,10201758]]]@709863295:
>> java.util.ConcurrentModificationException
>> at java.util.HashMap$HashIterator.nextNode(HashMap.java:1442)
>> [rt.jar:1.8.0_152]
>> at java.util.HashMap$EntryIterator.next(HashMap.java:1476)
>> [rt.jar:1.8.0_152]
>> at java.util.HashMap$EntryIterator.next(HashMap.java:1474)
>> [rt.jar:1.8.0_152]
>> at java.util.HashMap.putMapEntries(HashMap.java:512)
>> [rt.jar:1.8.0_152]
>> at java.util.HashMap.<init>(HashMap.java:490) [rt.jar:1.8.0_152]
>> at
>> org.apache.activemq.artemis.utils.collections.TypedProperties.<init>(
>> T
>> ypedProperties.java:83)
>> [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
>> at
>> org.apache.activemq.artemis.core.message.impl.CoreMessage.<init>(Core
>> M
>> essage.java:347)
>> [artemis-core-client-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
>> at
>> org.apache.activemq.artemis.core.message.impl.CoreMessage.<init>(Core
>> M
>> essage.java:321)
>> [artemis-core-client-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
>> at
>> org.apache.activemq.artemis.core.message.impl.CoreMessage.copy(CoreMe
>> s
>> sage.java:374) [artemis-core-client-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
>> at
>> org.apache.activemq.artemis.core.server.cluster.impl.ClusterConnectio
>> n
>> Bridge.beforeForward(ClusterConnectionBridge.java:168)
>> [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
>> at
>> org.apache.activemq.artemis.core.server.cluster.impl.BridgeImpl.handl
>> e
>> (BridgeImpl.java:581)
>> [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
>> at
>> org.apache.activemq.artemis.core.server.impl.QueueImpl.handle(QueueIm
>> p
>> l.java:2939) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
>> at
>> org.apache.activemq.artemis.core.server.impl.QueueImpl.deliver(QueueI
>> m
>> pl.java:2309) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
>> at
>> org.apache.activemq.artemis.core.server.impl.QueueImpl.access$2000(Qu
>> e
>> ueImpl.java:105) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
>> at
>> org.apache.activemq.artemis.core.server.impl.QueueImpl$DeliverRunner.
>> r
>> un(QueueImpl.java:3165)
>> [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
>> at
>> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(Order
>> e
>> dExecutor.java:42) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
>> at
>> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(Order
>> e
>> dExecutor.java:31) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
>> at
>> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePending
>> T
>> asks(ProcessorBase.java:66)
>> [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
>> j
>> ava:1149)
>> [rt.jar:1.8.0_152]
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.
>> java:624)
>> [rt.jar:1.8.0_152]
>> at
>> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveM
>> Q
>> ThreadFactory.java:118)
>> [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
>>
>> 15:06:08,002 ERROR [org.apache.activemq.artemis.core.server] AMQ224041:
>> Failed to deliver: java.util.NoSuchElementException
>> at
>> org.apache.activemq.artemis.utils.collections.PriorityLinkedListImpl$
>> P
>> riorityLinkedListIterator.repeat(PriorityLinkedListImpl.java:161)
>> [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
>> at
>> org.apache.activemq.artemis.core.server.impl.QueueImpl.deliver(QueueI
>> m
>> pl.java:2327) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
>> at
>> org.apache.activemq.artemis.core.server.impl.QueueImpl.access$2000(Qu
>> e
>> ueImpl.java:105) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
>> at
>> org.apache.activemq.artemis.core.server.impl.QueueImpl$DeliverRunner.
>> r
>> un(QueueImpl.java:3165)
>> [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
>> at
>> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(Order
>> e
>> dExecutor.java:42) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
>> at
>> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(Order
>> e
>> dExecutor.java:31) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
>> at
>> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePending
>> T
>> asks(ProcessorBase.java:66)
>> [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
>> j
>> ava:1149)
>> [rt.jar:1.8.0_152]
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.
>> java:624)
>> [rt.jar:1.8.0_152]
>> at
>> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveM
>> Q
>> ThreadFactory.java:118)
>> [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
>>
>>
>>
>>
>>
>> -----Original Message-----
>> From: Ilkka Virolainen [mailto:ilkka.virolai...@bitwise.fi]
>> Sent: 1. maaliskuuta 2018 16:59
>> To: users@activemq.apache.org
>> Subject: RE: Artemis 2.4.0 - Issues with memory leaks and JMS message
>> redistribution
>>
>> An update on this: I have replicated the memory and expiration issues
>> on current 2.5.0-SNAPSHOT with included client libraries and a one
>> node broker by modifying an existing artemis example. As messages are
>> routed to DLQ, paged and expired, memory consumption keeps increasing
>> and eventually leads to heap space exhaustion rendering the broker unable to
>> route messages.
>> What should happen is that the memory consumption should stay
>> reasonable even without expiration due to paging to disk but doubly
>> so because having expired, the messages shouldn't consume any resources.
>>
>> I'm not certain if the two issues (erroneous statistics on expiration
>> and the memory leak) are connected but they both do appear at the
>> same time raising suspicion. A possible cause could be that filtered
>> message expiration behaves differently than some other means of
>> expiration: it uses a private expiration method that takes a
>> transaction as a parameter. Unlike the nontransacted expiration
>> method, it checks for empty bindings separately but doesn't seem to
>> decrement counters appropriately in this case. Even though I have set
>> a null expiry-address (<expiry-address />) it is seen as nonnull in
>> expiration. Then as the expiry address is not null but bindings are
>> not found, the warning about dropping the message is logged. However,
>> it seems that the message is never acknowledged and the deliveringCount is
>> never decreased so delivery metrics end up being wrong.
>>
>> Shouldn't there be an acknowledgment of the message reference
>> following the logging when the following condition is matched?
>>
>> https://github.com/apache/activemq-artemis/blob/master/artemis-server
>> /
>> src/main/java/org/apache/activemq/artemis/core/server/impl/QueueImpl.
>> j
>> ava#L2735
>>
>> Also, why is the acknowledgment reason here not expiry but normal?
>> One would imagine it should be acknowledge(tx, ref,
>> AckReason.EXPIRED) instead of the default overload so that the
>> appropriate counters end up being
>> incremented:
>>
>> https://github.com/apache/activemq-artemis/blob/master/artemis-server
>> /
>> src/main/java/org/apache/activemq/artemis/core/server/impl/QueueImpl.
>> j
>> ava#L2747
>>
>> Best regards,
>> - Ilkka
>>
>> -----Original Message-----
>> From: Ilkka Virolainen [mailto:ilkka.virolai...@bitwise.fi]
>> Sent: 27. helmikuuta 2018 15:20
>> To: users@activemq.apache.org
>> Subject: RE: Artemis 2.4.0 - Issues with memory leaks and JMS message
>> redistribution
>>
>> Hello,
>>
>> - I don't have consumers on the DLQ and neither are any listed in its
>> JMX attributes
>> - The messages are being sent to the DLQ by the broker after a
>> delivery failure on another queue. The delivery failure is expected
>> and caused by a transactional rollback on the consumer.
>> - I am setting the expiry delay on the broker's DLQ address-settings
>> (not in message attributes). I'm setting an empty expiry-address in
>> the same place.
>> - I have a set of broker settings and a small springboot application
>> with which I was able to replicate the issue. Would you like me to
>> provide it for you somehow?
>>
>> It seems like there's a some kind of hiccup in message expiration.
>> When the messages routed to the DLQ start expiring, the broker logs:
>>
>> AMQ222146: Message has expired. No bindings for Expiry Address so
>> dropping it
>>
>> but when reviewing the DLQ statistics via JMX the ExpiredMessages
>> counter is not incremented, but the DeliveringCount is. As messages
>> keep expiring the deliverincount keeps increasing. This feels a lot
>> like the issue I've been having. Could it be that this process leaks
>> memory/resources or is it just that the expiration statistics always
>> assume that expiration results in redelivery thereby causing erroneous
>> numbers to be reported?
>>
>> Best regards,
>> - Ilkka
>>
>>
>> -----Original Message-----
>> From: Justin Bertram [mailto:jbert...@apache.org]
>> Sent: 23. helmikuuta 2018 16:51
>> To: users@activemq.apache.org
>> Subject: Re: Artemis 2.4.0 - Issues with memory leaks and JMS message
>> redistribution
>>
>> Couple of questions:
>>
>> - Do you have any consumers on the DLQ?
>> - Are messages being sent to the DLQ by the broker automatically (e.g.
>> based on delivery attempt failures) or is that being done by your
>> application?
>> - How are you setting the expiry delay?
>> - Do you have a reproducible test-case?
>>
>>
>> Justin
>>
>> On Fri, Feb 23, 2018 at 4:38 AM, Ilkka Virolainen <
>> ilkka.virolai...@bitwise.fi> wrote:
>>
>> > I'm still facing an issue with somewhat confusing behavior
>> > regarding message expiration in the DLQ, maybe related to the
>> > memory issues I've been having. My aim is to have messages routed
>> > to DLQ expire and dropped in one hour. To achieve this, I've set an
>> > empty expiry-address and the appropriate expiry-delay. The problem
>> > is, most of the messages routed to DLQ end up in an in-delivery
>> > state - they are not expiring and I cannot remove them via JMX.
>> > Messagecount in the DLQ is slightly higher than the deliveringcount
>> > and attempting to remove all messages only removes a number of
>> > messages that is equal to the difference between deliveringcount
>> > and messagecount which is approximately a few thousand messages
>> > while the messagecount is tens of thousands and
>> increasing as message delivery failures occur.
>> >
>> > What could be the reason for this behavior and how could it be avoided?
>> >
>> > -----Original Message-----
>> > From: Ilkka Virolainen [mailto:ilkka.virolai...@bitwise.fi]
>> > Sent: 22. helmikuuta 2018 13:38
>> > To: users@activemq.apache.org
>> > Subject: RE: Artemis 2.4.0 - Issues with memory leaks and JMS
>> > message redistribution
>> >
>> > To answer my own question in case anyone else is wondering about a
>> > similar issue, turns out the change in addressing is referred in
>> > ticket [1] and adding the multicastPrefix and anycastPrefix
>> > described in the ticket to my broker acceptors seems to have fixed my
>> > problem.
>> > If the issue regarding memory leaks persists I will try to provide
>> > a
>> reproducible test case.
>> >
>> > Thank you for your help, Justin.
>> >
>> > Best regards,
>> > - Ilkka
>> >
>> > [1] https://issues.apache.org/jira/browse/ARTEMIS-1644
>> >
>> >
>> > -----Original Message-----
>> > From: Ilkka Virolainen [mailto:ilkka.virolai...@bitwise.fi]
>> > Sent: 22. helmikuuta 2018 12:33
>> > To: users@activemq.apache.org
>> > Subject: RE: Artemis 2.4.0 - Issues with memory leaks and JMS
>> > message redistribution
>> >
>> > Having removed the address configuration and having switched from
>> > 2.4.0 to yesterday's snapshot of 2.5.0 it seems like the
>> > redistribution of messages is now working, but there also seems to
>> > have been a change in addressing between the versions causing
>> > another problem related to jms.queue / jms.topic prefixing. While
>> > the NMS clients listen and artemis jms clients send to the same
>> > topics as described in the previous message, Artemis 2.5.0 prefixes
>> > the addresses with jms.topic. While the messages are being sent to e.g.
>> > A.B.f64dd592-a8fb-442e-826d-927834d566f4.C.D they are only received
>> > if I explicitly prefix the listening address with jms.topic, for
>> > example topic://jms.topic.A.B.*.C.D. Can this somehow be avoided in
>> > the broker
>> configuration?
>> >
>> > Best regards
>> >
>> > -----Original Message-----
>> > From: Justin Bertram [mailto:jbert...@apache.org]
>> > Sent: 21. helmikuuta 2018 15:19
>> > To: users@activemq.apache.org
>> > Subject: Re: Artemis 2.4.0 - Issues with memory leaks and JMS
>> > message redistribution
>> >
>> > Your first issue is probably a misconfiguration. Your
>> > cluster-connection is using an "address" value of '*' which I
>> > assume is supposed to mean "all addresses," but the "address"
>> > element doesn't
>> support wildcards like this.
>> > Just leave it empty to match all addresses. See the documentation
>> > [1] for more details.
>> >
>> > Even after you fix that configuration issue you may run into issues.
>> > These may be fixed already via ARTEMIS-1523 and/or ARTEMIS-1680.
>> > If you have a reproducible test-case then you can verify using the
>> > head of the master branch.
>> >
>> > For the memory issue it would be helpful to have some heap dumps or
>> > something to actually see what's actually consuming the memory.
>> > Better yet would be a reproducible test-case. Do you have either?
>> >
>> >
>> > Justin
>> >
>> > [1] https://activemq.apache.org/artemis/docs/latest/clusters.html
>> >
>> >
>> >
>> > On Wed, Feb 21, 2018 at 5:39 AM, Ilkka Virolainen <
>> > ilkka.virolai...@bitwise.fi> wrote:
>> >
>> > > Hello,
>> > >
>> > > I am using Artemis 2.4.0 to broker messages through JMS
>> > > queues/topics between a set of clients. Some are Apache NMS 1.7.2
>> > > ActiveMQ clients and others are using Artemis JMS client 1.5.4
>> > > included in Spring Boot
>> > 1.5.3.
>> > > Broker topology is a symmetric cluster of two live nodes with
>> > > static connectors, both nodes having been setup as replicating
>> > > colocated backup pairs with scale down. I have two quite
>> > > frustrating issues at the
>> > moment:
>> > > message redistribution not working correctly and a memory leak
>> > > causing eventual thread death.
>> > >
>> > > ISSUE #1 - Message redistribution / load balancing not working:
>> > >
>> > > Client 1 (NMS) connects to broker a and starts listening, artemis
>> > > creates the following address:
>> > >
>> > > (Broker a):
>> > > A.B.*.C.D
>> > > |-queues
>> > > |-multicast
>> > > |-f64dd592-a8fb-442e-826d-927834d566f4
>> > >
>> > > Server 1 (artemis-jms-client) connects to broker b and sends a
>> > > message to
>> > > topic: A.B.f64dd592-a8fb-442e-826d-927834d566f4.C.D - this should
>> > > be routed to broker a since the corresponding queue has no
>> > > consumers on broker b (the queue does not exist). This however
>> > > does not happen and the client receives no messages. Broker b has
>> > > some other clients connected, causing similar (but not the same)
>> > > queues having been
>> created:
>> > >
>> > > (Broker b):
>> > > A.B.*.C.D
>> > > |-queues
>> > > |-multicast
>> > > |-1eb48079-7fd8-40e9-b822-bcc25695ced0
>> > > |-9f295257-c352-4ae6-b74b-d5994f330485
>> > >
>> > >
>> > > ISSUE #2: - Memory leak and eventual thread death
>> > >
>> > > Artemis broker has 4GB allocated heap space and global-max-size
>> > > is set up as half of that (being the default setting).
>> > > Address-full-policy is set to PAGE for all addresses and some
>> > > individual addresses have small max-size-bytes values set e.g.
>> > > 104857600. As far as I know the paging settings should limit
>> > > memory usage but what happens is that at times Artemis uses the
>> > > whole heap space, encounters an out of memory error and
>> > > dies:
>> > >
>> > > 05:39:29,510 WARN [org.eclipse.jetty.util.thread.QueuedThreadPool] :
>> > > java.lang.OutOfMemoryError: Java heap space
>> > > 05:39:16,646 WARN [io.netty.channel.ChannelInitializer] Failed
>> > > to initialize a channel. Closing: [id: ...]: java.lang.OutOfMemoryError:
>> > > Java heap space
>> > > 05:41:05,597 WARN
>> > > [org.eclipse.jetty.util.thread.QueuedThreadPool]
>> > > Unexpected thread death: org.eclipse.jetty.util.thread.
>> > > QueuedThreadPool$2@5ffaba31 in
>> > > qtp20111564{STARTED,8<=8<=200,i=2,q=0}
>> > >
>> > > Are these known issues in Artemis or misconfigurations in the brokers?
>> > >
>> > > The broker configurations are as follows. Broker b has an
>> > > identical configuration excluding that the cluster connector's
>> > > connector-ref and static-connector connector-ref refer to broker
>> > > b and broker a
>> > respectively.
>> > >
>> > > Best regards,
>> > >
>> > > broker.xml (broker a):
>> > >
>> > > <?xml version='1.0'?>
>> > > <configuration xmlns="urn:activemq" xmlns:xsi="http://www.w3.org/
>> > > 2001/XMLSchema-instance" xsi:schemaLocation="urn:activemq
>> > > /schema/artemis-configuration.xsd">
>> > > <core xmlns="urn:activemq:core" xmlns:xsi="http://www.w3.org/
>> > > 2001/XMLSchema-instance" xsi:schemaLocation="urn:activemq:core ">
>> > > <name>[broker-a-ip]</name>
>> > > <persistence-enabled>true</persistence-enabled>
>> > >
>> > > <journal-type>NIO</journal-type>
>> > >
>> > > <paging-directory>...</paging-directory>
>> > > <bindings-directory>...</bindings-directory>
>> > > <journal-directory>...</journal-directory>
>> > > <large-messages-directory>...</large-messages-directory>
>> > >
>> > > <journal-datasync>true</journal-datasync>
>> > > <journal-min-files>2</journal-min-files>
>> > > <journal-pool-files>-1</journal-pool-files>
>> > > <journal-buffer-timeout>788000</journal-buffer-timeout>
>> > > <disk-scan-period>5000</disk-scan-period>
>> > >
>> > > <max-disk-usage>97</max-disk-usage>
>> > >
>> > > <critical-analyzer>true</critical-analyzer>
>> > > <critical-analyzer-timeout>120000</critical-analyzer-timeout>
>> > > <critical-analyzer-check-period>60000</critical-
>> > > analyzer-check-period>
>> > > <critical-analyzer-policy>HALT</critical-analyzer-policy>
>> > >
>> > > <acceptors>
>> > > <acceptor name="invm-acceptor">vm://0</acceptor>
>> > > <acceptor name="artemis">tcp://0.0.0.0:61616</acceptor>
>> > > <acceptor
>> > > name="ssl">tcp://0.0.0.0:61617?sslEnabled=true;
>> > > keyStorePath=...;keyStorePassword=...</acceptor>
>> > > </acceptors>
>> > > <connectors>
>> > > <connector name="invm-connector">vm://0</connector>
>> > > <connector name="netty-connector">tcp://[
>> > > broker-a-ip]:61616</connector>
>> > > <connector name="broker-b-connector">[
>> > > broker-b-ip]:61616</connector>
>> > > </connectors>
>> > >
>> > > <cluster-connections>
>> > > <cluster-connection name="cluster-name">
>> > > <address>*</address>
>> > > <connector-ref>netty-connector</connector-ref>
>> > > <retry-interval>500</retry-interval>
>> > > <reconnect-attempts>5</reconnect-attempts>
>> > > <use-duplicate-detection>true</use-duplicate-detection>
>> > > <message-load-balancing>ON_DEMAND</message-load-
>> > balancing>
>> > > <max-hops>1</max-hops>
>> > > <static-connectors>
>> > > <connector-ref>broker-b-connector</connector-ref>
>> > > </static-connectors>
>> > > </cluster-connection>
>> > > </cluster-connections>
>> > >
>> > > <ha-policy>
>> > > <replication>
>> > > <colocated>
>> > >
>> > > <backup-request-retry-interval>5000</backup-request-
>> > > retry-interval>
>> > > <max-backups>3</max-backups>
>> > > <request-backup>true</request-backup>
>> > > <backup-port-offset>100</backup-port-offset>
>> > > <excludes>
>> > > <connector-ref>invm-connector</connector-ref>
>> > > <connector-ref>netty-connector</connector-ref>
>> > > </excludes>
>> > > <master>
>> > > <check-for-live-server>true</
>> > > check-for-live-server>
>> > > </master>
>> > > <slave>
>> > > <restart-backup>false</restart-backup>
>> > > <scale-down />
>> > > </slave>
>> > > </colocated>
>> > > </replication>
>> > > </ha-policy>
>> > >
>> > > <cluster-user>ARTEMIS.CLUSTER.ADMIN.USER</cluster-user>
>> > > <cluster-password>[the shared cluster
>> > > password]</cluster-password>
>> > >
>> > > <security-settings>
>> > > <security-setting match="#">
>> > > <permission type="createDurableQueue" roles="amq,
>> > > other-role" />
>> > > <permission type="deleteDurableQueue" roles="amq,
>> > > other-role" />
>> > > <permission type="createNonDurableQueue"
>> > > roles="amq, other-role" />
>> > > <permission type="createAddress" roles="amq,
>> other-role"
>> > />
>> > > <permission type="deleteNonDurableQueue"
>> > > roles="amq, other-role" />
>> > > <permission type="deleteAddress" roles="amq,
>> other-role"
>> > />
>> > > <permission type="consume" roles="amq, other-role" />
>> > > <permission type="browse" roles="amq, other-role" />
>> > > <permission type="send" roles="amq, other-role" />
>> > > <permission type="manage" roles="amq" />
>> > > </security-setting>
>> > > <security-setting match="A.some.queue">
>> > > <permission type="createNonDurableQueue"
>> > > roles="amq, other-role" />
>> > > <permission type="deleteNonDurableQueue"
>> > > roles="amq, other-role" />
>> > > <permission type="createDurableQueue" roles="amq,
>> > > other-role" />
>> > > <permission type="deleteDurableQueue" roles="amq,
>> > > other-role" />
>> > > <permission type="createAddress" roles="amq,
>> other-role"
>> > />
>> > > <permission type="deleteAddress" roles="amq,
>> other-role"
>> > />
>> > > <permission type="consume" roles="amq, other-role" />
>> > > <permission type="browse" roles="amq, other-role" />
>> > > <permission type="send" roles="amq, other-role" />
>> > > </security-setting>
>> > > <security-setting match="A.some.other.queue">
>> > > <permission type="createNonDurableQueue"
>> > > roles="amq, other-role" />
>> > > <permission type="deleteNonDurableQueue"
>> > > roles="amq, other-role" />
>> > > <permission type="createDurableQueue" roles="amq,
>> > > other-role" />
>> > > <permission type="deleteDurableQueue" roles="amq,
>> > > other-role" />
>> > > <permission type="createAddress" roles="amq,
>> other-role"
>> > />
>> > > <permission type="deleteAddress" roles="amq,
>> other-role"
>> > />
>> > > <permission type="consume" roles="amq, other-role" />
>> > > <permission type="browse" roles="amq, other-role" />
>> > > <permission type="send" roles="amq, other-role" />
>> > > </security-setting>
>> > > ...
>> > > ... etc.
>> > > ...
>> > > </security-settings>
>> > >
>> > > <address-settings>
>> > > <address-setting match="activemq.management#">
>> > > <dead-letter-address>DLQ</dead-letter-address>
>> > > <expiry-address>ExpiryQueue</expiry-address>
>> > > <redelivery-delay>0</redelivery-delay>
>> > > <max-size-bytes>-1</max-size-bytes>
>> > >
>> > > <message-counter-history-day-limit>10</message-counter-
>> > > history-day-limit>
>> > > <address-full-policy>PAGE</address-full-policy>
>> > > </address-setting>
>> > > <!--default for catch all -->
>> > > <address-setting match="#">
>> > > <dead-letter-address>DLQ</dead-letter-address>
>> > > <expiry-address>ExpiryQueue</expiry-address>
>> > > <redelivery-delay>0</redelivery-delay>
>> > > <max-size-bytes>-1</max-size-bytes>
>> > >
>> > > <message-counter-history-day-limit>10</message-counter-
>> > > history-day-limit>
>> > > <address-full-policy>PAGE</address-full-policy>
>> > > <redistribution-delay>1000</redistribution-delay>
>> > > </address-setting>
>> > > <address-setting match="DLQ">
>> > > <!-- 100 * 1024 * 1024 -> 100MB -->
>> > > <max-size-bytes>104857600</max-size-bytes>
>> > > <!-- 1000 * 60 * 60 -> 1h -->
>> > > <expiry-delay>3600000</expiry-delay>
>> > > <expiry-address />
>> > > </address-setting>
>> > > <address-setting match="A.some.queue">
>> > >
>> > > <redelivery-delay-multiplier>1.0</redelivery-delay-
>> > > multiplier>
>> > > <redelivery-delay>0</redelivery-delay>
>> > > <max-redelivery-delay>10</max-redelivery-delay>
>> > > </address-setting>
>> > > <address-setting match="A.some.other.queue">
>> > >
>> > > <redelivery-delay-multiplier>1.0</redelivery-delay-
>> > > multiplier>
>> > > <redelivery-delay>0</redelivery-delay>
>> > > <max-redelivery-delay>10</max-redelivery-delay>
>> > > <max-delivery-attempts>1</max-delivery-attempts>
>> > > <max-size-bytes>104857600</max-size-bytes>
>> > > </address-setting>
>> > > ...
>> > > ... etc.
>> > > ...
>> > > </address-settings>
>> > >
>> > > <addresses>
>> > > <address name="DLQ">
>> > > <anycast>
>> > > <queue name="DLQ" />
>> > > </anycast>
>> > > </address>
>> > > <address name="ExpiryQueue">
>> > > <anycast>
>> > > <queue name="ExpiryQueue" />
>> > > </anycast>
>> > > </address>
>> > > <address name="A.some.queue">
>> > > <anycast>
>> > > <queue name="A.some.queue">
>> > > <durable>true</durable>
>> > > </queue>
>> > > </anycast>
>> > > </address>
>> > > <address name="A.some.other.queue">
>> > > <anycast>
>> > > <queue name="A.some.other.queue">
>> > > <durable>true</durable>
>> > > </queue>
>> > > </anycast>
>> > > </address>
>> > > ...
>> > > ... etc.
>> > > ...
>> > > </addresses>
>> > > </core>
>> > > </configuration>
>> > >
>> >
>>
> --
> Clebert Suconic
--
Clebert Suconic
