yuebao created ARTEMIS-3105:
-------------------------------
Summary: Large message file not closed on backup side
Key: ARTEMIS-3105
URL: https://issues.apache.org/jira/browse/ARTEMIS-3105
Project: ActiveMQ Artemis
Issue Type: Bug
Components: Broker
Affects Versions: 2.16.0
Reporter: yuebao
We send a large number of large messages to the cluster, on the backup side, we
found error message of too many open files. The detailed log is as follows:
{panel:title=My title}
2021-02-06 17:40:29,793 WARN [org.apache.activemq.artemis.core.server]
AMQ222010: Critical IO Error, shutting down the server. file=NIOSequentialFile
/opt/apache/activemq/brokers/habroker01/./data/large-messages/21475041553.msg,
message=/opt/apache/activemq/brokers/habroker01/./data/large-messages/21475041553.msg
(Too many open files): ActiveMQIOErrorException[errorType=IO_ERROR
message=/opt/apache/activemq/brokers/habroker01/./data/large-messages/21475041553.msg
(Too many open files)]
at
org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:87)
[artemis-journal-2.4.0.jar:2.4.0]
at
org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:73)
[artemis-journal-2.4.0.jar:2.4.0]
at
org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl.openFile(LargeServerMessageImpl.java:385)
[artemis-server-2.4.0.jar:2.4.0]
at
org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl.validateFile(LargeServerMessageImpl.java:364)
[artemis-server-2.4.0.jar:2.4.0]
at
org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl.addBytes(LargeServerMessageImpl.java:120)
[artemis-server-2.4.0.jar:2.4.0]
at
org.apache.activemq.artemis.core.replication.ReplicationEndpoint.handleLargeMessageWrite(ReplicationEndpoint.java:623)
[artemis-server-2.4.0.jar:2.4.0]
at
org.apache.activemq.artemis.core.replication.ReplicationEndpoint.handlePacket(ReplicationEndpoint.java:216)
[artemis-server-2.4.0.jar:2.4.0]
at
org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:638)
[artemis-core-client-2.4.0.jar:2.4.0]
at
org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:392)
[artemis-core-client-2.4.0.jar:2.4.0]
at
org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:374)
[artemis-core-client-2.4.0.jar:2.4.0]
at
org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1148)
[artemis-core-client-2.4.0.jar:2.4.0]
at
org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:68)
[artemis-core-client-2.4.0.jar:2.4.0]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
[netty-all-4.1.16.Final.jar:4.1.16.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
[netty-all-4.1.16.Final.jar:4.1.16.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
[netty-all-4.1.16.Final.jar:4.1.16.Final]
at
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
[netty-all-4.1.16.Final.jar:4.1.16.Final]
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)
[netty-all-4.1.16.Final.jar:4.1.16.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
[netty-all-4.1.16.Final.jar:4.1.16.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
[netty-all-4.1.16.Final.jar:4.1.16.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
[netty-all-4.1.16.Final.jar:4.1.16.Final]
at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
[netty-all-4.1.16.Final.jar:4.1.16.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
[netty-all-4.1.16.Final.jar:4.1.16.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
[netty-all-4.1.16.Final.jar:4.1.16.Final]
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
[netty-all-4.1.16.Final.jar:4.1.16.Final]
at
io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:797)
[netty-all-4.1.16.Final.jar:4.1.16.Final]
at
io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:404)
[netty-all-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:304)
[netty-all-4.1.16.Final.jar:4.1.16.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
[netty-all-4.1.16.Final.jar:4.1.16.Final]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_92-SUNINGJVM]
Caused by: java.io.FileNotFoundException:
/opt/apache/activemq/brokers/habroker01/./data/large-messages/21475041553.msg
(Too many open files)
at java.io.RandomAccessFile.open0(Native Method)
[rt.jar:1.8.0_92-SUNINGJVM]
at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
[rt.jar:1.8.0_92-SUNINGJVM]
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)
[rt.jar:1.8.0_92-SUNINGJVM]
at
org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:79)
[artemis-journal-2.4.0.jar:2.4.0]
... 28 more
{panel}
We think that large message is released on live side but not released on the
backup size, when receive the last chunk.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
