[
https://issues.apache.org/jira/browse/AMQ-7366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17342489#comment-17342489
]
Erwin Dondorp commented on AMQ-7366:
------------------------------------
[~levin wong] we are facing a problem with roughly the same symptoms. did you
have any new information or insight since you filed this bug?
> ActiveMQ broker always hangs up at intervals in the case of high concurrent
> connections
> ---------------------------------------------------------------------------------------
>
> Key: AMQ-7366
> URL: https://issues.apache.org/jira/browse/AMQ-7366
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker, Connector, MQTT
> Affects Versions: 5.15.0
> Environment: activeMQ version:5.15.0
> linux OS version：Red Hat Enterprise Linux Server release 6.9 (Santiago)
> linux server configuration: Memery:32G、CPU:16、disk:200G
>
> Reporter: zhiwei wang
> Priority: Blocker
> Labels: performance
> Fix For: 5.15.0
>
>
> My broker JVM configuration is "{color:#FF0000}- xms16g - xmx16g - xss256k -
> XX: metaspacesize = 128M{color}".
> The transport connector in activemq.xml is configured as follows:
> {color:#FF0000}<transportConnector name="sslmqtt"
> uri="mqtt+ssl://0.0.0.0:1883?maximumConnections=200000&wireFormat.maxFrameSize=104857600&jms.watchTopicAdvisories=false&transport.defaultKeepAlive=30000"/>{color}
> *At present, it can support 15000 client connections. But every two days or
> less, ActiveMQ broker will hang up*. When I check the logs of ActiveMQ
> broker, I find the following errors:
> 1.
> {color:#FF0000}org.apache.activemq.transport.InactivityIOException: Channel
> was inactive for too (>60000){color} long:
> tcp://180.162.0.129:50014org.apache.activemq.transport.InactivityIOException:
> Channel was inactive for too (>60000) long: tcp://180.162.0.129:50014 at
> org.apache.activemq.transport.mqtt.MQTTInactivityMonitor$1$1.run(MQTTInactivityMonitor.java:82)[activemq-mqtt-5.15.0.jar:5.15.0]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)[:1.8.0_151]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)[:1.8.0_151]
> at java.lang.Thread.run(Thread.java:748)[:1.8.0_151]2019-12-17 13:48:28,123
> | DEBUG | No CONNECT frame received in time for
> tcp:///101.80.38.31:4751@1883! Throwing InactivityIOException. |
> org.apache.activemq.transport.mqtt.MQTTInactivityMonitor | InactivityMonitor
> ReadCheck2019-12-17 13:48:28,124 | DEBUG | Transport Connection to:
> tcp://101.88.157.200:6892 failed:
> org.apache.activemq.transport.InactivityIOException: Channel was inactive for
> too (>60000) long: tcp://101.88.157.200:6892 |
> org.apache.activemq.broker.TransportConnection.Transport |
> MQTTInactivityMonitor Async Task:
> java.util.concurrent.ThreadPoolExecutor$Worker@ddca0bb[State = -1, empty
> queue]org.apache.activemq.transport.InactivityIOException: Channel was
> inactive for too (>60000) long: tcp://101.88.157.200:6892 at
> org.apache.activemq.transport.mqtt.MQTTInactivityMonitor$1$1.run(MQTTInactivityMonitor.java:82)[activemq-mqtt-5.15.0.jar:5.15.0]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)[:1.8.0_151]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)[:1.8.0_151]
> at java.lang.Thread.run(Thread.java:748)[:1.8.0_151]2019-12-17 13:48:28,136
> | DEBUG | No CONNECT frame received in time for
> tcp:///114.83.88.242:36665@1883! Throwing InactivityIOException. |
> org.apache.activemq.transport.mqtt.MQTTInactivityMonitor | InactivityMonitor
> ReadCheck
> 2.
> {color:#FF0000}javax.net.ssl.SSLHandshakeException: Remote host closed
> connection during handshake{color}
> javax.net.ssl.SSLHandshakeException: Remote host closed connection during
> handshake at
> sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1002)[:1.8.0_151]
> at
> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)[:1.8.0_151]
> at
> sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:938)[:1.8.0_151]
> at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)[:1.8.0_151]
> at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)[activemq-client-5.15.0.jar:5.15.0]
> at
> org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:634)[activemq-client-5.15.0.jar:5.15.0]
> at
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:59)[activemq-client-5.15.0.jar:5.15.0]
> at
> org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:619)[activemq-client-5.15.0.jar:5.15.0]
> at java.io.DataInputStream.readByte(DataInputStream.java:265)[:1.8.0_151] at
> org.apache.activemq.transport.mqtt.MQTTWireFormat.unmarshal(MQTTWireFormat.java:86)[activemq-mqtt-5.15.0.jar:5.15.0]
> at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:240)[activemq-client-5.15.0.jar:5.15.0]
> at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:232)[activemq-client-5.15.0.jar:5.15.0]
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)[activemq-client-5.15.0.jar:5.15.0]
> at java.lang.Thread.run(Thread.java:748)[:1.8.0_151]
> 3.
> {color:#FF0000} java.lang.OutOfMemoryError：unable to create new native
> thread{color}
>
> *At the same time, the number of threads in ActiveMQ service keeps
> skyrocketing, reaching 60000, resulting in memory overflow。{color:#FF0000}I
> found that when the channel was inactive for too long is kicked out by the
> broker, the old thread is not destroyed. At this time, new requests come in,
> and the thread will still be created continuously, resulting in memory
> overflow{color}.*
> *My requirement is that ActiveMQ broker can support 30000 client
> connections.How can I solve this problem? Can someone help me*
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
