[
https://issues.apache.org/jira/browse/AMQ-9482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17849786#comment-17849786
]
Matt Pavlovich commented on AMQ-9482:
-------------------------------------
[~tomtichy] Keep in mind that the out-of-the-box defaults are configured for
developers testing on their workstation. Messaging workflows come in all shapes
and sizes the combination of number of connections, number of destinations,
message sizes and message flow rate all determine how the broker needs to be
configured to shape that traffic. This is true for all compute systems --
databases, caches, web servers, etc.
In this case, when I refer to "tuning", I'm covering managing available system
resources to support the workload. Your workload has 7,000 connections, and the
broker needs to be appropriately setup to manage that workload for the amount
of server resources you are running it on.
> Broker crashes after runaway threads spawn
> ------------------------------------------
>
> Key: AMQ-9482
> URL: https://issues.apache.org/jira/browse/AMQ-9482
> Project: ActiveMQ Classic
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.17.6, 6.0.1
> Environment: Bitnami created AMI in AWS
> Reporter: Tom Tichy
> Priority: Major
> Attachments: activemq.tdump, brokerInfo-after-crash-redacted.json,
> thread_dump.txt
>
>
> Running on Bitnami created AMI in AWS. The broker has about 7000 devices
> connected via MQTT. Each devices has its own topic name.
> Broker stays up for about 4-5 days before being hobbled and unable to create
> any new tasks/accept any new connections.
> (There is identical setup for staging environment with about 100 devices
> connected. It runs without any issues.)
> I have troubleshot the cause to be the systemd task limit. The current
> `TasksMax` is 18100. When running normally, the number of tasks is about 300.
> Then (every 4-5 days) there is a quick spike to the max 18100 tasks and it
> stays there never coming back down. The result is that the broker just sits
> there, does nothing useful and keeps logging the following message
>
> {code:java}
> [659914.788s][warning][os,thread] Failed to start thread "Unknown thread" -
> pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, g
> uardsize: 0k, detached.
> [659914.788s][warning][os,thread] Failed to start the native thread for
> java.lang.Thread "ActiveMQ BrokerService[localhost] Task-281805"
> ERROR | Scheduled task error
> java.lang.OutOfMemoryError: unable to create native thread: possibly out of
> memory or process/resource limits reached
> at java.lang.Thread.start0(Native Method) ~[?:?]
> at java.lang.Thread.start(Thread.java:809) ~[?:?]
> at
> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:945)
> ~[?:?]
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1364)
> ~[?:?]
> at
> org.apache.activemq.thread.TaskRunnerFactory.execute(TaskRunnerFactory.java:173)
> ~[activemq-client-6.0.1.jar:6.0.1]
> at
> org.apache.activemq.thread.TaskRunnerFactory.execute(TaskRunnerFactory.java:165)
> ~[activemq-client-6.0.1.jar:6.0.1]
> at org.apache.activemq.broker.region.Topic$7.run(Topic.java:820)
> ~[activemq-broker-6.0.1.jar:6.0.1]
> at
> org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:39)
> ~[activemq-client-6.0.1.jar:6.0.1]
> at java.util.TimerThread.mainLoop(Timer.java:566) ~[?:?]
> at java.util.TimerThread.run(Timer.java:516) ~[?:?]
> Exception in thread "ActiveMQ Broker[localhost] Scheduler"
> java.lang.OutOfMemoryError: unable to create native thread: possibly out of
> memory or process/resource limits reached
> at java.base/java.lang.Thread.start0(Native Method)
> at java.base/java.lang.Thread.start(Thread.java:809)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:945)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1364)
> at
> org.apache.activemq.thread.TaskRunnerFactory.execute(TaskRunnerFactory.java:173)
> at
> org.apache.activemq.thread.TaskRunnerFactory.execute(TaskRunnerFactory.java:165)
> at org.apache.activemq.broker.region.Topic$7.run(Topic.java:820)
> at
> org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:39)
> at java.base/java.util.TimerThread.mainLoop(Timer.java:566)
> at java.base/java.util.TimerThread.run(Timer.java:516)
> {code}
>
> The start command is
> {code:java}
> /opt/bitnami/java/bin/java -Xms2G -Xmx4G
> -Djava.util.logging.config.file=logging.properties
> -Djava.security.auth.login.config=/opt/bitnami/activemq/conf/login.config
> -Dorg.apache.activemq.UseDedicatedTaskRunner=false
> -Dcom.sun.management.jmxremote -Djava.awt.headless=true
> -Djava.io.tmpdir=/opt/bitnami/activemq/tmp --add-reads=java.xml=java.logging
> --add-opens java.base/java.security=ALL-UNNAMED --add-opens
> java.base/java.net=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED
> --add-opens java.base/java.util=ALL-UNNAMED --add-opens
> java.naming/javax.naming.spi=ALL-UNNAMED --add-opens
> java.rmi/sun.rmi.transport.tcp=ALL-UNNAMED --add-opens
> java.base/java.util.concurrent=ALL-UNNAMED --add-opens
> java.base/java.util.concurrent.atomic=ALL-UNNAMED
> --add-exports=java.base/sun.net.www.protocol.http=ALL-UNNAMED
> --add-exports=java.base/sun.net.www.protocol.https=ALL-UNNAMED
> --add-exports=java.base/sun.net.www.protocol.jar=ALL-UNNAMED
> --add-exports=jdk.xml.dom/org.w3c.dom.html=ALL-UNNAMED
> --add-exports=jdk.naming.rmi/com.sun.jndi.url.rmi=ALL-UNNAMED
> -Dactivemq.classpath=/opt/bitnami/activemq/conf:/opt/bitnami/activemq/../lib/:
> -Dactivemq.home=/opt/bitnami/activemq -Dactivemq.base=/opt/bitnami/activemq
> -Dactivemq.conf=/opt/bitnami/activemq/conf
> -Dactivemq.data=/opt/bitnami/activemq/data
> -Djolokia.conf=file:/opt/bitnami/activemq/conf/jolokia-access.xml -jar
> /opt/bitnami/activemq/bin/activemq.jar start {code}
> During the error condition, I am able to collect broker information via
> jolokia: [^brokerInfo-after-crash-redacted.json]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
For further information, visit: https://activemq.apache.org/contact
