[jira] [Work logged] (AMQ-9494) Upgrade to maven-source-plugin 3.3.1
[ https://issues.apache.org/jira/browse/AMQ-9494?focusedWorklogId=920864=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-920864 ] ASF GitHub Bot logged work on AMQ-9494: --- Author: ASF GitHub Bot Created on: 25/May/24 05:45 Start Date: 25/May/24 05:45 Worklog Time Spent: 10m Work Description: jbonofre merged PR #1224: URL: https://github.com/apache/activemq/pull/1224 Issue Time Tracking --- Worklog Id: (was: 920864) Time Spent: 20m (was: 10m) > Upgrade to maven-source-plugin 3.3.1 > > > Key: AMQ-9494 > URL: https://issues.apache.org/jira/browse/AMQ-9494 > Project: ActiveMQ Classic > Issue Type: Dependency upgrade >Reporter: Jean-Baptiste Onofré >Assignee: Jean-Baptiste Onofré >Priority: Major > Fix For: 6.2.0 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (AMQ-9494) Upgrade to maven-source-plugin 3.3.1
[ https://issues.apache.org/jira/browse/AMQ-9494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849428#comment-17849428 ] ASF subversion and git services commented on AMQ-9494: -- Commit d1ab8ef93d8807faf4fb5c9d28fb0a5a2c5b015d in activemq's branch refs/heads/main from JB Onofré [ https://gitbox.apache.org/repos/asf?p=activemq.git;h=d1ab8ef93 ] Merge pull request #1224 from jbonofre/AMQ-9494 AMQ-9494: Upgrade to maven-source-plugin 3.3.1 > Upgrade to maven-source-plugin 3.3.1 > > > Key: AMQ-9494 > URL: https://issues.apache.org/jira/browse/AMQ-9494 > Project: ActiveMQ Classic > Issue Type: Dependency upgrade >Reporter: Jean-Baptiste Onofré >Assignee: Jean-Baptiste Onofré >Priority: Major > Fix For: 6.2.0 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (AMQ-9494) Upgrade to maven-source-plugin 3.3.1
[ https://issues.apache.org/jira/browse/AMQ-9494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849427#comment-17849427 ] ASF subversion and git services commented on AMQ-9494: -- Commit 7bc8b67fe03a4695c6dd4ed0e333c418b2c99a74 in activemq's branch refs/heads/main from JB Onofré [ https://gitbox.apache.org/repos/asf?p=activemq.git;h=7bc8b67fe ] AMQ-9494: Upgrade to maven-source-plugin 3.3.1 > Upgrade to maven-source-plugin 3.3.1 > > > Key: AMQ-9494 > URL: https://issues.apache.org/jira/browse/AMQ-9494 > Project: ActiveMQ Classic > Issue Type: Dependency upgrade >Reporter: Jean-Baptiste Onofré >Assignee: Jean-Baptiste Onofré >Priority: Major > Fix For: 6.2.0 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (AMQ-9494) Upgrade to maven-source-plugin 3.3.1
[ https://issues.apache.org/jira/browse/AMQ-9494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Baptiste Onofré resolved AMQ-9494. --- Resolution: Fixed > Upgrade to maven-source-plugin 3.3.1 > > > Key: AMQ-9494 > URL: https://issues.apache.org/jira/browse/AMQ-9494 > Project: ActiveMQ Classic > Issue Type: Dependency upgrade >Reporter: Jean-Baptiste Onofré >Assignee: Jean-Baptiste Onofré >Priority: Major > Fix For: 6.2.0 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (AMQ-9505) ActiveMQ classic with postgres data source gives errors on start
[
https://issues.apache.org/jira/browse/AMQ-9505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jean-Baptiste Onofré reassigned AMQ-9505:
-
Assignee: Jean-Baptiste Onofré
> ActiveMQ classic with postgres data source gives errors on start
>
>
> Key: AMQ-9505
> URL: https://issues.apache.org/jira/browse/AMQ-9505
> Project: ActiveMQ Classic
> Issue Type: Bug
>Affects Versions: 6.1.1
> Environment: Fresh 6.1.1 installation on Ubuntu + Java 17
>Reporter: Susinda Perera
>Assignee: Jean-Baptiste Onofré
>Priority: Major
>
> I get this error when I start ActiveMQ
> {noformat}
> 2024-05-21 23:59:32,959 | INFO | Using Persistence Adapter:
> JDBCPersistenceAdapter(HikariDataSource (null)) |
> org.apache.activemq.broker.BrokerService | main
> 2024-05-21 23:59:32,968 | INFO | Starting Persistence Adapter:
> JDBCPersistenceAdapter(HikariDataSource (null)) |
> org.apache.activemq.broker.BrokerService | main
> 2024-05-21 23:59:32,969 | INFO | HikariPool-1 - Starting... |
> com.zaxxer.hikari.HikariDataSource | main
> 2024-05-21 23:59:35,211 | INFO | HikariPool-1 - Added connection
> org.postgresql.jdbc.PgConnection@70101687 | com.zaxxer.hikari.pool.HikariPool
> | main
> 2024-05-21 23:59:35,214 | INFO | HikariPool-1 - Start completed. |
> com.zaxxer.hikari.HikariDataSource | main
> 2024-05-21 23:59:35,433 | INFO | Database adapter driver override recognized
> for : [postgresql_jdbc_driver] - adapter: class
> org.apache.activemq.store.jdbc.adapter.PostgresqlJDBCAdapter |
> org.apache.activemq.store.jdbc.JDBCPersistenceAdapter | main
> 2024-05-21 23:59:38,423 | WARN | Could not create JDBC tables; they could
> already exist. Failure was: ALTER TABLE ACTIVEMQ_ACKS DROP CONSTRAINT
> "activemq_acks_pkey" Message: ERROR: constraint "activemq_acks_pkey" of
> relation "activemq_acks" does not exist SQLState: 42704 Vendor code: 0 |
> org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter | main
> 2024-05-21 23:59:38,423 | WARN | Failure details: ERROR: constraint
> "activemq_acks_pkey" of relation "activemq_acks" does not exist |
> org.apache.activemq.store.jdbc.JDBCPersistenceAdapter | main
> org.postgresql.util.PSQLException: ERROR: constraint "activemq_acks_pkey" of
> relation "activemq_acks" does not exist
>at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2725)
>at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2412)
>at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:371)
>at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:502)
>at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:419)
>at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:341)
>at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:326)
>at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:302)
>at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:297)
>at com.zaxxer.hikari.pool.ProxyStatement.execute(ProxyStatement.java:94)
>at
> com.zaxxer.hikari.pool.HikariProxyStatement.execute(HikariProxyStatement.java)
>at
> org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter.executeStatement(DefaultJDBCAdapter.java:112)
>at
> org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter.doCreateTables(DefaultJDBCAdapter.java:90)
>at
> org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.init(JDBCPersistenceAdapter.java:318)
>at
> org.apache.activemq.broker.LockableServiceSupport.preStart(LockableServiceSupport.java:99)
>at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:54)
>at
> org.apache.activemq.broker.BrokerService.doStartPersistenceAdapter(BrokerService.java:681)
>at
> org.apache.activemq.broker.BrokerService.startPersistenceAdapter(BrokerService.java:663)
>at org.apache.activemq.broker.BrokerService.start(BrokerService.java:627)
>at
> org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:73)
>at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
>at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
>at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>at java.base/java.lang.reflect.Method.invoke(Method.java:568)
>at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1890)
>at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1843)
>at
>
[jira] [Created] (ARTEMIS-4780) Artemis web console does not ever set 'artemisJmxDomain' in localStorage
Josh Byster created ARTEMIS-4780:
Summary: Artemis web console does not ever set 'artemisJmxDomain'
in localStorage
Key: ARTEMIS-4780
URL: https://issues.apache.org/jira/browse/ARTEMIS-4780
Project: ActiveMQ Artemis
Issue Type: Bug
Affects Versions: 2.32.0
Reporter: Josh Byster
We relocate our artemis packages as part of our shading process.
I did set up:
{code:java}
aActiveMQServer.getConfiguration().setJMXDomain("com.abc.shaded.org.apache.activemq.artemis");
{code}
However the web console does not show up. I noticed in the code it's because we
try to read in localStorage['artemisJmxDomain'] but never actually set it as
far as I can tell. Thus we always default to org.apache.activemq.artemis.
Everything works properly when setting localStorage['artemisJmxDomain'] =
"com.abc.shaded.org.apache.activemq.artemis"
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (ARTEMIS-4306) Add authn/z metrics
[
https://issues.apache.org/jira/browse/ARTEMIS-4306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849328#comment-17849328
]
Justin Bertram commented on ARTEMIS-4306:
-
I think support for {{CaffeineStatsCounter}} would be pretty straight-forward
to implement. I've got some ideas. I'll send a PR soon.
> Add authn/z metrics
> ---
>
> Key: ARTEMIS-4306
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4306
> Project: ActiveMQ Artemis
> Issue Type: Improvement
>Reporter: Justin Bertram
>Priority: Major
>
> It would be useful to have metrics for authn/z successes and failures as well
> as for metrics related to the corresponding caches.
> See this discussion on the users mailing list for more details:
> https://lists.apache.org/thread/g6ygyo4kb3xhygq8hpw7vsl3l2g5qt92
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (ARTEMIS-4306) Add authn/z metrics
[
https://issues.apache.org/jira/browse/ARTEMIS-4306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849099#comment-17849099
]
Mike Artz edited comment on ARTEMIS-4306 at 5/24/24 2:45 PM:
-
[~jbertram] - I made this [PR for micrometer to allow prefixing in the
CacheMeterBinder|https://github.com/micrometer-metrics/micrometer/pull/4048],
so that we could for example add the *"artemis.authentication."*
prefix but the PR _kind of_ got stuck, and the PR got closed.
Coming back to this now, _[if we use the
CacheMeterBinder|https://github.com/micrometer-metrics/micrometer/blob/37883fa6fb4a6d3f83d01f6b53101cc9f52b3f78/micrometer-core/src/main/java/io/micrometer/core/instrument/binder/cache/CacheMeterBinder.java],_
would it be ok to not have *"artemis.authentication."* prefixes, and adding
*authentication* and *authorization* as Tags? i.e. {{{}Tag("cacheName",
"authentication"){}}}and {{{}Tag("{}}}{{{}cacheName{}}}{{{}",
"authorization"){}}} It seems this might be more of a standard too for
micrometer users possibly.
However, now that we are using the
[CaffeineCache|https://github.com/ben-manes/caffeine/blob/b4cedbc411130b8e78c51effdd527756bdf1ff55/caffeine/src/main/java/com/github/benmanes/caffeine/cache/Cache.java],
I see there are two concrete classes -
[CaffeineCacheMetrics|https://github.com/micrometer-metrics/micrometer/blob/37883fa6fb4a6d3f83d01f6b53101cc9f52b3f78/micrometer-core/src/main/java/io/micrometer/core/instrument/binder/cache/CaffeineCacheMetrics.java]
and
[CaffeineStatsCounter|https://github.com/micrometer-metrics/micrometer/blob/37883fa6fb4a6d3f83d01f6b53101cc9f52b3f78/micrometer-core/src/main/java/io/micrometer/core/instrument/binder/cache/CaffeineStatsCounter.java].
Both of these look ok at first (more or less), but I am stuck at this
decision.
[CaffeineStatsCounter|https://github.com/micrometer-metrics/micrometer/blob/37883fa6fb4a6d3f83d01f6b53101cc9f52b3f78/micrometer-core/src/main/java/io/micrometer/core/instrument/binder/cache/CaffeineStatsCounter.java]
offers more detailed statistics than
[CaffeineCacheMetrics|https://github.com/micrometer-metrics/micrometer/blob/37883fa6fb4a6d3f83d01f6b53101cc9f52b3f78/micrometer-core/src/main/java/io/micrometer/core/instrument/binder/cache/CaffeineCacheMetrics.java]
and it allows name prefixes. However,
[CaffeineStatsCounter|https://github.com/micrometer-metrics/micrometer/blob/37883fa6fb4a6d3f83d01f6b53101cc9f52b3f78/micrometer-core/src/main/java/io/micrometer/core/instrument/binder/cache/CaffeineStatsCounter.java]
needs to pass the meterRegistry to the builder Caffeine object {_}before the
cache is built with the build() call{_}. There is no direct support to add a
[CaffeineStatsCounter|https://github.com/micrometer-metrics/micrometer/blob/37883fa6fb4a6d3f83d01f6b53101cc9f52b3f78/micrometer-core/src/main/java/io/micrometer/core/instrument/binder/cache/CaffeineStatsCounter.java]
to a Caffeine Cache after the cache is built. So if you use the
[CaffeineStatsCounter|https://github.com/micrometer-metrics/micrometer/blob/37883fa6fb4a6d3f83d01f6b53101cc9f52b3f78/micrometer-core/src/main/java/io/micrometer/core/instrument/binder/cache/CaffeineStatsCounter.java]
you might have to do something like call the
[MetricsConfiguration|https://github.com/apache/activemq-artemis/blob/c47713454caeece82df29a0a7fd4a2a39000576b/artemis-server/src/main/java/org/apache/activemq/artemis/core/config/MetricsConfiguration.java]
from the
*[SecurityStoreImpl|https://github.com/apache/activemq-artemis/blob/main/artemis-server/src/main/java/org/apache/activemq/artemis/core/security/impl/SecurityStoreImpl.java#L105-L108]*
like
{code:java}
MeterRegistry registry = metricsConfiguration.getPlugin().getRegistry();
authenticationCache = Caffeine.newBuilder()
.maximumSize(authenticationCacheSize)
.expireAfterWrite(invalidationInterval, TimeUnit.MILLISECONDS)
.recordStats(() -> new CaffeineStatsCounter(registry, "authentication"))
.build();{code}
And that doesnt make any sense because *MetricsConfiguration* happens after the
*SecurityStoreImpl.* So this doesnt seem like the best option.
*Other Options*
* Use
[CaffeineStatsCounter|https://github.com/micrometer-metrics/micrometer/blob/37883fa6fb4a6d3f83d01f6b53101cc9f52b3f78/micrometer-core/src/main/java/io/micrometer/core/instrument/binder/cache/CaffeineStatsCounter.java]
could initialize the authn cache (and authz cache) from the MetricsManager
class _(Couples the SecurityStoreImpl to the MetricsManager but maybe 2nd best
option)_
* Use
[CaffeineCacheMetrics|https://github.com/micrometer-metrics/micrometer/blob/37883fa6fb4a6d3f83d01f6b53101cc9f52b3f78/micrometer-core/src/main/java/io/micrometer/core/instrument/binder/cache/CaffeineCacheMetrics.java]
_(Seems like best option)_
* Create another concrete class (or maybe even extend the concrete class
[jira] [Commented] (AMQ-9482) Broker crashes after runaway threads spawn
[
https://issues.apache.org/jira/browse/AMQ-9482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849318#comment-17849318
]
Matt Pavlovich commented on AMQ-9482:
-
There are a ton of environment and use case specific details to get a tuning
exercise corrected-- very difficult to resolve over a JIRA.
> Broker crashes after runaway threads spawn
> --
>
> Key: AMQ-9482
> URL: https://issues.apache.org/jira/browse/AMQ-9482
> Project: ActiveMQ Classic
> Issue Type: Bug
> Components: Broker
>Affects Versions: 5.17.6, 6.0.1
> Environment: Bitnami created AMI in AWS
>Reporter: Tom Tichy
>Priority: Major
> Attachments: activemq.tdump, brokerInfo-after-crash-redacted.json,
> thread_dump.txt
>
>
> Running on Bitnami created AMI in AWS. The broker has about 7000 devices
> connected via MQTT. Each devices has its own topic name.
> Broker stays up for about 4-5 days before being hobbled and unable to create
> any new tasks/accept any new connections.
> (There is identical setup for staging environment with about 100 devices
> connected. It runs without any issues.)
> I have troubleshot the cause to be the systemd task limit. The current
> `TasksMax` is 18100. When running normally, the number of tasks is about 300.
> Then (every 4-5 days) there is a quick spike to the max 18100 tasks and it
> stays there never coming back down. The result is that the broker just sits
> there, does nothing useful and keeps logging the following message
>
> {code:java}
> [659914.788s][warning][os,thread] Failed to start thread "Unknown thread" -
> pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, g
> uardsize: 0k, detached.
> [659914.788s][warning][os,thread] Failed to start the native thread for
> java.lang.Thread "ActiveMQ BrokerService[localhost] Task-281805"
> ERROR | Scheduled task error
> java.lang.OutOfMemoryError: unable to create native thread: possibly out of
> memory or process/resource limits reached
> at java.lang.Thread.start0(Native Method) ~[?:?]
> at java.lang.Thread.start(Thread.java:809) ~[?:?]
> at
> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:945)
> ~[?:?]
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1364)
> ~[?:?]
> at
> org.apache.activemq.thread.TaskRunnerFactory.execute(TaskRunnerFactory.java:173)
> ~[activemq-client-6.0.1.jar:6.0.1]
> at
> org.apache.activemq.thread.TaskRunnerFactory.execute(TaskRunnerFactory.java:165)
> ~[activemq-client-6.0.1.jar:6.0.1]
> at org.apache.activemq.broker.region.Topic$7.run(Topic.java:820)
> ~[activemq-broker-6.0.1.jar:6.0.1]
> at
> org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:39)
> ~[activemq-client-6.0.1.jar:6.0.1]
> at java.util.TimerThread.mainLoop(Timer.java:566) ~[?:?]
> at java.util.TimerThread.run(Timer.java:516) ~[?:?]
> Exception in thread "ActiveMQ Broker[localhost] Scheduler"
> java.lang.OutOfMemoryError: unable to create native thread: possibly out of
> memory or process/resource limits reached
> at java.base/java.lang.Thread.start0(Native Method)
> at java.base/java.lang.Thread.start(Thread.java:809)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:945)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1364)
> at
> org.apache.activemq.thread.TaskRunnerFactory.execute(TaskRunnerFactory.java:173)
> at
> org.apache.activemq.thread.TaskRunnerFactory.execute(TaskRunnerFactory.java:165)
> at org.apache.activemq.broker.region.Topic$7.run(Topic.java:820)
> at
> org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:39)
> at java.base/java.util.TimerThread.mainLoop(Timer.java:566)
> at java.base/java.util.TimerThread.run(Timer.java:516)
> {code}
>
> The start command is
> {code:java}
> /opt/bitnami/java/bin/java -Xms2G -Xmx4G
> -Djava.util.logging.config.file=logging.properties
> -Djava.security.auth.login.config=/opt/bitnami/activemq/conf/login.config
> -Dorg.apache.activemq.UseDedicatedTaskRunner=false
> -Dcom.sun.management.jmxremote -Djava.awt.headless=true
> -Djava.io.tmpdir=/opt/bitnami/activemq/tmp --add-reads=java.xml=java.logging
> --add-opens java.base/java.security=ALL-UNNAMED --add-opens
> java.base/java.net=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED
> --add-opens java.base/java.util=ALL-UNNAMED --add-opens
> java.naming/javax.naming.spi=ALL-UNNAMED --add-opens
> java.rmi/sun.rmi.transport.tcp=ALL-UNNAMED --add-opens
> java.base/java.util.concurrent=ALL-UNNAMED --add-opens
> java.base/java.util.concurrent.atomic=ALL-UNNAMED
>
[jira] [Commented] (AMQ-9482) Broker crashes after runaway threads spawn
[
https://issues.apache.org/jira/browse/AMQ-9482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849248#comment-17849248
]
Tom Tichy commented on AMQ-9482:
Hi,
I did add the `soTimeout` and `soWriteTimeout`, but alas no joy.
I have a better thread dump with about 11.5k identical threads that are in the
WAITING state
Fastthread.io analysis says
{quote}
h1. 11440 threads with same stack trace
!https://fastthread.io/assets/globally-shared/images/icon-error.svg! 11440
threads are WAITING on *_park()_* method in *_jdk.internal.misc.Unsafe_* file
and they all have same stack trace. If multiple threads exhibit same stack
trace, you might want to examine their stack trace. (Note: If your application
is unresponsive or poorly responding, it might be caused because these threads).
{panel}
h2. {color:#cc3300}ActiveMQ BrokerService[localhost] Task-68826{color}
PRIORITY : 5
THREAD ID : 0X7F1E3010FB90
NATIVE ID : 0XE3ABE
NATIVE ID (DECIMAL) : 932542
STATE : TIMED_WAITING
stackTrace:
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park({color:#80}java.base@{*}17.0.9/Native
Method{*}{color}{*}){*}
- parking to wait for *<0x000700232d10>* (a
java.util.concurrent.SynchronousQueue$TransferStack)
at
java.util.concurrent.locks.LockSupport.parkNanos({color:#80}java.base@{*}17.0.9/LockSupport.java:252{*}{color}{*}){*}
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer({color:#80}java.base@{*}17.0.9/SynchronousQueue.java:401{*}{color}{*}){*}
at
java.util.concurrent.SynchronousQueue.poll({color:#80}java.base@{*}17.0.9/SynchronousQueue.java:903{*}{color}{*}){*}
at
java.util.concurrent.ThreadPoolExecutor.getTask({color:#80}java.base@{*}17.0.9/ThreadPoolExecutor.java:1061{*}{color}{*}){*}
at
java.util.concurrent.ThreadPoolExecutor.runWorker({color:#80}java.base@{*}17.0.9/ThreadPoolExecutor.java:1122{*}{color}{*}){*}
at
java.util.concurrent.ThreadPoolExecutor$Worker.run({color:#80}java.base@{*}17.0.9/ThreadPoolExecutor.java:635{*}{color}{*}){*}
at
java.lang.Thread.run({color:#80}java.base@{*}17.0.9/Thread.java:840{*}{color}{*}){*}
Locked ownable synchronizers:
- None{panel}
{quote}
Here is another piece of information that may be relevant. The devices that
connect to our brokers do something silly and open a new MQTT connection
everytime they want to send something. The MQTT spec allows this and ActiveMQ
dutifully logs
{code:java}
WARN | Stealing link for clientId {code}
[^thread_dump.txt]
> Broker crashes after runaway threads spawn
> --
>
> Key: AMQ-9482
> URL: https://issues.apache.org/jira/browse/AMQ-9482
> Project: ActiveMQ Classic
> Issue Type: Bug
> Components: Broker
>Affects Versions: 5.17.6, 6.0.1
> Environment: Bitnami created AMI in AWS
>Reporter: Tom Tichy
>Priority: Major
> Attachments: activemq.tdump, brokerInfo-after-crash-redacted.json,
> thread_dump.txt
>
>
> Running on Bitnami created AMI in AWS. The broker has about 7000 devices
> connected via MQTT. Each devices has its own topic name.
> Broker stays up for about 4-5 days before being hobbled and unable to create
> any new tasks/accept any new connections.
> (There is identical setup for staging environment with about 100 devices
> connected. It runs without any issues.)
> I have troubleshot the cause to be the systemd task limit. The current
> `TasksMax` is 18100. When running normally, the number of tasks is about 300.
> Then (every 4-5 days) there is a quick spike to the max 18100 tasks and it
> stays there never coming back down. The result is that the broker just sits
> there, does nothing useful and keeps logging the following message
>
> {code:java}
> [659914.788s][warning][os,thread] Failed to start thread "Unknown thread" -
> pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, g
> uardsize: 0k, detached.
> [659914.788s][warning][os,thread] Failed to start the native thread for
> java.lang.Thread "ActiveMQ BrokerService[localhost] Task-281805"
> ERROR | Scheduled task error
> java.lang.OutOfMemoryError: unable to create native thread: possibly out of
> memory or process/resource limits reached
> at java.lang.Thread.start0(Native Method) ~[?:?]
> at java.lang.Thread.start(Thread.java:809) ~[?:?]
> at
> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:945)
> ~[?:?]
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1364)
> ~[?:?]
> at
> org.apache.activemq.thread.TaskRunnerFactory.execute(TaskRunnerFactory.java:173)
> ~[activemq-client-6.0.1.jar:6.0.1]
> at
> org.apache.activemq.thread.TaskRunnerFactory.execute(TaskRunnerFactory.java:165)
> ~[activemq-client-6.0.1.jar:6.0.1]
> at
[jira] [Updated] (AMQ-9482) Broker crashes after runaway threads spawn
[
https://issues.apache.org/jira/browse/AMQ-9482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tom Tichy updated AMQ-9482:
---
Attachment: thread_dump.txt
> Broker crashes after runaway threads spawn
> --
>
> Key: AMQ-9482
> URL: https://issues.apache.org/jira/browse/AMQ-9482
> Project: ActiveMQ Classic
> Issue Type: Bug
> Components: Broker
>Affects Versions: 5.17.6, 6.0.1
> Environment: Bitnami created AMI in AWS
>Reporter: Tom Tichy
>Priority: Major
> Attachments: activemq.tdump, brokerInfo-after-crash-redacted.json,
> thread_dump.txt
>
>
> Running on Bitnami created AMI in AWS. The broker has about 7000 devices
> connected via MQTT. Each devices has its own topic name.
> Broker stays up for about 4-5 days before being hobbled and unable to create
> any new tasks/accept any new connections.
> (There is identical setup for staging environment with about 100 devices
> connected. It runs without any issues.)
> I have troubleshot the cause to be the systemd task limit. The current
> `TasksMax` is 18100. When running normally, the number of tasks is about 300.
> Then (every 4-5 days) there is a quick spike to the max 18100 tasks and it
> stays there never coming back down. The result is that the broker just sits
> there, does nothing useful and keeps logging the following message
>
> {code:java}
> [659914.788s][warning][os,thread] Failed to start thread "Unknown thread" -
> pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, g
> uardsize: 0k, detached.
> [659914.788s][warning][os,thread] Failed to start the native thread for
> java.lang.Thread "ActiveMQ BrokerService[localhost] Task-281805"
> ERROR | Scheduled task error
> java.lang.OutOfMemoryError: unable to create native thread: possibly out of
> memory or process/resource limits reached
> at java.lang.Thread.start0(Native Method) ~[?:?]
> at java.lang.Thread.start(Thread.java:809) ~[?:?]
> at
> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:945)
> ~[?:?]
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1364)
> ~[?:?]
> at
> org.apache.activemq.thread.TaskRunnerFactory.execute(TaskRunnerFactory.java:173)
> ~[activemq-client-6.0.1.jar:6.0.1]
> at
> org.apache.activemq.thread.TaskRunnerFactory.execute(TaskRunnerFactory.java:165)
> ~[activemq-client-6.0.1.jar:6.0.1]
> at org.apache.activemq.broker.region.Topic$7.run(Topic.java:820)
> ~[activemq-broker-6.0.1.jar:6.0.1]
> at
> org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:39)
> ~[activemq-client-6.0.1.jar:6.0.1]
> at java.util.TimerThread.mainLoop(Timer.java:566) ~[?:?]
> at java.util.TimerThread.run(Timer.java:516) ~[?:?]
> Exception in thread "ActiveMQ Broker[localhost] Scheduler"
> java.lang.OutOfMemoryError: unable to create native thread: possibly out of
> memory or process/resource limits reached
> at java.base/java.lang.Thread.start0(Native Method)
> at java.base/java.lang.Thread.start(Thread.java:809)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:945)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1364)
> at
> org.apache.activemq.thread.TaskRunnerFactory.execute(TaskRunnerFactory.java:173)
> at
> org.apache.activemq.thread.TaskRunnerFactory.execute(TaskRunnerFactory.java:165)
> at org.apache.activemq.broker.region.Topic$7.run(Topic.java:820)
> at
> org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:39)
> at java.base/java.util.TimerThread.mainLoop(Timer.java:566)
> at java.base/java.util.TimerThread.run(Timer.java:516)
> {code}
>
> The start command is
> {code:java}
> /opt/bitnami/java/bin/java -Xms2G -Xmx4G
> -Djava.util.logging.config.file=logging.properties
> -Djava.security.auth.login.config=/opt/bitnami/activemq/conf/login.config
> -Dorg.apache.activemq.UseDedicatedTaskRunner=false
> -Dcom.sun.management.jmxremote -Djava.awt.headless=true
> -Djava.io.tmpdir=/opt/bitnami/activemq/tmp --add-reads=java.xml=java.logging
> --add-opens java.base/java.security=ALL-UNNAMED --add-opens
> java.base/java.net=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED
> --add-opens java.base/java.util=ALL-UNNAMED --add-opens
> java.naming/javax.naming.spi=ALL-UNNAMED --add-opens
> java.rmi/sun.rmi.transport.tcp=ALL-UNNAMED --add-opens
> java.base/java.util.concurrent=ALL-UNNAMED --add-opens
> java.base/java.util.concurrent.atomic=ALL-UNNAMED
> --add-exports=java.base/sun.net.www.protocol.http=ALL-UNNAMED
> --add-exports=java.base/sun.net.www.protocol.https=ALL-UNNAMED
>
