Hitesh Jha created ARTEMIS-5484:
-----------------------------------
Summary: Artemis Critical IO error when persistence over postgresDB
Key: ARTEMIS-5484
URL: https://issues.apache.org/jira/browse/ARTEMIS-5484
Project: ActiveMQ Artemis
Issue Type: Improvement
Components: ActiveMQ-Artemis-Native
Affects Versions: 2.33.0
Reporter: Hitesh Jha
Assignee: Clebert Suconic
Artemis after few days keeps restarting with below IO error. We understand that
node manager lock is lost, but it does not seem due to DB disconnection. It
seems during high throughput node manager lease lock gets halted. We compared
the behavior with oracle DB with same config but never saw any restarts.
2025-01-25 04:05:54,930 ERROR
[org.apache.activemq.artemis.core.server.impl.jdbc.ActiveMQScheduledLeaseLock]
primary lock successful renew tooks 2008 ms, while is supposed to take <2000 ms
2025-01-25 06:51:55,590 ERROR
[org.apache.activemq.artemis.core.server.impl.jdbc.ActiveMQScheduledLeaseLock]
primary lock successful renew tooks 2005 ms, while is supposed to take <2000 ms
2025-01-26 04:18:58,342 ERROR
[org.apache.activemq.artemis.core.server.impl.jdbc.ActiveMQScheduledLeaseLock]
primary lock successful renew tooks 2005 ms, while is supposed to take <2000 ms
2025-01-26 12:10:24,259 ERROR
[org.apache.activemq.artemis.core.server.impl.jdbc.ActiveMQScheduledLeaseLock]
primary lock successful renew tooks 2076 ms, while is supposed to take <2000 ms
2025-01-26 14:10:29,031 ERROR [org.apache.activemq.artemis.core.server]
AMQ222010: Critical IO Error, shutting down the server. file=Lost NodeManager
lock, message=NULL
java.io.IOException: lost lock
at
org.apache.activemq.artemis.core.server.impl.SharedStorePrimaryActivation.lambda$registerActiveLockListener$0(SharedStorePrimaryActivation.java:124)
~[artemis-server-2.33.0.jar:2.33.0]
at
org.apache.activemq.artemis.core.server.NodeManager.lambda$notifyLostLock$0(NodeManager.java:172)
~[artemis-server-2.33.0.jar:2.33.0]
at java.base/java.lang.Iterable.forEach(Iterable.java:75) ~[?:?]
at
org.apache.activemq.artemis.core.server.NodeManager.notifyLostLock(NodeManager.java:170)
~[artemis-server-2.33.0.jar:2.33.0]
at
org.apache.activemq.artemis.core.server.impl.jdbc.JdbcNodeManager.notifyLostLock(JdbcNodeManager.java:158)
~[artemis-server-2.33.0.jar:2.33.0]
at
org.apache.activemq.artemis.core.server.impl.jdbc.ActiveMQScheduledLeaseLock.run(ActiveMQScheduledLeaseLock.java:115)
~[artemis-server-2.33.0.jar:2.33.0]
at
org.apache.activemq.artemis.core.server.ActiveMQScheduledComponent.runForExecutor(ActiveMQScheduledComponent.java:314)
~[artemis-commons-2.33.0.jar:2.33.0]
at
org.apache.activemq.artemis.core.server.ActiveMQScheduledComponent.lambda$bookedRunForScheduler$2(ActiveMQScheduledComponent.java:321)
~[artemis-commons-2.33.0.jar:2.33.0]
at
org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
~[artemis-commons-2.33.0.jar:2.33.0]
at
org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
~[artemis-commons-2.33.0.jar:2.33.0]
at
org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
~[artemis-commons-2.33.0.jar:2.33.0]
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[?:?]
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[?:?]
at
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
[artemis-commons-2.33.0.jar:2.33.0]
2025-01-26 14:10:29,043 ERROR
[org.apache.activemq.artemis.core.server.impl.jdbc.ActiveMQScheduledLeaseLock]
primary lock failed renew tooks 2377 ms, while is supposed to take <2000 ms
2025-01-26 14:10:29,357 ERROR [org.apache.activemq.artemis.core.server]
AMQ224068: Unable to stop component:
org.apache.activemq.artemis.core.server.impl.SharedStorePrimaryActivation
java.lang.IllegalStateException: the node manager is supposed to be started
at
org.apache.activemq.artemis.core.server.NodeManager.checkStarted(NodeManager.java:162)
~[artemis-server-2.33.0.jar:2.33.0]
at
org.apache.activemq.artemis.core.server.impl.jdbc.JdbcNodeManager.crashPrimaryServer(JdbcNodeManager.java:543)
~[artemis-server-2.33.0.jar:2.33.0]
at
org.apache.activemq.artemis.core.server.impl.SharedStorePrimaryActivation.close(SharedStorePrimaryActivation.java:150)
~[artemis-server-2.33.0.jar:2.33.0]
at
org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stop(ActiveMQServerImpl.java:1474)
[artemis-server-2.33.0.jar:2.33.0]
at
org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stop(ActiveMQServerImpl.java:1267)
[artemis-server-2.33.0.jar:2.33.0]
at
org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stop(ActiveMQServerImpl.java:1260)
[artemis-server-2.33.0.jar:2.33.0]
at
org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5.run(ActiveMQServerImpl.java:1040)
[artemis-server-2.33.0.jar:2.33.0]
2025-01-26 14:10:29,359 INFO [org.apache.activemq.artemis] AMQ241005: Stopping
embedded web server
2025-01-26 14:10:29,401 INFO [io.hawt.HawtioContextListener] Destroying hawtio
services
2025-01-26 14:10:29,410 INFO
[org.apache.activemq.hawtio.plugin.PluginContextListener] Destroyed
artemis-plugin plugin
2025-01-26 14:10:29,413 INFO
[org.apache.activemq.hawtio.branding.PluginContextListener] Destroyed
activemq-branding plugin
2025-01-26 14:10:29,478 INFO [org.apache.activemq.artemis] AMQ241006: Stopped
embedded web server
2025-01-26 14:10:29,478 INFO
[org.apache.activemq.artemis.protocol.amqp.connect.mirror.AckManagerProvider]
Removing ActiveMQServerImpl::name=artemis-primary-0
2025-01-26 14:10:29,480 INFO [org.apache.activemq.artemis.core.server]
AMQ221002: Apache ActiveMQ Artemis Message Broker version 2.33.0
[3fa9c14f-0f29-11ef-accf-c2fcb3d9bfbe] stopped, uptime 3 days 23 hours
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@activemq.apache.org
For additional commands, e-mail: issues-h...@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
