[jira] [Updated] (QPID-8274) [Broker-J][BDB HA] Broker can fail to become active when BDB HA virtual host times out to join the group
[
https://issues.apache.org/jira/browse/QPID-8274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Rudyy updated QPID-8274:
-
Description:
Broker containing a BDB HA Virtual Host node (belonging to the cluster
consisting of several nodes) can fail to start when BDB HA Virtual Host node
times out to join the group. The broker cannot complete activation (transition
into an ACTIVE state).
The stack traces like bellow are reported on BDB HA VHN timeout:
{noformat}
ERROR [Broker-Config] (o.a.q.s.m.AbstractConfiguredObject) - Failed to open
object with name 'node2'. Object will be put into ERROR state.
java.lang.RuntimeException: JE replicated environment creation took too long
(permitted time 18ms)
at
org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacade.createEnvironmentInSeparateThread(ReplicatedEnvironmentFacade.java:1577)
at
org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacade.createEnvironment(ReplicatedEnvironmentFacade.java:1521)
at
org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacade.(ReplicatedEnvironmentFacade.java:287)
at
org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacadeFactory.createEnvironmentFacade(ReplicatedEnvironmentFacadeFactory.java:130)
at
org.apache.qpid.server.store.berkeleydb.BDBConfigurationStore.init(BDBConfigurationStore.java:122)
at
org.apache.qpid.server.virtualhostnode.berkeleydb.BDBHAVirtualHostNodeImpl.activate(BDBHAVirtualHostNodeImpl.java:338)
at
org.apache.qpid.server.virtualhostnode.AbstractVirtualHostNode.doActivate(AbstractVirtualHostNode.java:162)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.qpid.server.model.AbstractConfiguredObject.attainState(AbstractConfiguredObject.java:1524)
at
org.apache.qpid.server.model.AbstractConfiguredObject.attainState(AbstractConfiguredObject.java:1503)
at
org.apache.qpid.server.model.AbstractConfiguredObject$8.onSuccess(AbstractConfiguredObject.java:1070)
at
org.apache.qpid.server.model.AbstractConfiguredObject$8.onSuccess(AbstractConfiguredObject.java:1064)
at
org.apache.qpid.server.model.AbstractConfiguredObject$22$1.run(AbstractConfiguredObject.java:2639)
at
org.apache.qpid.server.model.AbstractConfiguredObject$22$1.run(AbstractConfiguredObject.java:2635)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at
org.apache.qpid.server.model.AbstractConfiguredObject$22.onSuccess(AbstractConfiguredObject.java:2634)
at
com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1237)
at
org.apache.qpid.server.configuration.updater.TaskExecutorImpl$ImmediateIfSameThreadExecutor.execute(TaskExecutorImpl.java:400)
at
org.apache.qpid.server.configuration.updater.TaskExecutorImpl.execute(TaskExecutorImpl.java:183)
at
com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:911)
at
com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:645)
at
com.google.common.util.concurrent.AbstractFuture$TrustedFuture.addListener(AbstractFuture.java:101)
at
com.google.common.util.concurrent.Futures.addCallback(Futures.java:1209)
at
org.apache.qpid.server.model.AbstractConfiguredObject.addFutureCallback(AbstractConfiguredObject.java:2629)
at
org.apache.qpid.server.model.AbstractConfiguredObject.doAttainState(AbstractConfiguredObject.java:1063)
at
org.apache.qpid.server.model.AbstractConfiguredObject.access$600(AbstractConfiguredObject.java:95)
at
org.apache.qpid.server.model.AbstractConfiguredObject$7.performAction(AbstractConfiguredObject.java:1048)
at
org.apache.qpid.server.model.AbstractConfiguredObject$7.performAction(AbstractConfiguredObject.java:1038)
at
org.apache.qpid.server.model.AbstractConfiguredObject.applyToChildren(AbstractConfiguredObject.java:1311)
at
org.apache.qpid.server.model.AbstractConfiguredObject.doAttainState(AbstractConfiguredObject.java:1037)
at
org.apache.qpid.server.model.AbstractConfiguredObject.access$600(AbstractConfiguredObject.java:95)
at
org.apache.qpid.server.model.AbstractConfiguredObject$1.execute(AbstractConfiguredObject.java:589)
at
org.apache.qpid.server.model.AbstractConfiguredObject$1.execute(AbstractConfiguredObject.java:576)
at
org.apache.qpid.server.model.AbstractConfiguredObject$2.execute(AbstractConfiguredObject.java:637
[jira] [Commented] (QPID-8274) [Broker-J][BDB HA] Broker can fail to become active when BDB HA virtual host times out to join the group
[
https://issues.apache.org/jira/browse/QPID-8274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16768523#comment-16768523
]
Alex Rudyy commented on QPID-8274:
--
The JE {{Environment}} creation can hang as per stack trace below
{noformat}
at com.sleepycat.bind.tuple.TupleBinding.(TupleBinding.java:53)
at
com.sleepycat.je.cleaner.ExpirationProfile.populateCache(ExpirationProfile.java:193)
- locked java.util.HashMap@56d1b799
at
com.sleepycat.je.recovery.RecoveryManager.recover(RecoveryManager.java:434)
at com.sleepycat.je.dbi.EnvironmentImpl.finishInit(EnvironmentImpl.java:841)
- locked com.sleepycat.je.rep.impl.RepImpl@57b9bac8
at com.sleepycat.je.dbi.DbEnvPool.getEnvironment(DbEnvPool.java:222)
at com.sleepycat.je.Environment.makeEnvironmentImpl(Environment.java:267)
at com.sleepycat.je.Environment.(Environment.java:252)
at
com.sleepycat.je.rep.ReplicatedEnvironment.(ReplicatedEnvironment.java:607)
at
com.sleepycat.je.rep.ReplicatedEnvironment.(ReplicatedEnvironment.java:466)
at
com.sleepycat.je.rep.ReplicatedEnvironment.(ReplicatedEnvironment.java:540)
at
org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacade.createEnvironment(ReplicatedEnvironmentFacade.java:1589)
at
org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacade.access$1700(ReplicatedEnvironmentFacade.java:86)
at
org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacade$14.call(ReplicatedEnvironmentFacade.java:1537)
at
org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacade$14.call(ReplicatedEnvironmentFacade.java:1533)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{noformat}
> [Broker-J][BDB HA] Broker can fail to become active when BDB HA virtual host
> times out to join the group
>
>
> Key: QPID-8274
> URL: https://issues.apache.org/jira/browse/QPID-8274
> Project: Qpid
> Issue Type: Bug
> Components: Broker-J
>Affects Versions: qpid-java-6.1.6, qpid-java-broker-7.0.3,
> qpid-java-broker-7.0.2, qpid-java-6.0, qpid-java-6.0.1, qpid-java-6.0.2,
> qpid-java-6.0.3, qpid-java-6.0.4, qpid-java-6.0.5, qpid-java-6.1,
> qpid-java-6.0.6, qpid-java-6.1.1, qpid-java-6.1.2, qpid-java-6.0.7,
> qpid-java-6.1.3, qpid-java-6.0.8, qpid-java-6.1.4, qpid-java-broker-7.0.0,
> qpid-java-6.1.5, qpid-java-broker-7.0.1, qpid-java-6.1.7,
> qpid-java-broker-7.1.0, qpid-java-broker-7.0.4, qpid-java-broker-7.0.5,
> qpid-java-broker-7.0.6
>Reporter: Alex Rudyy
>Priority: Major
> Fix For: qpid-java-broker-7.0.7, qpid-java-broker-7.1.1
>
>
> Broker containing a BDB HA Virtual Host node (belonging to the cluster
> consisting of several nodes) can fail to start when BDB HA Virtual Host node
> times out to join the group. The broker cannot complete activation
> (transition into an ACTIVE state).
> The stack traces like bellow are reported on BDB HA VHN timeout:
> {noformat}
> ERROR [Broker-Config] (o.a.q.s.m.AbstractConfiguredObject) - Failed to open
> object with name 'node2'. Object will be put into ERROR state.
> java.lang.RuntimeException: JE replicated environment creation took too long
> (permitted time 18ms)
> at
> org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacade.createEnvironmentInSeparateThread(ReplicatedEnvironmentFacade.java:1577)
> at
> org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacade.createEnvironment(ReplicatedEnvironmentFacade.java:1521)
> at
> org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacade.(ReplicatedEnvironmentFacade.java:287)
> at
> org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacadeFactory.createEnvironmentFacade(ReplicatedEnvironmentFacadeFactory.java:130)
> at
> org.apache.qpid.server.store.berkeleydb.BDBConfigurationStore.init(BDBConfigurationStore.java:122)
> at
> org.apache.qpid.server.virtualhostnode.berkeleydb.BDBHAVirtualHostNodeImpl.activate(BDBHAVirtualHostNodeImpl.java:338)
> at
> org.apache.qpid.server.virtualhostnode.AbstractVirtualHostNode.doActivate(AbstractVirtualHostNode.java:162)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> a
[jira] [Comment Edited] (QPID-8274) [Broker-J][BDB HA] Broker can fail to become active when BDB HA virtual host times out to join the group
[
https://issues.apache.org/jira/browse/QPID-8274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16768808#comment-16768808
]
Alex Rudyy edited comment on QPID-8274 at 2/14/19 11:35 PM:
Whilst creation of environment for a replica node hang on loading of class
{{TupleBinding}}, a creation of another non-HA virtual host hang at the same
time on invocation of BDB JE method {{IntegerBinding.entryToInt(key)}}. Here is
the stack trace:
{noformat}
at
org.apache.qpid.server.store.berkeleydb.upgrade.Upgrader.getSourceVersion(Upgrader.java:119)
at
org.apache.qpid.server.store.berkeleydb.upgrade.Upgrader.upgradeIfNecessary(Upgrader.java:81)
at
org.apache.qpid.server.store.berkeleydb.StandardEnvironmentFacade.upgradeIfNecessary(StandardEnvironmentFacade.java:413)
at
org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore.upgradeStoreStructure(AbstractBDBMessageStore.java:113)
at
org.apache.qpid.server.virtualhost.AbstractVirtualHost.onActivate(AbstractVirtualHost.java:2588)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.qpid.server.model.AbstractConfiguredObject.attainState(AbstractConfiguredObject.java:1524)
at
org.apache.qpid.server.model.AbstractConfiguredObject.attainState(AbstractConfiguredObject.java:1503)
at
org.apache.qpid.server.model.AbstractConfiguredObject$8.onSuccess(AbstractConfiguredObject.java:1070)
at
org.apache.qpid.server.model.AbstractConfiguredObject$8.onSuccess(AbstractConfiguredObject.java:1064)
at
org.apache.qpid.server.model.AbstractConfiguredObject$22$1.run(AbstractConfiguredObject.java:2639)
at
org.apache.qpid.server.model.AbstractConfiguredObject$22$1.run(AbstractConfiguredObject.java:2635)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at
org.apache.qpid.server.model.AbstractConfiguredObject$22.onSuccess(AbstractConfiguredObject.java:2634)
at
com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1237)
at
org.apache.qpid.server.configuration.updater.TaskExecutorImpl$ImmediateIfSameThreadExecutor.execute(TaskExecutorImpl.java:400)
at
org.apache.qpid.server.configuration.updater.TaskExecutorImpl.execute(TaskExecutorImpl.java:183)
at
com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:911)
at
com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:645)
at
com.google.common.util.concurrent.AbstractFuture$TrustedFuture.addListener(AbstractFuture.java:101)
at com.google.common.util.concurrent.Futures.addCallback(Futures.java:1209)
at
org.apache.qpid.server.model.AbstractConfiguredObject.addFutureCallback(AbstractConfiguredObject.java:2629)
at
org.apache.qpid.server.model.AbstractConfiguredObject.doAttainState(AbstractConfiguredObject.java:1063)
at
org.apache.qpid.server.model.AbstractConfiguredObject.access$600(AbstractConfiguredObject.java:95)
at
org.apache.qpid.server.model.AbstractConfiguredObject$1.execute(AbstractConfiguredObject.java:589)
at
org.apache.qpid.server.model.AbstractConfiguredObject$1.execute(AbstractConfiguredObject.java:576)
at
org.apache.qpid.server.model.AbstractConfiguredObject$2.execute(AbstractConfiguredObject.java:637)
at
org.apache.qpid.server.model.AbstractConfiguredObject$2.execute(AbstractConfiguredObject.java:630)
at
org.apache.qpid.server.configuration.updater.TaskExecutorImpl$TaskLoggingWrapper.execute(TaskExecutorImpl.java:248)
at
org.apache.qpid.server.configuration.updater.TaskExecutorImpl$CallableWrapper$1.run(TaskExecutorImpl.java:320)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at
org.apache.qpid.server.configuration.updater.TaskExecutorImpl$CallableWrapper.call(TaskExecutorImpl.java:313)
at
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:111)
at
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:58)
at
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:75)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at
org.apache.qpid.server.bytebuffer.QpidByteBufferFactory.lambda$null$0(QpidByteBufferFactory.java:464)
at
org.apache.qpid.server.bytebuffer.QpidByteBufferFactory$$Lambda$3/174887633
[jira] [Commented] (QPID-8274) [Broker-J][BDB HA] Broker can fail to become active when BDB HA virtual host times out to join the group
[
https://issues.apache.org/jira/browse/QPID-8274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16768808#comment-16768808
]
Alex Rudyy commented on QPID-8274:
--
Whilst creation of environment for a replica node hang on loading of class
{{TupleBinding}} on attempt to , a creation of another non-HA virtual host hang
at the same time on invocation of BDB JE method
{{IntegerBinding.entryToInt(key)}}. Here is the stack trace:
{noformat}
at
org.apache.qpid.server.store.berkeleydb.upgrade.Upgrader.getSourceVersion(Upgrader.java:119)
at
org.apache.qpid.server.store.berkeleydb.upgrade.Upgrader.upgradeIfNecessary(Upgrader.java:81)
at
org.apache.qpid.server.store.berkeleydb.StandardEnvironmentFacade.upgradeIfNecessary(StandardEnvironmentFacade.java:413)
at
org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore.upgradeStoreStructure(AbstractBDBMessageStore.java:113)
at
org.apache.qpid.server.virtualhost.AbstractVirtualHost.onActivate(AbstractVirtualHost.java:2588)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.qpid.server.model.AbstractConfiguredObject.attainState(AbstractConfiguredObject.java:1524)
at
org.apache.qpid.server.model.AbstractConfiguredObject.attainState(AbstractConfiguredObject.java:1503)
at
org.apache.qpid.server.model.AbstractConfiguredObject$8.onSuccess(AbstractConfiguredObject.java:1070)
at
org.apache.qpid.server.model.AbstractConfiguredObject$8.onSuccess(AbstractConfiguredObject.java:1064)
at
org.apache.qpid.server.model.AbstractConfiguredObject$22$1.run(AbstractConfiguredObject.java:2639)
at
org.apache.qpid.server.model.AbstractConfiguredObject$22$1.run(AbstractConfiguredObject.java:2635)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at
org.apache.qpid.server.model.AbstractConfiguredObject$22.onSuccess(AbstractConfiguredObject.java:2634)
at
com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1237)
at
org.apache.qpid.server.configuration.updater.TaskExecutorImpl$ImmediateIfSameThreadExecutor.execute(TaskExecutorImpl.java:400)
at
org.apache.qpid.server.configuration.updater.TaskExecutorImpl.execute(TaskExecutorImpl.java:183)
at
com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:911)
at
com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:645)
at
com.google.common.util.concurrent.AbstractFuture$TrustedFuture.addListener(AbstractFuture.java:101)
at com.google.common.util.concurrent.Futures.addCallback(Futures.java:1209)
at
org.apache.qpid.server.model.AbstractConfiguredObject.addFutureCallback(AbstractConfiguredObject.java:2629)
at
org.apache.qpid.server.model.AbstractConfiguredObject.doAttainState(AbstractConfiguredObject.java:1063)
at
org.apache.qpid.server.model.AbstractConfiguredObject.access$600(AbstractConfiguredObject.java:95)
at
org.apache.qpid.server.model.AbstractConfiguredObject$1.execute(AbstractConfiguredObject.java:589)
at
org.apache.qpid.server.model.AbstractConfiguredObject$1.execute(AbstractConfiguredObject.java:576)
at
org.apache.qpid.server.model.AbstractConfiguredObject$2.execute(AbstractConfiguredObject.java:637)
at
org.apache.qpid.server.model.AbstractConfiguredObject$2.execute(AbstractConfiguredObject.java:630)
at
org.apache.qpid.server.configuration.updater.TaskExecutorImpl$TaskLoggingWrapper.execute(TaskExecutorImpl.java:248)
at
org.apache.qpid.server.configuration.updater.TaskExecutorImpl$CallableWrapper$1.run(TaskExecutorImpl.java:320)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at
org.apache.qpid.server.configuration.updater.TaskExecutorImpl$CallableWrapper.call(TaskExecutorImpl.java:313)
at
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:111)
at
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:58)
at
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:75)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at
org.apache.qpid.server.bytebuffer.QpidByteBufferFactory.lambda$null$0(QpidByteBufferFactory.java:464)
at
org.apache.qpid.server.bytebuffer.QpidByteBufferFactory$$Lambda$3/1748876332.run(Unknown
Source)
at java.l
[jira] [Updated] (QPID-8274) [Broker-J][BDB HA] Broker can fail to become active when BDB HA virtual host times out to join the group
[
https://issues.apache.org/jira/browse/QPID-8274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Rudyy updated QPID-8274:
-
Attachment: thread-dump.txt
> [Broker-J][BDB HA] Broker can fail to become active when BDB HA virtual host
> times out to join the group
>
>
> Key: QPID-8274
> URL: https://issues.apache.org/jira/browse/QPID-8274
> Project: Qpid
> Issue Type: Bug
> Components: Broker-J
>Affects Versions: qpid-java-6.1.6, qpid-java-broker-7.0.3,
> qpid-java-broker-7.0.2, qpid-java-6.0, qpid-java-6.0.1, qpid-java-6.0.2,
> qpid-java-6.0.3, qpid-java-6.0.4, qpid-java-6.0.5, qpid-java-6.1,
> qpid-java-6.0.6, qpid-java-6.1.1, qpid-java-6.1.2, qpid-java-6.0.7,
> qpid-java-6.1.3, qpid-java-6.0.8, qpid-java-6.1.4, qpid-java-broker-7.0.0,
> qpid-java-6.1.5, qpid-java-broker-7.0.1, qpid-java-6.1.7,
> qpid-java-broker-7.1.0, qpid-java-broker-7.0.4, qpid-java-broker-7.0.5,
> qpid-java-broker-7.0.6
>Reporter: Alex Rudyy
>Priority: Major
> Fix For: qpid-java-broker-7.0.7, qpid-java-broker-7.1.1
>
> Attachments: thread-dump.txt
>
>
> Broker containing a BDB HA Virtual Host node (belonging to the cluster
> consisting of several nodes) can fail to start when BDB HA Virtual Host node
> times out to join the group. The broker cannot complete activation
> (transition into an ACTIVE state).
> The stack traces like bellow are reported on BDB HA VHN timeout:
> {noformat}
> ERROR [Broker-Config] (o.a.q.s.m.AbstractConfiguredObject) - Failed to open
> object with name 'node2'. Object will be put into ERROR state.
> java.lang.RuntimeException: JE replicated environment creation took too long
> (permitted time 18ms)
> at
> org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacade.createEnvironmentInSeparateThread(ReplicatedEnvironmentFacade.java:1577)
> at
> org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacade.createEnvironment(ReplicatedEnvironmentFacade.java:1521)
> at
> org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacade.(ReplicatedEnvironmentFacade.java:287)
> at
> org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacadeFactory.createEnvironmentFacade(ReplicatedEnvironmentFacadeFactory.java:130)
> at
> org.apache.qpid.server.store.berkeleydb.BDBConfigurationStore.init(BDBConfigurationStore.java:122)
> at
> org.apache.qpid.server.virtualhostnode.berkeleydb.BDBHAVirtualHostNodeImpl.activate(BDBHAVirtualHostNodeImpl.java:338)
> at
> org.apache.qpid.server.virtualhostnode.AbstractVirtualHostNode.doActivate(AbstractVirtualHostNode.java:162)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.qpid.server.model.AbstractConfiguredObject.attainState(AbstractConfiguredObject.java:1524)
> at
> org.apache.qpid.server.model.AbstractConfiguredObject.attainState(AbstractConfiguredObject.java:1503)
> at
> org.apache.qpid.server.model.AbstractConfiguredObject$8.onSuccess(AbstractConfiguredObject.java:1070)
> at
> org.apache.qpid.server.model.AbstractConfiguredObject$8.onSuccess(AbstractConfiguredObject.java:1064)
> at
> org.apache.qpid.server.model.AbstractConfiguredObject$22$1.run(AbstractConfiguredObject.java:2639)
> at
> org.apache.qpid.server.model.AbstractConfiguredObject$22$1.run(AbstractConfiguredObject.java:2635)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:360)
> at
> org.apache.qpid.server.model.AbstractConfiguredObject$22.onSuccess(AbstractConfiguredObject.java:2634)
> at
> com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1237)
> at
> org.apache.qpid.server.configuration.updater.TaskExecutorImpl$ImmediateIfSameThreadExecutor.execute(TaskExecutorImpl.java:400)
> at
> org.apache.qpid.server.configuration.updater.TaskExecutorImpl.execute(TaskExecutorImpl.java:183)
> at
> com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:911)
> at
> com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:645)
> at
> com.google.common.util.concurrent.AbstractFuture$TrustedFuture.addListener(AbstractFuture.java:101)
> at
> com.google.common.util.concurrent.Futures.addCallback(Futures.java:1209)
> at
> or
[jira] [Updated] (QPID-8274) [Broker-J][BDB HA] Broker can fail to become active when BDB HA virtual host times out to join the group
[
https://issues.apache.org/jira/browse/QPID-8274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Rudyy updated QPID-8274:
-
Environment:
[Broker-Config] (q.m.b.platform) - [Broker] BRK-1010 : Platform : JVM : Oracle
Corporation version: 1.8.0_161-b12 OS : Linux version:
3.10.0-514.6.1.el7.x86_64 arch: amd64 cores: 40
> [Broker-J][BDB HA] Broker can fail to become active when BDB HA virtual host
> times out to join the group
>
>
> Key: QPID-8274
> URL: https://issues.apache.org/jira/browse/QPID-8274
> Project: Qpid
> Issue Type: Bug
> Components: Broker-J
>Affects Versions: qpid-java-6.1.6, qpid-java-broker-7.0.3,
> qpid-java-broker-7.0.2, qpid-java-6.0, qpid-java-6.0.1, qpid-java-6.0.2,
> qpid-java-6.0.3, qpid-java-6.0.4, qpid-java-6.0.5, qpid-java-6.1,
> qpid-java-6.0.6, qpid-java-6.1.1, qpid-java-6.1.2, qpid-java-6.0.7,
> qpid-java-6.1.3, qpid-java-6.0.8, qpid-java-6.1.4, qpid-java-broker-7.0.0,
> qpid-java-6.1.5, qpid-java-broker-7.0.1, qpid-java-6.1.7,
> qpid-java-broker-7.1.0, qpid-java-broker-7.0.4, qpid-java-broker-7.0.5,
> qpid-java-broker-7.0.6
> Environment: [Broker-Config] (q.m.b.platform) - [Broker] BRK-1010 :
> Platform : JVM : Oracle Corporation version: 1.8.0_161-b12 OS : Linux
> version: 3.10.0-514.6.1.el7.x86_64 arch: amd64 cores: 40
>Reporter: Alex Rudyy
>Priority: Major
> Fix For: qpid-java-broker-7.0.7, qpid-java-broker-7.1.1
>
> Attachments: thread-dump.txt
>
>
> Broker containing a BDB HA Virtual Host node (belonging to the cluster
> consisting of several nodes) can fail to start when BDB HA Virtual Host node
> times out to join the group. The broker cannot complete activation
> (transition into an ACTIVE state).
> The stack traces like bellow are reported on BDB HA VHN timeout:
> {noformat}
> ERROR [Broker-Config] (o.a.q.s.m.AbstractConfiguredObject) - Failed to open
> object with name 'node2'. Object will be put into ERROR state.
> java.lang.RuntimeException: JE replicated environment creation took too long
> (permitted time 18ms)
> at
> org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacade.createEnvironmentInSeparateThread(ReplicatedEnvironmentFacade.java:1577)
> at
> org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacade.createEnvironment(ReplicatedEnvironmentFacade.java:1521)
> at
> org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacade.(ReplicatedEnvironmentFacade.java:287)
> at
> org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacadeFactory.createEnvironmentFacade(ReplicatedEnvironmentFacadeFactory.java:130)
> at
> org.apache.qpid.server.store.berkeleydb.BDBConfigurationStore.init(BDBConfigurationStore.java:122)
> at
> org.apache.qpid.server.virtualhostnode.berkeleydb.BDBHAVirtualHostNodeImpl.activate(BDBHAVirtualHostNodeImpl.java:338)
> at
> org.apache.qpid.server.virtualhostnode.AbstractVirtualHostNode.doActivate(AbstractVirtualHostNode.java:162)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.qpid.server.model.AbstractConfiguredObject.attainState(AbstractConfiguredObject.java:1524)
> at
> org.apache.qpid.server.model.AbstractConfiguredObject.attainState(AbstractConfiguredObject.java:1503)
> at
> org.apache.qpid.server.model.AbstractConfiguredObject$8.onSuccess(AbstractConfiguredObject.java:1070)
> at
> org.apache.qpid.server.model.AbstractConfiguredObject$8.onSuccess(AbstractConfiguredObject.java:1064)
> at
> org.apache.qpid.server.model.AbstractConfiguredObject$22$1.run(AbstractConfiguredObject.java:2639)
> at
> org.apache.qpid.server.model.AbstractConfiguredObject$22$1.run(AbstractConfiguredObject.java:2635)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:360)
> at
> org.apache.qpid.server.model.AbstractConfiguredObject$22.onSuccess(AbstractConfiguredObject.java:2634)
> at
> com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1237)
> at
> org.apache.qpid.server.configuration.updater.TaskExecutorImpl$ImmediateIfSameThreadExecutor.execute(TaskExecutorImpl.java:400)
> at
> org.apache.qpid.server.configuration.updater.TaskExecutorImpl.execute(TaskExecutorImpl.java:183)
> at
> com.google.common.util.concurrent
