[jira] [Updated] (QPID-8274) [Broker-J][BDB HA] Broker can fail to become active when BDB HA virtual host times out to join the group

2019-02-14 Thread Alex Rudyy (JIRA)


 [ 
https://issues.apache.org/jira/browse/QPID-8274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alex Rudyy updated QPID-8274:
-
Environment: 
[Broker-Config] (q.m.b.platform) - [Broker] BRK-1010 : Platform : JVM : Oracle 
Corporation version: 1.8.0_161-b12 OS : Linux version: 
3.10.0-514.6.1.el7.x86_64 arch: amd64 cores: 40


> [Broker-J][BDB HA] Broker can fail to become active when BDB HA virtual host 
> times out to join the group
> 
>
> Key: QPID-8274
> URL: https://issues.apache.org/jira/browse/QPID-8274
> Project: Qpid
>  Issue Type: Bug
>  Components: Broker-J
>Affects Versions: qpid-java-6.1.6, qpid-java-broker-7.0.3, 
> qpid-java-broker-7.0.2, qpid-java-6.0, qpid-java-6.0.1, qpid-java-6.0.2, 
> qpid-java-6.0.3, qpid-java-6.0.4, qpid-java-6.0.5, qpid-java-6.1, 
> qpid-java-6.0.6, qpid-java-6.1.1, qpid-java-6.1.2, qpid-java-6.0.7, 
> qpid-java-6.1.3, qpid-java-6.0.8, qpid-java-6.1.4, qpid-java-broker-7.0.0, 
> qpid-java-6.1.5, qpid-java-broker-7.0.1, qpid-java-6.1.7, 
> qpid-java-broker-7.1.0, qpid-java-broker-7.0.4, qpid-java-broker-7.0.5, 
> qpid-java-broker-7.0.6
> Environment: [Broker-Config] (q.m.b.platform) - [Broker] BRK-1010 : 
> Platform : JVM : Oracle Corporation version: 1.8.0_161-b12 OS : Linux 
> version: 3.10.0-514.6.1.el7.x86_64 arch: amd64 cores: 40
>Reporter: Alex Rudyy
>Priority: Major
> Fix For: qpid-java-broker-7.0.7, qpid-java-broker-7.1.1
>
> Attachments: thread-dump.txt
>
>
> Broker containing a BDB HA Virtual Host node (belonging to the cluster 
> consisting of several nodes) can fail to start when BDB HA Virtual Host node 
> times out to join the group. The broker cannot complete activation 
> (transition into an ACTIVE state).
> The stack traces like bellow are reported on BDB HA VHN timeout:
> {noformat}
> ERROR [Broker-Config] (o.a.q.s.m.AbstractConfiguredObject) - Failed to open 
> object with name 'node2'.  Object will be put into ERROR state.
> java.lang.RuntimeException: JE replicated environment creation took too long 
> (permitted time 18ms)
> at 
> org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacade.createEnvironmentInSeparateThread(ReplicatedEnvironmentFacade.java:1577)
> at 
> org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacade.createEnvironment(ReplicatedEnvironmentFacade.java:1521)
> at 
> org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacade.(ReplicatedEnvironmentFacade.java:287)
> at 
> org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacadeFactory.createEnvironmentFacade(ReplicatedEnvironmentFacadeFactory.java:130)
> at 
> org.apache.qpid.server.store.berkeleydb.BDBConfigurationStore.init(BDBConfigurationStore.java:122)
> at 
> org.apache.qpid.server.virtualhostnode.berkeleydb.BDBHAVirtualHostNodeImpl.activate(BDBHAVirtualHostNodeImpl.java:338)
> at 
> org.apache.qpid.server.virtualhostnode.AbstractVirtualHostNode.doActivate(AbstractVirtualHostNode.java:162)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.apache.qpid.server.model.AbstractConfiguredObject.attainState(AbstractConfiguredObject.java:1524)
> at 
> org.apache.qpid.server.model.AbstractConfiguredObject.attainState(AbstractConfiguredObject.java:1503)
> at 
> org.apache.qpid.server.model.AbstractConfiguredObject$8.onSuccess(AbstractConfiguredObject.java:1070)
> at 
> org.apache.qpid.server.model.AbstractConfiguredObject$8.onSuccess(AbstractConfiguredObject.java:1064)
> at 
> org.apache.qpid.server.model.AbstractConfiguredObject$22$1.run(AbstractConfiguredObject.java:2639)
> at 
> org.apache.qpid.server.model.AbstractConfiguredObject$22$1.run(AbstractConfiguredObject.java:2635)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:360)
> at 
> org.apache.qpid.server.model.AbstractConfiguredObject$22.onSuccess(AbstractConfiguredObject.java:2634)
> at 
> com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1237)
> at 
> org.apache.qpid.server.configuration.updater.TaskExecutorImpl$ImmediateIfSameThreadExecutor.execute(TaskExecutorImpl.java:400)
> at 
> org.apache.qpid.server.configuration.updater.TaskExecutorImpl.execute(TaskExecutorImpl.java:183)
> at 
> 

[jira] [Updated] (QPID-8274) [Broker-J][BDB HA] Broker can fail to become active when BDB HA virtual host times out to join the group

2019-02-14 Thread Alex Rudyy (JIRA)


 [ 
https://issues.apache.org/jira/browse/QPID-8274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alex Rudyy updated QPID-8274:
-
Attachment: thread-dump.txt

> [Broker-J][BDB HA] Broker can fail to become active when BDB HA virtual host 
> times out to join the group
> 
>
> Key: QPID-8274
> URL: https://issues.apache.org/jira/browse/QPID-8274
> Project: Qpid
>  Issue Type: Bug
>  Components: Broker-J
>Affects Versions: qpid-java-6.1.6, qpid-java-broker-7.0.3, 
> qpid-java-broker-7.0.2, qpid-java-6.0, qpid-java-6.0.1, qpid-java-6.0.2, 
> qpid-java-6.0.3, qpid-java-6.0.4, qpid-java-6.0.5, qpid-java-6.1, 
> qpid-java-6.0.6, qpid-java-6.1.1, qpid-java-6.1.2, qpid-java-6.0.7, 
> qpid-java-6.1.3, qpid-java-6.0.8, qpid-java-6.1.4, qpid-java-broker-7.0.0, 
> qpid-java-6.1.5, qpid-java-broker-7.0.1, qpid-java-6.1.7, 
> qpid-java-broker-7.1.0, qpid-java-broker-7.0.4, qpid-java-broker-7.0.5, 
> qpid-java-broker-7.0.6
>Reporter: Alex Rudyy
>Priority: Major
> Fix For: qpid-java-broker-7.0.7, qpid-java-broker-7.1.1
>
> Attachments: thread-dump.txt
>
>
> Broker containing a BDB HA Virtual Host node (belonging to the cluster 
> consisting of several nodes) can fail to start when BDB HA Virtual Host node 
> times out to join the group. The broker cannot complete activation 
> (transition into an ACTIVE state).
> The stack traces like bellow are reported on BDB HA VHN timeout:
> {noformat}
> ERROR [Broker-Config] (o.a.q.s.m.AbstractConfiguredObject) - Failed to open 
> object with name 'node2'.  Object will be put into ERROR state.
> java.lang.RuntimeException: JE replicated environment creation took too long 
> (permitted time 18ms)
> at 
> org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacade.createEnvironmentInSeparateThread(ReplicatedEnvironmentFacade.java:1577)
> at 
> org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacade.createEnvironment(ReplicatedEnvironmentFacade.java:1521)
> at 
> org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacade.(ReplicatedEnvironmentFacade.java:287)
> at 
> org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacadeFactory.createEnvironmentFacade(ReplicatedEnvironmentFacadeFactory.java:130)
> at 
> org.apache.qpid.server.store.berkeleydb.BDBConfigurationStore.init(BDBConfigurationStore.java:122)
> at 
> org.apache.qpid.server.virtualhostnode.berkeleydb.BDBHAVirtualHostNodeImpl.activate(BDBHAVirtualHostNodeImpl.java:338)
> at 
> org.apache.qpid.server.virtualhostnode.AbstractVirtualHostNode.doActivate(AbstractVirtualHostNode.java:162)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.apache.qpid.server.model.AbstractConfiguredObject.attainState(AbstractConfiguredObject.java:1524)
> at 
> org.apache.qpid.server.model.AbstractConfiguredObject.attainState(AbstractConfiguredObject.java:1503)
> at 
> org.apache.qpid.server.model.AbstractConfiguredObject$8.onSuccess(AbstractConfiguredObject.java:1070)
> at 
> org.apache.qpid.server.model.AbstractConfiguredObject$8.onSuccess(AbstractConfiguredObject.java:1064)
> at 
> org.apache.qpid.server.model.AbstractConfiguredObject$22$1.run(AbstractConfiguredObject.java:2639)
> at 
> org.apache.qpid.server.model.AbstractConfiguredObject$22$1.run(AbstractConfiguredObject.java:2635)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:360)
> at 
> org.apache.qpid.server.model.AbstractConfiguredObject$22.onSuccess(AbstractConfiguredObject.java:2634)
> at 
> com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1237)
> at 
> org.apache.qpid.server.configuration.updater.TaskExecutorImpl$ImmediateIfSameThreadExecutor.execute(TaskExecutorImpl.java:400)
> at 
> org.apache.qpid.server.configuration.updater.TaskExecutorImpl.execute(TaskExecutorImpl.java:183)
> at 
> com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:911)
> at 
> com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:645)
> at 
> com.google.common.util.concurrent.AbstractFuture$TrustedFuture.addListener(AbstractFuture.java:101)
> at 
> com.google.common.util.concurrent.Futures.addCallback(Futures.java:1209)
> at 
> 

[jira] [Commented] (QPID-8274) [Broker-J][BDB HA] Broker can fail to become active when BDB HA virtual host times out to join the group

2019-02-14 Thread Alex Rudyy (JIRA)


[ 
https://issues.apache.org/jira/browse/QPID-8274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16768808#comment-16768808
 ] 

Alex Rudyy commented on QPID-8274:
--

Whilst creation of environment for a replica node hang on loading of class 
{{TupleBinding}} on attempt to , a creation of another non-HA virtual host hang 
at the same time on invocation of BDB JE method 
{{IntegerBinding.entryToInt(key)}}. Here is the stack trace:

{noformat}
at 
org.apache.qpid.server.store.berkeleydb.upgrade.Upgrader.getSourceVersion(Upgrader.java:119)
at 
org.apache.qpid.server.store.berkeleydb.upgrade.Upgrader.upgradeIfNecessary(Upgrader.java:81)
at 
org.apache.qpid.server.store.berkeleydb.StandardEnvironmentFacade.upgradeIfNecessary(StandardEnvironmentFacade.java:413)
at 
org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore.upgradeStoreStructure(AbstractBDBMessageStore.java:113)
at 
org.apache.qpid.server.virtualhost.AbstractVirtualHost.onActivate(AbstractVirtualHost.java:2588)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.qpid.server.model.AbstractConfiguredObject.attainState(AbstractConfiguredObject.java:1524)
at 
org.apache.qpid.server.model.AbstractConfiguredObject.attainState(AbstractConfiguredObject.java:1503)
at 
org.apache.qpid.server.model.AbstractConfiguredObject$8.onSuccess(AbstractConfiguredObject.java:1070)
at 
org.apache.qpid.server.model.AbstractConfiguredObject$8.onSuccess(AbstractConfiguredObject.java:1064)
at 
org.apache.qpid.server.model.AbstractConfiguredObject$22$1.run(AbstractConfiguredObject.java:2639)
at 
org.apache.qpid.server.model.AbstractConfiguredObject$22$1.run(AbstractConfiguredObject.java:2635)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at 
org.apache.qpid.server.model.AbstractConfiguredObject$22.onSuccess(AbstractConfiguredObject.java:2634)
at 
com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1237)
at 
org.apache.qpid.server.configuration.updater.TaskExecutorImpl$ImmediateIfSameThreadExecutor.execute(TaskExecutorImpl.java:400)
at 
org.apache.qpid.server.configuration.updater.TaskExecutorImpl.execute(TaskExecutorImpl.java:183)
at 
com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:911)
at 
com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:645)
at 
com.google.common.util.concurrent.AbstractFuture$TrustedFuture.addListener(AbstractFuture.java:101)
at com.google.common.util.concurrent.Futures.addCallback(Futures.java:1209)
at 
org.apache.qpid.server.model.AbstractConfiguredObject.addFutureCallback(AbstractConfiguredObject.java:2629)
at 
org.apache.qpid.server.model.AbstractConfiguredObject.doAttainState(AbstractConfiguredObject.java:1063)
at 
org.apache.qpid.server.model.AbstractConfiguredObject.access$600(AbstractConfiguredObject.java:95)
at 
org.apache.qpid.server.model.AbstractConfiguredObject$1.execute(AbstractConfiguredObject.java:589)
at 
org.apache.qpid.server.model.AbstractConfiguredObject$1.execute(AbstractConfiguredObject.java:576)
at 
org.apache.qpid.server.model.AbstractConfiguredObject$2.execute(AbstractConfiguredObject.java:637)
at 
org.apache.qpid.server.model.AbstractConfiguredObject$2.execute(AbstractConfiguredObject.java:630)
at 
org.apache.qpid.server.configuration.updater.TaskExecutorImpl$TaskLoggingWrapper.execute(TaskExecutorImpl.java:248)
at 
org.apache.qpid.server.configuration.updater.TaskExecutorImpl$CallableWrapper$1.run(TaskExecutorImpl.java:320)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at 
org.apache.qpid.server.configuration.updater.TaskExecutorImpl$CallableWrapper.call(TaskExecutorImpl.java:313)
at 
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:111)
at 
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:58)
at 
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:75)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at 
org.apache.qpid.server.bytebuffer.QpidByteBufferFactory.lambda$null$0(QpidByteBufferFactory.java:464)
at 
org.apache.qpid.server.bytebuffer.QpidByteBufferFactory$$Lambda$3/1748876332.run(Unknown
 Source)
at 

[jira] [Comment Edited] (QPID-8274) [Broker-J][BDB HA] Broker can fail to become active when BDB HA virtual host times out to join the group

2019-02-14 Thread Alex Rudyy (JIRA)


[ 
https://issues.apache.org/jira/browse/QPID-8274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16768808#comment-16768808
 ] 

Alex Rudyy edited comment on QPID-8274 at 2/14/19 11:35 PM:


Whilst creation of environment for a replica node hang on loading of class 
{{TupleBinding}}, a creation of another non-HA virtual host hang at the same 
time on invocation of BDB JE method {{IntegerBinding.entryToInt(key)}}. Here is 
the stack trace:

{noformat}
at 
org.apache.qpid.server.store.berkeleydb.upgrade.Upgrader.getSourceVersion(Upgrader.java:119)
at 
org.apache.qpid.server.store.berkeleydb.upgrade.Upgrader.upgradeIfNecessary(Upgrader.java:81)
at 
org.apache.qpid.server.store.berkeleydb.StandardEnvironmentFacade.upgradeIfNecessary(StandardEnvironmentFacade.java:413)
at 
org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore.upgradeStoreStructure(AbstractBDBMessageStore.java:113)
at 
org.apache.qpid.server.virtualhost.AbstractVirtualHost.onActivate(AbstractVirtualHost.java:2588)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.qpid.server.model.AbstractConfiguredObject.attainState(AbstractConfiguredObject.java:1524)
at 
org.apache.qpid.server.model.AbstractConfiguredObject.attainState(AbstractConfiguredObject.java:1503)
at 
org.apache.qpid.server.model.AbstractConfiguredObject$8.onSuccess(AbstractConfiguredObject.java:1070)
at 
org.apache.qpid.server.model.AbstractConfiguredObject$8.onSuccess(AbstractConfiguredObject.java:1064)
at 
org.apache.qpid.server.model.AbstractConfiguredObject$22$1.run(AbstractConfiguredObject.java:2639)
at 
org.apache.qpid.server.model.AbstractConfiguredObject$22$1.run(AbstractConfiguredObject.java:2635)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at 
org.apache.qpid.server.model.AbstractConfiguredObject$22.onSuccess(AbstractConfiguredObject.java:2634)
at 
com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1237)
at 
org.apache.qpid.server.configuration.updater.TaskExecutorImpl$ImmediateIfSameThreadExecutor.execute(TaskExecutorImpl.java:400)
at 
org.apache.qpid.server.configuration.updater.TaskExecutorImpl.execute(TaskExecutorImpl.java:183)
at 
com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:911)
at 
com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:645)
at 
com.google.common.util.concurrent.AbstractFuture$TrustedFuture.addListener(AbstractFuture.java:101)
at com.google.common.util.concurrent.Futures.addCallback(Futures.java:1209)
at 
org.apache.qpid.server.model.AbstractConfiguredObject.addFutureCallback(AbstractConfiguredObject.java:2629)
at 
org.apache.qpid.server.model.AbstractConfiguredObject.doAttainState(AbstractConfiguredObject.java:1063)
at 
org.apache.qpid.server.model.AbstractConfiguredObject.access$600(AbstractConfiguredObject.java:95)
at 
org.apache.qpid.server.model.AbstractConfiguredObject$1.execute(AbstractConfiguredObject.java:589)
at 
org.apache.qpid.server.model.AbstractConfiguredObject$1.execute(AbstractConfiguredObject.java:576)
at 
org.apache.qpid.server.model.AbstractConfiguredObject$2.execute(AbstractConfiguredObject.java:637)
at 
org.apache.qpid.server.model.AbstractConfiguredObject$2.execute(AbstractConfiguredObject.java:630)
at 
org.apache.qpid.server.configuration.updater.TaskExecutorImpl$TaskLoggingWrapper.execute(TaskExecutorImpl.java:248)
at 
org.apache.qpid.server.configuration.updater.TaskExecutorImpl$CallableWrapper$1.run(TaskExecutorImpl.java:320)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at 
org.apache.qpid.server.configuration.updater.TaskExecutorImpl$CallableWrapper.call(TaskExecutorImpl.java:313)
at 
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:111)
at 
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:58)
at 
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:75)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at 
org.apache.qpid.server.bytebuffer.QpidByteBufferFactory.lambda$null$0(QpidByteBufferFactory.java:464)
at 
org.apache.qpid.server.bytebuffer.QpidByteBufferFactory$$Lambda$3/1748876332.run(Unknown
 

[jira] [Commented] (QPID-8274) [Broker-J][BDB HA] Broker can fail to become active when BDB HA virtual host times out to join the group

2019-02-14 Thread Alex Rudyy (JIRA)


[ 
https://issues.apache.org/jira/browse/QPID-8274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16768523#comment-16768523
 ] 

Alex Rudyy commented on QPID-8274:
--

The JE {{Environment}} creation can hang as per stack trace below
{noformat}
 at com.sleepycat.bind.tuple.TupleBinding.(TupleBinding.java:53)
at 
com.sleepycat.je.cleaner.ExpirationProfile.populateCache(ExpirationProfile.java:193)
-  locked java.util.HashMap@56d1b799
at 
com.sleepycat.je.recovery.RecoveryManager.recover(RecoveryManager.java:434)
at com.sleepycat.je.dbi.EnvironmentImpl.finishInit(EnvironmentImpl.java:841)
-  locked com.sleepycat.je.rep.impl.RepImpl@57b9bac8
at com.sleepycat.je.dbi.DbEnvPool.getEnvironment(DbEnvPool.java:222)
at com.sleepycat.je.Environment.makeEnvironmentImpl(Environment.java:267)
at com.sleepycat.je.Environment.(Environment.java:252)
at 
com.sleepycat.je.rep.ReplicatedEnvironment.(ReplicatedEnvironment.java:607)
at 
com.sleepycat.je.rep.ReplicatedEnvironment.(ReplicatedEnvironment.java:466)
at 
com.sleepycat.je.rep.ReplicatedEnvironment.(ReplicatedEnvironment.java:540)
at 
org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacade.createEnvironment(ReplicatedEnvironmentFacade.java:1589)
at 
org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacade.access$1700(ReplicatedEnvironmentFacade.java:86)
at 
org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacade$14.call(ReplicatedEnvironmentFacade.java:1537)
at 
org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacade$14.call(ReplicatedEnvironmentFacade.java:1533)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{noformat}

> [Broker-J][BDB HA] Broker can fail to become active when BDB HA virtual host 
> times out to join the group
> 
>
> Key: QPID-8274
> URL: https://issues.apache.org/jira/browse/QPID-8274
> Project: Qpid
>  Issue Type: Bug
>  Components: Broker-J
>Affects Versions: qpid-java-6.1.6, qpid-java-broker-7.0.3, 
> qpid-java-broker-7.0.2, qpid-java-6.0, qpid-java-6.0.1, qpid-java-6.0.2, 
> qpid-java-6.0.3, qpid-java-6.0.4, qpid-java-6.0.5, qpid-java-6.1, 
> qpid-java-6.0.6, qpid-java-6.1.1, qpid-java-6.1.2, qpid-java-6.0.7, 
> qpid-java-6.1.3, qpid-java-6.0.8, qpid-java-6.1.4, qpid-java-broker-7.0.0, 
> qpid-java-6.1.5, qpid-java-broker-7.0.1, qpid-java-6.1.7, 
> qpid-java-broker-7.1.0, qpid-java-broker-7.0.4, qpid-java-broker-7.0.5, 
> qpid-java-broker-7.0.6
>Reporter: Alex Rudyy
>Priority: Major
> Fix For: qpid-java-broker-7.0.7, qpid-java-broker-7.1.1
>
>
> Broker containing a BDB HA Virtual Host node (belonging to the cluster 
> consisting of several nodes) can fail to start when BDB HA Virtual Host node 
> times out to join the group. The broker cannot complete activation 
> (transition into an ACTIVE state).
> The stack traces like bellow are reported on BDB HA VHN timeout:
> {noformat}
> ERROR [Broker-Config] (o.a.q.s.m.AbstractConfiguredObject) - Failed to open 
> object with name 'node2'.  Object will be put into ERROR state.
> java.lang.RuntimeException: JE replicated environment creation took too long 
> (permitted time 18ms)
> at 
> org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacade.createEnvironmentInSeparateThread(ReplicatedEnvironmentFacade.java:1577)
> at 
> org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacade.createEnvironment(ReplicatedEnvironmentFacade.java:1521)
> at 
> org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacade.(ReplicatedEnvironmentFacade.java:287)
> at 
> org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacadeFactory.createEnvironmentFacade(ReplicatedEnvironmentFacadeFactory.java:130)
> at 
> org.apache.qpid.server.store.berkeleydb.BDBConfigurationStore.init(BDBConfigurationStore.java:122)
> at 
> org.apache.qpid.server.virtualhostnode.berkeleydb.BDBHAVirtualHostNodeImpl.activate(BDBHAVirtualHostNodeImpl.java:338)
> at 
> org.apache.qpid.server.virtualhostnode.AbstractVirtualHostNode.doActivate(AbstractVirtualHostNode.java:162)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at 

[jira] [Updated] (QPID-8274) [Broker-J][BDB HA] Broker can fail to become active when BDB HA virtual host times out to join the group

2019-02-14 Thread Alex Rudyy (JIRA)


 [ 
https://issues.apache.org/jira/browse/QPID-8274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alex Rudyy updated QPID-8274:
-
Description: 
Broker containing a BDB HA Virtual Host node (belonging to the cluster 
consisting of several nodes) can fail to start when BDB HA Virtual Host node 
times out to join the group. The broker cannot complete activation (transition 
into an ACTIVE state).

The stack traces like bellow are reported on BDB HA VHN timeout:

{noformat}
ERROR [Broker-Config] (o.a.q.s.m.AbstractConfiguredObject) - Failed to open 
object with name 'node2'.  Object will be put into ERROR state.
java.lang.RuntimeException: JE replicated environment creation took too long 
(permitted time 18ms)
at 
org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacade.createEnvironmentInSeparateThread(ReplicatedEnvironmentFacade.java:1577)
at 
org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacade.createEnvironment(ReplicatedEnvironmentFacade.java:1521)
at 
org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacade.(ReplicatedEnvironmentFacade.java:287)
at 
org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacadeFactory.createEnvironmentFacade(ReplicatedEnvironmentFacadeFactory.java:130)
at 
org.apache.qpid.server.store.berkeleydb.BDBConfigurationStore.init(BDBConfigurationStore.java:122)
at 
org.apache.qpid.server.virtualhostnode.berkeleydb.BDBHAVirtualHostNodeImpl.activate(BDBHAVirtualHostNodeImpl.java:338)
at 
org.apache.qpid.server.virtualhostnode.AbstractVirtualHostNode.doActivate(AbstractVirtualHostNode.java:162)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.qpid.server.model.AbstractConfiguredObject.attainState(AbstractConfiguredObject.java:1524)
at 
org.apache.qpid.server.model.AbstractConfiguredObject.attainState(AbstractConfiguredObject.java:1503)
at 
org.apache.qpid.server.model.AbstractConfiguredObject$8.onSuccess(AbstractConfiguredObject.java:1070)
at 
org.apache.qpid.server.model.AbstractConfiguredObject$8.onSuccess(AbstractConfiguredObject.java:1064)
at 
org.apache.qpid.server.model.AbstractConfiguredObject$22$1.run(AbstractConfiguredObject.java:2639)
at 
org.apache.qpid.server.model.AbstractConfiguredObject$22$1.run(AbstractConfiguredObject.java:2635)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at 
org.apache.qpid.server.model.AbstractConfiguredObject$22.onSuccess(AbstractConfiguredObject.java:2634)
at 
com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1237)
at 
org.apache.qpid.server.configuration.updater.TaskExecutorImpl$ImmediateIfSameThreadExecutor.execute(TaskExecutorImpl.java:400)
at 
org.apache.qpid.server.configuration.updater.TaskExecutorImpl.execute(TaskExecutorImpl.java:183)
at 
com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:911)
at 
com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:645)
at 
com.google.common.util.concurrent.AbstractFuture$TrustedFuture.addListener(AbstractFuture.java:101)
at 
com.google.common.util.concurrent.Futures.addCallback(Futures.java:1209)
at 
org.apache.qpid.server.model.AbstractConfiguredObject.addFutureCallback(AbstractConfiguredObject.java:2629)
at 
org.apache.qpid.server.model.AbstractConfiguredObject.doAttainState(AbstractConfiguredObject.java:1063)
at 
org.apache.qpid.server.model.AbstractConfiguredObject.access$600(AbstractConfiguredObject.java:95)
at 
org.apache.qpid.server.model.AbstractConfiguredObject$7.performAction(AbstractConfiguredObject.java:1048)
at 
org.apache.qpid.server.model.AbstractConfiguredObject$7.performAction(AbstractConfiguredObject.java:1038)
at 
org.apache.qpid.server.model.AbstractConfiguredObject.applyToChildren(AbstractConfiguredObject.java:1311)
at 
org.apache.qpid.server.model.AbstractConfiguredObject.doAttainState(AbstractConfiguredObject.java:1037)
at 
org.apache.qpid.server.model.AbstractConfiguredObject.access$600(AbstractConfiguredObject.java:95)
at 
org.apache.qpid.server.model.AbstractConfiguredObject$1.execute(AbstractConfiguredObject.java:589)
at 
org.apache.qpid.server.model.AbstractConfiguredObject$1.execute(AbstractConfiguredObject.java:576)
at