[jira] [Commented] (QPID-8274) [Broker-J][BDB HA] Broker can fail to become active when BDB HA virtual host times out to join the group
[
https://issues.apache.org/jira/browse/QPID-8274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16769307#comment-16769307
]
Alex Rudyy commented on QPID-8274:
--
It looks like a defect in BDB JE.
A number of JDK defect reports
(JDK-[6301579|https://bugs.openjdk.java.net/browse/JDK-6301579],
[JDK-8037567|https://bugs.openjdk.java.net/browse/JDK-8037567]) was raised
about the same problem with class initialization, but, they have been closed as
"not an issue".
It seems Java spec does not allow static initializers to reference subclasses.
The TupleBinding is parent of IntegerBinding. In static block of TupleBinding
an instance of IntegerBinding is created. It seems that on attempt to
initialize IntegerBinding from multiple threads a deadlock occurs.
I raised a discussion on BDB JE forum about this problem
[https://community.oracle.com/thread/4201425|https://community.oracle.com/thread/4201425]
I am inclined to close this JIRA as not an issue, but before doing that, I
would prefer to wait for any reply in Oracle BDB JE forum.
> [Broker-J][BDB HA] Broker can fail to become active when BDB HA virtual host
> times out to join the group
>
>
> Key: QPID-8274
> URL: https://issues.apache.org/jira/browse/QPID-8274
> Project: Qpid
> Issue Type: Bug
> Components: Broker-J
>Affects Versions: qpid-java-6.1.6, qpid-java-broker-7.0.3,
> qpid-java-broker-7.0.2, qpid-java-6.0, qpid-java-6.0.1, qpid-java-6.0.2,
> qpid-java-6.0.3, qpid-java-6.0.4, qpid-java-6.0.5, qpid-java-6.1,
> qpid-java-6.0.6, qpid-java-6.1.1, qpid-java-6.1.2, qpid-java-6.0.7,
> qpid-java-6.1.3, qpid-java-6.0.8, qpid-java-6.1.4, qpid-java-broker-7.0.0,
> qpid-java-6.1.5, qpid-java-broker-7.0.1, qpid-java-6.1.7,
> qpid-java-broker-7.1.0, qpid-java-broker-7.0.4, qpid-java-broker-7.0.5,
> qpid-java-broker-7.0.6
> Environment: [Broker-Config] (q.m.b.platform) - [Broker] BRK-1010 :
> Platform : JVM : Oracle Corporation version: 1.8.0_161-b12 OS : Linux
> version: 3.10.0-514.6.1.el7.x86_64 arch: amd64 cores: 40
>Reporter: Alex Rudyy
>Priority: Major
> Fix For: qpid-java-broker-7.0.7, qpid-java-broker-7.1.1
>
> Attachments: thread-dump.txt
>
>
> Broker containing a BDB HA Virtual Host node (belonging to the cluster
> consisting of several nodes) can fail to start when BDB HA Virtual Host node
> times out to join the group. The broker cannot complete activation
> (transition into an ACTIVE state).
> The stack traces like bellow are reported on BDB HA VHN timeout:
> {noformat}
> ERROR [Broker-Config] (o.a.q.s.m.AbstractConfiguredObject) - Failed to open
> object with name 'node2'. Object will be put into ERROR state.
> java.lang.RuntimeException: JE replicated environment creation took too long
> (permitted time 18ms)
> at
> org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacade.createEnvironmentInSeparateThread(ReplicatedEnvironmentFacade.java:1577)
> at
> org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacade.createEnvironment(ReplicatedEnvironmentFacade.java:1521)
> at
> org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacade.(ReplicatedEnvironmentFacade.java:287)
> at
> org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacadeFactory.createEnvironmentFacade(ReplicatedEnvironmentFacadeFactory.java:130)
> at
> org.apache.qpid.server.store.berkeleydb.BDBConfigurationStore.init(BDBConfigurationStore.java:122)
> at
> org.apache.qpid.server.virtualhostnode.berkeleydb.BDBHAVirtualHostNodeImpl.activate(BDBHAVirtualHostNodeImpl.java:338)
> at
> org.apache.qpid.server.virtualhostnode.AbstractVirtualHostNode.doActivate(AbstractVirtualHostNode.java:162)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.qpid.server.model.AbstractConfiguredObject.attainState(AbstractConfiguredObject.java:1524)
> at
> org.apache.qpid.server.model.AbstractConfiguredObject.attainState(AbstractConfiguredObject.java:1503)
> at
> org.apache.qpid.server.model.AbstractConfiguredObject$8.onSuccess(AbstractConfiguredObject.java:1070)
> at
> org.apache.qpid.server.model.AbstractConfiguredObject$8.onSuccess(AbstractConfiguredObject.java:1064)
> at
> org.apache.qpid.server.model.AbstractConfiguredObject$22$1.run(AbstractConfiguredObject.java:2639)
> at
> org.apache.qpid.
[jira] [Commented] (QPID-8274) [Broker-J][BDB HA] Broker can fail to become active when BDB HA virtual host times out to join the group
[
https://issues.apache.org/jira/browse/QPID-8274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16768808#comment-16768808
]
Alex Rudyy commented on QPID-8274:
--
Whilst creation of environment for a replica node hang on loading of class
{{TupleBinding}} on attempt to , a creation of another non-HA virtual host hang
at the same time on invocation of BDB JE method
{{IntegerBinding.entryToInt(key)}}. Here is the stack trace:
{noformat}
at
org.apache.qpid.server.store.berkeleydb.upgrade.Upgrader.getSourceVersion(Upgrader.java:119)
at
org.apache.qpid.server.store.berkeleydb.upgrade.Upgrader.upgradeIfNecessary(Upgrader.java:81)
at
org.apache.qpid.server.store.berkeleydb.StandardEnvironmentFacade.upgradeIfNecessary(StandardEnvironmentFacade.java:413)
at
org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore.upgradeStoreStructure(AbstractBDBMessageStore.java:113)
at
org.apache.qpid.server.virtualhost.AbstractVirtualHost.onActivate(AbstractVirtualHost.java:2588)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.qpid.server.model.AbstractConfiguredObject.attainState(AbstractConfiguredObject.java:1524)
at
org.apache.qpid.server.model.AbstractConfiguredObject.attainState(AbstractConfiguredObject.java:1503)
at
org.apache.qpid.server.model.AbstractConfiguredObject$8.onSuccess(AbstractConfiguredObject.java:1070)
at
org.apache.qpid.server.model.AbstractConfiguredObject$8.onSuccess(AbstractConfiguredObject.java:1064)
at
org.apache.qpid.server.model.AbstractConfiguredObject$22$1.run(AbstractConfiguredObject.java:2639)
at
org.apache.qpid.server.model.AbstractConfiguredObject$22$1.run(AbstractConfiguredObject.java:2635)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at
org.apache.qpid.server.model.AbstractConfiguredObject$22.onSuccess(AbstractConfiguredObject.java:2634)
at
com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1237)
at
org.apache.qpid.server.configuration.updater.TaskExecutorImpl$ImmediateIfSameThreadExecutor.execute(TaskExecutorImpl.java:400)
at
org.apache.qpid.server.configuration.updater.TaskExecutorImpl.execute(TaskExecutorImpl.java:183)
at
com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:911)
at
com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:645)
at
com.google.common.util.concurrent.AbstractFuture$TrustedFuture.addListener(AbstractFuture.java:101)
at com.google.common.util.concurrent.Futures.addCallback(Futures.java:1209)
at
org.apache.qpid.server.model.AbstractConfiguredObject.addFutureCallback(AbstractConfiguredObject.java:2629)
at
org.apache.qpid.server.model.AbstractConfiguredObject.doAttainState(AbstractConfiguredObject.java:1063)
at
org.apache.qpid.server.model.AbstractConfiguredObject.access$600(AbstractConfiguredObject.java:95)
at
org.apache.qpid.server.model.AbstractConfiguredObject$1.execute(AbstractConfiguredObject.java:589)
at
org.apache.qpid.server.model.AbstractConfiguredObject$1.execute(AbstractConfiguredObject.java:576)
at
org.apache.qpid.server.model.AbstractConfiguredObject$2.execute(AbstractConfiguredObject.java:637)
at
org.apache.qpid.server.model.AbstractConfiguredObject$2.execute(AbstractConfiguredObject.java:630)
at
org.apache.qpid.server.configuration.updater.TaskExecutorImpl$TaskLoggingWrapper.execute(TaskExecutorImpl.java:248)
at
org.apache.qpid.server.configuration.updater.TaskExecutorImpl$CallableWrapper$1.run(TaskExecutorImpl.java:320)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at
org.apache.qpid.server.configuration.updater.TaskExecutorImpl$CallableWrapper.call(TaskExecutorImpl.java:313)
at
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:111)
at
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:58)
at
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:75)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at
org.apache.qpid.server.bytebuffer.QpidByteBufferFactory.lambda$null$0(QpidByteBufferFactory.java:464)
at
org.apache.qpid.server.bytebuffer.QpidByteBufferFactory$$Lambda$3/1748876332.run(Unknown
Source)
at java.l
[jira] [Commented] (QPID-8274) [Broker-J][BDB HA] Broker can fail to become active when BDB HA virtual host times out to join the group
[
https://issues.apache.org/jira/browse/QPID-8274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16768523#comment-16768523
]
Alex Rudyy commented on QPID-8274:
--
The JE {{Environment}} creation can hang as per stack trace below
{noformat}
at com.sleepycat.bind.tuple.TupleBinding.(TupleBinding.java:53)
at
com.sleepycat.je.cleaner.ExpirationProfile.populateCache(ExpirationProfile.java:193)
- locked java.util.HashMap@56d1b799
at
com.sleepycat.je.recovery.RecoveryManager.recover(RecoveryManager.java:434)
at com.sleepycat.je.dbi.EnvironmentImpl.finishInit(EnvironmentImpl.java:841)
- locked com.sleepycat.je.rep.impl.RepImpl@57b9bac8
at com.sleepycat.je.dbi.DbEnvPool.getEnvironment(DbEnvPool.java:222)
at com.sleepycat.je.Environment.makeEnvironmentImpl(Environment.java:267)
at com.sleepycat.je.Environment.(Environment.java:252)
at
com.sleepycat.je.rep.ReplicatedEnvironment.(ReplicatedEnvironment.java:607)
at
com.sleepycat.je.rep.ReplicatedEnvironment.(ReplicatedEnvironment.java:466)
at
com.sleepycat.je.rep.ReplicatedEnvironment.(ReplicatedEnvironment.java:540)
at
org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacade.createEnvironment(ReplicatedEnvironmentFacade.java:1589)
at
org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacade.access$1700(ReplicatedEnvironmentFacade.java:86)
at
org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacade$14.call(ReplicatedEnvironmentFacade.java:1537)
at
org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacade$14.call(ReplicatedEnvironmentFacade.java:1533)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{noformat}
> [Broker-J][BDB HA] Broker can fail to become active when BDB HA virtual host
> times out to join the group
>
>
> Key: QPID-8274
> URL: https://issues.apache.org/jira/browse/QPID-8274
> Project: Qpid
> Issue Type: Bug
> Components: Broker-J
>Affects Versions: qpid-java-6.1.6, qpid-java-broker-7.0.3,
> qpid-java-broker-7.0.2, qpid-java-6.0, qpid-java-6.0.1, qpid-java-6.0.2,
> qpid-java-6.0.3, qpid-java-6.0.4, qpid-java-6.0.5, qpid-java-6.1,
> qpid-java-6.0.6, qpid-java-6.1.1, qpid-java-6.1.2, qpid-java-6.0.7,
> qpid-java-6.1.3, qpid-java-6.0.8, qpid-java-6.1.4, qpid-java-broker-7.0.0,
> qpid-java-6.1.5, qpid-java-broker-7.0.1, qpid-java-6.1.7,
> qpid-java-broker-7.1.0, qpid-java-broker-7.0.4, qpid-java-broker-7.0.5,
> qpid-java-broker-7.0.6
>Reporter: Alex Rudyy
>Priority: Major
> Fix For: qpid-java-broker-7.0.7, qpid-java-broker-7.1.1
>
>
> Broker containing a BDB HA Virtual Host node (belonging to the cluster
> consisting of several nodes) can fail to start when BDB HA Virtual Host node
> times out to join the group. The broker cannot complete activation
> (transition into an ACTIVE state).
> The stack traces like bellow are reported on BDB HA VHN timeout:
> {noformat}
> ERROR [Broker-Config] (o.a.q.s.m.AbstractConfiguredObject) - Failed to open
> object with name 'node2'. Object will be put into ERROR state.
> java.lang.RuntimeException: JE replicated environment creation took too long
> (permitted time 18ms)
> at
> org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacade.createEnvironmentInSeparateThread(ReplicatedEnvironmentFacade.java:1577)
> at
> org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacade.createEnvironment(ReplicatedEnvironmentFacade.java:1521)
> at
> org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacade.(ReplicatedEnvironmentFacade.java:287)
> at
> org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacadeFactory.createEnvironmentFacade(ReplicatedEnvironmentFacadeFactory.java:130)
> at
> org.apache.qpid.server.store.berkeleydb.BDBConfigurationStore.init(BDBConfigurationStore.java:122)
> at
> org.apache.qpid.server.virtualhostnode.berkeleydb.BDBHAVirtualHostNodeImpl.activate(BDBHAVirtualHostNodeImpl.java:338)
> at
> org.apache.qpid.server.virtualhostnode.AbstractVirtualHostNode.doActivate(AbstractVirtualHostNode.java:162)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> a
