[
https://issues.apache.org/jira/browse/AMQ-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shi Lei updated AMQ-5435:
-------------------------
Description:
I am using jdbc master/slave with lease database lock.
<amq:broker id="broker" startAsync="true">
I found if I call broker.stop to stop a slave broker service (which means it
tries to get a lease locker and has not got yet), its Persistence Adapter
Starting Thread is still alive. If I create and start a new broker in the same
java VM, there will be 2 Persistence Adapter Starting Threads inside the same
java VM. At this time, if the master broker is down, the stopped broker will
get the database lease locker, but somehow it cannot start broker. Now I have 2
broker service in the same VM. One has got the locker, but cannot start broker,
the other one is still requesting the locker.
The root cause is that after stopping broker, LeaseDatabaseLocker.isStopping()
is false, LeaseDatabaseLocker.isStopped() is true,
In LeaseDatabaseLocker.doStart
while (!isStopping()) {
Connection connection = null;
PreparedStatement statement = null;
try {
connection = getConnection();
initTimeDiff(connection);
statement = connection.prepareStatement(sql);
setQueryTimeout(statement);
now = System.currentTimeMillis() + diffFromCurrentTime;
statement.setString(1, getLeaseHolderId());
statement.setLong(2, now + lockAcquireSleepInterval);
statement.setLong(3, now);
int result = statement.executeUpdate();
if (result == 1) {
// we got the lease, verify we still have it
if (keepAlive()) {
break;
}
}
reportLeasOwnerShipAndDuration(connection);
} catch (Exception e) {
LOG.debug(getLeaseHolderId() + " lease acquire failure: "+ e,
e);
if (isStopping()) {
throw new Exception(
"Cannot start broker as being asked to shut down. "
+ "Interrupted attempt to acquire lock: "
+ e, e);
}
if (handleStartException) {
lockable.getBrokerService().handleIOException(IOExceptionSupport.create(e));
}
} finally {
close(statement);
close(connection);
}
LOG.info(getLeaseHolderId() + " failed to acquire lease. Sleeping
for " + lockAcquireSleepInterval + " milli(s) before trying again...");
TimeUnit.MILLISECONDS.sleep(lockAcquireSleepInterval);
}
if (isStopping()) {
throw new RuntimeException(getLeaseHolderId() + " failing lease
acquire due to stop");
}
LOG.info(getLeaseHolderId() + ", becoming master with lease expiry " +
new Date(now) + " on dataSource: " + dataSource);
}
I think we should replace isStopping() with isStopping() or isStopped().
was:
I am using jdbc master/slave with lease database lock.
I found if I call broker.stop to stop a slave broker service (which means it
tries to get a lease locker and has not got yet), its Persistence Adapter
Starting Thread is still alive. If I create and start a new broker in the same
java VM, there will be 2 Persistence Adapter Starting Threads inside the same
java VM. At this time, if the master broker is down, the stopped broker will
get the database lease locker, but somehow it cannot start broker. Now I have 2
broker service in the same VM. One has got the locker, but cannot start broker,
the other one is still requesting the locker.
The root cause is that after stopping broker, LeaseDatabaseLocker.isStopping()
is false, LeaseDatabaseLocker.isStopped() is true,
In LeaseDatabaseLocker.doStart
while (!isStopping()) {
Connection connection = null;
PreparedStatement statement = null;
try {
connection = getConnection();
initTimeDiff(connection);
statement = connection.prepareStatement(sql);
setQueryTimeout(statement);
now = System.currentTimeMillis() + diffFromCurrentTime;
statement.setString(1, getLeaseHolderId());
statement.setLong(2, now + lockAcquireSleepInterval);
statement.setLong(3, now);
int result = statement.executeUpdate();
if (result == 1) {
// we got the lease, verify we still have it
if (keepAlive()) {
break;
}
}
reportLeasOwnerShipAndDuration(connection);
} catch (Exception e) {
LOG.debug(getLeaseHolderId() + " lease acquire failure: "+ e,
e);
if (isStopping()) {
throw new Exception(
"Cannot start broker as being asked to shut down. "
+ "Interrupted attempt to acquire lock: "
+ e, e);
}
if (handleStartException) {
lockable.getBrokerService().handleIOException(IOExceptionSupport.create(e));
}
} finally {
close(statement);
close(connection);
}
LOG.info(getLeaseHolderId() + " failed to acquire lease. Sleeping
for " + lockAcquireSleepInterval + " milli(s) before trying again...");
TimeUnit.MILLISECONDS.sleep(lockAcquireSleepInterval);
}
if (isStopping()) {
throw new RuntimeException(getLeaseHolderId() + " failing lease
acquire due to stop");
}
LOG.info(getLeaseHolderId() + ", becoming master with lease expiry " +
new Date(now) + " on dataSource: " + dataSource);
}
I think we should replace isStopping() with isStopping() or isStopped().
> Persistence Adapter Starting Thread is still alive after stopping a slave
> broker with lease database locker
> -----------------------------------------------------------------------------------------------------------
>
> Key: AMQ-5435
> URL: https://issues.apache.org/jira/browse/AMQ-5435
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.10.0
> Environment: Windows, JDK7
> Reporter: Shi Lei
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> I am using jdbc master/slave with lease database lock.
> <amq:broker id="broker" startAsync="true">
> I found if I call broker.stop to stop a slave broker service (which means it
> tries to get a lease locker and has not got yet), its Persistence Adapter
> Starting Thread is still alive. If I create and start a new broker in the
> same java VM, there will be 2 Persistence Adapter Starting Threads inside the
> same java VM. At this time, if the master broker is down, the stopped broker
> will get the database lease locker, but somehow it cannot start broker. Now I
> have 2 broker service in the same VM. One has got the locker, but cannot
> start broker, the other one is still requesting the locker.
> The root cause is that after stopping broker,
> LeaseDatabaseLocker.isStopping() is false, LeaseDatabaseLocker.isStopped() is
> true,
> In LeaseDatabaseLocker.doStart
> while (!isStopping()) {
> Connection connection = null;
> PreparedStatement statement = null;
> try {
> connection = getConnection();
> initTimeDiff(connection);
> statement = connection.prepareStatement(sql);
> setQueryTimeout(statement);
> now = System.currentTimeMillis() + diffFromCurrentTime;
> statement.setString(1, getLeaseHolderId());
> statement.setLong(2, now + lockAcquireSleepInterval);
> statement.setLong(3, now);
> int result = statement.executeUpdate();
> if (result == 1) {
> // we got the lease, verify we still have it
> if (keepAlive()) {
> break;
> }
> }
> reportLeasOwnerShipAndDuration(connection);
> } catch (Exception e) {
> LOG.debug(getLeaseHolderId() + " lease acquire failure: "+ e,
> e);
> if (isStopping()) {
> throw new Exception(
> "Cannot start broker as being asked to shut down.
> "
> + "Interrupted attempt to acquire lock: "
> + e, e);
> }
> if (handleStartException) {
>
> lockable.getBrokerService().handleIOException(IOExceptionSupport.create(e));
> }
> } finally {
> close(statement);
> close(connection);
> }
> LOG.info(getLeaseHolderId() + " failed to acquire lease.
> Sleeping for " + lockAcquireSleepInterval + " milli(s) before trying
> again...");
> TimeUnit.MILLISECONDS.sleep(lockAcquireSleepInterval);
> }
> if (isStopping()) {
> throw new RuntimeException(getLeaseHolderId() + " failing lease
> acquire due to stop");
> }
> LOG.info(getLeaseHolderId() + ", becoming master with lease expiry "
> + new Date(now) + " on dataSource: " + dataSource);
> }
> I think we should replace isStopping() with isStopping() or isStopped().
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
