merlimat opened a new pull request, #4926:
URL: https://github.com/apache/bookkeeper/pull/4926

   ### Problem
   
   `Auditor.shutdown()` shuts its executor down and waits up to 30 seconds for 
it to terminate before interrupting the
   running task (`Executor not shutting down, interrupting`). It waits, up to 
the whole 30 seconds, when:
   
   - an audit waits for the ledger replication to be enabled: the bookie check, 
a lost bookie audit and a
     `lostBookieRecoveryDelay` change all start with 
`waitIfLedgerReplicationDisabled()`, which waits until an operator
     enables it again (e.g. during a rolling restart done with the auto 
recovery disabled);
   - the audit of a lost bookie is delayed by `lostBookieRecoveryDelay`: a 
`ScheduledThreadPoolExecutor` still runs
     delayed tasks after `shutdown()`, so the auditor waits for the delay, and 
when it is shorter than 30 seconds, runs
     the audit after it was shut down.
   
   ### Example
   
   ```java
   // Auditor.shutdown(), master
   executor.shutdown();   // keeps running the delayed audit, doesn't interrupt 
a blocked one
   try {
       while (!executor.awaitTermination(30, TimeUnit.SECONDS)) {
           log.warn("Executor not shutting down, interrupting");
           executor.shutdownNow();
       }
   ```
   
   To reproduce: disable the auto recovery (`bin/bookkeeper shell autorecovery 
-disable`), stop a bookie that is not the
   auditor (the auditor's audit of the lost bookie now waits for the 
replication to be enabled), then stop the auditor's
   bookie: it takes 30 seconds.
   
   ### Change
   
   `shutdown()` calls `shutdownNow()` instead of `shutdown()`: it interrupts 
the running task and drops the pending and
   delayed ones right away, then waits for the executor to terminate as before.
   
   ```java
   // Interrupt the running audit rather than wait for it: an audit can wait 
for the ledger
   // replication to be enabled for as long as it stays disabled, and 
shutdown() would still
   // run the audit delayed for a lost bookie. The next auditor runs the audits 
again.
   executor.shutdownNow();
   ```
   
   An audit in progress is cut short instead of getting up to 30 seconds to 
finish: the next auditor runs its bookie
   check as soon as it starts and the periodic checks on their schedule, and 
audits longer than 30 seconds were already
   interrupted. Before BOOKKEEPER-496 the auditor ran on its own thread and 
`shutdown()` interrupted it right away.
   
   ### Testing
   
   - New `AuditorPeriodicCheckTest.testShutdownWithDelayedBookieAudit`: an 
auditor whose audit of a lost bookie is
     delayed by `lostBookieRecoveryDelay` (`Integer.MAX_VALUE`) must shut down 
within 10 seconds. Fails on master
     (`TimeoutException`).
   - New `AuditorPeriodicCheckTest.testShutdownWithAuditWaitingForReplication`: 
with the ledger replication disabled, an
     auditor whose bookie check waits for it must shut down within 10 seconds. 
Fails on master (`TimeoutException`).
   - The `org.apache.bookkeeper.replication` tests (19 classes) plus 
`BookKeeperAdminTest`, `BookieDecommissionTest`,
     `TestHttpService` and `BookieInitializationTest` pass, except 4 
`BookieInitializationTest` tests that fail locally
     on master too (they can't resolve the machine's hostname). Checkstyle and 
SpotBugs pass.
   
   Those 23 classes on master and with this change, one run each (other test 
runs were going on at the same time; the
   waits removed have fixed lengths):
   
   | Class | master (s) | this change (s) |
   |---|---|---|
   | `AuditorRollingRestartTest` | 96.1 | 5.9 |
   | `AuditorPeriodicCheckTest` | 269.9 | 102.9 |
   | `TestAutoRecoveryAlongWithBookieServers` | 42.4 | 13.7 |
   | all 23 classes | 1,777 | 1,522 |
   
   `Executor not shutting down, interrupting` appears 7 times in master's logs 
of these classes, and never with this
   change. On its own, this change makes `AuditorLedgerCheckerTest` slower (212 
to 274 s): the auditor now stops at once,
   so the tests more often stop the next elector before it has become the 
auditor, and that stop waits 10 seconds, which
   #4925 fixes. With both changes the logs have neither wait, and the 23 
classes take 919 s instead of 1,777 s:
   
   | Class | master (s) | #4925 (s) | this change (s) | both (s) |
   |---|---|---|---|---|
   | `AuditorBookieTest` | 217.2 | 21.1 | 177.2 | 20.8 |
   | `AuditorPeriodicCheckTest` | 269.9 | 198.8 | 102.9 | 51.1 |
   | `AuditorLedgerCheckerTest` | 212.1 | 77.6 | 274.0 | 72.5 |
   | `AuditorRollingRestartTest` | 96.1 | 96.1 | 5.9 | 5.9 |
   | `BookieAutoRecoveryTest` | 136.3 | 68.8 | 128.6 | 68.9 |
   | `BookieDecommissionTest` | 221.0 | 151.2 | 221.4 | 150.8 |
   | `BookKeeperAdminTest` | 59.3 | 29.3 | 60.5 | 28.9 |
   | `TestAutoRecoveryAlongWithBookieServers` | 42.4 | 13.4 | 13.7 | 13.5 |
   | all 23 classes | 1,777 | 1,169 | 1,522 | 919 |
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to