hangc0276 opened a new pull request, #3809:
URL: https://github.com/apache/bookkeeper/pull/3809

   Fix #3808 
   
   ### Motivation
   When the tests run in the following configurations
   - The BookKeeper cluster has multiple bookies (more than 2 bookies)
   - Enabled auto-recovery
   
   When we shut down the bookkeeper cluster, it will take more than 30s. The 
root cause is that we don't shut down **all** the auto-recovery service before 
we shut down the bookies. 
   
   
https://github.com/apache/bookkeeper/blob/80d3aac30f69d2a10f06dca4f9c2a62b8d01220a/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/BookKeeperClusterTestCase.java#L949-L972
   
   When we shut down one bookie, the auditor may change from one bookie to 
another and start to do all the checking tasks, which will block the following 
shutdown process. The shutdown process will wait at most 30s and then will call 
`executor.shutdownNow()` to shut down all the tasks.
   
https://github.com/apache/bookkeeper/blob/80d3aac30f69d2a10f06dca4f9c2a62b8d01220a/bookkeeper-server/src/main/java/org/apache/bookkeeper/replication/Auditor.java#L627-L632
   
   The logs of the slow tests.
   ```
   2023-02-26T09:43:07,576 - INFO  - [Time-limited test:AutoRecoveryMain@147] - 
Shutting down auto recovery: 0
   2023-02-26T09:43:07,577 - INFO  - [Time-limited test:AutoRecoveryMain@151] - 
Shutting down AutoRecovery
   2023-02-26T09:43:07,578 - INFO  - [Time-limited test:Auditor@626] - Shutting 
down auditor
   2023-02-26T09:43:07,578 - INFO  - 
[AuditorElector-127.0.0.1:62826:ZkLedgerAuditorManager@176] - Shutting down 
AuditorElector
   2023-02-26T09:43:07,586 - INFO  - 
[AuditorElector-127.0.0.1:62830:Auditor@203] - AuthProvider used by the Auditor 
is null
   2023-02-26T09:43:07,588 - INFO  - 
[AuditorElector-127.0.0.1:62830:ZkLedgerUnderreplicationManager@823] - 
lostBookieRecoveryDelay Znode is already present, so using existing 
lostBookieRecoveryDelay Znode value
   2023-02-26T09:43:07,588 - INFO  - 
[AuditorElector-127.0.0.1:62830:Auditor@210] - Valid lostBookieRecoveryDelay 
zNode is available, so not creating lostBookieRecoveryDelay zNode as part of 
Auditor initialization 
   2023-02-26T09:43:07,588 - INFO  - 
[AuditorElector-127.0.0.1:62830:Auditor@380] - I'm starting as Auditor Bookie. 
ID: 127.0.0.1:62830
   2023-02-26T09:43:07,590 - INFO  - 
[AuditorElector-127.0.0.1:62830:Auditor@431] - Auditor periodic bookie checking 
enabled 'auditorPeriodicBookieCheckInterval' 86400 seconds
   2023-02-26T09:43:07,590 - INFO  - 
[AuditorElector-127.0.0.1:62830:Auditor@441] - Auditor periodic ledger checking 
enabled 'auditorPeriodicCheckInterval' 604800 seconds
   2023-02-26T09:43:07,591 - INFO  - 
[AuditorElector-127.0.0.1:62830:Auditor@470] - checkAllLedgers scheduling info. 
 checkAllLedgersLastExecutedCTime: 1677375779900 
durationSinceLastExecutionInSecs: 7 initialDelay: 604793 interval: 604800
   2023-02-26T09:43:07,591 - INFO  - 
[AuditorBookie-127.0.0.1:62830:AuditorBookieCheckTask@119] - Starting 
auditBookies
   2023-02-26T09:43:07,591 - INFO  - 
[AuditorElector-127.0.0.1:62830:Auditor@521] - Periodic placementPolicy check 
disabled
   2023-02-26T09:43:07,591 - INFO  - 
[AuditorElector-127.0.0.1:62830:Auditor@529] - Periodic replicas check disabled
   2023-02-26T09:43:07,595 - INFO  - 
[AuditorBookie-127.0.0.1:62830:AuditorBookieCheckTask@156] - Completed 
auditBookies
   2023-02-26T09:43:07,850 - INFO  - 
[AutoRecoveryDeathWatcher-62826:AutoRecoveryMain$AutoRecoveryDeathWatcher@235] 
- AutoRecoveryDeathWatcher noticed the AutoRecovery is not running any 
more,exiting the watch loop!
   2023-02-26T09:43:07,850 - INFO  - 
[AutoRecoveryDeathWatcher-62826:AutoRecoveryMain$AutoRecoveryDeathWatcher@219] 
- AutoRecoveryDeathWatcher exited loop due to uncaught exception from thread 
AutoRecoveryDeathWatcher-62826
   java.lang.RuntimeException: AutoRecovery is not running any more
        at 
org.apache.bookkeeper.replication.AutoRecoveryMain$AutoRecoveryDeathWatcher.run(AutoRecoveryMain.java:245)
 ~[classes/:?]
   2023-02-26T09:43:07,851 - INFO  - 
[AutoRecoveryDeathWatcher-62826:AutoRecoveryMain@147] - Shutting down auto 
recovery: 0
   2023-02-26T09:43:37,501 - WARN  - [Time-limited test:Auditor@630] - Executor 
not shutting down, interrupting
   2023-02-26T09:43:37,503 - INFO  - [Time-limited 
test:AuditorCheckAllLedgersTask@130] - Shutting down AuditorCheckAllLedgersTask
   2023-02-26T09:43:37,503 - INFO  - [Time-limited test:ReplicationWorker@728] 
- Shutting down replication worker
   2023-02-26T09:43:37,503 - INFO  - [Time-limited test:ReplicationWorker@736] 
- Shutting down ReplicationWorker
   ```
   
   ### Modifications
   Shutdown the auto-recovery service before shutting down bookies in the test 
cluster.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to