hangc0276 opened a new pull request, #3809: URL: https://github.com/apache/bookkeeper/pull/3809
Fix #3808 ### Motivation When the tests run in the following configurations - The BookKeeper cluster has multiple bookies (more than 2 bookies) - Enabled auto-recovery When we shut down the bookkeeper cluster, it will take more than 30s. The root cause is that we don't shut down **all** the auto-recovery service before we shut down the bookies. https://github.com/apache/bookkeeper/blob/80d3aac30f69d2a10f06dca4f9c2a62b8d01220a/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/BookKeeperClusterTestCase.java#L949-L972 When we shut down one bookie, the auditor may change from one bookie to another and start to do all the checking tasks, which will block the following shutdown process. The shutdown process will wait at most 30s and then will call `executor.shutdownNow()` to shut down all the tasks. https://github.com/apache/bookkeeper/blob/80d3aac30f69d2a10f06dca4f9c2a62b8d01220a/bookkeeper-server/src/main/java/org/apache/bookkeeper/replication/Auditor.java#L627-L632 The logs of the slow tests. ``` 2023-02-26T09:43:07,576 - INFO - [Time-limited test:AutoRecoveryMain@147] - Shutting down auto recovery: 0 2023-02-26T09:43:07,577 - INFO - [Time-limited test:AutoRecoveryMain@151] - Shutting down AutoRecovery 2023-02-26T09:43:07,578 - INFO - [Time-limited test:Auditor@626] - Shutting down auditor 2023-02-26T09:43:07,578 - INFO - [AuditorElector-127.0.0.1:62826:ZkLedgerAuditorManager@176] - Shutting down AuditorElector 2023-02-26T09:43:07,586 - INFO - [AuditorElector-127.0.0.1:62830:Auditor@203] - AuthProvider used by the Auditor is null 2023-02-26T09:43:07,588 - INFO - [AuditorElector-127.0.0.1:62830:ZkLedgerUnderreplicationManager@823] - lostBookieRecoveryDelay Znode is already present, so using existing lostBookieRecoveryDelay Znode value 2023-02-26T09:43:07,588 - INFO - [AuditorElector-127.0.0.1:62830:Auditor@210] - Valid lostBookieRecoveryDelay zNode is available, so not creating lostBookieRecoveryDelay zNode as part of Auditor initialization 2023-02-26T09:43:07,588 - INFO - [AuditorElector-127.0.0.1:62830:Auditor@380] - I'm starting as Auditor Bookie. ID: 127.0.0.1:62830 2023-02-26T09:43:07,590 - INFO - [AuditorElector-127.0.0.1:62830:Auditor@431] - Auditor periodic bookie checking enabled 'auditorPeriodicBookieCheckInterval' 86400 seconds 2023-02-26T09:43:07,590 - INFO - [AuditorElector-127.0.0.1:62830:Auditor@441] - Auditor periodic ledger checking enabled 'auditorPeriodicCheckInterval' 604800 seconds 2023-02-26T09:43:07,591 - INFO - [AuditorElector-127.0.0.1:62830:Auditor@470] - checkAllLedgers scheduling info. checkAllLedgersLastExecutedCTime: 1677375779900 durationSinceLastExecutionInSecs: 7 initialDelay: 604793 interval: 604800 2023-02-26T09:43:07,591 - INFO - [AuditorBookie-127.0.0.1:62830:AuditorBookieCheckTask@119] - Starting auditBookies 2023-02-26T09:43:07,591 - INFO - [AuditorElector-127.0.0.1:62830:Auditor@521] - Periodic placementPolicy check disabled 2023-02-26T09:43:07,591 - INFO - [AuditorElector-127.0.0.1:62830:Auditor@529] - Periodic replicas check disabled 2023-02-26T09:43:07,595 - INFO - [AuditorBookie-127.0.0.1:62830:AuditorBookieCheckTask@156] - Completed auditBookies 2023-02-26T09:43:07,850 - INFO - [AutoRecoveryDeathWatcher-62826:AutoRecoveryMain$AutoRecoveryDeathWatcher@235] - AutoRecoveryDeathWatcher noticed the AutoRecovery is not running any more,exiting the watch loop! 2023-02-26T09:43:07,850 - INFO - [AutoRecoveryDeathWatcher-62826:AutoRecoveryMain$AutoRecoveryDeathWatcher@219] - AutoRecoveryDeathWatcher exited loop due to uncaught exception from thread AutoRecoveryDeathWatcher-62826 java.lang.RuntimeException: AutoRecovery is not running any more at org.apache.bookkeeper.replication.AutoRecoveryMain$AutoRecoveryDeathWatcher.run(AutoRecoveryMain.java:245) ~[classes/:?] 2023-02-26T09:43:07,851 - INFO - [AutoRecoveryDeathWatcher-62826:AutoRecoveryMain@147] - Shutting down auto recovery: 0 2023-02-26T09:43:37,501 - WARN - [Time-limited test:Auditor@630] - Executor not shutting down, interrupting 2023-02-26T09:43:37,503 - INFO - [Time-limited test:AuditorCheckAllLedgersTask@130] - Shutting down AuditorCheckAllLedgersTask 2023-02-26T09:43:37,503 - INFO - [Time-limited test:ReplicationWorker@728] - Shutting down replication worker 2023-02-26T09:43:37,503 - INFO - [Time-limited test:ReplicationWorker@736] - Shutting down ReplicationWorker ``` ### Modifications Shutdown the auto-recovery service before shutting down bookies in the test cluster. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
