[Sts-sponsors] [Bug 1874075] Re: rabbitmq-server startup timeouts differ between SysV and systemd
Performing verification for rabbitmq-server in -proposed for xenial. I installed 3.5.7-1ubuntu0.16.04.4 from -proposed, and followed the instructions I wrote in comment #59. I sent a message to host 1, checked that host 1 was master, and host 2 was the slave. I shut host 1 down, and then sent a message to host 2. Host 2 became the new master. I shut host 2 down. I then started host 1 up again. The rabbit service came up in a "activating" state, due to ExecStartPost=/usr/lib/rabbitmq/bin/rabbitmq-server-wait kicking in. The rabbitmq in xenial does not offer verbose logging in its /var/log/rabbitmq/rabbit.log file about waiting for host 2, so instead I focused on the systemd service. The service will time out after 60 seconds, something I noted and explained previously, which myself and @ddstreet find to be an acceptable timeout. When the timeout occurs, /usr/lib/rabbitmq/bin/rabbitmq-server-wait exits with an error code, which causes systemd to restart the service, as expected. After a 10 second wait, the service starts in the activating state, waiting for host 2 to come up. I left the service for 15 minutes, and it timed out and restarted successfully 15 times. I then started host 2 up, and host 1 rejoined the cluster. Both hosts have the rabbit service in the active state. The package is behaving as expected on xenial, and I am happy to mark this as verified. ** Tags removed: verification-needed verification-needed-xenial ** Tags added: verification-done-xenial -- You received this bug notification because you are a member of STS Sponsors, which is subscribed to the bug report. https://bugs.launchpad.net/bugs/1874075 Title: rabbitmq-server startup timeouts differ between SysV and systemd Status in rabbitmq-server package in Ubuntu: Fix Released Status in rabbitmq-server source package in Xenial: Fix Committed Status in rabbitmq-server source package in Bionic: Fix Committed Status in rabbitmq-server source package in Eoan: Won't Fix Status in rabbitmq-server source package in Focal: Fix Released Status in rabbitmq-server source package in Groovy: Fix Released Status in rabbitmq-server package in Debian: New Bug description: The startup timeouts were recently adjusted and synchronized between the SysV and systemd startup files. https://github.com/rabbitmq/rabbitmq-server-release/pull/129 The new startup files should be included in this package. [Impact] After starting the RabbitMQ server process, the startup script will wait for the server to start by calling `rabbitmqctl wait` and will time out after 10 s. The startup time of the server depends on how quickly the Mnesia database becomes available and the server will time out after `mnesia_table_loading_retry_timeout` ms times `mnesia_table_loading_retry_limit` retries. By default this wait is 30,000 ms times 10 retries, i.e. 300 s. The mismatch between these two timeout values might lead to the startup script failing prematurely while the server is still waiting for the Mnesia tables. This change introduces variable `RABBITMQ_STARTUP_TIMEOUT` and the `--timeout` option into the startup script. The default value for this timeout is set to 10 minutes (600 seconds). This change also updates the systemd service file to match the timeout values between the two service management methods. [Scope] Upstream patch: https://github.com/rabbitmq/rabbitmq-server- release/pull/129 * Fix is not included in the Debian package * Fix is not included in any Ubuntu series * Groovy and Focal can apply the upstream patch as is * Bionic and Xenial need an additional fix in the systemd service file to set the `RABBITMQ_STARTUP_TIMEOUT` variable for the `rabbitmq-server-wait` helper script. [Test Case] In a clustered setup with two nodes, A and B. 1. create queue on A 2. shut down B 3. shut down A 4. boot B The broker on B will wait for A. The systemd service will wait for 10 seconds and then fail. Boot A and the rabbitmq-server process on B will complete startup. [Regression Potential] This change alters the behavior of the startup scripts when the Mnesia database takes long to become available. This might lead to failures further down the service dependency chain. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/rabbitmq-server/+bug/1874075/+subscriptions -- Mailing list: https://launchpad.net/~sts-sponsors Post to : sts-sponsors@lists.launchpad.net Unsubscribe : https://launchpad.net/~sts-sponsors More help : https://help.launchpad.net/ListHelp
[Sts-sponsors] [Bug 1874075] Re: rabbitmq-server startup timeouts differ between SysV and systemd
Performing verification for rabbitmq-server in -proposed for bionic. I installed 3.6.10-1ubuntu0.3 from -proposed, and followed the instructions I wrote in comment #59. I sent a message to host 1, checked that host 1 was master, and host 2 was the slave. I shut host 1 down, and then sent a message to host 2. Host 2 became the new master. I shut host 2 down. I then started host 1 up again. The rabbit service came up in a "activating" state, due to ExecStartPost=/usr/lib/rabbitmq/bin/rabbitmq-server-wait kicking in. I looked at /var/log/rabbitmq/host1.log, and rabbit logged its attempts to sync with host 2 every 30 seconds, for 10 tries. At this point, /usr/lib/rabbitmq/bin/rabbitmq-server-wait exits with an error code since it has timed out, and systemd then restarts the service after a 10 second delay. This is all as expected. The service comes back up in the activating state and continues to wait for host 2. I left it like this for 15 minutes, and the rabbitmq-server-wait timeout, restart and waiting all works correctly. I then started host 2 up, and host 1 rejoined the cluster. Both hosts have the rabbit service in the active state. The package is behaving as expected on bionic, and I am happy to mark this as verified. ** Tags removed: verification-needed-bionic ** Tags added: verification-done-bionic -- You received this bug notification because you are a member of STS Sponsors, which is subscribed to the bug report. https://bugs.launchpad.net/bugs/1874075 Title: rabbitmq-server startup timeouts differ between SysV and systemd Status in rabbitmq-server package in Ubuntu: Fix Released Status in rabbitmq-server source package in Xenial: Fix Committed Status in rabbitmq-server source package in Bionic: Fix Committed Status in rabbitmq-server source package in Eoan: Won't Fix Status in rabbitmq-server source package in Focal: Fix Released Status in rabbitmq-server source package in Groovy: Fix Released Status in rabbitmq-server package in Debian: New Bug description: The startup timeouts were recently adjusted and synchronized between the SysV and systemd startup files. https://github.com/rabbitmq/rabbitmq-server-release/pull/129 The new startup files should be included in this package. [Impact] After starting the RabbitMQ server process, the startup script will wait for the server to start by calling `rabbitmqctl wait` and will time out after 10 s. The startup time of the server depends on how quickly the Mnesia database becomes available and the server will time out after `mnesia_table_loading_retry_timeout` ms times `mnesia_table_loading_retry_limit` retries. By default this wait is 30,000 ms times 10 retries, i.e. 300 s. The mismatch between these two timeout values might lead to the startup script failing prematurely while the server is still waiting for the Mnesia tables. This change introduces variable `RABBITMQ_STARTUP_TIMEOUT` and the `--timeout` option into the startup script. The default value for this timeout is set to 10 minutes (600 seconds). This change also updates the systemd service file to match the timeout values between the two service management methods. [Scope] Upstream patch: https://github.com/rabbitmq/rabbitmq-server- release/pull/129 * Fix is not included in the Debian package * Fix is not included in any Ubuntu series * Groovy and Focal can apply the upstream patch as is * Bionic and Xenial need an additional fix in the systemd service file to set the `RABBITMQ_STARTUP_TIMEOUT` variable for the `rabbitmq-server-wait` helper script. [Test Case] In a clustered setup with two nodes, A and B. 1. create queue on A 2. shut down B 3. shut down A 4. boot B The broker on B will wait for A. The systemd service will wait for 10 seconds and then fail. Boot A and the rabbitmq-server process on B will complete startup. [Regression Potential] This change alters the behavior of the startup scripts when the Mnesia database takes long to become available. This might lead to failures further down the service dependency chain. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/rabbitmq-server/+bug/1874075/+subscriptions -- Mailing list: https://launchpad.net/~sts-sponsors Post to : sts-sponsors@lists.launchpad.net Unsubscribe : https://launchpad.net/~sts-sponsors More help : https://help.launchpad.net/ListHelp
[Sts-sponsors] [Bug 1874075] Re: rabbitmq-server startup timeouts differ between SysV and systemd
Performing verification for rabbitmq-server in -proposed for focal. I know the focal package is released to -updates, but I will verify again regardless. I installed 3.8.2-0ubuntu1.1 from -proposed, and followed the instructions I wrote in comment #59. I sent a message to host 1, checked that host 1 was master, and host 2 was the slave. I shut host 1 down, and then sent a message to host 2. Host 2 became the new master. I shut host 2 down. I then started host 1 up again. When I ran "sudo systemctl status rabbitmq-server.service", rabbit was in the "activating" state, due to having its systemd service as type=notify. I let it sit on "activating" for an entire 10 minutes, at which point it timed out as planned, and the service restarted as expected after 10 seconds. The service came up again in the "activating" state as expected. I then started host 2 up again, and after it had come up, both rabbitmq-server services were in the active state, and servicing requests. The package is behaving as expected on focal, and I am happy to mark this as verified. -- You received this bug notification because you are a member of STS Sponsors, which is subscribed to the bug report. https://bugs.launchpad.net/bugs/1874075 Title: rabbitmq-server startup timeouts differ between SysV and systemd Status in rabbitmq-server package in Ubuntu: Fix Released Status in rabbitmq-server source package in Xenial: Fix Committed Status in rabbitmq-server source package in Bionic: Fix Committed Status in rabbitmq-server source package in Eoan: Won't Fix Status in rabbitmq-server source package in Focal: Fix Released Status in rabbitmq-server source package in Groovy: Fix Released Status in rabbitmq-server package in Debian: New Bug description: The startup timeouts were recently adjusted and synchronized between the SysV and systemd startup files. https://github.com/rabbitmq/rabbitmq-server-release/pull/129 The new startup files should be included in this package. [Impact] After starting the RabbitMQ server process, the startup script will wait for the server to start by calling `rabbitmqctl wait` and will time out after 10 s. The startup time of the server depends on how quickly the Mnesia database becomes available and the server will time out after `mnesia_table_loading_retry_timeout` ms times `mnesia_table_loading_retry_limit` retries. By default this wait is 30,000 ms times 10 retries, i.e. 300 s. The mismatch between these two timeout values might lead to the startup script failing prematurely while the server is still waiting for the Mnesia tables. This change introduces variable `RABBITMQ_STARTUP_TIMEOUT` and the `--timeout` option into the startup script. The default value for this timeout is set to 10 minutes (600 seconds). This change also updates the systemd service file to match the timeout values between the two service management methods. [Scope] Upstream patch: https://github.com/rabbitmq/rabbitmq-server- release/pull/129 * Fix is not included in the Debian package * Fix is not included in any Ubuntu series * Groovy and Focal can apply the upstream patch as is * Bionic and Xenial need an additional fix in the systemd service file to set the `RABBITMQ_STARTUP_TIMEOUT` variable for the `rabbitmq-server-wait` helper script. [Test Case] In a clustered setup with two nodes, A and B. 1. create queue on A 2. shut down B 3. shut down A 4. boot B The broker on B will wait for A. The systemd service will wait for 10 seconds and then fail. Boot A and the rabbitmq-server process on B will complete startup. [Regression Potential] This change alters the behavior of the startup scripts when the Mnesia database takes long to become available. This might lead to failures further down the service dependency chain. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/rabbitmq-server/+bug/1874075/+subscriptions -- Mailing list: https://launchpad.net/~sts-sponsors Post to : sts-sponsors@lists.launchpad.net Unsubscribe : https://launchpad.net/~sts-sponsors More help : https://help.launchpad.net/ListHelp
[Sts-sponsors] [Bug 1874075] Re: rabbitmq-server startup timeouts differ between SysV and systemd
This bug was fixed in the package rabbitmq-server - 3.8.2-0ubuntu1.1 --- rabbitmq-server (3.8.2-0ubuntu1.1) focal; urgency=medium * Fix timeout discrepancy between SysV and systemd (LP: #1874075) upstream, rabbitmq-server-release - 694540270c8 * d/rabbitmq-server.init - Add RABBITMQ_STARTUP_TIMEOUT and default to 600 * d/rabbitmq-server.service - Default TimeoutStartSec to 600 -- Nicolas Bock Tue, 21 Apr 2020 06:37:55 -0600 ** Changed in: rabbitmq-server (Ubuntu Focal) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of STS Sponsors, which is subscribed to the bug report. https://bugs.launchpad.net/bugs/1874075 Title: rabbitmq-server startup timeouts differ between SysV and systemd Status in rabbitmq-server package in Ubuntu: Fix Released Status in rabbitmq-server source package in Xenial: Fix Committed Status in rabbitmq-server source package in Bionic: Fix Committed Status in rabbitmq-server source package in Eoan: Won't Fix Status in rabbitmq-server source package in Focal: Fix Released Status in rabbitmq-server source package in Groovy: Fix Released Status in rabbitmq-server package in Debian: New Bug description: The startup timeouts were recently adjusted and synchronized between the SysV and systemd startup files. https://github.com/rabbitmq/rabbitmq-server-release/pull/129 The new startup files should be included in this package. [Impact] After starting the RabbitMQ server process, the startup script will wait for the server to start by calling `rabbitmqctl wait` and will time out after 10 s. The startup time of the server depends on how quickly the Mnesia database becomes available and the server will time out after `mnesia_table_loading_retry_timeout` ms times `mnesia_table_loading_retry_limit` retries. By default this wait is 30,000 ms times 10 retries, i.e. 300 s. The mismatch between these two timeout values might lead to the startup script failing prematurely while the server is still waiting for the Mnesia tables. This change introduces variable `RABBITMQ_STARTUP_TIMEOUT` and the `--timeout` option into the startup script. The default value for this timeout is set to 10 minutes (600 seconds). This change also updates the systemd service file to match the timeout values between the two service management methods. [Scope] Upstream patch: https://github.com/rabbitmq/rabbitmq-server- release/pull/129 * Fix is not included in the Debian package * Fix is not included in any Ubuntu series * Groovy and Focal can apply the upstream patch as is * Bionic and Xenial need an additional fix in the systemd service file to set the `RABBITMQ_STARTUP_TIMEOUT` variable for the `rabbitmq-server-wait` helper script. [Test Case] In a clustered setup with two nodes, A and B. 1. create queue on A 2. shut down B 3. shut down A 4. boot B The broker on B will wait for A. The systemd service will wait for 10 seconds and then fail. Boot A and the rabbitmq-server process on B will complete startup. [Regression Potential] This change alters the behavior of the startup scripts when the Mnesia database takes long to become available. This might lead to failures further down the service dependency chain. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/rabbitmq-server/+bug/1874075/+subscriptions -- Mailing list: https://launchpad.net/~sts-sponsors Post to : sts-sponsors@lists.launchpad.net Unsubscribe : https://launchpad.net/~sts-sponsors More help : https://help.launchpad.net/ListHelp
[Sts-sponsors] [Bug 1874075] Update Released
The verification of the Stable Release Update for rabbitmq-server has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions. -- You received this bug notification because you are a member of STS Sponsors, which is subscribed to the bug report. https://bugs.launchpad.net/bugs/1874075 Title: rabbitmq-server startup timeouts differ between SysV and systemd Status in rabbitmq-server package in Ubuntu: Fix Released Status in rabbitmq-server source package in Xenial: Fix Committed Status in rabbitmq-server source package in Bionic: Fix Committed Status in rabbitmq-server source package in Eoan: Won't Fix Status in rabbitmq-server source package in Focal: Fix Released Status in rabbitmq-server source package in Groovy: Fix Released Status in rabbitmq-server package in Debian: New Bug description: The startup timeouts were recently adjusted and synchronized between the SysV and systemd startup files. https://github.com/rabbitmq/rabbitmq-server-release/pull/129 The new startup files should be included in this package. [Impact] After starting the RabbitMQ server process, the startup script will wait for the server to start by calling `rabbitmqctl wait` and will time out after 10 s. The startup time of the server depends on how quickly the Mnesia database becomes available and the server will time out after `mnesia_table_loading_retry_timeout` ms times `mnesia_table_loading_retry_limit` retries. By default this wait is 30,000 ms times 10 retries, i.e. 300 s. The mismatch between these two timeout values might lead to the startup script failing prematurely while the server is still waiting for the Mnesia tables. This change introduces variable `RABBITMQ_STARTUP_TIMEOUT` and the `--timeout` option into the startup script. The default value for this timeout is set to 10 minutes (600 seconds). This change also updates the systemd service file to match the timeout values between the two service management methods. [Scope] Upstream patch: https://github.com/rabbitmq/rabbitmq-server- release/pull/129 * Fix is not included in the Debian package * Fix is not included in any Ubuntu series * Groovy and Focal can apply the upstream patch as is * Bionic and Xenial need an additional fix in the systemd service file to set the `RABBITMQ_STARTUP_TIMEOUT` variable for the `rabbitmq-server-wait` helper script. [Test Case] In a clustered setup with two nodes, A and B. 1. create queue on A 2. shut down B 3. shut down A 4. boot B The broker on B will wait for A. The systemd service will wait for 10 seconds and then fail. Boot A and the rabbitmq-server process on B will complete startup. [Regression Potential] This change alters the behavior of the startup scripts when the Mnesia database takes long to become available. This might lead to failures further down the service dependency chain. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/rabbitmq-server/+bug/1874075/+subscriptions -- Mailing list: https://launchpad.net/~sts-sponsors Post to : sts-sponsors@lists.launchpad.net Unsubscribe : https://launchpad.net/~sts-sponsors More help : https://help.launchpad.net/ListHelp
[Sts-sponsors] [Bug 1874075] Re: rabbitmq-server startup timeouts differ between SysV and systemd
marking verification-done-focal as i mentioned in comment 71 ** Tags removed: verification-needed-focal ** Tags added: verification-done-focal -- You received this bug notification because you are a member of STS Sponsors, which is subscribed to the bug report. https://bugs.launchpad.net/bugs/1874075 Title: rabbitmq-server startup timeouts differ between SysV and systemd Status in rabbitmq-server package in Ubuntu: Fix Released Status in rabbitmq-server source package in Xenial: Fix Committed Status in rabbitmq-server source package in Bionic: Fix Committed Status in rabbitmq-server source package in Eoan: Won't Fix Status in rabbitmq-server source package in Focal: Fix Committed Status in rabbitmq-server source package in Groovy: Fix Released Status in rabbitmq-server package in Debian: New Bug description: The startup timeouts were recently adjusted and synchronized between the SysV and systemd startup files. https://github.com/rabbitmq/rabbitmq-server-release/pull/129 The new startup files should be included in this package. [Impact] After starting the RabbitMQ server process, the startup script will wait for the server to start by calling `rabbitmqctl wait` and will time out after 10 s. The startup time of the server depends on how quickly the Mnesia database becomes available and the server will time out after `mnesia_table_loading_retry_timeout` ms times `mnesia_table_loading_retry_limit` retries. By default this wait is 30,000 ms times 10 retries, i.e. 300 s. The mismatch between these two timeout values might lead to the startup script failing prematurely while the server is still waiting for the Mnesia tables. This change introduces variable `RABBITMQ_STARTUP_TIMEOUT` and the `--timeout` option into the startup script. The default value for this timeout is set to 10 minutes (600 seconds). This change also updates the systemd service file to match the timeout values between the two service management methods. [Scope] Upstream patch: https://github.com/rabbitmq/rabbitmq-server- release/pull/129 * Fix is not included in the Debian package * Fix is not included in any Ubuntu series * Groovy and Focal can apply the upstream patch as is * Bionic and Xenial need an additional fix in the systemd service file to set the `RABBITMQ_STARTUP_TIMEOUT` variable for the `rabbitmq-server-wait` helper script. [Test Case] In a clustered setup with two nodes, A and B. 1. create queue on A 2. shut down B 3. shut down A 4. boot B The broker on B will wait for A. The systemd service will wait for 10 seconds and then fail. Boot A and the rabbitmq-server process on B will complete startup. [Regression Potential] This change alters the behavior of the startup scripts when the Mnesia database takes long to become available. This might lead to failures further down the service dependency chain. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/rabbitmq-server/+bug/1874075/+subscriptions -- Mailing list: https://launchpad.net/~sts-sponsors Post to : sts-sponsors@lists.launchpad.net Unsubscribe : https://launchpad.net/~sts-sponsors More help : https://help.launchpad.net/ListHelp
