[Wikidata-bugs] [Maniphest] T274270: WDQS servers taking up to 30 minutes to reboot

bking edited projects, added Discovery-Search; removed Discovery-Search (Current work). TASK DETAIL https://phabricator.wikimedia.org/T274270 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: bking, RKemper, Gehel, Aklapper, Astuthiodit_1,

[Wikidata-bugs] [Maniphest] T274270: WDQS servers taking up to 30 minutes to reboot

Gehel edited projects, added Discovery-Search (Current work); removed Wikidata-Query-Service. TASK DETAIL https://phabricator.wikimedia.org/T274270 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: bking, RKemper, Gehel, Aklapper,

[Wikidata-bugs] [Maniphest] T274270: WDQS servers taking up to 30 minutes to reboot

Gehel removed a project: Discovery-Search (Current work). TASK DETAIL https://phabricator.wikimedia.org/T274270 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: bking, RKemper, Gehel, Aklapper, Astuthiodit_1, AWesterinen, karapayneWMDE,

[Wikidata-bugs] [Maniphest] T274270: WDQS servers taking up to 30 minutes to reboot

Gehel added a project: Discovery-Search (Current work). TASK DETAIL https://phabricator.wikimedia.org/T274270 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: bking, RKemper, Gehel, Aklapper, Astuthiodit_1, AWesterinen, karapayneWMDE,

[Wikidata-bugs] [Maniphest] T274270: WDQS servers taking up to 30 minutes to reboot

Gehel removed a project: Discovery-Search. TASK DETAIL https://phabricator.wikimedia.org/T274270 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: bking, RKemper, Gehel, Aklapper, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot,

[Wikidata-bugs] [Maniphest] T274270: WDQS servers taking up to 30 minutes to reboot

bking added a comment. Unfortunately, the systemd workaround listed above did **not** work. We will try adjusting some other unit file values when time permits. TASK DETAIL https://phabricator.wikimedia.org/T274270 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T274270: WDQS servers taking up to 30 minutes to reboot

bking added a comment. Correction: both MDRAID and LVM servers have this problem. Both services' systemd unit files have the same "Conflicts=shutdown.target" directive. Still haven't tried the systemd workaround though, will test that today. TASK DETAIL

[Wikidata-bugs] [Maniphest] T274270: WDQS servers taking up to 30 minutes to reboot

bking added a comment. Another piece of the puzzle, some wdqs hosts use MDRAID for their /srv partition, some use LVM . Working assumption is that only the LVM hosts will take forever to reboot. TASK DETAIL

[Wikidata-bugs] [Maniphest] T274270: WDQS servers taking up to 30 minutes to reboot

bking added a comment. Actions tried so far: disabling swap via systemd before rebooting. Worked on `wdqs2007`, did not work on `wdqs2002`. Also worth noting is that we had previously rebooted `wdqs2007` within the last 30 minutes, so a minor kernel update (from 4.19.0-16-amd64 to

[Wikidata-bugs] [Maniphest] T274270: WDQS servers taking up to 30 minutes to reboot

bking added a comment. This is still happening, @RKemper found some interesting links that could explain this behavior: https://wiki.freedesktop.org/www/Software/systemd/Debugging/#diagnosingshutdownproblems

[Wikidata-bugs] [Maniphest] T274270: WDQS servers taking up to 30 minutes to reboot

Gehel edited projects, added Discovery-Search; removed Discovery-Search (Current work). TASK DETAIL https://phabricator.wikimedia.org/T274270 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: RKemper, Gehel, Aklapper, karapayneWMDE,

[Wikidata-bugs] [Maniphest] T274270: WDQS servers taking up to 30 minutes to reboot

MPhamWMF set the point value for this task to "8". TASK DETAIL https://phabricator.wikimedia.org/T274270 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: MPhamWMF Cc: RKemper, Gehel, Aklapper, MPhamWMF, CBogen, Akuckartz, Nandana, Namenlos314, Lahi,

[Wikidata-bugs] [Maniphest] T274270: WDQS servers taking up to 30 minutes to reboot

Gehel moved this task from Operations to Current work on the Wikidata-Query-Service board. Gehel added a project: Discovery-Search (Current work). TASK DETAIL https://phabricator.wikimedia.org/T274270 WORKBOARD https://phabricator.wikimedia.org/project/board/891/ EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T274270: WDQS servers taking up to 30 minutes to reboot

Gehel triaged this task as "High" priority. Gehel moved this task from All WDQS-related tasks to Operations on the Wikidata-Query-Service board. TASK DETAIL https://phabricator.wikimedia.org/T274270 WORKBOARD https://phabricator.wikimedia.org/project/board/891/ EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T274270: WDQS servers taking up to 30 minutes to reboot

Gehel updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T274270 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Gehel, Aklapper, MPhamWMF, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86,

[Wikidata-bugs] [Maniphest] T274270: WDQS servers taking up to 30 minutes to reboot

Gehel created this task. Gehel added a project: Wikidata-Query-Service. Restricted Application added a subscriber: Aklapper. Restricted Application added a project: Wikidata. TASK DESCRIPTION As an administrator of WDQS, I want reboots to be fast so that I can run cluster wide operations in a