[Wikidata-bugs] [Maniphest] T294961: Resolve kernel hang on wcqs* instances

[Gehel]

Gehel closed this task as "Resolved".
Gehel claimed this task.

TASK DETAIL
  https://phabricator.wikimedia.org/T294961

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Gehel
Cc: Legoktm, Dzahn, MoritzMuehlenhoff, RKemper, Aklapper, Gehel, CBogen, 
ttaylor, Zache, Fuzheado, So9q, GFontenelle_WMF, EBernhardson, joanna_borun, 
Invadibot, MPhamWMF, Devnull, maantietaja, lmata, Muchiri124, Akuckartz, 
RhinosF1, Legado_Shulgin, ReaperDawn, Nandana, Namenlos314, Davinaclare77, 
Techguru.pc, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, 
QZanden, EBjune, merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, 
Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, faidon, Addshore, Mbch331, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294961: Resolve kernel hang on wcqs* instances

[Maintenance_bot]

Maintenance_bot removed a project: Patch-For-Review.

TASK DETAIL
  https://phabricator.wikimedia.org/T294961

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Maintenance_bot
Cc: Legoktm, Dzahn, MoritzMuehlenhoff, RKemper, Aklapper, Gehel, CBogen, 
ttaylor, Zache, Fuzheado, So9q, GFontenelle_WMF, EBernhardson, joanna_borun, 
Invadibot, MPhamWMF, Devnull, maantietaja, lmata, Muchiri124, Akuckartz, 
RhinosF1, Legado_Shulgin, ReaperDawn, Nandana, Namenlos314, Davinaclare77, 
Techguru.pc, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, 
QZanden, EBjune, merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, 
Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, faidon, Addshore, Mbch331, Jay8g, fgiunchedi, 786, Suran38, 
Biggs657, Lalamarie69, Juan90264, Alter-paule, Beast1978, Un1tY, Hook696, 
Kent7301, joker88john, CucyNoiD, Gaboe420, Giuliamocci, Cpaulf30, Af420, 
Bsandipan, Lewizho99, Maathavan
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294961: Resolve kernel hang on wcqs* instances

[gerritbot]

gerritbot added a comment.


  Change 743223 **merged** by Ryan Kemper:
  
  [operations/puppet@production] Switch WCQS to profile::base::linux510
  
  https://gerrit.wikimedia.org/r/743223

TASK DETAIL
  https://phabricator.wikimedia.org/T294961

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: Legoktm, Dzahn, MoritzMuehlenhoff, RKemper, Aklapper, Gehel, CBogen, 
ttaylor, Zache, Fuzheado, So9q, GFontenelle_WMF, EBernhardson, 786, Suran38, 
Biggs657, joanna_borun, Invadibot, Lalamarie69, MPhamWMF, Devnull, maantietaja, 
lmata, Juan90264, Muchiri124, Alter-paule, Beast1978, Un1tY, Akuckartz, 
Hook696, Kent7301, RhinosF1, joker88john, Legado_Shulgin, ReaperDawn, CucyNoiD, 
Nandana, Namenlos314, Gaboe420, Giuliamocci, Davinaclare77, Cpaulf30, 
Techguru.pc, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Lewizho99, 
Zppix, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, 
Addshore, Mbch331, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294961: Resolve kernel hang on wcqs* instances

[gerritbot]

gerritbot added a project: Patch-For-Review.

TASK DETAIL
  https://phabricator.wikimedia.org/T294961

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: Legoktm, Dzahn, MoritzMuehlenhoff, RKemper, Aklapper, Gehel, CBogen, 
ttaylor, Zache, Fuzheado, So9q, GFontenelle_WMF, EBernhardson, 786, Suran38, 
Biggs657, joanna_borun, Invadibot, Lalamarie69, MPhamWMF, Devnull, maantietaja, 
lmata, Juan90264, Muchiri124, Alter-paule, Beast1978, Un1tY, Akuckartz, 
Hook696, Kent7301, RhinosF1, joker88john, Legado_Shulgin, ReaperDawn, CucyNoiD, 
Nandana, Namenlos314, Gaboe420, Giuliamocci, Davinaclare77, Cpaulf30, 
Techguru.pc, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Lewizho99, 
Zppix, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, 
Addshore, Mbch331, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294961: Resolve kernel hang on wcqs* instances

[gerritbot]

gerritbot added a comment.


  Change 743223 had a related patch set uploaded (by Ryan Kemper; author: Ryan 
Kemper):
  
  [operations/puppet@production] Switch WCQS to profile::base::linux510
  
  https://gerrit.wikimedia.org/r/743223

TASK DETAIL
  https://phabricator.wikimedia.org/T294961

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: Legoktm, Dzahn, MoritzMuehlenhoff, RKemper, Aklapper, Gehel, CBogen, 
ttaylor, Zache, Fuzheado, So9q, GFontenelle_WMF, EBernhardson, joanna_borun, 
Invadibot, MPhamWMF, Devnull, maantietaja, lmata, Muchiri124, Akuckartz, 
RhinosF1, Legado_Shulgin, ReaperDawn, Nandana, Namenlos314, Davinaclare77, 
Techguru.pc, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, 
QZanden, EBjune, merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, 
Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, faidon, Addshore, Mbch331, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294961: Resolve kernel hang on wcqs* instances

[Maintenance_bot]

Maintenance_bot removed a project: Patch-For-Review.

TASK DETAIL
  https://phabricator.wikimedia.org/T294961

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Maintenance_bot
Cc: Legoktm, Dzahn, MoritzMuehlenhoff, RKemper, Aklapper, Gehel, CBogen, 
ttaylor, Zache, Fuzheado, So9q, GFontenelle_WMF, EBernhardson, joanna_borun, 
Invadibot, MPhamWMF, Devnull, maantietaja, lmata, Muchiri124, Akuckartz, 
RhinosF1, Legado_Shulgin, ReaperDawn, Nandana, Namenlos314, Davinaclare77, 
Techguru.pc, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, 
QZanden, EBjune, merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, 
Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, faidon, Addshore, Mbch331, Jay8g, fgiunchedi, 786, Suran38, 
Biggs657, Lalamarie69, Juan90264, Alter-paule, Beast1978, Un1tY, Hook696, 
Kent7301, joker88john, CucyNoiD, Gaboe420, Giuliamocci, Cpaulf30, Af420, 
Bsandipan, Lewizho99, Maathavan
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294961: Resolve kernel hang on wcqs* instances

[gerritbot]

gerritbot added a comment.


  Change 742729 **merged** by Ryan Kemper:
  
  [operations/puppet@production] Add profile::base::linux419 to the WCQS role
  
  https://gerrit.wikimedia.org/r/742729

TASK DETAIL
  https://phabricator.wikimedia.org/T294961

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: Legoktm, Dzahn, MoritzMuehlenhoff, RKemper, Aklapper, Gehel, CBogen, 
ttaylor, Zache, Fuzheado, So9q, GFontenelle_WMF, EBernhardson, 786, Suran38, 
Biggs657, joanna_borun, Invadibot, Lalamarie69, MPhamWMF, Devnull, maantietaja, 
lmata, Juan90264, Muchiri124, Alter-paule, Beast1978, Un1tY, Akuckartz, 
Hook696, Kent7301, RhinosF1, joker88john, Legado_Shulgin, ReaperDawn, CucyNoiD, 
Nandana, Namenlos314, Gaboe420, Giuliamocci, Davinaclare77, Cpaulf30, 
Techguru.pc, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Lewizho99, 
Zppix, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, 
Addshore, Mbch331, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294961: Resolve kernel hang on wcqs* instances

[gerritbot]

gerritbot added a project: Patch-For-Review.

TASK DETAIL
  https://phabricator.wikimedia.org/T294961

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: Legoktm, Dzahn, MoritzMuehlenhoff, RKemper, Aklapper, Gehel, CBogen, 
ttaylor, Zache, Fuzheado, So9q, GFontenelle_WMF, EBernhardson, 786, Suran38, 
Biggs657, joanna_borun, Invadibot, Lalamarie69, MPhamWMF, Devnull, maantietaja, 
lmata, Juan90264, Muchiri124, Alter-paule, Beast1978, Un1tY, Akuckartz, 
Hook696, Kent7301, RhinosF1, joker88john, Legado_Shulgin, ReaperDawn, CucyNoiD, 
Nandana, Namenlos314, Gaboe420, Giuliamocci, Davinaclare77, Cpaulf30, 
Techguru.pc, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Lewizho99, 
Zppix, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, 
Addshore, Mbch331, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294961: Resolve kernel hang on wcqs* instances

[gerritbot]

gerritbot added a comment.


  Change 742729 had a related patch set uploaded (by Muehlenhoff; author: 
Muehlenhoff):
  
  [operations/puppet@production] Add profile::base::linux419 to the WCQS role
  
  https://gerrit.wikimedia.org/r/742729

TASK DETAIL
  https://phabricator.wikimedia.org/T294961

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: Legoktm, Dzahn, MoritzMuehlenhoff, RKemper, Aklapper, Gehel, CBogen, 
ttaylor, Zache, Fuzheado, So9q, GFontenelle_WMF, EBernhardson, joanna_borun, 
Invadibot, MPhamWMF, Devnull, maantietaja, lmata, Muchiri124, Akuckartz, 
RhinosF1, Legado_Shulgin, ReaperDawn, Nandana, Namenlos314, Davinaclare77, 
Techguru.pc, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, 
QZanden, EBjune, merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, 
Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, faidon, Addshore, Mbch331, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294961: Resolve kernel hang on wcqs* instances

[MoritzMuehlenhoff]

MoritzMuehlenhoff added a comment.


  In T294961#7534879 , 
@EBernhardson wrote:
  
  > Another round of import tests completed, nothing fell over. Calling this 
done for now.
  
  We still to add profile::base::linux419 to the WCQS roles, otherwise with the 
next reimage they'd get installed with Linux 4.9 again.

TASK DETAIL
  https://phabricator.wikimedia.org/T294961

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: MoritzMuehlenhoff
Cc: Legoktm, Dzahn, MoritzMuehlenhoff, RKemper, Aklapper, Gehel, CBogen, 
ttaylor, Zache, Fuzheado, So9q, GFontenelle_WMF, EBernhardson, joanna_borun, 
Invadibot, MPhamWMF, Devnull, maantietaja, lmata, Muchiri124, Akuckartz, 
RhinosF1, Legado_Shulgin, ReaperDawn, Nandana, Namenlos314, Davinaclare77, 
Techguru.pc, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, 
QZanden, EBjune, merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, 
Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, faidon, Addshore, Mbch331, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294961: Resolve kernel hang on wcqs* instances

[EBernhardson]

EBernhardson moved this task from Waiting to Needs Reporting on the 
Discovery-Search (Current work) board.
EBernhardson added a comment.


  Another round of import tests completed, nothing fell over. Calling this done 
for now.

TASK DETAIL
  https://phabricator.wikimedia.org/T294961

WORKBOARD
  https://phabricator.wikimedia.org/project/board/1227/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: EBernhardson
Cc: Legoktm, Dzahn, MoritzMuehlenhoff, RKemper, Aklapper, Gehel, CBogen, 
ttaylor, Zache, Fuzheado, So9q, GFontenelle_WMF, EBernhardson, joanna_borun, 
Invadibot, MPhamWMF, Devnull, maantietaja, lmata, Muchiri124, Akuckartz, 
RhinosF1, Legado_Shulgin, ReaperDawn, Nandana, Namenlos314, Davinaclare77, 
Techguru.pc, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, 
QZanden, EBjune, merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, 
Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, faidon, Addshore, Mbch331, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294961: Resolve kernel hang on wcqs* instances

[EBernhardson]

EBernhardson added a comment.


  Started another round of imports today to see how it goes. If it doesn't fall 
over might as well call this done for now.

TASK DETAIL
  https://phabricator.wikimedia.org/T294961

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: EBernhardson
Cc: Legoktm, Dzahn, MoritzMuehlenhoff, RKemper, Aklapper, Gehel, CBogen, 
ttaylor, Zache, Fuzheado, So9q, GFontenelle_WMF, EBernhardson, joanna_borun, 
Bongo-Cat, Invadibot, MPhamWMF, Devnull, maantietaja, lmata, Muchiri124, 
Akuckartz, RhinosF1, Legado_Shulgin, ReaperDawn, Nandana, Namenlos314, 
Davinaclare77, Techguru.pc, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles, faidon, Addshore, Mbch331, Jay8g, 
fgiunchedi
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294961: Resolve kernel hang on wcqs* instances

[colewhite]

colewhite triaged this task as "Medium" priority.

TASK DETAIL
  https://phabricator.wikimedia.org/T294961

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: colewhite
Cc: Dzahn, MoritzMuehlenhoff, RKemper, Aklapper, Gehel, CBogen, ttaylor, Zache, 
Fuzheado, So9q, GFontenelle_WMF, EBernhardson, joanna_borun, Invadibot, 
MPhamWMF, Devnull, maantietaja, lmata, Muchiri124, Akuckartz, RhinosF1, 
Legado_Shulgin, ReaperDawn, Nandana, Namenlos314, Davinaclare77, Techguru.pc, 
Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, QZanden, EBjune, 
merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, 
faidon, Addshore, Mbch331, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294961: Resolve kernel hang on wcqs* instances

[EBernhardson]

EBernhardson added a comment.


  The import that caused everything to fall over last time completed. I'm not 
sure that's enough to declare this fixed (it ran once before as well) but after 
putting the puppet patch in place we can probably wait on this one to see if it 
reoccurs.

TASK DETAIL
  https://phabricator.wikimedia.org/T294961

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: EBernhardson
Cc: Dzahn, MoritzMuehlenhoff, RKemper, Aklapper, Gehel, CBogen, ttaylor, Zache, 
Fuzheado, So9q, GFontenelle_WMF, EBernhardson, joanna_borun, Invadibot, 
MPhamWMF, Devnull, maantietaja, lmata, Muchiri124, Akuckartz, RhinosF1, 
Legado_Shulgin, ReaperDawn, Nandana, Namenlos314, Davinaclare77, Techguru.pc, 
Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, QZanden, EBjune, 
merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, 
faidon, Addshore, Mbch331, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294961: Resolve kernel hang on wcqs* instances

[RKemper]

RKemper added a comment.


  In T294961#7489817 , 
@Gehel wrote:
  
  > Let's make sure the kernel version is pinned somewhere in our puppet code! 
Then we can wait to see if the problem is reproduced or not.
  
  
  
  >  ryankemper, ebernhardson: yeah, let's install the kernel package 
manually for testing and if it fixes the issue, then we can apply the 
profile::base::linux510
  
  So specifically we'll apply `profile::base::linux510`

TASK DETAIL
  https://phabricator.wikimedia.org/T294961

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Dzahn, MoritzMuehlenhoff, RKemper, Aklapper, Gehel, CBogen, ttaylor, Zache, 
Fuzheado, So9q, GFontenelle_WMF, EBernhardson, joanna_borun, Invadibot, 
MPhamWMF, Devnull, maantietaja, lmata, Muchiri124, Akuckartz, RhinosF1, 
Legado_Shulgin, ReaperDawn, Nandana, Namenlos314, Davinaclare77, Techguru.pc, 
Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, QZanden, EBjune, 
merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, 
faidon, Addshore, Mbch331, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294961: Resolve kernel hang on wcqs* instances

[RKemper]

RKemper merged a task: T294865: wcqs1002 and wcqs2001 unresponsive.
RKemper added a subscriber: Dzahn.

TASK DETAIL
  https://phabricator.wikimedia.org/T294961

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Dzahn, MoritzMuehlenhoff, RKemper, Aklapper, Gehel, CBogen, ttaylor, Zache, 
Fuzheado, So9q, GFontenelle_WMF, EBernhardson, joanna_borun, Invadibot, 
MPhamWMF, Devnull, maantietaja, lmata, Muchiri124, Akuckartz, RhinosF1, 
Legado_Shulgin, ReaperDawn, Nandana, Namenlos314, Davinaclare77, Techguru.pc, 
Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, QZanden, EBjune, 
merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, 
faidon, Addshore, Mbch331, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294961: Resolve kernel hang on wcqs* instances

[Gehel]

Gehel added a comment.


  Let's make sure the kernel version is pinned somewhere in our puppet code! 
Then we can wait to see if the problem is reproduced or not.

TASK DETAIL
  https://phabricator.wikimedia.org/T294961

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Gehel
Cc: Dzahn, MoritzMuehlenhoff, RKemper, Aklapper, Gehel, CBogen, ttaylor, Zache, 
Fuzheado, So9q, GFontenelle_WMF, EBernhardson, joanna_borun, Invadibot, 
MPhamWMF, Devnull, maantietaja, lmata, Muchiri124, Akuckartz, RhinosF1, 
Legado_Shulgin, ReaperDawn, Nandana, Namenlos314, Davinaclare77, Techguru.pc, 
Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, QZanden, EBjune, 
merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, 
faidon, Addshore, Mbch331, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294961: Resolve kernel hang on wcqs* instances

[Stashbot]

Stashbot added a comment.


  Mentioned in SAL (#wikimedia-operations) [2021-11-04T17:23:57Z]  
T294961  [WCQS] Installed kernel 
version `Linux 5.10.0-0.bpo.9-amd64` on all wcqs* hosts

TASK DETAIL
  https://phabricator.wikimedia.org/T294961

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Stashbot
Cc: MoritzMuehlenhoff, RKemper, Aklapper, Gehel, CBogen, ttaylor, Zache, 
Fuzheado, So9q, GFontenelle_WMF, EBernhardson, joanna_borun, Invadibot, 
MPhamWMF, Devnull, maantietaja, lmata, Muchiri124, Akuckartz, RhinosF1, 
Legado_Shulgin, ReaperDawn, Nandana, Namenlos314, Davinaclare77, Techguru.pc, 
Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, QZanden, EBjune, 
merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, 
faidon, Addshore, Mbch331, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294961: Resolve kernel hang on wcqs* instances

[Stashbot]

Stashbot added a comment.


  Mentioned in SAL (#wikimedia-operations) [2021-11-04T16:48:45Z]  
T294961  [WCQS] Power cycled all 6 
wcqs* hosts via the mgmt console (`racadm serveraction powercycle`)

TASK DETAIL
  https://phabricator.wikimedia.org/T294961

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Stashbot
Cc: MoritzMuehlenhoff, RKemper, Aklapper, Gehel, CBogen, ttaylor, Zache, 
Fuzheado, So9q, GFontenelle_WMF, EBernhardson, joanna_borun, Invadibot, 
MPhamWMF, Devnull, maantietaja, lmata, Muchiri124, Akuckartz, RhinosF1, 
Legado_Shulgin, ReaperDawn, Nandana, Namenlos314, Davinaclare77, Techguru.pc, 
Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, QZanden, EBjune, 
merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, 
faidon, Addshore, Mbch331, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294961: Resolve kernel hang on wcqs* instances

[MoritzMuehlenhoff]

MoritzMuehlenhoff added a comment.


  In T294961#7479399 , 
@EBernhardson wrote:
  
  > Without having better information, i would guess we are triggering a 
deadlock somewhere in the kernel related to disk writes? But no particular 
locking is mentioned in the traces so perhaps not. As it affected all 6 
instances it seems fairly reproducable, although perhaps we should try and take 
the application out of the picture and try to reproduce with some simple tools 
to generate disk writes? This might be fairly tedious, the error took more than 
a day to trigger and the only idea I have for reproducing externally is to 
generate random write loads of similar size.
  >
  > @MoritzMuehlenhoff as someone who deals with the kernel often, any 
suggestions for where to investigate?
  
  It looks to be deadlocking somewhere deep in the I/O layer, there are some 
sysctls and kernel settings that we could fine-tune, but given that this only 
happens after a full day run, that'll be a slow going process.
  
  I see two next steps that we should try:
  
  1. Looking at Netbox these were purchased in March, but that doesn't 
necessarily mean that the system firmware is up-to-date. Often these are 
delivered with the firmware version once the specific server model was 
originally shipped. We could ask DC ops to upgrade one of the servers to the 
latest versions and re-test.
  2. These are Buster systems, but we can try the 5.10 kernel available from 
Debian backports (installable with "apt-get install 
linux-image-5.10.0-0.bpo.9"). We have a handful of services which also the 
kernel on buster already (e.g. the Hadoop and stat* hosts with the AMD GPUs).

TASK DETAIL
  https://phabricator.wikimedia.org/T294961

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: MoritzMuehlenhoff
Cc: MoritzMuehlenhoff, RKemper, Aklapper, Gehel, CBogen, ttaylor, Zache, 
Fuzheado, So9q, GFontenelle_WMF, EBernhardson, joanna_borun, Invadibot, 
MPhamWMF, Devnull, maantietaja, lmata, Muchiri124, Akuckartz, RhinosF1, 
Legado_Shulgin, ReaperDawn, Nandana, Namenlos314, Davinaclare77, Techguru.pc, 
Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, QZanden, EBjune, 
merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, 
faidon, Addshore, Mbch331, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294961: Resolve kernel hang on wcqs* instances

[Maintenance_bot]

Maintenance_bot removed a project: Patch-For-Review.

TASK DETAIL
  https://phabricator.wikimedia.org/T294961

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Maintenance_bot
Cc: MoritzMuehlenhoff, RKemper, Aklapper, Gehel, CBogen, ttaylor, Zache, 
Fuzheado, So9q, GFontenelle_WMF, EBernhardson, joanna_borun, Invadibot, 
MPhamWMF, Devnull, maantietaja, lmata, Muchiri124, Akuckartz, RhinosF1, 
Legado_Shulgin, ReaperDawn, Nandana, Namenlos314, Davinaclare77, Techguru.pc, 
Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, QZanden, EBjune, 
merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, 
faidon, Addshore, Mbch331, Jay8g, fgiunchedi, Suran38, Biggs657, Lalamarie69, 
Juan90264, Alter-paule, Beast1978, Un1tY, Hook696, Kent7301, joker88john, 
CucyNoiD, Gaboe420, Giuliamocci, Cpaulf30, Af420, Bsandipan, Lewizho99, 
Maathavan
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294961: Resolve kernel hang on wcqs* instances

[Stashbot]

Stashbot added a comment.


  Mentioned in SAL (#wikimedia-operations) [2021-11-03T21:56:00Z]  
T294961  [WCQS] Forcing recheck of 
`PyBal IPVS diff check` and `PyBal backends health check`

TASK DETAIL
  https://phabricator.wikimedia.org/T294961

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Stashbot
Cc: MoritzMuehlenhoff, RKemper, Aklapper, Gehel, CBogen, ttaylor, Zache, 
Fuzheado, So9q, GFontenelle_WMF, EBernhardson, Suran38, Biggs657, joanna_borun, 
Invadibot, Lalamarie69, MPhamWMF, Devnull, maantietaja, lmata, Juan90264, 
Muchiri124, Alter-paule, Beast1978, Un1tY, Akuckartz, Hook696, Kent7301, 
RhinosF1, joker88john, Legado_Shulgin, ReaperDawn, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Davinaclare77, Cpaulf30, Techguru.pc, Lahi, 
Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, 
QZanden, EBjune, merbst, LawExplorer, Lewizho99, Zppix, Maathavan, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles, faidon, Addshore, Mbch331, Jay8g, 
fgiunchedi
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294961: Resolve kernel hang on wcqs* instances

[Stashbot]

Stashbot added a comment.


  Mentioned in SAL (#wikimedia-operations) [2021-11-03T21:53:32Z]  
T294961  [WCQS] Merged 
https://gerrit.wikimedia.org/r/c/operations/puppet/+/736564 and successfully 
ran `ryankemper@cumin1001:~$ sudo cumin 'A:icinga or A:dns-auth' 
run-puppet-agent`

TASK DETAIL
  https://phabricator.wikimedia.org/T294961

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Stashbot
Cc: MoritzMuehlenhoff, RKemper, Aklapper, Gehel, CBogen, ttaylor, Zache, 
Fuzheado, So9q, GFontenelle_WMF, EBernhardson, Suran38, Biggs657, joanna_borun, 
Invadibot, Lalamarie69, MPhamWMF, Devnull, maantietaja, lmata, Juan90264, 
Muchiri124, Alter-paule, Beast1978, Un1tY, Akuckartz, Hook696, Kent7301, 
RhinosF1, joker88john, Legado_Shulgin, ReaperDawn, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Davinaclare77, Cpaulf30, Techguru.pc, Lahi, 
Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, 
QZanden, EBjune, merbst, LawExplorer, Lewizho99, Zppix, Maathavan, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles, faidon, Addshore, Mbch331, Jay8g, 
fgiunchedi
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294961: Resolve kernel hang on wcqs* instances

[gerritbot]

gerritbot added a comment.


  Change 736564 **merged** by Ryan Kemper:
  
  [operations/puppet@production] wcqs: state change production->lvs_setup
  
  https://gerrit.wikimedia.org/r/736564

TASK DETAIL
  https://phabricator.wikimedia.org/T294961

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: MoritzMuehlenhoff, RKemper, Aklapper, Gehel, CBogen, ttaylor, Zache, 
Fuzheado, So9q, GFontenelle_WMF, EBernhardson, Suran38, Biggs657, joanna_borun, 
Invadibot, Lalamarie69, MPhamWMF, Devnull, maantietaja, lmata, Juan90264, 
Muchiri124, Alter-paule, Beast1978, Un1tY, Akuckartz, Hook696, Kent7301, 
RhinosF1, joker88john, Legado_Shulgin, ReaperDawn, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Davinaclare77, Cpaulf30, Techguru.pc, Lahi, 
Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, 
QZanden, EBjune, merbst, LawExplorer, Lewizho99, Zppix, Maathavan, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles, faidon, Addshore, Mbch331, Jay8g, 
fgiunchedi
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294961: Resolve kernel hang on wcqs* instances

[Stashbot]

Stashbot added a comment.


  Mentioned in SAL (#wikimedia-operations) [2021-11-03T21:47:45Z]  
T294961  [WCQS] DNS changes rolled 
out, proceeding to the `lvs_setup` step: 
https://gerrit.wikimedia.org/r/c/operations/puppet/+/736564

TASK DETAIL
  https://phabricator.wikimedia.org/T294961

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Stashbot
Cc: MoritzMuehlenhoff, RKemper, Aklapper, Gehel, CBogen, ttaylor, Zache, 
Fuzheado, So9q, GFontenelle_WMF, EBernhardson, Suran38, Biggs657, joanna_borun, 
Invadibot, Lalamarie69, MPhamWMF, Devnull, maantietaja, lmata, Juan90264, 
Muchiri124, Alter-paule, Beast1978, Un1tY, Akuckartz, Hook696, Kent7301, 
RhinosF1, joker88john, Legado_Shulgin, ReaperDawn, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Davinaclare77, Cpaulf30, Techguru.pc, Lahi, 
Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, 
QZanden, EBjune, merbst, LawExplorer, Lewizho99, Zppix, Maathavan, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles, faidon, Addshore, Mbch331, Jay8g, 
fgiunchedi
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294961: Resolve kernel hang on wcqs* instances

[RKemper]

RKemper added a comment.


  DNS changes from https://gerrit.wikimedia.org/r/c/operations/dns/+/736585
  -
  
ryankemper@authdns1001:~$ sudo -i authdns-update
Updating authdns1001.wikimedia.org (self)...
Pulling the current revision from 
https://gerrit.wikimedia.org/r/operations/dns.git
Reviewing 2c9042d86630ef640aaf1ab101c9f98bda1824e0...

 templates/wmnet| 1 -
 utils/mock_etc/discovery-geo-resources | 1 -
 2 files changed, 2 deletions(-)

diff --git templates/wmnet templates/wmnet
index c89c1610..041fc7b9 100644
--- templates/wmnet
+++ templates/wmnet
@@ -654,7 +654,6 @@ push-notifications 300/10 IN DYNA 
geoip!disc-push-notifications
 recommendation-api 300/10 IN DYNA geoip!disc-recommendation-api
 wdqs   300/10 IN DYNA geoip!disc-wdqs
 wdqs-internal  300/10 IN DYNA geoip!disc-wdqs-internal
-wcqs   300/10 IN DYNA geoip!disc-wcqs
 zotero 300/10 IN DYNA geoip!disc-zotero
 blubberoid 300/10 IN DYNA geoip!disc-blubberoid
 eventgate-analytics 300/10 IN DYNA geoip!disc-eventgate-analytics
diff --git utils/mock_etc/discovery-geo-resources 
utils/mock_etc/discovery-geo-resources
index 64e612b3..41a50fc4 100644
--- utils/mock_etc/discovery-geo-resources
+++ utils/mock_etc/discovery-geo-resources
@@ -66,4 +66,3 @@ disc-shellbox-media  => { map => mock, dcmap => { 
mock => 192.0.2.1 } }
 disc-shellbox-syntaxhighlight => { map => mock, dcmap => { mock => 
192.0.2.1 } }
 disc-shellbox-timeline   => { map => mock, dcmap => { mock => 192.0.2.1 } }
 disc-tegola-vector-tiles => { map => mock, dcmap => { mock => 192.0.2.1 } }
-disc-wcqs=> { map => mock, dcmap => { mock => 192.0.2.1 } }

Merge these changes? (yes/no)? yes
Updating 7a86417b..2c9042d8
Fast-forward
 templates/wmnet| 1 -
 utils/mock_etc/discovery-geo-resources | 1 -
 2 files changed, 2 deletions(-)
Deploying via utils/deploy-check.py...
Assembling and testing data in /tmp/dns-check.oenx7otf
 -- Generating zonefiles from zone templates
 -- Processed 211 zones into directory /tmp/dns-check.oenx7otf/zones
OK: No tabs
Summary of violations:
W001|MISSING_IP_FOR_NAME_AND_PTR: 37
W002|MISSING_PTR_FOR_NAME_AND_IP: 27
W103|MISSING_MGMT_FOR_NAME: 65
W105|TOO_MANY_PUBLIC_NAMES: 3
RESULT: 0 Errors, 132 Warnings, 0 Ignored violations, 0 Ignored lines
 -- Copying automatically generated zone files under target tree
 -- Copying repo-driven real config files and admin_state
 -- Copying puppetized config and GeoIP from /etc/gdnsd
 -- Checking for illegal tabs in zonefiles
 -- Running zone_validator to check WMF rules
 -- Running /usr/sbin/gdnsd checkconf on /tmp/dns-check.oenx7otf
 -- Preflight checkconf is OK
Deploying from /tmp/dns-check.oenx7otf to system dirs
 -- Zone changed: wmnet
 -- Descending to subdirectory: netbox
 -- Done with subdir: netbox
Reloading gdnsd zonefiles
info: Zone data reloaded
OK
---

authdns2001.wikimedia.org,dns[1001-1002,2001-2002,3001-3002,4001-4002,5001-5002].wikimedia.org
 (11)
---
OK - authdns updated successfully

OK - authdns-update successful on all nodes!

TASK DETAIL
  https://phabricator.wikimedia.org/T294961

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: MoritzMuehlenhoff, RKemper, Aklapper, Gehel, CBogen, ttaylor, Zache, 
Fuzheado, So9q, GFontenelle_WMF, EBernhardson, Suran38, Biggs657, joanna_borun, 
Invadibot, Lalamarie69, MPhamWMF, Devnull, maantietaja, lmata, Juan90264, 
Muchiri124, Alter-paule, Beast1978, Un1tY, Akuckartz, Hook696, Kent7301, 
RhinosF1, joker88john, Legado_Shulgin, ReaperDawn, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Davinaclare77, Cpaulf30, Techguru.pc, Lahi, 
Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, 
QZanden, EBjune, merbst, LawExplorer, Lewizho99, Zppix, Maathavan, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles, faidon, Addshore, Mbch331, Jay8g, 
fgiunchedi
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294961: Resolve kernel hang on wcqs* instances

[Stashbot]

Stashbot added a comment.


  Mentioned in SAL (#wikimedia-operations) [2021-11-03T21:45:46Z]  
T294961  [WCQS] Merged 
https://gerrit.wikimedia.org/r/c/operations/dns/+/736585, running 
`ryankemper@authdns1001:~$ sudo -i authdns-update`

TASK DETAIL
  https://phabricator.wikimedia.org/T294961

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Stashbot
Cc: MoritzMuehlenhoff, RKemper, Aklapper, Gehel, CBogen, ttaylor, Zache, 
Fuzheado, So9q, GFontenelle_WMF, EBernhardson, Suran38, Biggs657, joanna_borun, 
Invadibot, Lalamarie69, MPhamWMF, Devnull, maantietaja, lmata, Juan90264, 
Muchiri124, Alter-paule, Beast1978, Un1tY, Akuckartz, Hook696, Kent7301, 
RhinosF1, joker88john, Legado_Shulgin, ReaperDawn, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Davinaclare77, Cpaulf30, Techguru.pc, Lahi, 
Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, 
QZanden, EBjune, merbst, LawExplorer, Lewizho99, Zppix, Maathavan, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles, faidon, Addshore, Mbch331, Jay8g, 
fgiunchedi
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294961: Resolve kernel hang on wcqs* instances

[gerritbot]

gerritbot added a comment.


  Change 736585 **merged** by Ryan Kemper:
  
  [operations/dns@master] Revert \"wcqs: add discovery record\"
  
  https://gerrit.wikimedia.org/r/736585

TASK DETAIL
  https://phabricator.wikimedia.org/T294961

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: MoritzMuehlenhoff, RKemper, Aklapper, Gehel, CBogen, ttaylor, Zache, 
Fuzheado, So9q, GFontenelle_WMF, EBernhardson, Suran38, Biggs657, joanna_borun, 
Invadibot, Lalamarie69, MPhamWMF, Devnull, maantietaja, lmata, Juan90264, 
Muchiri124, Alter-paule, Beast1978, Un1tY, Akuckartz, Hook696, Kent7301, 
RhinosF1, joker88john, Legado_Shulgin, ReaperDawn, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Davinaclare77, Cpaulf30, Techguru.pc, Lahi, 
Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, 
QZanden, EBjune, merbst, LawExplorer, Lewizho99, Zppix, Maathavan, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles, faidon, Addshore, Mbch331, Jay8g, 
fgiunchedi
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294961: Resolve kernel hang on wcqs* instances

[EBernhardson]

EBernhardson updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T294961

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: EBernhardson
Cc: MoritzMuehlenhoff, RKemper, Aklapper, Gehel, CBogen, ttaylor, Zache, 
Fuzheado, So9q, GFontenelle_WMF, EBernhardson, Suran38, Biggs657, joanna_borun, 
Invadibot, Lalamarie69, MPhamWMF, Devnull, maantietaja, lmata, Juan90264, 
Muchiri124, Alter-paule, Beast1978, Un1tY, Akuckartz, Hook696, Kent7301, 
RhinosF1, joker88john, Legado_Shulgin, ReaperDawn, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Davinaclare77, Cpaulf30, Techguru.pc, Lahi, 
Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, 
QZanden, EBjune, merbst, LawExplorer, Lewizho99, Zppix, Maathavan, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles, faidon, Addshore, Mbch331, Jay8g, 
fgiunchedi
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294961: Resolve kernel hang on wcqs* instances

[gerritbot]

gerritbot added a comment.


  Change 736585 had a related patch set uploaded (by Ryan Kemper; author: Ryan 
Kemper):
  
  [operations/dns@master] Revert \"wcqs: add discovery record\"
  
  https://gerrit.wikimedia.org/r/736585

TASK DETAIL
  https://phabricator.wikimedia.org/T294961

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: MoritzMuehlenhoff, RKemper, Aklapper, Gehel, CBogen, ttaylor, Zache, 
Fuzheado, So9q, GFontenelle_WMF, EBernhardson, Suran38, Biggs657, joanna_borun, 
Invadibot, Lalamarie69, MPhamWMF, Devnull, maantietaja, lmata, Juan90264, 
Muchiri124, Alter-paule, Beast1978, Un1tY, Akuckartz, Hook696, Kent7301, 
RhinosF1, joker88john, Legado_Shulgin, ReaperDawn, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Davinaclare77, Cpaulf30, Techguru.pc, Lahi, 
Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, 
QZanden, EBjune, merbst, LawExplorer, Lewizho99, Zppix, Maathavan, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles, faidon, Addshore, Mbch331, Jay8g, 
fgiunchedi
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294961: Resolve kernel hang on wcqs* instances

[gerritbot]

gerritbot added a project: Patch-For-Review.

TASK DETAIL
  https://phabricator.wikimedia.org/T294961

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: MoritzMuehlenhoff, RKemper, Aklapper, Gehel, CBogen, ttaylor, Zache, 
Fuzheado, So9q, GFontenelle_WMF, EBernhardson, Suran38, Biggs657, joanna_borun, 
Invadibot, Lalamarie69, MPhamWMF, Devnull, maantietaja, lmata, Juan90264, 
Muchiri124, Alter-paule, Beast1978, Un1tY, Akuckartz, Hook696, Kent7301, 
RhinosF1, joker88john, Legado_Shulgin, ReaperDawn, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Davinaclare77, Cpaulf30, Techguru.pc, Lahi, 
Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, 
QZanden, EBjune, merbst, LawExplorer, Lewizho99, Zppix, Maathavan, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles, faidon, Addshore, Mbch331, Jay8g, 
fgiunchedi
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294961: Resolve kernel hang on wcqs* instances

[gerritbot]

gerritbot added a comment.


  Change 736564 had a related patch set uploaded (by Ryan Kemper; author: Ryan 
Kemper):
  
  [operations/puppet@production] wcqs: state change production->lvs_setup
  
  https://gerrit.wikimedia.org/r/736564

TASK DETAIL
  https://phabricator.wikimedia.org/T294961

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: MoritzMuehlenhoff, RKemper, Aklapper, Gehel, CBogen, ttaylor, Zache, 
Fuzheado, So9q, GFontenelle_WMF, EBernhardson, joanna_borun, Invadibot, 
MPhamWMF, Devnull, maantietaja, lmata, Muchiri124, Akuckartz, RhinosF1, 
Legado_Shulgin, ReaperDawn, Nandana, Namenlos314, Davinaclare77, Techguru.pc, 
Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, QZanden, EBjune, 
merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, 
faidon, Addshore, Mbch331, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294961: Resolve kernel hang on wcqs* instances

[RKemper]

RKemper added a parent task: T294865: wcqs1002 and wcqs2001 unresponsive.

TASK DETAIL
  https://phabricator.wikimedia.org/T294961

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: MoritzMuehlenhoff, RKemper, Aklapper, Gehel, CBogen, ttaylor, Zache, 
Fuzheado, So9q, GFontenelle_WMF, EBernhardson, joanna_borun, Invadibot, 
MPhamWMF, Devnull, maantietaja, lmata, Muchiri124, Akuckartz, RhinosF1, 
Legado_Shulgin, ReaperDawn, Nandana, Namenlos314, Davinaclare77, Techguru.pc, 
Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, QZanden, EBjune, 
merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, 
faidon, Addshore, Mbch331, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294961: Resolve kernel hang on wcqs* instances

[Stashbot]

Stashbot added a comment.


  Mentioned in SAL (#wikimedia-operations) [2021-11-03T19:35:49Z]  
depooled wcqs2003 (pooled=inactive) because Icinga alerts that servers are down 
but pooled. not in production yet but issues (T294961 
)

TASK DETAIL
  https://phabricator.wikimedia.org/T294961

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Stashbot
Cc: MoritzMuehlenhoff, RKemper, Aklapper, Gehel, CBogen, ttaylor, Zache, 
Fuzheado, So9q, GFontenelle_WMF, EBernhardson, joanna_borun, Invadibot, 
MPhamWMF, Devnull, maantietaja, lmata, Muchiri124, Akuckartz, RhinosF1, 
Legado_Shulgin, ReaperDawn, Nandana, Namenlos314, Davinaclare77, Techguru.pc, 
Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, QZanden, EBjune, 
merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, 
faidon, Addshore, Mbch331, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294961: Resolve kernel hang on wcqs* instances

[EBernhardson]

EBernhardson added a subscriber: MoritzMuehlenhoff.
EBernhardson added a comment.


  Some random info i looked up:
  
  - grafana reports free memory of at least 90G across instances. This is 
typical, the application leans heavily on the linux disk cache to keep it's 
data in memory.
  - Instances were all generally doing 4-6k write iops with a throughput of 
~50MB/s for 24hr+. Minimal reads.
  - Disk utilization reports around 15%, all instances similar
  - Filesystem for application data mounted to /srv, ~15% used.
  - raid reports all drives healthy (U):
  
ebernhardson@wcqs1003:~$ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid10] [raid6] 
[raid5] [raid4] 
md0 : active raid10 sda2[0] sdb2[1] sdd2[3] sdc2[2]
  3749898240 blocks super 1.2 512K chunks 2 near-copies [4/4] []
  bitmap: 1/28 pages [4KB], 65536KB chunk

unused devices: 
  
  Reading from md0 works as expected:
  
ebernhardson@wcqs1003:~$ sudo dd if=/dev/md0 of=/dev/null bs=1048576 
count=10 status=progress
104048099328 bytes (104 GB, 97 GiB) copied, 80 s, 1.3 GB/s
10+0 records in
10+0 records out
10485760 bytes (105 GB, 98 GiB) copied, 80.7095 s, 1.3 GB/s
  
  Attempting to sync a file to disk hangs:
  
root@wcqs1003:~# echo 'foo' > test
root@wcqs1003:~# sync test
  
  Without having better information, i would guess we are triggering a deadlock 
somewhere in the kernel related to disk writes. As it affected all 6 instances 
it seems fairly reproducable, although perhaps we should try and take the 
application out of the picture and try to reproduce with some simple tools to 
generate disk writes?
  
  @MoritzMuehlenhoff as someone who deals with the kernel often, any 
suggestions for where to investigate?

TASK DETAIL
  https://phabricator.wikimedia.org/T294961

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: EBernhardson
Cc: MoritzMuehlenhoff, RKemper, Aklapper, Gehel, CBogen, ttaylor, Zache, 
Fuzheado, So9q, GFontenelle_WMF, EBernhardson, joanna_borun, Invadibot, 
MPhamWMF, Devnull, maantietaja, lmata, Muchiri124, Akuckartz, RhinosF1, 
Legado_Shulgin, ReaperDawn, Nandana, Namenlos314, Davinaclare77, Techguru.pc, 
Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Hfbn0, QZanden, EBjune, 
merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, 
faidon, Addshore, Mbch331, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294961: Resolve kernel hang on wcqs* instances

[EBernhardson]

EBernhardson updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T294961

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: EBernhardson
Cc: Aklapper, Gehel, CBogen, ttaylor, Zache, Fuzheado, So9q, GFontenelle_WMF, 
EBernhardson, joanna_borun, Invadibot, MPhamWMF, Devnull, maantietaja, lmata, 
Muchiri124, Akuckartz, RhinosF1, Legado_Shulgin, ReaperDawn, Nandana, 
Namenlos314, Davinaclare77, Techguru.pc, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles, faidon, Addshore, Mbch331, Jay8g, 
fgiunchedi
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294961: Resolve kernel hang on wcqs* instances

[EBernhardson]

EBernhardson added a project: SRE.
Restricted Application added a project: wdwb-tech.

TASK DETAIL
  https://phabricator.wikimedia.org/T294961

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: EBernhardson
Cc: Aklapper, Gehel, CBogen, ttaylor, Zache, Fuzheado, So9q, GFontenelle_WMF, 
EBernhardson, joanna_borun, Invadibot, MPhamWMF, Devnull, maantietaja, lmata, 
Muchiri124, Akuckartz, RhinosF1, Legado_Shulgin, ReaperDawn, Nandana, 
Namenlos314, Davinaclare77, Techguru.pc, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles, faidon, Addshore, Mbch331, Jay8g, 
fgiunchedi
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T294961: Resolve kernel hang on wcqs* instances

[EBernhardson]

EBernhardson created this task.
EBernhardson added projects: Wikidata, Wikidata-Query-Service, Discovery-Search 
(Current work).

TASK DESCRIPTION
  While setting up the new wcqs service on wcqs[12]00[123] we started an 
import, the same import process that we have run many times on the wcqs-beta 
instance. After two days half of the instances became unresponsive, and the 
remaining half appear to have stopped importing but have not 100% hung the 
instance (yet).
  
  Attached is the dmesg for all instances that were still accessible, in all 
cases there is a single moment where the kernel starts complaining about hung 
tasks related to disk IO.
  
  wcqs1001: P17667 
  wcqs1003: P17668 
  wcqs2002: P17669 

TASK DETAIL
  https://phabricator.wikimedia.org/T294961

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: EBernhardson
Cc: Aklapper, Gehel, CBogen, ttaylor, Zache, Fuzheado, So9q, GFontenelle_WMF, 
EBernhardson, Invadibot, MPhamWMF, maantietaja, Akuckartz, Nandana, 
Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, 
EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org