[Wikidata-bugs] [Maniphest] T260281: mw* servers memory leaks (12 Aug)

2020-08-20 Thread jijiki 
jijiki updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T260281

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Joe, jijiki
Cc: eprodromou, Michael, NullPointer, Platonides, hashar, Addshore, Majavah, 
Ladsgroup, JMeybohm, ema, Joe, RhinosF1, ArielGlenn, jijiki, Aklapper, CDanis, 
lmata, wkandek, Akuckartz, darthmon_wmde, WDoranWMF, holger.knust, 
EvanProdromou, Legado_Shulgin, Nandana, Klaas_Z4us_V, Davinaclare77, Qtn1293, 
Techguru.pc, Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, 
LawExplorer, Zppix, elukey, _jensen, rosalieper, Agabi10, Scott_WUaS, Pchelolo, 
Wong128hk, Wikidata-bugs, aude, faidon, Mbch331, Rxy, Jay8g, fgiunchedi, Dzahn
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T260281: mw* servers memory leaks (12 Aug)

2020-08-20 Thread Joe 
Joe closed this task as "Resolved".
Joe added a comment.


  Reporting here in brief:
  
  - We confirmed the problem had to do with activating firejail for all 
executions of external programs. That triggered a kernel bug
  - This kernel bug can be bypassed by disabling kernel memory accounting in 
cgroups, or by moving to a newer kernel like 4.19
  - For now we've gone through a full rolling reboot of our mediawiki servers 
to disable kernel memory accounting in cgroups
  - We might move to 4.19 at a later stage.
  
  The issue is resolved and we're seeing ~ stable memory usage across all 
clusters, so the issue can be considered resolved.
  
  TLDR for third parties: if you run MediaWiki with `$wgShellRestrictionMethod 
= 'firejail';` you should do so with a relatively recent kernel, 4.19+ or 5.3+ 
IIRC.

TASK DETAIL
  https://phabricator.wikimedia.org/T260281

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Joe
Cc: eprodromou, Michael, NullPointer, Platonides, hashar, Addshore, Majavah, 
Ladsgroup, JMeybohm, ema, Joe, RhinosF1, ArielGlenn, jijiki, Aklapper, CDanis, 
lmata, wkandek, Akuckartz, darthmon_wmde, WDoranWMF, holger.knust, 
EvanProdromou, Legado_Shulgin, Nandana, Klaas_Z4us_V, Davinaclare77, Qtn1293, 
Techguru.pc, Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, 
LawExplorer, Zppix, elukey, _jensen, rosalieper, Agabi10, Scott_WUaS, Pchelolo, 
Wong128hk, Wikidata-bugs, aude, faidon, Mbch331, Rxy, Jay8g, fgiunchedi, Dzahn
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T260281: mw* servers memory leaks (12 Aug)

2020-08-19 Thread Joe 
Joe claimed this task.

TASK DETAIL
  https://phabricator.wikimedia.org/T260281

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Joe
Cc: eprodromou, Michael, NullPointer, Platonides, hashar, Addshore, Majavah, 
Ladsgroup, JMeybohm, ema, Joe, RhinosF1, ArielGlenn, jijiki, Aklapper, CDanis, 
lmata, wkandek, Akuckartz, darthmon_wmde, WDoranWMF, holger.knust, 
EvanProdromou, Legado_Shulgin, Nandana, Klaas_Z4us_V, Davinaclare77, Qtn1293, 
Techguru.pc, Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, 
LawExplorer, Zppix, elukey, _jensen, rosalieper, Agabi10, Scott_WUaS, Pchelolo, 
Wong128hk, Wikidata-bugs, aude, faidon, Mbch331, Rxy, Jay8g, fgiunchedi, Dzahn
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T260281: mw* servers memory leaks (12 Aug)

2020-08-18 Thread Joe 
Joe closed subtask Restricted Task as Resolved.

TASK DETAIL
  https://phabricator.wikimedia.org/T260281

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Joe
Cc: eprodromou, Michael, NullPointer, Platonides, hashar, Addshore, Majavah, 
Ladsgroup, JMeybohm, ema, Joe, RhinosF1, ArielGlenn, jijiki, Aklapper, CDanis, 
lmata, wkandek, Akuckartz, darthmon_wmde, WDoranWMF, holger.knust, 
EvanProdromou, Legado_Shulgin, Nandana, Klaas_Z4us_V, Davinaclare77, Qtn1293, 
Techguru.pc, Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, 
LawExplorer, Zppix, elukey, _jensen, rosalieper, Agabi10, Scott_WUaS, Pchelolo, 
Wong128hk, Wikidata-bugs, aude, faidon, Mbch331, Rxy, Jay8g, fgiunchedi, Dzahn
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T260281: mw* servers memory leaks (12 Aug)

2020-08-18 Thread eprodromou 
eprodromou moved this task from Inbox to Tracking/Watching on the Platform 
Engineering board.
eprodromou added a comment.


  We're tracking this, but unsure as to next steps. Let us know if more active 
investigation from Platform team is needed.

TASK DETAIL
  https://phabricator.wikimedia.org/T260281

WORKBOARD
  https://phabricator.wikimedia.org/project/board/3654/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: eprodromou
Cc: eprodromou, Michael, NullPointer, Platonides, hashar, Addshore, Majavah, 
Ladsgroup, JMeybohm, ema, Joe, RhinosF1, ArielGlenn, jijiki, Aklapper, CDanis, 
lmata, wkandek, Akuckartz, darthmon_wmde, WDoranWMF, holger.knust, 
EvanProdromou, Legado_Shulgin, Nandana, Klaas_Z4us_V, Davinaclare77, Qtn1293, 
Techguru.pc, Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, 
LawExplorer, Zppix, elukey, _jensen, rosalieper, Agabi10, Scott_WUaS, Pchelolo, 
Wong128hk, Wikidata-bugs, aude, faidon, Mbch331, Rxy, Jay8g, fgiunchedi, Dzahn
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T260281: mw* servers memory leaks (12 Aug)

2020-08-17 Thread ops-monitoring-bot 
ops-monitoring-bot added a comment.


  Completed auto-reimage of hosts:
  
['mw1359.eqiad.wmnet']
  
  and were **ALL** successful.

TASK DETAIL
  https://phabricator.wikimedia.org/T260281

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ops-monitoring-bot
Cc: Michael, NullPointer, Platonides, hashar, Addshore, Majavah, Ladsgroup, 
JMeybohm, ema, Joe, RhinosF1, ArielGlenn, jijiki, Aklapper, CDanis, lmata, 
wkandek, Akuckartz, darthmon_wmde, WDoranWMF, holger.knust, EvanProdromou, 
Legado_Shulgin, Nandana, Klaas_Z4us_V, Davinaclare77, Qtn1293, Techguru.pc, 
Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, LawExplorer, Zppix, 
elukey, _jensen, rosalieper, Agabi10, Scott_WUaS, Pchelolo, Wong128hk, 
Wikidata-bugs, aude, faidon, Mbch331, Rxy, Jay8g, fgiunchedi, Dzahn
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T260281: mw* servers memory leaks (12 Aug)

2020-08-17 Thread ops-monitoring-bot 
ops-monitoring-bot added a comment.


  Script wmf-auto-reimage was launched by cdanis on cumin1001.eqiad.wmnet for 
hosts:
  
mw1359.eqiad.wmnet
  
  The log can be found in 
`/var/log/wmf-auto-reimage/202008171607_cdanis_15670_mw1359_eqiad_wmnet.log`.

TASK DETAIL
  https://phabricator.wikimedia.org/T260281

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ops-monitoring-bot
Cc: Michael, NullPointer, Platonides, hashar, Addshore, Majavah, Ladsgroup, 
JMeybohm, ema, Joe, RhinosF1, ArielGlenn, jijiki, Aklapper, CDanis, lmata, 
wkandek, Akuckartz, darthmon_wmde, WDoranWMF, holger.knust, EvanProdromou, 
Legado_Shulgin, Nandana, Klaas_Z4us_V, Davinaclare77, Qtn1293, Techguru.pc, 
Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, LawExplorer, Zppix, 
elukey, _jensen, rosalieper, Agabi10, Scott_WUaS, Pchelolo, Wong128hk, 
Wikidata-bugs, aude, faidon, Mbch331, Rxy, Jay8g, fgiunchedi, Dzahn
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T260281: mw* servers memory leaks (12 Aug)

2020-08-14 Thread RhinosF1 
RhinosF1 added a comment.


  In T260281#6385334 , 
@NullPointer wrote:
  
  > I suggest setting this a security issue since this may cause people to 
//intentionally// make memory leaks to damage servers using this software.
  
  If it's related to Score/Lillypond as speculated, then no third party sight 
should be running it anyway due to the security issues found,

TASK DETAIL
  https://phabricator.wikimedia.org/T260281

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RhinosF1
Cc: NullPointer, Platonides, hashar, Addshore, Majavah, Ladsgroup, JMeybohm, 
ema, Joe, RhinosF1, ArielGlenn, jijiki, Aklapper, CDanis, lmata, wkandek, 
Akuckartz, darthmon_wmde, WDoranWMF, holger.knust, EvanProdromou, 
Legado_Shulgin, Nandana, Klaas_Z4us_V, Davinaclare77, Qtn1293, Techguru.pc, 
Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, LawExplorer, Zppix, 
elukey, _jensen, rosalieper, Agabi10, Scott_WUaS, Pchelolo, Wong128hk, 
Wikidata-bugs, aude, faidon, Mbch331, Rxy, Jay8g, fgiunchedi, Dzahn
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T260281: mw* servers memory leaks (12 Aug)

2020-08-14 Thread NullPointer 
NullPointer added a comment.


  I suggest setting this a security issue since this may cause people to 
//intentionally// make memory leaks to damage servers using this software.

TASK DETAIL
  https://phabricator.wikimedia.org/T260281

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: NullPointer
Cc: NullPointer, Platonides, hashar, Addshore, Majavah, Ladsgroup, JMeybohm, 
ema, Joe, RhinosF1, ArielGlenn, jijiki, Aklapper, CDanis, lmata, wkandek, 
Akuckartz, darthmon_wmde, WDoranWMF, holger.knust, EvanProdromou, 
Legado_Shulgin, Nandana, Klaas_Z4us_V, Davinaclare77, Qtn1293, Techguru.pc, 
Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, LawExplorer, Zppix, 
elukey, _jensen, rosalieper, Agabi10, Scott_WUaS, Pchelolo, Wong128hk, 
Wikidata-bugs, aude, faidon, Mbch331, Rxy, Jay8g, fgiunchedi, Dzahn
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T260281: mw* servers memory leaks (12 Aug)

2020-08-13 Thread Stashbot 
Stashbot added a comment.


  Mentioned in SAL (#wikimedia-operations) [2020-08-13T14:45:55Z]  repool 
mw1382 with kernel memory accounting disabled T260281 


TASK DETAIL
  https://phabricator.wikimedia.org/T260281

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Stashbot
Cc: Addshore, Majavah, Ladsgroup, JMeybohm, ema, Joe, RhinosF1, ArielGlenn, 
jijiki, Aklapper, CDanis, lmata, wkandek, Akuckartz, darthmon_wmde, WDoranWMF, 
holger.knust, EvanProdromou, Legado_Shulgin, Nandana, Klaas_Z4us_V, 
Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, 
Hfbn0, QZanden, LawExplorer, Zppix, elukey, _jensen, rosalieper, Agabi10, 
Scott_WUaS, Pchelolo, Wong128hk, Wikidata-bugs, aude, faidon, Mbch331, Rxy, 
Jay8g, fgiunchedi, Dzahn
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T260281: mw* servers memory leaks (12 Aug)

2020-08-13 Thread Stashbot 
Stashbot added a comment.


  Mentioned in SAL (#wikimedia-operations) [2020-08-13T14:38:52Z]  reboot 
mw1382 with kernel memory accounting disabled T260281 


TASK DETAIL
  https://phabricator.wikimedia.org/T260281

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Stashbot
Cc: Addshore, Majavah, Ladsgroup, JMeybohm, ema, Joe, RhinosF1, ArielGlenn, 
jijiki, Aklapper, CDanis, lmata, wkandek, Akuckartz, darthmon_wmde, WDoranWMF, 
holger.knust, EvanProdromou, Legado_Shulgin, Nandana, Klaas_Z4us_V, 
Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, 
Hfbn0, QZanden, LawExplorer, Zppix, elukey, _jensen, rosalieper, Agabi10, 
Scott_WUaS, Pchelolo, Wong128hk, Wikidata-bugs, aude, faidon, Mbch331, Rxy, 
Jay8g, fgiunchedi, Dzahn
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T260281: mw* servers memory leaks (12 Aug)

2020-08-13 Thread ema 
ema added a comment.


  `node_vmstat_nr_slab_unreclaimable` is going up indefinitely on nodes 
affected by the issue, following a pattern that matches the general memory 
usage. However, the actual amount of "lost" memory does not match the size of 
unreclaimable slabs, which is only ~2G on mw1357:
  
root@mw1357:~# grep SUnreclaim /proc/meminfo 
SUnreclaim:  2097548 kB
  
  F32102914: Screenshot from 2020-08-13 15-04-23.png 

  
  We discussed two things to try out next:
  
  - reboot 4.9 with memory accounting disabled (`cgroup.memory=nokmem`) to see 
if we ran into 
https://lwn.net/ml/linux-kernel/20190611231813.3148843-1-g...@fb.com/, in 
particular considering that mem_cgroup_try_charge is featured often in the 
`memcg_schedule_kmem_cache_create` traces that @cdanis suggested to gather (see 
P12251  and P12252 
)
  - reboot with 4.19 from stretch-backports because it's a low hanging fruit 
and 4.9 is very old

TASK DETAIL
  https://phabricator.wikimedia.org/T260281

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ema
Cc: Addshore, Majavah, Ladsgroup, JMeybohm, ema, Joe, RhinosF1, ArielGlenn, 
jijiki, Aklapper, CDanis, lmata, wkandek, Akuckartz, darthmon_wmde, WDoranWMF, 
holger.knust, EvanProdromou, Legado_Shulgin, Nandana, Klaas_Z4us_V, 
Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, 
Hfbn0, QZanden, LawExplorer, Zppix, elukey, _jensen, rosalieper, Agabi10, 
Scott_WUaS, Pchelolo, Wong128hk, Wikidata-bugs, aude, faidon, Mbch331, Rxy, 
Jay8g, fgiunchedi, Dzahn
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T260281: mw* servers memory leaks (12 Aug)

2020-08-13 Thread ema 
ema added a comment.


  In T260281#6382529 , @ema 
wrote:
  
  > I've installed systemtap on mw1357
  
  Nevermind, I've seen only now that mw1357 is depooled. Here's some 
preliminary results from mw1359:  P12251 
. Note that the calls happen in 
bursts but rarely (12:16:30, 12:16:45, 12:17:31). Also note that the traces 
include calls to `perf_trace_run_bpf_submit`, it might be wise to run the two 
different instrumentation tools on different hosts for clarity.

TASK DETAIL
  https://phabricator.wikimedia.org/T260281

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ema
Cc: Majavah, Ladsgroup, JMeybohm, ema, Joe, RhinosF1, ArielGlenn, jijiki, 
Aklapper, CDanis, lmata, wkandek, Akuckartz, darthmon_wmde, WDoranWMF, 
holger.knust, EvanProdromou, Legado_Shulgin, Nandana, Klaas_Z4us_V, 
Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, 
Hfbn0, QZanden, LawExplorer, Zppix, elukey, _jensen, rosalieper, Agabi10, 
Scott_WUaS, Pchelolo, Wong128hk, Wikidata-bugs, aude, faidon, Mbch331, Rxy, 
Jay8g, fgiunchedi, Dzahn
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T260281: mw* servers memory leaks (12 Aug)

2020-08-13 Thread ema 
ema added a comment.


  In T260281#6381768 , 
@CDanis wrote:
  
  > attach a tracepoint to `memcg_schedule_kmem_cache_create` and gather 
calling stacktraces.  That's the function that creates the work item that 
results in a worker thread calling `memcg_create_kmem_cache`, as seen in the 
stack traces we saw for 32-byte mallocs.
  
  I've installed systemtap on mw1357 and started instrumenting 
`memcg_create_kmem_cache`. The function does not seem to be called at present? 
Same goes for `memcg_create_kmem_cache`.
  
root@mw1357:~# stap -ve 'probe 
kernel.function("memcg_schedule_kmem_cache_create") { print_backtrace() ; 
exit() }'
Pass 1: parsed user script and 477 library scripts using 
153836virt/86440res/6576shr/80088data kb, in 210usr/20sys/241real ms.
Pass 2: analyzed script: 1 probe, 2 functions, 0 embeds, 0 globals using 
195668virt/129392res/7460shr/121920data kb, in 560usr/70sys/620real ms.
Pass 3: using cached 
/root/.systemtap/cache/8a/stap_8aaac3604a689cbbd6ae1226de2f15db_1317.c
Pass 4: using cached 
/root/.systemtap/cache/8a/stap_8aaac3604a689cbbd6ae1226de2f15db_1317.ko
Pass 5: starting run.
# ema's note: output stops here
  
  To make sure that nothing basic is wrong with the systemtap setup I looked 
for `icmp_reply`, that does print a backtrace.

TASK DETAIL
  https://phabricator.wikimedia.org/T260281

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ema
Cc: Majavah, Ladsgroup, JMeybohm, ema, Joe, RhinosF1, ArielGlenn, jijiki, 
Aklapper, CDanis, lmata, wkandek, Akuckartz, darthmon_wmde, WDoranWMF, 
holger.knust, EvanProdromou, Legado_Shulgin, Nandana, Klaas_Z4us_V, 
Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, 
Hfbn0, QZanden, LawExplorer, Zppix, elukey, _jensen, rosalieper, Agabi10, 
Scott_WUaS, Pchelolo, Wong128hk, Wikidata-bugs, aude, faidon, Mbch331, Rxy, 
Jay8g, fgiunchedi, Dzahn
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T260281: mw* servers memory leaks (12 Aug)

2020-08-13 Thread ema 
ema updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T260281

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: ema
Cc: Ladsgroup, JMeybohm, ema, Joe, RhinosF1, ArielGlenn, jijiki, Aklapper, 
CDanis, lmata, wkandek, Akuckartz, darthmon_wmde, WDoranWMF, holger.knust, 
EvanProdromou, Legado_Shulgin, Nandana, Klaas_Z4us_V, Davinaclare77, Qtn1293, 
Techguru.pc, Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, 
LawExplorer, Zppix, elukey, _jensen, rosalieper, Agabi10, Scott_WUaS, Pchelolo, 
Wong128hk, Wikidata-bugs, aude, faidon, Mbch331, Rxy, Jay8g, fgiunchedi, Dzahn
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T260281: mw* servers memory leaks (12 Aug)

2020-08-13 Thread Stashbot 
Stashbot added a comment.


  Mentioned in SAL (#wikimedia-operations) [2020-08-13T08:43:31Z] <_joe_> 
downgrading imagemagick on mw1378 T260281 


TASK DETAIL
  https://phabricator.wikimedia.org/T260281

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Stashbot
Cc: JMeybohm, ema, Joe, RhinosF1, ArielGlenn, jijiki, Aklapper, CDanis, lmata, 
wkandek, Akuckartz, darthmon_wmde, WDoranWMF, holger.knust, EvanProdromou, 
Legado_Shulgin, Nandana, Klaas_Z4us_V, Davinaclare77, Qtn1293, Techguru.pc, 
Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, LawExplorer, Zppix, 
elukey, _jensen, rosalieper, Agabi10, Scott_WUaS, Pchelolo, Wong128hk, 
Wikidata-bugs, aude, faidon, Mbch331, Rxy, Jay8g, fgiunchedi, Dzahn
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T260281: mw* servers memory leaks (12 Aug)

2020-08-12 Thread Joe 
Joe triaged this task as "Unbreak Now!" priority.
Joe added projects: Platform Engineering, Wikidata.
Joe added a comment.


  I'm not 100% sure that slabs are the problem here, but I'll try to followup 
later.
  
  In the meantime, the servers we've rebooted yesterday are definitely showing 
the same behaviour again. This means that in a week we'll have to reboot them 
all again.
  
  I am therefore:
  1 - raising this task to UBN! priority, as this has the potential to disrupt 
the work of all SREs constantly
  2 - focusing on looking at what was changed on August 4th in the morning, 
mostly in code, and try to rule out stuff. This might result in some reverts.
  3 - Trying to build better automation around the task of rebooting a large 
fleet of servers.
  
  Also adding tags for platform engineering and wikidata as the two potentially 
worrisome code changes regard Score's enablement and a change to Wikibase.

TASK DETAIL
  https://phabricator.wikimedia.org/T260281

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Joe
Cc: Joe, RhinosF1, ArielGlenn, jijiki, Aklapper, CDanis, lmata, wkandek, 
JMeybohm, Akuckartz, darthmon_wmde, WDoranWMF, holger.knust, EvanProdromou, 
Legado_Shulgin, Nandana, Klaas_Z4us_V, Davinaclare77, Qtn1293, Techguru.pc, 
Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, LawExplorer, Zppix, 
elukey, _jensen, rosalieper, Agabi10, Scott_WUaS, Pchelolo, Wong128hk, 
Wikidata-bugs, aude, faidon, Mbch331, Rxy, Jay8g, fgiunchedi, Dzahn
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs