BBlack added a comment.
Current State: - cp3007 and cp1045 are depooled from user traffic, icinga-downtimed for several days, and have puppet disabled. Please do not re-enable puppet on these! They also have confd shut down, and are running custom configs to continue debugging this issue under varnish4. - The rest of cache_misc is reverted to varnish3, which should temporarily resolve this issue for user traffic over the weekend and into next week while we continued isolated investigation using the two nodes above. - Please do **not** resolve this ticket - this is a bandaid, and we still have a lot of digging to do. - Please **do** report any similar user-facing failures from here forward, as there shouldn't be any while the cluster is reverted to varnish3. TASK DETAIL https://phabricator.wikimedia.org/T134989 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: BBlack Cc: Ronarts12, Krenair, Dzahn, GWicke, Smalyshev, Heather, Nirzar, ZMcCune, ema, Stashbot, Luke081515, matmarex, TerraCodes, Urbanecm, KDDLB, hashar, Jonas, gerritbot, BBlack, Aklapper, Zppix, Lydia_Pintscher, Gehel, Avner, Lewizho99, Maathavan, debt, D3r1ck01, FloNight, Izno, jkroll, Wikidata-bugs, Jdouglas, aude, Deskana, Manybubbles, Mbch331, Jay8g _______________________________________________ Wikidata-bugs mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
