gerritbot added a comment.
Change 301374 merged by BBlack:
Revert "Revert "Revert "cache_misc: do not deliver expired cached objects"""
https://gerrit.wikimedia.org/r/301374TASK DETAILhttps://phabricator.wikimedia.org/T134989EMAIL
gerritbot added a comment.
Change 301374 had a related patch set uploaded (by BBlack):
Revert "Revert "Revert "cache_misc: do not deliver expired cached objects"""
https://gerrit.wikimedia.org/r/301374TASK DETAILhttps://phabricator.wikimedia.org/T134989EMAIL
ema added a comment.
I've upgraded cache_misc to Varnish 4 again. We're now running a patched
version of Varnish (4.1.2-1wm5) including the following patches:
- 0005-handle-eof-http1.1.patch
Stashbot added a comment.
Mentioned in SAL [2016-05-17T09:02:23Z] upgrading esams cache_misc to
varnish 4 (https://phabricator.wikimedia.org/T126206,
https://phabricator.wikimedia.org/T134989)
TASK DETAIL
https://phabricator.wikimedia.org/T134989
EMAIL PREFERENCES
Stashbot added a comment.
Mentioned in SAL [2016-05-17T08:31:56Z] upgrading codfw cache_misc to
varnish 4 (https://phabricator.wikimedia.org/T126206,
https://phabricator.wikimedia.org/T134989)
TASK DETAIL
https://phabricator.wikimedia.org/T134989
EMAIL PREFERENCES
gerritbot added a comment.
Change 289148 merged by Ema:
Revert "cache_misc: downgrade almost all to varnish3"
https://gerrit.wikimedia.org/r/289148
TASK DETAIL
https://phabricator.wikimedia.org/T134989
EMAIL PREFERENCES
gerritbot added a comment.
Change 289148 had a related patch set uploaded (by Ema):
Revert "cache_misc: downgrade almost all to varnish3"
https://gerrit.wikimedia.org/r/289148
TASK DETAIL
https://phabricator.wikimedia.org/T134989
EMAIL PREFERENCES
BBlack added a comment.
cache_maps cluster switched to the new varnish package today
TASK DETAIL
https://phabricator.wikimedia.org/T134989
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: BBlack
Cc: jeremyb, Ronarts12, Krenair, Dzahn, GWicke,
gerritbot added a comment.
Change 288817 merged by Ema:
Revert 'Allow grace-hits on hit-for-pass objects'
https://gerrit.wikimedia.org/r/288817
TASK DETAIL
https://phabricator.wikimedia.org/T134989
EMAIL PREFERENCES
gerritbot added a comment.
Change 288817 had a related patch set uploaded (by Ema):
Revert 'Allow grace-hits on hit-for-pass objects'
https://gerrit.wikimedia.org/r/288817
TASK DETAIL
https://phabricator.wikimedia.org/T134989
EMAIL PREFERENCES
ema added a comment.
I haven't been able to reproduce the bug so far after reverting the commit
mentioned above. cp1045 and cp3007 are both running fine with an updated
version of varnish.
Please use cp3007:80 if you want to play around, it will only hit itself and
cp1045 to ensure
ema added a comment.
Update: setting default_grace to 0 makes the issue unreproducible. 4.1.0 is
not affected, while 4.1.1 is, so I started looking for grace-related commits
between the two versions.
Looks like the bug was introduced by this one:
ema added a comment.
The issue can be reproduced running two varnishd on the same host, without
any VCL. A web server listening on port 80 is also needed, serving any type of
file. I've used a 612 bytes HTML page.
/usr/sbin/varnishd -a :3128 -b localhost:80 -p default_ttl=5 -p
BBlack added a comment.
Current State:
- cp3007 and cp1045 are depooled from user traffic, icinga-downtimed for
several days, and have puppet disabled. Please do not re-enable puppet on
these! They also have confd shut down, and are running custom configs to
continue debugging this
BBlack added a comment.
I forgot one of our temporary hacks in the list above in
https://phabricator.wikimedia.org/T134989#2290254:
4. https://gerrit.wikimedia.org/r/#/c/288656/ - we also enabled a critical
small bit here in v4 vcl_hit. I reverted this for now during the varnish3
gerritbot added a comment.
Change 288655 had a related patch set uploaded (by BBlack):
cache_misc: downgrade almost all to varnish3
https://gerrit.wikimedia.org/r/288655
TASK DETAIL
https://phabricator.wikimedia.org/T134989
EMAIL PREFERENCES
gerritbot added a comment.
Change 288655 merged by BBlack:
cache_misc: downgrade almost all to varnish3
https://gerrit.wikimedia.org/r/288655
TASK DETAIL
https://phabricator.wikimedia.org/T134989
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
Stashbot added a comment.
Mentioned in SAL [2016-05-13T13:13:56Z] depooling cp3007 to try
reproducing https://phabricator.wikimedia.org/T134989
TASK DETAIL
https://phabricator.wikimedia.org/T134989
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
ema added a comment.
In https://phabricator.wikimedia.org/T134989#2292155, @Jonas wrote:
> Issue appearing again
[...]
> < Age: 58
> < X-Cache: cp1045 hit+miss(0), cp3009 hit+miss(0), cp3007 frontend hit(4)
On @Gehel's repro from this morning
Jonas added a comment.
Issue appearing again
curl -v 'https://query.wikidata.org/i18n/en.json'
* Hostname was NOT found in DNS cache
* Trying 91.198.174.217...
* Connected to query.wikidata.org (91.198.174.217) port 443 (#0)
* successfully set certificate verify
Stashbot added a comment.
Mentioned in SAL [2016-05-13T10:27:01Z] depooling cp1061 trying to
reproduce https://phabricator.wikimedia.org/T134989
TASK DETAIL
https://phabricator.wikimedia.org/T134989
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
ema added a comment.
Issue reproduced: https://phabricator.wikimedia.org/P3067
TASK DETAIL
https://phabricator.wikimedia.org/T134989
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: ema
Cc: Smalyshev, Heather, Nirzar, ZMcCune, ema, Stashbot,
ema added a comment.
In https://phabricator.wikimedia.org/T134989#2290254, @BBlack wrote:
> 1. We've got 2x upstream bugfixes applied to our varnishd on cache_misc:
https://github.com/varnishcache/varnish-cache/commit/d828a042b3fc2c2b4f1fea83021f0d5508649e50
+
Smalyshev added a comment.
I don't see any weirdness anymore on query.wikidata.org/ @Jonas ?
TASK DETAIL
https://phabricator.wikimedia.org/T134989
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Smalyshev
Cc: Smalyshev, Heather, Nirzar, ZMcCune,
BBlack added a comment.
Has anyone been able to reproduce any of the problems in the tickets merged
into here, since roughly the timestamp of the above message?
TASK DETAIL
https://phabricator.wikimedia.org/T134989
EMAIL PREFERENCES
BBlack added a comment.
So we're currently have several experiments in play trying to figure this out:
1. We've got 2x upstream bugfixes applied to our varnishd on cache_misc:
https://github.com/varnishcache/varnish-cache/commit/d828a042b3fc2c2b4f1fea83021f0d5508649e50
+
Stashbot added a comment.
Mentioned in SAL [2016-05-12T15:52:43Z] testing file storage on misc
eqiad+esams https://phabricator.wikimedia.org/T134989
TASK DETAIL
https://phabricator.wikimedia.org/T134989
EMAIL PREFERENCES
Kghbln added a comment.
In https://phabricator.wikimedia.org/T134989#2289372, @BBlack wrote:
> due to missing character encoding supposedly, but it's entirely likely it's
missing everything and that's just the first thing it notices.
I believe that this observation is correct.
BBlack added a comment.
In the merged ticket above, it's browser access to status.wm.o, and the
browser's getting a 304 Not Modified and complaining about it (due to missing
character encoding supposedly, but it's entirely likely it's missing everything
and that's just the first thing it
Stashbot added a comment.
Mentioned in SAL [2016-05-12T13:34:23Z] upgrading misc to varnish
4.1.2-1wm4 and wiping caches (https://phabricator.wikimedia.org/T134989)
TASK DETAIL
https://phabricator.wikimedia.org/T134989
EMAIL PREFERENCES
gerritbot added a comment.
Change 288387 merged by Ema:
Add debian/patches/0006-IMS-duplicate-headers-proper-handling.patch
https://gerrit.wikimedia.org/r/288387
TASK DETAIL
https://phabricator.wikimedia.org/T134989
EMAIL PREFERENCES
gerritbot added a comment.
Change 288387 had a related patch set uploaded (by Ema):
Add debian/patches/0006-IMS-duplicate-headers-proper-handling.patch
https://gerrit.wikimedia.org/r/288387
TASK DETAIL
https://phabricator.wikimedia.org/T134989
EMAIL PREFERENCES
Jonas added a comment.
Now I get content-length 0 for query.wikidata.org
jonkr@C134:~$ curl -v 'https://query.wikidata.org/'
* Hostname was NOT found in DNS cache
* Trying 91.198.174.217...
* Connected to query.wikidata.org (91.198.174.217) port 443 (#0)
* successfully
Stashbot added a comment.
Mentioned in SAL [2016-05-12T08:33:24Z] wiping misc caches once again
(https://phabricator.wikimedia.org/T134989)
TASK DETAIL
https://phabricator.wikimedia.org/T134989
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To:
Stashbot added a comment.
Mentioned in SAL [2016-05-12T07:51:40Z] upgrading misc ulsfo to varnish
4.1.2-1wm3 and wiping caches (https://phabricator.wikimedia.org/T134989)
TASK DETAIL
https://phabricator.wikimedia.org/T134989
EMAIL PREFERENCES
Stashbot added a comment.
Mentioned in SAL [2016-05-12T07:47:03Z] upgrading misc codfw to
4.1.2-1wm3 and wiping caches (https://phabricator.wikimedia.org/T134989)
TASK DETAIL
https://phabricator.wikimedia.org/T134989
EMAIL PREFERENCES
Stashbot added a comment.
Mentioned in SAL [2016-05-12T07:05:38Z] running varnish 4.1.2-1wm3 in
misc esams (https://phabricator.wikimedia.org/T134989)
TASK DETAIL
https://phabricator.wikimedia.org/T134989
EMAIL PREFERENCES
gerritbot added a comment.
Change 288350 merged by Ema:
Add debian/patches/0005-handle-eof-http1.1.patch
https://gerrit.wikimedia.org/r/288350
TASK DETAIL
https://phabricator.wikimedia.org/T134989
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
gerritbot added a comment.
Change 288350 had a related patch set uploaded (by Ema):
Add debian/patches/0005-handle-eof-http1.1.patch
https://gerrit.wikimedia.org/r/288350
TASK DETAIL
https://phabricator.wikimedia.org/T134989
EMAIL PREFERENCES
BBlack added a comment.
Status update: we've been debugging this off and on all day. It's some kind
of bug fallout from cache_misc's upgrade to Varnish 4. It's a very complicated
bug, and we don't really understand it yet. We've made some band-aid fixes to
VCL for now which should keep
gerritbot added a comment.
Change 288213 merged by BBlack:
tlsproxy: use http/1.1 only in conjunction with varnish4
https://gerrit.wikimedia.org/r/288213
TASK DETAIL
https://phabricator.wikimedia.org/T134989
EMAIL PREFERENCES
gerritbot added a comment.
Change 288213 had a related patch set uploaded (by BBlack):
tlsproxy: use http/1.1 only in conjunction with varnish4
https://gerrit.wikimedia.org/r/288213
TASK DETAIL
https://phabricator.wikimedia.org/T134989
EMAIL PREFERENCES
Gehel added a comment.
Seems the issue (or a similar one is still happening), I'm trying to
reproduce...
TASK DETAIL
https://phabricator.wikimedia.org/T134989
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Gehel
Cc: BBlack, Aklapper, Zppix,
Gehel added a comment.
I can't reproduce the issue. Since it was random, I might just be lucky.
Closing this anyway for the moment, I'll reopen if I see the issue again.
TASK DETAIL
https://phabricator.wikimedia.org/T134989
EMAIL PREFERENCES
BBlack added a comment.
Assuming there was no transient issue (which became cached) on the wdqs end
of things, then this was likely a transient thing from nginx experiments or the
cache_misc varnish4 upgrade. I banned all wdqs objects from cache_misc and now
your test URL works fine. Can
45 matches
Mail list logo