[Wikidata-bugs] [Maniphest] [Commented On] T133866: Varnish seems to sometimes mangle uncompressed API results
Trung.anh.dinh added a comment. Thank you for the fix @BBlack :) TASK DETAIL https://phabricator.wikimedia.org/T133866 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: BBlack, Trung.anh.dinh Cc: gerritbot, mahmoud, Slaporte, Zppix, Ricordisamoa, Trung.anh.dinh, MZMcBride, Anomie, Yurivict, TerraCodes, Orlodrim, BBlack, akosiaris, zhuyifei1999, elukey, ema, Aklapper, hoo, Lewizho99, Maathavan, D3r1ck01, Izno, Wikidata-bugs, aude, Mbch331, Jay8g, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T133866: Varnish seems to sometimes mangle uncompressed API results
MZMcBride added a comment. This looks fixed to me. Thank you, @BBlack! TASK DETAIL https://phabricator.wikimedia.org/T133866 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: BBlack, MZMcBride Cc: gerritbot, mahmoud, Slaporte, Zppix, Ricordisamoa, Trung.anh.dinh, MZMcBride, Anomie, Yurivict, TerraCodes, Orlodrim, BBlack, akosiaris, zhuyifei1999, elukey, ema, Aklapper, hoo, Lewizho99, Maathavan, D3r1ck01, Izno, Wikidata-bugs, aude, Mbch331, Jay8g, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T133866: Varnish seems to sometimes mangle uncompressed API results
gerritbot added a comment. Change 287633 merged by BBlack: cache_misc: remove all do_stream=true https://gerrit.wikimedia.org/r/287633 TASK DETAIL https://phabricator.wikimedia.org/T133866 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: BBlack, gerritbot Cc: gerritbot, mahmoud, Slaporte, Zppix, Ricordisamoa, Trung.anh.dinh, MZMcBride, Anomie, Yurivict, TerraCodes, Orlodrim, BBlack, akosiaris, zhuyifei1999, elukey, ema, Aklapper, hoo, Lewizho99, Maathavan, D3r1ck01, Izno, Wikidata-bugs, aude, Mbch331, Jay8g, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T133866: Varnish seems to sometimes mangle uncompressed API results
gerritbot added a comment. Change 287633 had a related patch set uploaded (by BBlack): cache_misc: remove all do_stream=true https://gerrit.wikimedia.org/r/287633 TASK DETAIL https://phabricator.wikimedia.org/T133866 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: BBlack, gerritbot Cc: gerritbot, mahmoud, Slaporte, Zppix, Ricordisamoa, Trung.anh.dinh, MZMcBride, Anomie, Yurivict, TerraCodes, Orlodrim, BBlack, akosiaris, zhuyifei1999, elukey, ema, Aklapper, hoo, Lewizho99, Maathavan, D3r1ck01, Izno, Wikidata-bugs, aude, Mbch331, Jay8g, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T133866: Varnish seems to sometimes mangle uncompressed API results
gerritbot added a comment. Change 287626 merged by BBlack: Revert "Varnish: stream all pass traffic" https://gerrit.wikimedia.org/r/287626 TASK DETAIL https://phabricator.wikimedia.org/T133866 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: BBlack, gerritbot Cc: gerritbot, mahmoud, Slaporte, Zppix, Ricordisamoa, Trung.anh.dinh, MZMcBride, Anomie, Yurivict, TerraCodes, Orlodrim, BBlack, akosiaris, zhuyifei1999, elukey, ema, Aklapper, hoo, Lewizho99, Maathavan, D3r1ck01, Izno, Wikidata-bugs, aude, Mbch331, Jay8g, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T133866: Varnish seems to sometimes mangle uncompressed API results
gerritbot added a comment. Change 287624 merged by BBlack: Revert "text VCL: do_stream when creating hit-for-pass" https://gerrit.wikimedia.org/r/287624 TASK DETAIL https://phabricator.wikimedia.org/T133866 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: BBlack, gerritbot Cc: gerritbot, mahmoud, Slaporte, Zppix, Ricordisamoa, Trung.anh.dinh, MZMcBride, Anomie, Yurivict, TerraCodes, Orlodrim, BBlack, akosiaris, zhuyifei1999, elukey, ema, Aklapper, hoo, Lewizho99, Maathavan, D3r1ck01, Izno, Wikidata-bugs, aude, Mbch331, Jay8g, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T133866: Varnish seems to sometimes mangle uncompressed API results
gerritbot added a comment. Change 287623 merged by BBlack: Remove X-Pass-Stream support from cache_misc size stream VCL https://gerrit.wikimedia.org/r/287623 TASK DETAIL https://phabricator.wikimedia.org/T133866 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: BBlack, gerritbot Cc: gerritbot, mahmoud, Slaporte, Zppix, Ricordisamoa, Trung.anh.dinh, MZMcBride, Anomie, Yurivict, TerraCodes, Orlodrim, BBlack, akosiaris, zhuyifei1999, elukey, ema, Aklapper, hoo, Lewizho99, Maathavan, D3r1ck01, Izno, Wikidata-bugs, aude, Mbch331, Jay8g, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T133866: Varnish seems to sometimes mangle uncompressed API results
gerritbot added a comment. Change 287625 merged by BBlack: Revert "Varnish: protect against external streampass header setting" https://gerrit.wikimedia.org/r/287625 TASK DETAIL https://phabricator.wikimedia.org/T133866 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: BBlack, gerritbot Cc: gerritbot, mahmoud, Slaporte, Zppix, Ricordisamoa, Trung.anh.dinh, MZMcBride, Anomie, Yurivict, TerraCodes, Orlodrim, BBlack, akosiaris, zhuyifei1999, elukey, ema, Aklapper, hoo, Lewizho99, Maathavan, D3r1ck01, Izno, Wikidata-bugs, aude, Mbch331, Jay8g, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T133866: Varnish seems to sometimes mangle uncompressed API results
gerritbot added a comment. Change 287624 had a related patch set uploaded (by BBlack): Revert "text VCL: do_stream when creating hit-for-pass" https://gerrit.wikimedia.org/r/287624 TASK DETAIL https://phabricator.wikimedia.org/T133866 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: BBlack, gerritbot Cc: gerritbot, mahmoud, Slaporte, Zppix, Ricordisamoa, Trung.anh.dinh, MZMcBride, Anomie, Yurivict, TerraCodes, Orlodrim, BBlack, akosiaris, zhuyifei1999, elukey, ema, Aklapper, hoo, Lewizho99, Maathavan, D3r1ck01, Izno, Wikidata-bugs, aude, Mbch331, Jay8g, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T133866: Varnish seems to sometimes mangle uncompressed API results
gerritbot added a comment. Change 287626 had a related patch set uploaded (by BBlack): Revert "Varnish: stream all pass traffic" https://gerrit.wikimedia.org/r/287626 TASK DETAIL https://phabricator.wikimedia.org/T133866 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: BBlack, gerritbot Cc: gerritbot, mahmoud, Slaporte, Zppix, Ricordisamoa, Trung.anh.dinh, MZMcBride, Anomie, Yurivict, TerraCodes, Orlodrim, BBlack, akosiaris, zhuyifei1999, elukey, ema, Aklapper, hoo, Lewizho99, Maathavan, D3r1ck01, Izno, Wikidata-bugs, aude, Mbch331, Jay8g, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T133866: Varnish seems to sometimes mangle uncompressed API results
gerritbot added a comment. Change 287623 had a related patch set uploaded (by BBlack): Remove X-Pass-Stream support from cache_misc size stream VCL https://gerrit.wikimedia.org/r/287623 TASK DETAIL https://phabricator.wikimedia.org/T133866 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: BBlack, gerritbot Cc: gerritbot, mahmoud, Slaporte, Zppix, Ricordisamoa, Trung.anh.dinh, MZMcBride, Anomie, Yurivict, TerraCodes, Orlodrim, BBlack, akosiaris, zhuyifei1999, elukey, ema, Aklapper, hoo, D3r1ck01, Izno, Wikidata-bugs, aude, Mbch331, Jay8g, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T133866: Varnish seems to sometimes mangle uncompressed API results
gerritbot added a comment. Change 287625 had a related patch set uploaded (by BBlack): Revert "Varnish: protect against external streampass header setting" https://gerrit.wikimedia.org/r/287625 TASK DETAIL https://phabricator.wikimedia.org/T133866 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: BBlack, gerritbot Cc: gerritbot, mahmoud, Slaporte, Zppix, Ricordisamoa, Trung.anh.dinh, MZMcBride, Anomie, Yurivict, TerraCodes, Orlodrim, BBlack, akosiaris, zhuyifei1999, elukey, ema, Aklapper, hoo, Lewizho99, Maathavan, D3r1ck01, Izno, Wikidata-bugs, aude, Mbch331, Jay8g, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T133866: Varnish seems to sometimes mangle uncompressed API results
Yurivict added a comment. It also seems to be related to the size of the response. 32kB is the critical size. The respone is 32766=0x7FFE bytes. TASK DETAIL https://phabricator.wikimedia.org/T133866 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: BBlack, Yurivict Cc: Trung.anh.dinh, MZMcBride, Anomie, Yurivict, TerraCodes, Orlodrim, BBlack, akosiaris, zhuyifei1999, elukey, ema, Aklapper, hoo, D3r1ck01, Izno, Wikidata-bugs, aude, Mbch331, Jay8g, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T133866: Varnish seems to sometimes mangle uncompressed API results
Yurivict added a comment. It is 100% reproducible. TASK DETAIL https://phabricator.wikimedia.org/T133866 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: BBlack, Yurivict Cc: Trung.anh.dinh, MZMcBride, Anomie, Yurivict, TerraCodes, Orlodrim, BBlack, akosiaris, zhuyifei1999, elukey, ema, Aklapper, hoo, D3r1ck01, Izno, Wikidata-bugs, aude, Mbch331, Jay8g, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T133866: Varnish seems to sometimes mangle uncompressed API results
MZMcBride added a comment. Using the exact URL from the task description, I'm consistently getting a bad response: >>> from subprocess import check_output >>> args = ['curl', '-s', 'https://www.wikidata.org/w/api.php?action=wbgetentities=no=claims=json=Q18407428|Q18407424|Q18407421|Q18407418|Q18407262|Q18407245|Q18407242|Q18407240|Q18407236|Q18407234|Q18407233|Q18407228|Q18407225|Q18407224|Q18407221|Q18407218|Q18407127|Q18407044|Q18407038|Q18407009|Q18407008|Q18407006|Q18407005|Q18407004'] >>> output = check_output(args) >>> output[-30:] 'nk":"normal"}]}}},"succe\'5\xfe+\x0b\x7f' I got called away while testing. Using the same URL later, I'm now getting a different bad response: >>> output[-30:] 'nk":"normal"}]}}},"succe\'e>\x17\'\x7f' I thought both `api.php` request URLs and responses were typically not cached. However, if I append a cache-buster (in this case, `&4`), I get non-corrupt output: >>> from subprocess import check_output >>> args = ['curl', '-s', 'https://www.wikidata.org/w/api.php?action=wbgetentities=no=claims=json=Q18407428|Q18407424|Q18407421|Q18407418|Q18407262|Q18407245|Q18407242|Q18407240|Q18407236|Q18407234|Q18407233|Q18407228|Q18407225|Q18407224|Q18407221|Q18407218|Q18407127|Q18407044|Q18407038|Q18407009|Q18407008|Q18407006|Q18407005|Q18407004&4'] >>> output = check_output(args) >>> output[-30:] 'nk":"normal"}]}}},"success":1}' Weird. TASK DETAIL https://phabricator.wikimedia.org/T133866 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: BBlack, MZMcBride Cc: Trung.anh.dinh, MZMcBride, Anomie, Yurivict, TerraCodes, Orlodrim, BBlack, akosiaris, zhuyifei1999, elukey, ema, Aklapper, hoo, D3r1ck01, Izno, Wikidata-bugs, aude, Mbch331, Jay8g, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T133866: Varnish seems to sometimes mangle uncompressed API results
hoo added a comment. Another report about this (probably) on Wikitech: https://lists.wikimedia.org/pipermail/wikitech-l/2016-May/085526.html TASK DETAIL https://phabricator.wikimedia.org/T133866 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: BBlack, hoo Cc: elukey, ema, Aklapper, hoo, D3r1ck01, Izno, Wikidata-bugs, aude, Mbch331, Jay8g, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T133866: Varnish seems to sometimes mangle uncompressed API results
BBlack added a comment. Did some further testing on an isolated test machine, using our current varnish3 package. - Got 2833-byte test file from uncorrupted (--compressed) output on prod. This is the exact compressed content bytes emitted by MW/Apache for the broken test URL. - Configured a test backend (nginx) to serve static files, and to always set CE:gzip. - Placed the gzipped 2833 byte file in test directory, fetched over curl w/ --compressed, md5sum comes out right. - When fetched through our varnishd with a default config and do_gzip turned on, varnish does decompress this file for the curl client, and there is no corruption (still same md5sum). This rules out the possibility that this is some pure, data-sensitive varnish bug with gunzipping the content itself. However, the notable diff in this test from reality is that nginx serving a static pre-gzipped file is (a) not emitting it as TE:chunked and (b) even if it did, it probably wouldn't use the same chunk boundaries, nor is it likely to share any TE:chunked encoding bugs or varnish-bug-triggers... TASK DETAIL https://phabricator.wikimedia.org/T133866 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: BBlack Cc: elukey, ema, Aklapper, hoo, D3r1ck01, Izno, Wikidata-bugs, aude, Mbch331, Jay8g, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T133866: Varnish seems to sometimes mangle uncompressed API results
BBlack added a comment. Just jotting down the things I know so far from investigating this morning. I still don't have a good answer yet. Based on just the test URL, debugging it extensively at various layers: 1. The response size of that URL is in the ballpark of 32KB uncompressed, so this is not a large-objects issue. It also streams out of the backend reliably and quickly, there are no significant timing pauses. 2. Without a doubt, anytime the client doesn't use AE:gzip when talking to the public endpoint, the response is corrupt. 3. Slightly deeper, even when testing requests against a single layer of varnish internally, the response to a non-AE:gzip client request is corrupt. 4. It's definitely happening at the Varnish<->Apache/MediaWiki boundary (disagreement) or within a single varnishd process as it prepares the response. It's not a Varnish<->Varnish, Varnish<->Nginx, or Nginx<->client issue. 5. All of the gzip/chunked stuff looks basically correct in the headers at varnish/apache boundary and varnish/client boundary. We do send AE:gzip when expected, we do get CE:gzip when expected (only when asked for), the gzip-ness of MW/Apache's output always correctly follows its CE:gzip header (or lack thereof), etc. 6. Curl has no problem parsing the output of a direct fetch from Apache/MediaWiki, whether using `--compressed` to set AE:gzip or not, and the results hash the same (identical content). 7. Varnish emits the corrupt content for the non-AE:gzip client regardless of whether I tweak the test scenario to ensure that varnish is the one gzipping the content, or that Apache/Mediawiki are the ones gzipping the content. So it's not an error in gzip compression of the response by just one party or the other. The error happens when gunzipping the response for the non-AE:gzip client. 8. However, when I run through a similar set of fully-debugged test scenarios for https://en.wikipedia.org/wiki/Barack_Obama , which is ~1MB in uncompressed length, and similarly TE:chunked with backend gzip capabilities and do_gzip=true, and on the same cluster and VCL (and even same test machine), I don't get the corruption for a non-AE:gzip client, even though varnish is decompressing that on the fly as with the bad test URL. The obvious distinctions here between the Barack article and the failing test URL aren't much: api.svc vs appservers.svc shouldn't matter, right, they're both behind the same apache and hhvm configs. This leaves me guessing that there's something special about the specific output of the test URL that's causing this. There's almost certainly a varnish bug involved here, but the question is: is this a pure varnish gunzip bug that's sensitive to certain conditions which exist for the test URL but not the Barack one? Is the output of Apache/MW buggy in some way for the test URL such that it's tripping the bug (in which case it's still a varnish bug that it doesn't reject the buggy response and turn it into a 503 or similar), or is it non-buggy, but "special" in a way that trips a varnish bug? TASK DETAIL https://phabricator.wikimedia.org/T133866 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: BBlack Cc: elukey, ema, Aklapper, hoo, D3r1ck01, Izno, Wikidata-bugs, aude, Mbch331, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T133866: Varnish seems to sometimes mangle uncompressed API results
hoo added a comment. In https://phabricator.wikimedia.org/T133866#2259636, @BBlack wrote: > Do you know if some normal traffic is affected, such that we'd know a start date for a recent change in behavior? Or is it suspected that it was always this way? I'm not aware of normal traffic being affected, it possibly started around 2016-04-11 as per https://www.wikidata.org/w/index.php?oldid=330081884#Corrupted_response_end. TASK DETAIL https://phabricator.wikimedia.org/T133866 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: BBlack, hoo Cc: ema, Aklapper, hoo, D3r1ck01, Izno, Wikidata-bugs, aude, Mbch331, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T133866: Varnish seems to sometimes mangle uncompressed API results
BBlack added a comment. Do you know if some normal traffic is affected, such that we'd know a start date for a recent change in behavior? Or is it suspected that it was always this way? I've been digging through some debugging on this URL (which is an applayer chunked-response with no-cache headers), and it's definitely happening at the varnish<->MW boundary (as opposed to further up, at varnish<->varnish or nginx<->varnish), and only for non-AE:gzip requests. The length of the result is correct, but there's corruption in the trailing bytes. TASK DETAIL https://phabricator.wikimedia.org/T133866 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: BBlack Cc: ema, Aklapper, hoo, D3r1ck01, Izno, Wikidata-bugs, aude, Mbch331, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
