[Bug 59645] Lines with “nan” for “Request service time” column and empty HTTP status code in cache logs
https://bugzilla.wikimedia.org/show_bug.cgi?id=59645 Andrew Otto o...@wikimedia.org changed: What|Removed |Added Status|NEW |RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 59645] Lines with “nan” for “Request service time” column and empty HTTP status code in cache logs
https://bugzilla.wikimedia.org/show_bug.cgi?id=59645 Toby Negrin tneg...@wikimedia.org changed: What|Removed |Added Priority|Unprioritized |Normal -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 59645] Lines with “nan” for “Request service time” column and empty HTTP status code in cache logs
https://bugzilla.wikimedia.org/show_bug.cgi?id=59645 --- Comment #6 from Bingle bingle-ad...@wikimedia.org --- Prioritization and scheduling of this bug is tracked on Mingle card https://wikimedia.mingle.thoughtworks.com/projects/analytics/cards/cards/1391 -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 59645] Lines with “nan” for “Request service time” column and empty HTTP status code in cache logs
https://bugzilla.wikimedia.org/show_bug.cgi?id=59645 Toby Negrin tneg...@wikimedia.org changed: What|Removed |Added CC||ezac...@wikimedia.org, ||o...@wikimedia.org, ||tneg...@wikimedia.org --- Comment #2 from Toby Negrin tneg...@wikimedia.org --- Hi Christian, Erik -- what's the impact of this? Do we need to spend some time working with ops to track it down? thanks, -Toby -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 59645] Lines with “nan” for “Request service time” column and empty HTTP status code in cache logs
https://bugzilla.wikimedia.org/show_bug.cgi?id=59645 --- Comment #3 from Toby Negrin tneg...@wikimedia.org --- Per Andrew/Stefan -- this ends up causing problems with Hive. Magnus has created a patch; we hope to get this deployed this week. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 59645] Lines with “nan” for “Request service time” column and empty HTTP status code in cache logs
https://bugzilla.wikimedia.org/show_bug.cgi?id=59645 --- Comment #4 from christ...@quelltextlich.at --- (In reply to comment #2) what's the impact of this? It's not super critical. But since it seems to be more than just the nans, it may be more than the initial few lines that are affected. Do we need to spend some time working with ops to track it down? Ops already know and had a first look :-) It seems only the nans started on 2013-12-10. Some bogus requests were there before, but it seems they have been treated differently. Not sure yet. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 59645] Lines with “nan” for “Request service time” column and empty HTTP status code in cache logs
https://bugzilla.wikimedia.org/show_bug.cgi?id=59645 --- Comment #5 from christ...@quelltextlich.at --- (In reply to comment #3) Per Andrew/Stefan -- this ends up causing problems with Hive. Magnus has created a patch; we hope to get this deployed this week. Yes, but that'll only address the issue on the varnishkafka side. varnishncsa (hence udp2log, webstatscollector, tsvs) do not benefit from this fix. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 59645] Lines with “nan” for “Request service time” column and empty HTTP status code in cache logs
https://bugzilla.wikimedia.org/show_bug.cgi?id=59645 --- Comment #1 from christ...@quelltextlich.at --- The host name of the affected lines range over all hosts in upload-eqiad, upload-esams, and mobile-eqiad, but no other cache. The request method of the affected lines look mangled. HEAD+GET account for ~55% of the affected lines. The other 45% of the affected lines, the logged request method looks like being part of the http header [1]. The URL column of the affected lines is for about 8% the mobile version of jawiki and ptwiki. None of those 8% show a User Agent, Referrer, or Accept-Language. For 80% of the affected lines, the URL column is upload.wikimedia.org. For the remaining ~10% of affected lines, the URL column did not make too much sense. They are for example “http://-en-US,en;q=0.8”. So it's also header information that spilled over into this column. Requested URLs are not distributed all over the place, but rather isolated. So for example the requested URL for ~33% of the affected lines was: http://upload.wikimedia.orgHTTP/1.1 For ~12% it was http://upload.wikimedia.org/wikipedia/en/c/ca/2M_TV_(logo).gif . And those requests are not clustered around a certain date / time, but span the whole date / time range between 2013-12-10 and now. The referrers are “-” for ~60% of affected lines, and for ~32% some Url from mobile enwiki. The logged User-Agents, Accept-Languages look sane. [1] Here are some examples: Accept-Language: x-Mercedes-Benz_Three_Point_Star.jpg exagonal_Icon.svg.png .svg.png Referer: HTTP/1.1 To see more of them, run cut -f 8 /home/qchris/nan.tsv | sort | uniq -c | sort -n on stat1002. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
