[Bug 40514] "Couldn't resolve host 'ms-fe.pmtpa.wmnet'" CloudFiles errors
https://bugzilla.wikimedia.org/show_bug.cgi?id=40514 Aaron Schulz changed: What|Removed |Added Status|NEW |RESOLVED Resolution|--- |WORKSFORME --- Comment #12 from Aaron Schulz --- Not seeing these anymore -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 40514] "Couldn't resolve host 'ms-fe.pmtpa.wmnet'" CloudFiles errors
https://bugzilla.wikimedia.org/show_bug.cgi?id=40514 --- Comment #11 from Antoine "hashar" Musso --- This is still occurring from time to time :( $ zgrep -c 'resolve host' swift-backend.log-201312* swift-backend.log-20131201.gz:0 swift-backend.log-20131202.gz:0 swift-backend.log-20131203.gz:0 swift-backend.log-20131204.gz:0 swift-backend.log-20131205.gz:0 swift-backend.log-20131206.gz:0 swift-backend.log-20131207.gz:0 swift-backend.log-20131208.gz:0 swift-backend.log-20131209.gz:0 swift-backend.log-20131210.gz:51 swift-backend.log-20131211.gz:115 swift-backend.log-20131212.gz:35 swift-backend.log-20131213.gz:0 swift-backend.log-20131214.gz:0 swift-backend.log-20131215.gz:0 $ -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 40514] "Couldn't resolve host 'ms-fe.pmtpa.wmnet'" CloudFiles errors
https://bugzilla.wikimedia.org/show_bug.cgi?id=40514 --- Comment #10 from Aaron Schulz --- Tried that a long time ago, didn't work. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 40514] "Couldn't resolve host 'ms-fe.pmtpa.wmnet'" CloudFiles errors
https://bugzilla.wikimedia.org/show_bug.cgi?id=40514 --- Comment #9 from Andre Klapper --- (In reply to comment #6 by Faidon) > I've asked Aaron to isolate the code in question and produce some kind of > script that we can run repetively, reproduce and run under strace/gdb, rather > than trying to attach them on random jobrunners and hope we catch it. Faidon / Aaron: Has this happened yet? -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 40514] "Couldn't resolve host 'ms-fe.pmtpa.wmnet'" CloudFiles errors
https://bugzilla.wikimedia.org/show_bug.cgi?id=40514 --- Comment #8 from Andre Klapper --- (In reply to comment #6 by Faidon) > I've asked Aaron to isolate the code in question and produce some kind of > script that we can run repetively, reproduce and run under strace/gdb, rather > than trying to attach them on random jobrunners and hope we catch it. Faidon / Aaron: Has this happened yet? -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 40514] "Couldn't resolve host 'ms-fe.pmtpa.wmnet'" CloudFiles errors
https://bugzilla.wikimedia.org/show_bug.cgi?id=40514 --- Comment #7 from Andre Klapper --- (In reply to comment #6 by Faidon) > I've asked Aaron to isolate the code in question and produce some kind of > script that we can run repetively, reproduce and run under strace/gdb, rather > than trying to attach them on random jobrunners and hope we catch it. Faidon / Aaron: Has this happened yet? -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 40514] "Couldn't resolve host 'ms-fe.pmtpa.wmnet'" CloudFiles errors
https://bugzilla.wikimedia.org/show_bug.cgi?id=40514 --- Comment #6 from Faidon Liambotis --- Sorry for not updating the ticket earlier -- I've actually attempted to debug this and have chatted with Aaron last week or the one before that. I've verified that at the time errors were spawned, DNS replies were coming into the system. Also, it's peculiar how no other infrastructure seems to be affected, not even the application server ones (this apparently affects only job runners). It's also something that has manifested recently, possibly after the precise upgrade. I have some suspections that it may be curl-related (curl has an internal DNS cache that is enabled by default, so it's not just simple libc resolver calls). I've asked Aaron to isolate the code in question and produce some kind of script that we can run repetively, reproduce and run under strace/gdb, rather than trying to attach them on random jobrunners and hope we catch it. The issue happens on jobrunners, so it's under php cli anyway, so the environment won't be that different anyway. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 40514] "Couldn't resolve host 'ms-fe.pmtpa.wmnet'" CloudFiles errors
https://bugzilla.wikimedia.org/show_bug.cgi?id=40514 Andre Klapper changed: What|Removed |Added Keywords||ops Priority|Unprioritized |High Component|General/Unknown |DNS -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 40514] "Couldn't resolve host 'ms-fe.pmtpa.wmnet'" CloudFiles errors
https://bugzilla.wikimedia.org/show_bug.cgi?id=40514 Aaron Schulz changed: What|Removed |Added CC||fai...@wikimedia.org AssignedTo|aschulz4...@gmail.com |wikibugs-l@lists.wikimedia. ||org --- Comment #5 from Aaron Schulz 2012-12-03 19:43:53 UTC --- (In reply to comment #4) > Switched auth URL to an IP to avoid dns lookups for auth requests. I'll see if > this works around the dns problems or pushes them down the road. Can is down the road :) This really needs an ops person to look at. -- Configure bugmail: https://bugzilla.wikimedia.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 40514] "Couldn't resolve host 'ms-fe.pmtpa.wmnet'" CloudFiles errors
https://bugzilla.wikimedia.org/show_bug.cgi?id=40514 Aaron Schulz changed: What|Removed |Added AssignedTo|wikibugs-l@lists.wikimedia. |aschulz4...@gmail.com |org | --- Comment #4 from Aaron Schulz 2012-11-29 18:15:51 UTC --- Switched auth URL to an IP to avoid dns lookups for auth requests. I'll see if this works around the dns problems or pushes them down the road. -- Configure bugmail: https://bugzilla.wikimedia.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 40514] "Couldn't resolve host 'ms-fe.pmtpa.wmnet'" CloudFiles errors
https://bugzilla.wikimedia.org/show_bug.cgi?id=40514 Aaron Schulz changed: What|Removed |Added Summary|"Couldn't resolve host |"Couldn't resolve host |'ms-fe.pmtpa.wmnet'" and|'ms-fe.pmtpa.wmnet'" |"unexpected response" |CloudFiles errors |CloudFiles errors | --- Comment #3 from Aaron Schulz 2012-11-28 22:45:18 UTC --- I improved the errors for auth requests and they are now "Couldn't resolve host 'ms-fe.pmtpa.wmnet'", so these are all the same error. -- Configure bugmail: https://bugzilla.wikimedia.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
