It is related for some since Slack is in us-east-1 On Mon, Nov 23, 2015 at 11:52 AM, Chris Swingler <[email protected]> wrote:
> I'm not sure these are related (I'm seeing zero packet loss to Slack from > Chicago on Cogent) but Slack's definitely broken. Seemed to have been a > cascading failure, a colleague of mine complained over IRC he got booted > from his, then I got booted from one Slack team, then another. > > On Nov 23, 2015, at 1:49 PM, Justin Oeder via Outages <[email protected]> > wrote: > > Slack, which is hosted with Amazon is currently down. > > > > ------------------------------ > *From: *"Grant Ridder via Outages" <[email protected]> > *To: *"Mike Lyon" <[email protected]> > *Cc: *[email protected], "Tomoc Field" <[email protected]> > *Sent: *Monday, November 23, 2015 2:43:57 PM > *Subject: *Re: [outages] packet loss to us-east-1 > > Seeing this issue again > 4. AS46375 50.ae4.gw.pao1.sonic.net 0.0% 10 15.5 > 23.2 2.9 89.3 28.6 > 5. AS8121 199.127.107.189 0.0% 10 64.2 > 26.6 3.7 85.7 28.2 > 6. AS8121 xe0-0-0-7.core2.sv8.layer42.net 0.0% 10 66.6 > 24.8 3.5 102.1 33.4 > 7. AS16509 72.21.221.176 0.0% 10 64.2 > 26.9 3.0 99.1 32.4 > 8. AS??? 54.240.243.50 0.0% 10 147.4 > 115.9 88.2 176.9 27.9 > 9. AS??? 54.240.243.61 90.0% 10 78.4 > 78.4 78.4 78.4 0.0 > 10. AS??? 54.239.42.38 10.0% 10 130.4 > 106.9 71.8 167.6 31.8 > 11. AS16509 54.239.108.120 70.0% 10 178.9 > 125.2 90.0 178.9 47.2 > 12. AS16509 54.239.111.235 20.0% 10 143.1 > 99.0 70.8 158.1 34.7 > 13. AS16509 54.239.111.230 70.0% 10 89.5 > 93.9 88.7 103.5 8.3 > > -Grant > > On Fri, Nov 20, 2015 at 5:27 PM, Mike Lyon <[email protected]> wrote: > >> Hmmm, >> It shows as if it is getting there but look at that jump from 15ms to >> 85ms on hops 7 and 8: >> >> traceroute amazon.com >> >> traceroute: Warning: amazon.com has multiple addresses; using >> 176.32.103.205 >> >> traceroute to amazon.com (176.32.103.205), 64 hops max, 52 byte packets >> >> 1 192.168.20.1 (192.168.20.1) 0.708 ms 0.395 ms 0.328 ms >> >> 2 162.250.208.141 (162.250.208.141) 0.671 ms 0.622 ms 0.588 ms >> >> 3 10.25.0.1 (10.25.0.1) 1.985 ms 4.165 ms 21.626 ms >> >> 4 10.20.0.1 (10.20.0.1) 7.852 ms 7.626 ms 5.214 ms >> >> 5 10.0.0.1 (10.0.0.1) 8.041 ms 17.839 ms 34.089 ms >> >> 6 69.36.226.193 (69.36.226.193) 5.477 ms 14.228 ms 20.203 ms >> >> 7 xe0-2-0-4.core4.scl.layer42.net (65.50.198.237) 15.153 ms 13.475 >> ms 8.706 ms >> >> 8 205.251.229.145 (205.251.229.145) 85.231 ms >> >> 54.240.242.108 (54.240.242.108) 96.157 ms >> >> 205.251.229.145 (205.251.229.145) 175.633 ms >> >> 9 54.240.242.141 (54.240.242.141) 100.007 ms 234.602 ms >> >> 54.240.242.121 (54.240.242.121) 94.263 ms >> >> 10 54.239.109.102 (54.239.109.102) 110.966 ms >> >> 54.239.109.136 (54.239.109.136) 123.322 ms >> >> 54.239.110.50 (54.239.110.50) 105.029 ms >> >> 11 54.239.109.143 (54.239.109.143) 84.239 ms >> >> 54.239.110.23 (54.239.110.23) 91.540 ms >> >> 54.239.110.57 (54.239.110.57) 86.620 ms >> >> 12 54.239.109.143 (54.239.109.143) 91.382 ms >> >> 205.251.245.168 (205.251.245.168) 87.930 ms >> >> 54.239.109.143 (54.239.109.143) 76.879 ms >> >> 13 * 205.251.245.168 (205.251.245.168) 90.780 ms 96.386 ms >> >> On Fri, Nov 20, 2015 at 5:24 PM, Grant Ridder <[email protected]> >> wrote: >> >>> Looks like they solved it by routing around layer42 and go via cogent. >>> That in itself is a bad idea :/ >>> 4. AS46375 50.ae4.gw.pao1.sonic.net 1.2% 81 >>> 3.5 12.0 2.6 91.0 16.2 >>> 5. AS174 te0-0-0-15.ccr21.sjc04.atlas.cogentco.com 6.2% 81 >>> 6.6 10.5 3.2 79.9 11.7 >>> 6. AS174 be2016.ccr22.sfo01.atlas.cogentco.com 3.7% 81 >>> 15.3 14.6 4.2 98.9 16.3 >>> 7. AS174 be2133.ccr22.mci01.atlas.cogentco.com 4.9% 81 >>> 47.7 50.4 41.8 113.6 13.3 >>> 8. AS174 be2157.ccr42.ord01.atlas.cogentco.com 70.0% 81 >>> 54.1 67.2 53.9 140.2 21.3 >>> 9. AS174 be2217.ccr41.ord03.atlas.cogentco.com 33.3% 81 >>> 57.3 63.6 54.3 158.4 17.0 >>> 10. AS174 ntt.ord03.atlas.cogentco.com 66.2% 81 >>> 56.9 72.0 56.5 126.8 20.6 >>> 11. AS2914 ae-3.amazon.chcgil09.us.bb.gin.ntt.net 34.6% 81 >>> 61.5 61.8 55.9 89.7 8.1 >>> >>> -Grant >>> >>> On Fri, Nov 20, 2015 at 5:14 PM, Mike Lyon <[email protected]> wrote: >>> >>>> Oh thank god! I was seeing it too but wasn't sure if was Layer42 or >>>> Amazon that was having the issue... >>>> Thanks, >>>> Mike >>>> >>>> >>>> On Fri, Nov 20, 2015 at 5:09 PM, Grant Ridder via Outages < >>>> [email protected]> wrote: >>>> >>>>> Saw that, thanks! >>>>> -Grant >>>>> >>>>> On Fri, Nov 20, 2015 at 4:49 PM, Tomoc Field <[email protected]> >>>>> wrote: >>>>> >>>>>> Yes. 70%+ packet loss from 7065 to all Amazon services as far as I >>>>>> can tell. This was experienced via both layer42 and XO, but Amazon >>>>>> appears >>>>>> to have resolved their issues at this time. >>>>>> >>>>>> -Tomoc >>>>>> >>>>>> >>>>>> On Fri, Nov 20, 2015 at 04:34:42PM -0800, Grant Ridder via Outages >>>>>> wrote: >>>>>> > Anyone else seeing packet log into us-east-1? >>>>>> > >>>>>> > 54.240.243.121 (AS16509) is Amazon >>>>>> > >>>>>> > MTR report excerpt >>>>>> > 4. AS46375 50.ae4.gw.pao1.sonic.net 0.0% 10 >>>>>> 2.5 >>>>>> > 8.3 2.5 37.5 10.7 >>>>>> > 5. AS8121 199.127.107.189 0.0% 10 >>>>>> 3.8 >>>>>> > 7.9 3.4 34.9 9.5 >>>>>> > 6. AS8121 xe0-0-0-7.core2.sv8.layer42.net 0.0% 10 >>>>>> 3.9 >>>>>> > 6.0 3.4 17.6 4.6 >>>>>> > 7. AS16509 72.21.221.176 0.0% 10 >>>>>> 4.6 >>>>>> > 5.2 3.0 15.5 3.6 >>>>>> > 8. AS??? 54.240.243.110 0.0% 10 >>>>>> 98.6 >>>>>> > 96.4 86.9 104.4 6.2 >>>>>> > 9. AS??? 54.240.243.121 90.0% 10 >>>>>> 85.7 >>>>>> > 85.7 85.7 85.7 0.0 >>>>>> > 10. AS16509 205.251.229.104 20.0% 10 >>>>>> 87.5 >>>>>> > 85.2 82.5 87.8 2.0 >>>>>> > 11. AS??? 54.239.42.29 70.0% 10 >>>>>> 85.0 >>>>>> > 84.3 82.8 85.2 1.0 >>>>>> > 12. AS16509 54.239.109.238 30.0% 10 >>>>>> 87.4 >>>>>> > 96.8 87.4 105.2 6.5 >>>>>> > AS16509 54.239.110.16 >>>>>> > >>>>>> > >>>>>> > -Grant >>>>>> >>>>>> > _______________________________________________ >>>>>> > Outages mailing list >>>>>> > [email protected] >>>>>> > https://puck.nether.net/mailman/listinfo/outages >>>>>> >>>>>> >>>>> >>>>> _______________________________________________ >>>>> Outages mailing list >>>>> [email protected] >>>>> https://puck.nether.net/mailman/listinfo/outages >>>>> >>>>> >>>> >>>> >>>> -- >>>> Mike Lyon >>>> 408-621-4826 >>>> [email protected] >>>> >>>> http://www.linkedin.com/in/mlyon >>>> >>>> >>>> >>>> >>> >> >> >> -- >> Mike Lyon >> 408-621-4826 >> [email protected] >> >> http://www.linkedin.com/in/mlyon >> >> >> >> > > _______________________________________________ > Outages mailing list > [email protected] > https://puck.nether.net/mailman/listinfo/outages > _______________________________________________ > Outages mailing list > [email protected] > https://puck.nether.net/mailman/listinfo/outages > > >
_______________________________________________ Outages mailing list [email protected] https://puck.nether.net/mailman/listinfo/outages
