I'm not sure these are related (I'm seeing zero packet loss to Slack from Chicago on Cogent) but Slack's definitely broken. Seemed to have been a cascading failure, a colleague of mine complained over IRC he got booted from his, then I got booted from one Slack team, then another.
> On Nov 23, 2015, at 1:49 PM, Justin Oeder via Outages <[email protected]> > wrote: > > Slack, which is hosted with Amazon is currently down. > > > > From: "Grant Ridder via Outages" <[email protected]> > To: "Mike Lyon" <[email protected]> > Cc: [email protected], "Tomoc Field" <[email protected]> > Sent: Monday, November 23, 2015 2:43:57 PM > Subject: Re: [outages] packet loss to us-east-1 > > Seeing this issue again > 4. AS46375 50.ae4.gw.pao1.sonic.net <http://50.ae4.gw.pao1.sonic.net/> > 0.0% 10 15.5 23.2 2.9 89.3 28.6 > 5. AS8121 199.127.107.189 0.0% 10 64.2 > 26.6 3.7 85.7 28.2 > 6. AS8121 xe0-0-0-7.core2.sv8.layer42.net > <http://xe0-0-0-7.core2.sv8.layer42.net/> 0.0% 10 66.6 24.8 > 3.5 102.1 33.4 > 7. AS16509 72.21.221.176 0.0% 10 64.2 > 26.9 3.0 99.1 32.4 > 8. AS??? 54.240.243.50 0.0% 10 147.4 > 115.9 88.2 176.9 27.9 > 9. AS??? 54.240.243.61 90.0% 10 78.4 > 78.4 78.4 78.4 0.0 > 10. AS??? 54.239.42.38 10.0% 10 130.4 > 106.9 71.8 167.6 31.8 > 11. AS16509 54.239.108.120 70.0% 10 178.9 > 125.2 90.0 178.9 47.2 > 12. AS16509 54.239.111.235 20.0% 10 143.1 > 99.0 70.8 158.1 34.7 > 13. AS16509 54.239.111.230 70.0% 10 89.5 > 93.9 88.7 103.5 8.3 > > -Grant > > On Fri, Nov 20, 2015 at 5:27 PM, Mike Lyon <[email protected] > <mailto:[email protected]>> wrote: > Hmmm, > It shows as if it is getting there but look at that jump from 15ms to 85ms on > hops 7 and 8: > > traceroute amazon.com <http://amazon.com/> > traceroute: Warning: amazon.com <http://amazon.com/> has multiple addresses; > using 176.32.103.205 > > traceroute to amazon.com <http://amazon.com/> (176.32.103.205), 64 hops max, > 52 byte packets > > 1 192.168.20.1 (192.168.20.1) 0.708 ms 0.395 ms 0.328 ms > > 2 162.250.208.141 (162.250.208.141) 0.671 ms 0.622 ms 0.588 ms > > 3 10.25.0.1 (10.25.0.1) 1.985 ms 4.165 ms 21.626 ms > > 4 10.20.0.1 (10.20.0.1) 7.852 ms 7.626 ms 5.214 ms > > 5 10.0.0.1 (10.0.0.1) 8.041 ms 17.839 ms 34.089 ms > > 6 69.36.226.193 (69.36.226.193) 5.477 ms 14.228 ms 20.203 ms > > 7 xe0-2-0-4.core4.scl.layer42.net <http://xe0-2-0-4.core4.scl.layer42.net/> > (65.50.198.237) 15.153 ms 13.475 ms 8.706 ms > > 8 205.251.229.145 (205.251.229.145) 85.231 ms > > 54.240.242.108 (54.240.242.108) 96.157 ms > > 205.251.229.145 (205.251.229.145) 175.633 ms > > 9 54.240.242.141 (54.240.242.141) 100.007 ms 234.602 ms > > 54.240.242.121 (54.240.242.121) 94.263 ms > > 10 54.239.109.102 (54.239.109.102) 110.966 ms > > 54.239.109.136 (54.239.109.136) 123.322 ms > > 54.239.110.50 (54.239.110.50) 105.029 ms > > 11 54.239.109.143 (54.239.109.143) 84.239 ms > > 54.239.110.23 (54.239.110.23) 91.540 ms > > 54.239.110.57 (54.239.110.57) 86.620 ms > > 12 54.239.109.143 (54.239.109.143) 91.382 ms > > 205.251.245.168 (205.251.245.168) 87.930 ms > > 54.239.109.143 (54.239.109.143) 76.879 ms > > 13 * 205.251.245.168 (205.251.245.168) 90.780 ms 96.386 ms > > > On Fri, Nov 20, 2015 at 5:24 PM, Grant Ridder <[email protected] > <mailto:[email protected]>> wrote: > Looks like they solved it by routing around layer42 and go via cogent. That > in itself is a bad idea :/ > 4. AS46375 50.ae4.gw.pao1.sonic.net <http://50.ae4.gw.pao1.sonic.net/> > 1.2% 81 3.5 12.0 2.6 91.0 16.2 > 5. AS174 te0-0-0-15.ccr21.sjc04.atlas.cogentco.com > <http://te0-0-0-15.ccr21.sjc04.atlas.cogentco.com/> 6.2% 81 6.6 10.5 > 3.2 79.9 11.7 > 6. AS174 be2016.ccr22.sfo01.atlas.cogentco.com > <http://be2016.ccr22.sfo01.atlas.cogentco.com/> 3.7% 81 15.3 14.6 > 4.2 98.9 16.3 > 7. AS174 be2133.ccr22.mci01.atlas.cogentco.com > <http://be2133.ccr22.mci01.atlas.cogentco.com/> 4.9% 81 47.7 50.4 > 41.8 113.6 13.3 > 8. AS174 be2157.ccr42.ord01.atlas.cogentco.com > <http://be2157.ccr42.ord01.atlas.cogentco.com/> 70.0% 81 54.1 67.2 > 53.9 140.2 21.3 > 9. AS174 be2217.ccr41.ord03.atlas.cogentco.com > <http://be2217.ccr41.ord03.atlas.cogentco.com/> 33.3% 81 57.3 63.6 > 54.3 158.4 17.0 > 10. AS174 ntt.ord03.atlas.cogentco.com > <http://ntt.ord03.atlas.cogentco.com/> 66.2% 81 56.9 72.0 > 56.5 126.8 20.6 > 11. AS2914 ae-3.amazon.chcgil09.us.bb.gin.ntt.net > <http://ae-3.amazon.chcgil09.us.bb.gin.ntt.net/> 34.6% 81 61.5 61.8 > 55.9 89.7 8.1 > > -Grant > > On Fri, Nov 20, 2015 at 5:14 PM, Mike Lyon <[email protected] > <mailto:[email protected]>> wrote: > Oh thank god! I was seeing it too but wasn't sure if was Layer42 or Amazon > that was having the issue... > Thanks, > Mike > > > On Fri, Nov 20, 2015 at 5:09 PM, Grant Ridder via Outages > <[email protected] <mailto:[email protected]>> wrote: > Saw that, thanks! > -Grant > > On Fri, Nov 20, 2015 at 4:49 PM, Tomoc Field <[email protected] > <mailto:[email protected]>> wrote: > Yes. 70%+ packet loss from 7065 to all Amazon services as far as I can tell. > This was experienced via both layer42 and XO, but Amazon appears to have > resolved their issues at this time. > > -Tomoc > > > On Fri, Nov 20, 2015 at 04:34:42PM -0800, Grant Ridder via Outages wrote: > > Anyone else seeing packet log into us-east-1? > > > > 54.240.243.121 (AS16509) is Amazon > > > > MTR report excerpt > > 4. AS46375 50.ae4.gw.pao1.sonic.net <http://50.ae4.gw.pao1.sonic.net/> > > 0.0% 10 2.5 > > 8.3 2.5 37.5 10.7 <tel:8.3%20%20%202.5%20%2037.5%20%2010.7> > > 5. AS8121 199.127.107.189 0.0% 10 3.8 > > 7.9 3.4 34.9 9.5 > > 6. AS8121 xe0-0-0-7.core2.sv8.layer42.net > > <http://xe0-0-0-7.core2.sv8.layer42.net/> 0.0% 10 3.9 > > 6.0 3.4 17.6 4.6 > > 7. AS16509 72.21.221.176 0.0% 10 4.6 > > 5.2 3.0 15.5 3.6 > > 8. AS??? 54.240.243.110 0.0% 10 98.6 > > 96.4 86.9 104.4 6.2 > > 9. AS??? 54.240.243.121 90.0% 10 85.7 > > 85.7 85.7 85.7 0.0 > > 10. AS16509 205.251.229.104 20.0% 10 87.5 > > 85.2 82.5 87.8 2.0 > > 11. AS??? 54.239.42.29 70.0% 10 85.0 > > 84.3 82.8 85.2 1.0 > > 12. AS16509 54.239.109.238 30.0% 10 87.4 > > 96.8 87.4 105.2 6.5 > > AS16509 54.239.110.16 > > > > > > -Grant > > > _______________________________________________ > > Outages mailing list > > [email protected] <mailto:[email protected]> > > https://puck.nether.net/mailman/listinfo/outages > > <https://puck.nether.net/mailman/listinfo/outages> > > > > _______________________________________________ > Outages mailing list > [email protected] <mailto:[email protected]> > https://puck.nether.net/mailman/listinfo/outages > <https://puck.nether.net/mailman/listinfo/outages> > > > > > -- > Mike Lyon > 408-621-4826 <tel:408-621-4826> > [email protected] <mailto:[email protected]> > http://www.linkedin.com/in/mlyon <http://www.linkedin.com/in/mlyon> > > > > > > > > -- > Mike Lyon > 408-621-4826 <tel:408-621-4826> > [email protected] <mailto:[email protected]> > http://www.linkedin.com/in/mlyon <http://www.linkedin.com/in/mlyon> > > > > > > _______________________________________________ > Outages mailing list > [email protected] > https://puck.nether.net/mailman/listinfo/outages > _______________________________________________ > Outages mailing list > [email protected] > https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list [email protected] https://puck.nether.net/mailman/listinfo/outages
