Re: Facebook more specific via Level3 ?
On Sun, Apr 16, 2017 at 04:20:20PM +0300, Max Tulyev wrote: > got the same from Kiev, Ukraine: > > dig fbcdn.com > fbcdn.com.300 IN A 31.13.74.1 > which is slow and routed through USA > > and > dig fbcdn.com @8.8.8.8 > fbcdn.com.299 IN A 31.13.93.3 > which is fast and routed through Germany > > Same is for IPv6. > > Is there any solutions without dirty hacks? As mentioned in this thread, talk to the Facebook NOC: On Tue, Mar 21, 2017 at 05:41:23PM +, Doug Porter wrote: > > Please reach out to n...@fb.com in the future." > Kind regards, Job
Re: Facebook more specific via Level3 ?
Hi, got the same from Kiev, Ukraine: dig fbcdn.com fbcdn.com. 300 IN A 31.13.74.1 which is slow and routed through USA and dig fbcdn.com @8.8.8.8 fbcdn.com. 299 IN A 31.13.93.3 which is fast and routed through Germany Same is for IPv6. Is there any solutions without dirty hacks? On 22.03.17 12:02, Radu-Adrian Feurdean wrote: > Hi, the load-balancing definitely doesn't choose the *nearest* mirror. > We are in France and unless we do dirty tricks, we *always* get directed > to US sites (as far as LA), with horrible performance. Everything since > end of December. As a consequence we let the dirty tricks in place > (query facebook.com and fbcdn.com on 8.8.8.8 instead of regular > recursive resolving) and we get directed to Frankfurt or Amsterdam > (never London or Paris). >
Re: Facebook more specific via Level3 ?
On Wed, Mar 22, 2017, at 12:25, Mike Hammett wrote: > Are your DNS resolvers on your network? No DNS forwarding? Yes, DNS resolvers on our network. Forwarding only for facebook.com and fbcdn.com, in order to eliminate bad performance associated with "direct recursion".
Re: Facebook more specific via Level3 ?
Hi Mike, Im running some DNS on my own for a few hundred users from an private community project. But this issue is also affecting DNS from smaller/other ISPs which do NOT use any forwarder but the root DNS. Best regards Jürgen
Re: Facebook more specific via Level3 ?
Are your DNS resolvers on your network? No DNS forwarding? - Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP - Original Message - From: "Radu-Adrian Feurdean" To: "Jürgen Jaritsch" , "Doug Porter" , nanog@nanog.org Sent: Wednesday, March 22, 2017 5:02:12 AM Subject: Re: Facebook more specific via Level3 ? On Tue, Mar 21, 2017, at 20:38, Jürgen Jaritsch wrote: > I understand that FB is using some type of DNS geo-loadbalancing and other > mechanism to redirect users to (possibly) nearer mirrors. The used DNS is > directly requesting the root DNS and not any other public DNS (e.g. not > 8.8.8.8). Running some requests within 3 minutes gives me the below > results: > > www.facebook.com => star-mini.c10r.facebook.com. => 31.13.77.36 > www.facebook.com => star-mini.c10r.facebook.com. => 157.240.2.35 > www.facebook.com => star-mini.c10r.facebook.com. => 31.13.93.36 > www.facebook.com => star-mini.c10r.facebook.com. => 31.13.76.68 Hi, the load-balancing definitely doesn't choose the *nearest* mirror. We are in France and unless we do dirty tricks, we *always* get directed to US sites (as far as LA), with horrible performance. Everything since end of December. As a consequence we let the dirty tricks in place (query facebook.com and fbcdn.com on 8.8.8.8 instead of regular recursive resolving) and we get directed to Frankfurt or Amsterdam (never London or Paris).
AW: Facebook more specific via Level3 ?
Hi, > (query facebook.com and fbcdn.com on 8.8.8.8 instead of regular recursive > resolving) and we get directed to Frankfurt or Amsterdam (never London or > Paris). This is exactly what I've implemented yesterday on my end :). Best regards Jürgen -Ursprüngliche Nachricht- Von: Radu-Adrian Feurdean [mailto:na...@radu-adrian.feurdean.net] Gesendet: Mittwoch, 22. März 2017 11:02 An: Jürgen Jaritsch ; Doug Porter ; nanog@nanog.org Betreff: Re: Facebook more specific via Level3 ? On Tue, Mar 21, 2017, at 20:38, Jürgen Jaritsch wrote: > I understand that FB is using some type of DNS geo-loadbalancing and > other mechanism to redirect users to (possibly) nearer mirrors. The > used DNS is directly requesting the root DNS and not any other public > DNS (e.g. not 8.8.8.8). Running some requests within 3 minutes gives > me the below > results: > > www.facebook.com => star-mini.c10r.facebook.com. => 31.13.77.36 > www.facebook.com => star-mini.c10r.facebook.com. => 157.240.2.35 > www.facebook.com => star-mini.c10r.facebook.com. => 31.13.93.36 > www.facebook.com => star-mini.c10r.facebook.com. => 31.13.76.68 Hi, the load-balancing definitely doesn't choose the *nearest* mirror. We are in France and unless we do dirty tricks, we *always* get directed to US sites (as far as LA), with horrible performance. Everything since end of December. As a consequence we let the dirty tricks in place (query facebook.com and fbcdn.com on 8.8.8.8 instead of regular recursive resolving) and we get directed to Frankfurt or Amsterdam (never London or Paris).
Re: Facebook more specific via Level3 ?
On Tue, Mar 21, 2017, at 20:38, Jürgen Jaritsch wrote: > I understand that FB is using some type of DNS geo-loadbalancing and other > mechanism to redirect users to (possibly) nearer mirrors. The used DNS is > directly requesting the root DNS and not any other public DNS (e.g. not > 8.8.8.8). Running some requests within 3 minutes gives me the below > results: > > www.facebook.com => star-mini.c10r.facebook.com. => 31.13.77.36 > www.facebook.com => star-mini.c10r.facebook.com. => 157.240.2.35 > www.facebook.com => star-mini.c10r.facebook.com. => 31.13.93.36 > www.facebook.com => star-mini.c10r.facebook.com. => 31.13.76.68 Hi, the load-balancing definitely doesn't choose the *nearest* mirror. We are in France and unless we do dirty tricks, we *always* get directed to US sites (as far as LA), with horrible performance. Everything since end of December. As a consequence we let the dirty tricks in place (query facebook.com and fbcdn.com on 8.8.8.8 instead of regular recursive resolving) and we get directed to Frankfurt or Amsterdam (never London or Paris).
Re: Facebook more specific via Level3 ?
> looks like this is also affecting other prefixes: Many of our prefixes are only announced to peers in the metro they originate in. Please stop obsessing about this detail; it's not the problem. > I understand that FB is using some type of DNS > geo-loadbalancing and other mechanism to redirect users to > (possibly) nearer mirrors. We target traffic two ways. One is relatively traditional dns global load balancing, using the resolver ip. The other method---which steers the vast majority of our traffic---vends urls that send people to a specific PoP based on their client ip. It appears you're having a targeting problem. Please reach out to our NOC to get support. -- dsp
AW: Facebook more specific via Level3 ?
Hi Luke, please see https://mailman.nanog.org/pipermail/nanog/2017-March/090658.html ... I did some tests a few min ago and yes, I'm receiving the 31.13.77.x and 31.13.76.x via DNS for www.facebook.com. Best regards Jürgen -Ursprüngliche Nachricht- Von: Luke Guillory [mailto:lguill...@reservetele.com] Gesendet: Dienstag, 21. März 2017 20:38 An: Jürgen Jaritsch ; nanog@nanog.org Betreff: RE: Facebook more specific via Level3 ? Are they replying with that subnet via dns when requests are being made? Luke Guillory Network Operations Manager Tel:985.536.1212 Fax:985.536.0300 Email: lguill...@reservetele.com Reserve Telecommunications 100 RTC Dr Reserve, LA 70084 _ Disclaimer: The information transmitted, including attachments, is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or privileged material which should not disseminate, distribute or be copied. Please notify Luke Guillory immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. Luke Guillory therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. . -Original Message- From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Jürgen Jaritsch Sent: Tuesday, March 21, 2017 2:24 PM To: nanog@nanog.org Subject: Re: Facebook more specific via Level3 ? Hi, > This specific, and many others, are only announced to peers in the > metro they originate in. To receive this prefix directly you'll need > to peer with us in Los Angeles. the point is: Level3 is exporting this prefix to the EU since ~1 week … Telia is learning it from Level3 and they also started to re-export it: Telia Looking Glass (http://lg.telia.net/?query=bgp&protocol=IPv4&addr=31.13.71.0/24+exact&route r=Vienna) Command: show route protocol bgp 31.13.71.36 table inet.0 31.13.71.0/24 *[BGP/170] 18w5d 16:40:16, MED 0, localpref 150 AS path: 3356 32934 I, validation-state: unverified > to 80.239.128.178 via ae9.0 This is causing >120ms latency to Austrian (and some German ...) networks towards Facebook - current traceroute from source network 188.172.239.0/24: Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. router 0.0% 40.5 0.5 0.5 0.6 0.0 2. 192.168.8.3 0.0% 40.6 0.7 0.6 0.8 0.1 3. 37.252.236.880.0% 4 37.3 34.2 30.6 37.3 2.9 4. er-03.00-09-23.anx04.vie.at.anexia-it.com0.0% 4 21.1 31.9 21.1 50.9 13.3 5. cr-02.0v-08-72.anx03.vie.at.anexia-it.com0.0% 4 23.2 25.9 20.4 31.6 5.1 6. win-b4-link.telia.net0.0% 4 22.1 25.0 22.1 30.1 3.5 7. level-ic-1573273-wien-b4.c.telia.net 0.0% 3 22.4 22.3 21.1 23.3 1.1 8. ae-3-3611.edge2.dallas1.level3.net 0.0% 3 150.5 150.6 150.4 150.8 0.2 9. facebook-in.edge2.dallas1.level3.net 0.0% 3 156.6 152.9 151.0 156.6 3.2 10. po102.psw02.dft4.tfbnw.net 0.0% 3 151.4 151.6 151.4 152.0 0.4 11. 173.252.67.570.0% 3 151.6 152.3 151.6 153.4 0.9 12. edge-star-mini-shv-01-dft4.facebook.com 0.0% 3 159.3 158.4 152.5 163.6 5.6 The admins of AS42473 started to drop the more specific from Level3 and now they get it from Telia (which is the Level3 re-export they learned). I guess you guys should talk to Level3 and ask them what the hell they are doing? :). Thanks & best regards Jürgen
RE: Facebook more specific via Level3 ?
Are they replying with that subnet via dns when requests are being made? Luke Guillory Network Operations Manager Tel:985.536.1212 Fax:985.536.0300 Email: lguill...@reservetele.com Reserve Telecommunications 100 RTC Dr Reserve, LA 70084 _ Disclaimer: The information transmitted, including attachments, is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or privileged material which should not disseminate, distribute or be copied. Please notify Luke Guillory immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. Luke Guillory therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. . -Original Message- From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Jürgen Jaritsch Sent: Tuesday, March 21, 2017 2:24 PM To: nanog@nanog.org Subject: Re: Facebook more specific via Level3 ? Hi, > This specific, and many others, are only announced to peers in the > metro they originate in. To receive this prefix directly you'll need > to peer with us in Los Angeles. the point is: Level3 is exporting this prefix to the EU since ~1 week … Telia is learning it from Level3 and they also started to re-export it: Telia Looking Glass (http://lg.telia.net/?query=bgp&protocol=IPv4&addr=31.13.71.0/24+exact&route r=Vienna) Command: show route protocol bgp 31.13.71.36 table inet.0 31.13.71.0/24 *[BGP/170] 18w5d 16:40:16, MED 0, localpref 150 AS path: 3356 32934 I, validation-state: unverified > to 80.239.128.178 via ae9.0 This is causing >120ms latency to Austrian (and some German ...) networks towards Facebook - current traceroute from source network 188.172.239.0/24: Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. router 0.0% 40.5 0.5 0.5 0.6 0.0 2. 192.168.8.3 0.0% 40.6 0.7 0.6 0.8 0.1 3. 37.252.236.880.0% 4 37.3 34.2 30.6 37.3 2.9 4. er-03.00-09-23.anx04.vie.at.anexia-it.com0.0% 4 21.1 31.9 21.1 50.9 13.3 5. cr-02.0v-08-72.anx03.vie.at.anexia-it.com0.0% 4 23.2 25.9 20.4 31.6 5.1 6. win-b4-link.telia.net0.0% 4 22.1 25.0 22.1 30.1 3.5 7. level-ic-1573273-wien-b4.c.telia.net 0.0% 3 22.4 22.3 21.1 23.3 1.1 8. ae-3-3611.edge2.dallas1.level3.net 0.0% 3 150.5 150.6 150.4 150.8 0.2 9. facebook-in.edge2.dallas1.level3.net 0.0% 3 156.6 152.9 151.0 156.6 3.2 10. po102.psw02.dft4.tfbnw.net 0.0% 3 151.4 151.6 151.4 152.0 0.4 11. 173.252.67.570.0% 3 151.6 152.3 151.6 153.4 0.9 12. edge-star-mini-shv-01-dft4.facebook.com 0.0% 3 159.3 158.4 152.5 163.6 5.6 The admins of AS42473 started to drop the more specific from Level3 and now they get it from Telia (which is the Level3 re-export they learned). I guess you guys should talk to Level3 and ask them what the hell they are doing? :). Thanks & best regards Jürgen
AW: Facebook more specific via Level3 ?
Hi Doug, looks like this is also affecting other prefixes: 157.240.3.0/24 *[BGP/170] 18w5d 16:50:37, MED 0, localpref 150 AS path: 3356 32934 I, validation-state: unverified > to 80.239.128.178 via ae9.0 I understand that FB is using some type of DNS geo-loadbalancing and other mechanism to redirect users to (possibly) nearer mirrors. The used DNS is directly requesting the root DNS and not any other public DNS (e.g. not 8.8.8.8). Running some requests within 3 minutes gives me the below results: www.facebook.com => star-mini.c10r.facebook.com. => 31.13.77.36 www.facebook.com => star-mini.c10r.facebook.com. => 157.240.2.35 www.facebook.com => star-mini.c10r.facebook.com. => 31.13.93.36 www.facebook.com => star-mini.c10r.facebook.com. => 31.13.76.68 Trace to 31.13.77.36: Packets Pings Host Loss% Snt Last Avg Best Wrst StDev [...] 6. win-b4-link.telia.net0.0% 3 21.5 27.5 21.5 35.7 7.3 7. level-ic-1573273-wien-b4.c.telia.net 0.0% 3 21.7 20.8 20.0 21.7 0.8 8. ae-1-9.edge2.sanjose3.level3.net 0.0% 3 177.1 179.8 177.1 182.6 2.8 9. 4.53.210.78 0.0% 3 178.3 178.8 178.3 179.2 0.5 10. po131.asw04.sjc1.tfbnw.net 0.0% 3 184.6 180.0 177.6 184.6 4.0 11. po241.psw02.sjc2.tfbnw.net 0.0% 3 177.8 177.6 177.3 177.8 0.3 12. 173.252.67.790.0% 3 176.3 177.4 176.3 177.9 0.9 13. edge-star-mini-shv-01-sjc2.facebook.com 0.0% 3 177.6 177.4 177.1 177.6 0.3 Trace to 157.240.2.35: Packets Pings Host Loss% Snt Last Avg Best Wrst StDev [...] 6. win-b4-link.telia.net0.0% 3 20.9 25.8 20.8 35.8 8.6 7. level-ic-1573273-wien-b4.c.telia.net 0.0% 3 20.5 21.4 20.2 23.4 1.8 8. ??? 9. 4.15.85.102 0.0% 3 142.9 138.5 135.8 142.9 3.8 10. po121.asw01.ord3.tfbnw.net 0.0% 3 135.5 135.7 135.5 136.0 0.4 11. po211.psw01c.ort2.tfbnw.net 0.0% 2 136.1 135.7 135.3 136.1 0.6 12. 173.252.67.127 0.0% 2 151.6 147.6 143.6 151.6 5.7 13. edge-star-mini-shv-01-ort2.facebook.com 0.0% 2 136.0 135.9 135.8 136.0 0.2 Trace to 31.13.93.36 => this one is going via the FB DE-CIX peering and latency of 34ms is perfect: Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. router 0.0% 50.5 0.5 0.5 0.6 0.0 2. 192.168.8.3 0.0% 40.7 0.7 0.7 0.8 0.1 3. 37.252.236.880.0% 4 37.8 38.0 31.8 46.1 6.0 4. er-03.00-09-23.anx04.vie.at.anexia-it.com0.0% 4 25.3 25.2 23.1 29.2 2.9 5. cr-02.0v-08-72.anx03.vie.at.anexia-it.com0.0% 4 21.3 30.4 20.6 40.1 10.9 6. cr-01.0v-08-73.anx25.fra.de.anexia-it.com0.0% 4 45.7 39.1 33.1 45.7 5.3 7. ae1.br02.fra1.tfbnw.net 0.0% 4 33.0 34.6 32.1 40.3 3.8 8. po114.asw01.fra2.tfbnw.net 0.0% 4 33.6 35.6 33.3 41.7 4.0 9. po211.psw01d.fra3.tfbnw.net 0.0% 4 40.8 36.6 32.9 40.8 3.4 10. 173.252.67.171 0.0% 4 41.2 35.6 33.3 41.2 3.7 11. edge-star-mini-shv-01-fra3.facebook.com 0.0% 4 34.6 33.9 33.2 34.6 0.6 Trace to 31.13.76.68: Packets Pings Host Loss% Snt Last Avg Best Wrst StDev [...] 6. win-b4-link.telia.net0.0%14 28.9 24.7 20.3 29.3 3.5 7. level-ic-1573273-wien-b4.c.telia.net 0.0%14 22.8 22.5 20.0 30.9 3.7 8. ae-2-3613.edge1.seattle3.level3.net 0.0%13 195.2 189.2 186.0 195.2 3.5 9. 4.59.232.46 0.0%13 187.9 188.2 186.7 196.1 2.5 10. po101.psw03.sea1.tfbnw.net 0.0%13 194.7 188.3 186.5 194.7 2.0 11. 173.252.67.125 0.0%13 188.0 188.8 186.9 195.5 2.2 12. edge-star-mini-shv-01-sea1.facebook.com 0.0%13 186.6 189.0 186.2 196.0 3.1 Best regards Jürgen -Ursprüngliche Nachricht- Von: Doug Porter [mailto:d...@fb.com] Gesendet: Dienstag, 21. März 2017 18:41 An: Jürgen Jaritsch ; nanog@nanog.org Betreff: Re: Fac
Re: Facebook more specific via Level3 ?
Hi, > This specific, and many others, are only announced to peers in the > metro they originate in. To receive this prefix directly you'll > need to peer with us in Los Angeles. the point is: Level3 is exporting this prefix to the EU since ~1 week Telia is learning it from Level3 and they also started to re-export it: Telia Looking Glass (http://lg.telia.net/?query=bgp&protocol=IPv4&addr=31.13.71.0/24+exact&route r=Vienna) Command: show route protocol bgp 31.13.71.36 table inet.0 31.13.71.0/24 *[BGP/170] 18w5d 16:40:16, MED 0, localpref 150 AS path: 3356 32934 I, validation-state: unverified > to 80.239.128.178 via ae9.0 This is causing >120ms latency to Austrian (and some German ...) networks towards Facebook - current traceroute from source network 188.172.239.0/24: Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. router 0.0% 40.5 0.5 0.5 0.6 0.0 2. 192.168.8.3 0.0% 40.6 0.7 0.6 0.8 0.1 3. 37.252.236.880.0% 4 37.3 34.2 30.6 37.3 2.9 4. er-03.00-09-23.anx04.vie.at.anexia-it.com0.0% 4 21.1 31.9 21.1 50.9 13.3 5. cr-02.0v-08-72.anx03.vie.at.anexia-it.com0.0% 4 23.2 25.9 20.4 31.6 5.1 6. win-b4-link.telia.net0.0% 4 22.1 25.0 22.1 30.1 3.5 7. level-ic-1573273-wien-b4.c.telia.net 0.0% 3 22.4 22.3 21.1 23.3 1.1 8. ae-3-3611.edge2.dallas1.level3.net 0.0% 3 150.5 150.6 150.4 150.8 0.2 9. facebook-in.edge2.dallas1.level3.net 0.0% 3 156.6 152.9 151.0 156.6 3.2 10. po102.psw02.dft4.tfbnw.net 0.0% 3 151.4 151.6 151.4 152.0 0.4 11. 173.252.67.570.0% 3 151.6 152.3 151.6 153.4 0.9 12. edge-star-mini-shv-01-dft4.facebook.com 0.0% 3 159.3 158.4 152.5 163.6 5.6 The admins of AS42473 started to drop the more specific from Level3 and now they get it from Telia (which is the Level3 re-export they learned). I guess you guys should talk to Level3 and ask them what the hell they are doing? :). Thanks & best regards Jürgen
Re: Facebook more specific via Level3 ?
> 31.13.70.0/24 > > This more specific is only visible via AS3356 ... Facebook > isn’t even announcing it via direct peering. This specific, and many others, are only announced to peers in the metro they originate in. To receive this prefix directly you'll need to peer with us in Los Angeles. It appears you're in Austria though, which means there's likely no use in you peering with us in LA. We globally load balance people to the best point of presence. You shouldn't see much, if any, traffic to or from the above prefix. > Any Facebook admin on or off list available to get this > debugged and tracked down? Please reach out to n...@fb.com in the future. Regards, -- dsp
Re: Facebook more specific via Level3 ?
I see that specific route both of my upstreams and not going through level 3. NetworkNext HopMEDLocPrf Weight Path *>x 31.13.70.0/24 x.x.x.x 0 80 0 6461 32934 i *i 31.13.70.0/24 x.x.x.x 0 80 0 209 32934 i * 31.13.70.0/24 x.x.x.x 10 80 0 209 32934 i On Tue, Mar 21, 2017 at 12:56 PM, Jürgen Jaritsch wrote: > Hi, > > > > is anyone else receiving Facebook’s /24 more specific from Level3 (AS3356)? > > > > 31.13.70.0/24 *[BGP/170] 1w5d 17:21:28, MED 0, localpref 100, from > a.b.c.d > > AS path: 3356 32934 I, validation-state: unverified > > > > This more specific is only visible via AS3356 … Facebook isn’t even > announcing it via direct peering. > > > > Any Facebook admin on or off list available to get this debugged and > tracked > down? > > > > > > Thanks & best regards > > Jürgen > >
Facebook more specific via Level3 ?
Hi, is anyone else receiving Facebooks /24 more specific from Level3 (AS3356)? 31.13.70.0/24 *[BGP/170] 1w5d 17:21:28, MED 0, localpref 100, from a.b.c.d AS path: 3356 32934 I, validation-state: unverified This more specific is only visible via AS3356 Facebook isnt even announcing it via direct peering. Any Facebook admin on or off list available to get this debugged and tracked down? Thanks & best regards Jürgen
