Re: Verizon having a bad routing day today?
On Sat, Feb 9, 2019 at 9:09 PM Christopher Morrow wrote: > I wonder if there's a lurking verizon/701 engineer on-list who may have a > few moments to reach me out of band? :) I've got what looks like busted > routing (or > howdy! actually 3 different vz folk found me, explained what I'm seeing and ... mostly it's ok just unexpected by me :) -chris
Re: AT/as7018 now drops invalid prefixes from peers
On 11/Feb/19 16:53, Jay Borkenhagen wrote: > FYI: > > The AT/as7018 network is now dropping all RPKI-invalid route > announcements that we receive from our peers. > > We continue to accept invalid route announcements from our customers, > at least for now. We are communicating with our customers whose > invalid announcements we are propagating, informing them that these > routes will be accepted by fewer and fewer networks over time. > > Thanks to those of you who are publishing ROAs in the RPKI. We would > also like to encourage other networks to join us in taking this step > to improve the quality of routing information in the Internet. Well done! Mark.
Re: AT/as7018 now drops invalid prefixes from peers
Job Snijders writes: > Dear Jay, AT, > > On Mon, Feb 11, 2019 at 09:53:45AM -0500, Jay Borkenhagen wrote: > > The AT/as7018 network is now dropping all RPKI-invalid route > > announcements that we receive from our peers. > > Thanks for filtering us! :-) Any time! :-) > If you can share more about the experience in terms of load on the > support tiers in your organisation, or questions from peering partners, > that could perhaps be helpful information for others in their > preparations. > A few reports of resulting connectivity loss have come in through various channels: on Jared's outages mailing list, on IRC, through our customer care ticket system, etc. Thusfar I have been very pleased with the reactions folks have had when we described how our policy change caused us to lose their affected route announcement. Everyone so far has understood the purpose of the RPKI, they understood why the affected route announcements were deemed invalid and thus were dropped, and best of all -- they understood what they needed to do to fix things. We got some very good advice watching this video from your most recent NLNOG day: https://www.youtube.com/watch?v=vrzl__yGqLE ... but there is one place where I disagree with Niels. He advised against lowering the local-pref of invalid routes. I agree that this should not be anyone's target policy, but it is a useful step along the way. To set invalid routes a lower local-pref, one needs to establish RTR sessions from routers to relying party servers, and to configure a policy that takes validation state into account. The policy here can also set community based on the validation state, which can help with flow-based traffic analysis. Then, when you are comfortable operating RPs that talk RTR to your routers and you're ready to implement a meaningful policy, it's a simple matter of changing from: if validation-state = invalid then local-pref = $LOWER community = foo fi to: if validation-state = invalid then drop fi In short: C'mon in! The water's fine! :-) Thanks. Jay B.
Re: AT/as7018 now drops invalid prefixes from peers
valdis.kletni...@vt.edu writes: > On Mon, 11 Feb 2019 09:53:45 -0500, Jay Borkenhagen said: > > The AT/as7018 network is now dropping all RPKI-invalid route > > announcements that we receive from our peers. > > Congrats! Thanks! > Are you able to comment on what amount of routes are getting dropped? In round numbers, we dropped about 5000 invalid prefixes total between ipv4 and ipv6. Roughly half of those prefixes were covered by less-specific non-invalid routes, so connectivity should not have been affected for those prefixes (assuming an announcement yields reachability to all destinations within it). Flow analysis was showing just a couple Gbps of traffic to all invalid routes all across the country, and much less than that with those invalids having no covering less-specifics. Jay B.
Re: AT/as7018 now drops invalid prefixes from peers
Compton, Rich A writes: > That's great! Do you guys have plans to publish ROAs for your own > netblocks? If so, can you please share info on your process (tools, > pitfalls, etc.)? Thanks! > Hi Rich, We do have ROAs published for a not insignificant fraction of our address space. For example (and cherry-picking the representation most favorable to us) we're listed at #6 in the "25 Autonomous Systems with the most Address Space VALID by RPKI" at this NIST RPKI tracker: https://rpki-monitor.antd.nist.gov/#rpki_adopters We will publish more ROAs over time. Thusfar we have been utilizing ARIN's hosted model, but down the road ARIN's delegated model will be in our future. https://www.arin.net/resources/rpki/using_rpki.html Thanks. Jay B.
Re: AT/as7018 now drops invalid prefixes from peers
On Mon, 11 Feb 2019 09:53:45 -0500, Jay Borkenhagen said: > The AT/as7018 network is now dropping all RPKI-invalid route > announcements that we receive from our peers. Congrats! Are you able to comment on what amount of routes are getting dropped?
Re: AT/as7018 now drops invalid prefixes from peers
This is the best news today! Great job!! Cheers, Melchior On Mon, Feb 11, 2019 at 3:56 PM Jay Borkenhagen wrote: > > FYI: > > The AT/as7018 network is now dropping all RPKI-invalid route > announcements that we receive from our peers. > > We continue to accept invalid route announcements from our customers, > at least for now. We are communicating with our customers whose > invalid announcements we are propagating, informing them that these > routes will be accepted by fewer and fewer networks over time. > > Thanks to those of you who are publishing ROAs in the RPKI. We would > also like to encourage other networks to join us in taking this step > to improve the quality of routing information in the Internet. > > Thanks! > > Jay B. > > > >
Re: AT/as7018 now drops invalid prefixes from peers
That's great! Do you guys have plans to publish ROAs for your own netblocks? If so, can you please share info on your process (tools, pitfalls, etc.)? Thanks! On 2/11/19, 7:55 AM, "NANOG on behalf of Jay Borkenhagen" wrote: FYI: The AT/as7018 network is now dropping all RPKI-invalid route announcements that we receive from our peers. We continue to accept invalid route announcements from our customers, at least for now. We are communicating with our customers whose invalid announcements we are propagating, informing them that these routes will be accepted by fewer and fewer networks over time. Thanks to those of you who are publishing ROAs in the RPKI. We would also like to encourage other networks to join us in taking this step to improve the quality of routing information in the Internet. Thanks! Jay B. E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
Re: AT/as7018 now drops invalid prefixes from peers
Jay & everyone AT: I just want to say thank you. Kudos to your team for implementing and management for having the intestinal fortitude to do so. -- TTFN, patrick > On Feb 11, 2019, at 09:53, Jay Borkenhagen wrote: > > > FYI: > > The AT/as7018 network is now dropping all RPKI-invalid route > announcements that we receive from our peers. > > We continue to accept invalid route announcements from our customers, > at least for now. We are communicating with our customers whose > invalid announcements we are propagating, informing them that these > routes will be accepted by fewer and fewer networks over time. > > Thanks to those of you who are publishing ROAs in the RPKI. We would > also like to encourage other networks to join us in taking this step > to improve the quality of routing information in the Internet. > > Thanks! > > Jay B. > >
Re: AT/as7018 now drops invalid prefixes from peers
Dear Jay, AT, On Mon, Feb 11, 2019 at 09:53:45AM -0500, Jay Borkenhagen wrote: > The AT/as7018 network is now dropping all RPKI-invalid route > announcements that we receive from our peers. Thanks for filtering us! :-) AT doing origin validation combined with the peerlock-style AS_PATH filters this makes for a pretty strongly protected path between you and others. > We continue to accept invalid route announcements from our customers, > at least for now. We are communicating with our customers whose > invalid announcements we are propagating, informing them that these > routes will be accepted by fewer and fewer networks over time. I think this is a sensible strategy. > Thanks to those of you who are publishing ROAs in the RPKI. We would > also like to encourage other networks to join us in taking this step > to improve the quality of routing information in the Internet. Thank you for paving the way! If you can share more about the experience in terms of load on the support tiers in your organisation, or questions from peering partners, that could perhaps be helpful information for others in their preparations. Kind regards, Job
Re: AT/as7018 now drops invalid prefixes from peers
A round of applause to AT for leading the way! Best regards, Martijn On 2/11/19 3:53 PM, Jay Borkenhagen wrote: > FYI: > > The AT/as7018 network is now dropping all RPKI-invalid route > announcements that we receive from our peers. > > We continue to accept invalid route announcements from our customers, > at least for now. We are communicating with our customers whose > invalid announcements we are propagating, informing them that these > routes will be accepted by fewer and fewer networks over time. > > Thanks to those of you who are publishing ROAs in the RPKI. We would > also like to encourage other networks to join us in taking this step > to improve the quality of routing information in the Internet. > > Thanks! > > Jay B. > > >
Re: AT/as7018 now drops invalid prefixes from peers
On Mon, Feb 11, 2019 at 6:55 AM Jay Borkenhagen wrote: > > FYI: > > The AT/as7018 network is now dropping all RPKI-invalid route > announcements that we receive from our peers. > > We continue to accept invalid route announcements from our customers, > at least for now. We are communicating with our customers whose > invalid announcements we are propagating, informing them that these > routes will be accepted by fewer and fewer networks over time. > > Thanks to those of you who are publishing ROAs in the RPKI. We would > also like to encourage other networks to join us in taking this step > to improve the quality of routing information in the Internet. > > Thanks! > > Jay B. > > > Good move AT , thanks for taking this on >
AT/as7018 now drops invalid prefixes from peers
FYI: The AT/as7018 network is now dropping all RPKI-invalid route announcements that we receive from our peers. We continue to accept invalid route announcements from our customers, at least for now. We are communicating with our customers whose invalid announcements we are propagating, informing them that these routes will be accepted by fewer and fewer networks over time. Thanks to those of you who are publishing ROAs in the RPKI. We would also like to encourage other networks to join us in taking this step to improve the quality of routing information in the Internet. Thanks! Jay B.
Re: Last Mile Design
On 11/Feb/19 16:21, Mikael Abrahamsson wrote: > > Speaking of an Asia-Pac example, Thailand, the government owned telco. > > https://www.tot.co.th/%E0%B8%9A%E0%B8%A3%E0%B8%B4%E0%B8%81%E0%B8%B2%E0%B8%A3%E0%B8%AD%E0%B8%B4%E0%B8%99%E0%B9%80%E0%B8%97%E0%B8%AD%E0%B8%A3%E0%B9%8C%E0%B9%80%E0%B8%99%E0%B9%87%E0%B8%95/tot-fiber-2u > > > Typically there are 1-3 different FTTH providers if you live in > something that resembles a town, pay 50-100EUR installation fee, they > show up within days to pull your new fiber and now you can have 150/50 > for around 20EUR a month. > > The price level has remained the same the past 5-6 years, but speed > has gone up from 10/3 to 150/50 for the same monthly payment. Last > year the 150/50 price level offering was 100/20 instead. I can attest to this as I saw exactly the same thing in Malaysia, and more so after I've been away these past couple of years. The company I worked for at the time started rolling our GPON with the top speed at 50Mbps for several hundred RM/month back in 2010. 9 years later, they are selling 1Gbps @ RM99/month. I suppose we can choke it down to the cost of living in the North Western hemisphere being what it is, or some such reason :-). Mark.
Re: Last Mile Design
On Mon, 11 Feb 2019, Mark Tinka wrote: We have the same problem here in Africa too (and I saw it in Asia-Pac while I was there as well)... non-telco-centric companies that deployed Speaking of an Asia-Pac example, Thailand, the government owned telco. https://www.tot.co.th/%E0%B8%9A%E0%B8%A3%E0%B8%B4%E0%B8%81%E0%B8%B2%E0%B8%A3%E0%B8%AD%E0%B8%B4%E0%B8%99%E0%B9%80%E0%B8%97%E0%B8%AD%E0%B8%A3%E0%B9%8C%E0%B9%80%E0%B8%99%E0%B9%87%E0%B8%95/tot-fiber-2u Typically there are 1-3 different FTTH providers if you live in something that resembles a town, pay 50-100EUR installation fee, they show up within days to pull your new fiber and now you can have 150/50 for around 20EUR a month. The price level has remained the same the past 5-6 years, but speed has gone up from 10/3 to 150/50 for the same monthly payment. Last year the 150/50 price level offering was 100/20 instead. -- Mikael Abrahamssonemail: swm...@swm.pp.se
Re: Last Mile Design
On 11/Feb/19 15:55, Mikael Abrahamsson wrote: > > > If they had just stayed at the L1 level and provided dark fiber for > the amount of money mentioned before (for instance 10-15 EUR a month) > then a lot of the problems wouldn't be there. They could have used the > same organisation as before that now could do fiber as well, and > that's that. Simple product, can't go wrong in a lot of weird ways. We have the same problem here in Africa too (and I saw it in Asia-Pac while I was there as well)... non-telco-centric companies that deployed fibre to manage their non-telco infrastructure, now entering the telco space to make use of the excess capacity, or because they want to be part of the "next digital wave", with zero operational experience, and a single-mindedness about one thing - "We will never sell dark fibre to anyone". Ultimately, they wise up or get bought by an operator. It's just a question of how much patience you've got to spend. Mark.
Re: Last Mile Design
On Mon, 11 Feb 2019, Mark Tinka wrote: someone else" they will say "huh? what do you mean". There is an unfortunate common conflation between the fiber optic cable and the services offered on it. I get what you're saying, but sadly, someone has to take the risk to build out a network. Unless you are a large incumbent like Telia, chances are it will be company whose sole focus is just fibre network construction, and anything higher up in the layers is of no interest to them. The problem here is that it might be an energy company or someone who isn't really into datacom. Now they're going to have to operate an active network to provide this "bitstream access" with DHCP relays, BCP38 support and all that comes with it. The result is that right now, most of these networks do not support IPv6 and they do not support > 1 gigabit/s speed (some don't even support more than 100-500 either). If they had just stayed at the L1 level and provided dark fiber for the amount of money mentioned before (for instance 10-15 EUR a month) then a lot of the problems wouldn't be there. They could have used the same organisation as before that now could do fiber as well, and that's that. Simple product, can't go wrong in a lot of weird ways. -- Mikael Abrahamssonemail: swm...@swm.pp.se
Re: Last Mile Design
On 11/Feb/19 12:49, Mikael Abrahamsson wrote: > > In Sweden it's very common that people who live in detached house > areas have to pay 1500-3000EUR to get attached to the fiber network as > it's being built out. There are even bank loans you can get to pay for > this, and pay it off over time. It's considered to be a good deal > because it improves the value of the house as well as a huge > improvement over having satellite-dish/terrestrial TV and ADSL/LTE for > Internet access, now instead you can pay 30-40EUR a month to get a > everything over the fiber. Yes, makes sense, especially if you can get support to fund it. > > Now, I like the LLUB model where ISPs get access to the dark fiber all > the way to the customer, and we do have that here as well, just not as > commonly as I'd like. That's where https://www.bahnhof.se/villafiber/ > comes from where they offer 10GE for 50EUR a month. This is done on > Telia LLUB:ed dark fiber which costs around 15EUR a month (regulated > price). It's a great PR case for "dark fiber access rocks and > bitstream sucks". You get IPv6 in there as well, which isn't commonly > available on most of the bitstream access services (because not only > do we not do PON, we don't do PPPoE either here in Sweden). Cut the price of wine and meat and I'll move to this PPPoE-free land :-). > > So it's a mixed bag and pricing and functionality could definitely be > better, but the FTTH rollout has gone quite well here and it's as > usual 10-15 different factors contributing but the willingness of the > population who lives in houses to fork out 1500-3000EUR for fiber > install has made this a lot less cash flow misery for the ISPs that > roll this out. I just wish there would have been a requirement for > everybody to actually rent this dark fiber out (which there isn't > unless you're one of the biggest players) because after paying those > 1500-3000EUR and you ask the fiber installation company "who owns this > fiber?" they say "we do" and if you ask "ok, I'd like it connected to > someone else" they will say "huh? what do you mean". There is an > unfortunate common conflation between the fiber optic cable and the > services offered on it. I get what you're saying, but sadly, someone has to take the risk to build out a network. Unless you are a large incumbent like Telia, chances are it will be company whose sole focus is just fibre network construction, and anything higher up in the layers is of no interest to them. Mark.
Re: Last Mile Design
On Mon, 11 Feb 2019, Mark Tinka wrote: In any case, we are now building out our own fiber to cover the gaps left by TDC. Here the end user has to pay DKK 12,000 (USD 1,824 / EUR 1,608) one time fee and with that he gets everything including 5 years of free internet. This works out at DKK 200 / month including 25% VAT tax (USD 30 / EUR 27). Very interesting - don't you feel that an initial outlay like that could put some potential customers off? Then again, per capita income in Denmark, I'd imagine, could allow most to think about this. If all that buys me Internet access for 5 years before I have to shell out anymore wedge, I'd do it. In Sweden it's very common that people who live in detached house areas have to pay 1500-3000EUR to get attached to the fiber network as it's being built out. There are even bank loans you can get to pay for this, and pay it off over time. It's considered to be a good deal because it improves the value of the house as well as a huge improvement over having satellite-dish/terrestrial TV and ADSL/LTE for Internet access, now instead you can pay 30-40EUR a month to get a everything over the fiber. Now, I like the LLUB model where ISPs get access to the dark fiber all the way to the customer, and we do have that here as well, just not as commonly as I'd like. That's where https://www.bahnhof.se/villafiber/ comes from where they offer 10GE for 50EUR a month. This is done on Telia LLUB:ed dark fiber which costs around 15EUR a month (regulated price). It's a great PR case for "dark fiber access rocks and bitstream sucks". You get IPv6 in there as well, which isn't commonly available on most of the bitstream access services (because not only do we not do PON, we don't do PPPoE either here in Sweden). So it's a mixed bag and pricing and functionality could definitely be better, but the FTTH rollout has gone quite well here and it's as usual 10-15 different factors contributing but the willingness of the population who lives in houses to fork out 1500-3000EUR for fiber install has made this a lot less cash flow misery for the ISPs that roll this out. I just wish there would have been a requirement for everybody to actually rent this dark fiber out (which there isn't unless you're one of the biggest players) because after paying those 1500-3000EUR and you ask the fiber installation company "who owns this fiber?" they say "we do" and if you ask "ok, I'd like it connected to someone else" they will say "huh? what do you mean". There is an unfortunate common conflation between the fiber optic cable and the services offered on it. -- Mikael Abrahamssonemail: swm...@swm.pp.se
Re: Last Mile Design
On 11/Feb/19 11:31, Thomas Bellman wrote: > I assume this is targeted towards single-family detached houses, where > the family owns the house themselves. Then they likely will view that > as an investment in the house. If you want to sell your house a couple > of years later, and it doesn't have a fiber connection, buyers will be > less attracted to the house, and want to pay less. Makes sense. > It might also be more expensive to connect after the initial buildout > of an area. I believe that's how the commercial companies in Sweden > that build FTTH work. Cities also aren't keen on opening up streets again, e.t.c. > > I can also note that where I live (Linköping, Sweden), the municipal > fiber company charges ~2400 EUR to connect a single-family home to their > network. That does *not* include the laying of fiber on your property, > from the street to your house. And on top of that, you need to buy > Internet connectivity from a normal commercial ISP at a monthly cost; > the municipal fiber company only provides layer 2 connectivity between > the home and the ISPs (currently 19 different ISPs). Having an option, even though it could be pricey, is better than not having anything at all. Mark. signature.asc Description: OpenPGP digital signature
Re: Last Mile Design
On 2019-02-11 04:57 CET, Mark Tinka wrote: > On 10/Feb/19 17:46, Baldur Norddahl wrote: [...] >> In any case, we are now building out our own fiber to cover the gaps >> left by TDC. Here the end user has to pay DKK 12,000 (USD 1,824 / EUR >> 1,608) one time fee and with that he gets everything including 5 years >> of free internet. This works out at DKK 200 / month including 25% VAT >> tax (USD 30 / EUR 27). > Very interesting - don't you feel that an initial outlay like that could > put some potential customers off? Then again, per capita income in > Denmark, I'd imagine, could allow most to think about this. If all that > buys me Internet access for 5 years before I have to shell out anymore > wedge, I'd do it. I assume this is targeted towards single-family detached houses, where the family owns the house themselves. Then they likely will view that as an investment in the house. If you want to sell your house a couple of years later, and it doesn't have a fiber connection, buyers will be less attracted to the house, and want to pay less. It might also be more expensive to connect after the initial buildout of an area. I believe that's how the commercial companies in Sweden that build FTTH work. I can also note that where I live (Linköping, Sweden), the municipal fiber company charges ~2400 EUR to connect a single-family home to their network. That does *not* include the laying of fiber on your property, from the street to your house. And on top of that, you need to buy Internet connectivity from a normal commercial ISP at a monthly cost; the municipal fiber company only provides layer 2 connectivity between the home and the ISPs (currently 19 different ISPs). /Bellman signature.asc Description: OpenPGP digital signature
