Scott, Thank you for your input.
What do you recommend for network segmentation? A VLAN per Chassis, a VLAN per service, or a VLAN per customer/port? When you say qinq VLANs are you referring to the CVLAN model? I am really interested to know how the largest providers, like Comcast, AT&T, and Verizon setup their residential networks. Little information is available on the internet besides the access platforms they use. On Thu, Jul 31, 2014 at 1:51 PM, Scott Helms <khe...@zcorum.com> wrote: > > > On Thu, Jul 31, 2014 at 2:25 PM, Colton Conor <colton.co...@gmail.com> > wrote: > >> I have read both the Juniper MX and Cisco ASR9K do support this advanced >> BRAS functionality, what Juniper calls Subscriber Feature Management and >> what Cisco calls BGN. These software functions run on the router itself, >> however the are not free or included with the base chassis. To enable these >> you must pay a hefty fee. So you are saying that these advanced feature >> packs that the largest networking markers in the world sell are really not >> needed anymore due to advancements on the access vendor side of the house? >> From the reading I have done about these solutions, it is kind of like >> PPPoE with a radius setup, but instead DHCP option 82 with a radius setup. >> These routers are also capable of running a local DHCP server, but I am not >> sure if that is recommended. >> > > Yeah, that's it in a nutshell. There are several options, like matching > on Option 82 or redirecting to a web page, but at the end of the day I > don't believe they're worth the time or expense. Keep in mind that earlier > in my career I was a huge proponent of BRAS architecture and I've put in > everything from Nortel Shasta's to Lucent Terminators, to Redbacks, to > Juniper ERXs and several more models I can't remember. Once you get past > the whole lack of authentication, which was never very secure, and > understand that you can depend on Option 82 to tell you where a session > came from physically the rest is just finding away to count and account for > bits. > > Oh, and I never recommend running the DHCP daemon on a piece of networking > gear for service providers. > > >> >> The DPoE DOCSIS provisioning of your GPON network is interesting, but is >> that really relevant for a new provider if they don't have cable CMTS >> systems already deployed. Sure, it makes sense for the cable compaines who >> have already bought billing systems and are used to living in >> a DOCSIS world. But if you were starting fresh from the group up are you >> recommending we look at GPON providers like Calix because they support DPoE >> so we can buy DOCIS billing systems? That is an interesting concept. >> > > I'd strongly recommend finding a vendor that says they will support it on > the shelves you're going to buy even if they don't today. Even if you're > not doing DOCSIS cable modems and don't ever plan to the provisioning > paradigm (DHCP, TFTP, ToD) is much simpler than the proprietary north bound > (usually SOAP) API that direct integration requires. You can even build > your own provisioning system with a little scripting and there are many > more commercial options than there are for direct integration to the > shelves. > > >> >> >> >> >> >> On Thu, Jul 31, 2014 at 12:59 PM, Scott Helms <khe...@zcorum.com> wrote: >> >>> >>> On Thu, Jul 31, 2014 at 12:07 PM, Colton Conor <colton.co...@gmail.com> >>> wrote: >>> >>>> Scott, >>>> >>>> Thanks for the long post. >>>> >>>> We will use a layer 2 10G aggregation switch then to aggregate the >>>> chassis at the core location. Do you have any recommendations on 10G >>>> switches? >>>> >>> >>> Not really, just stick with one of the major brands and you _should_ be >>> fine. >>> >>> >>>> >>>> Yes I realize the math is a little backwards as this is >>>> all hypothetical at this point. We would provision each ONT as a shared >>>> 1Gbps offering similar to Google Fiber. We know there will be a large >>>> amount of oversubscription as no one really uses a full Gbps or anywhere >>>> close to it. I just wanted to stress the point that carrier redundancy at >>>> the 10G level would be a requirement for the core router, and it should of >>>> course have 10G links going to the uplinks on the aggregation switch. I >>>> think the Cisco ASR9k and the Juniper MX line will do well. Not sure if >>>> there are any others that can handle this level of traffic on the BGP side? >>>> >>> >>> That's reasonable IMO and yes, I think the Juniper MX can handle that as >>> well as some other functions for you related to subscriber management if >>> you want. The MX line has a full BRAS set of capabilities built into it >>> that it inherited from the older ERX line, but they're commonly deployed >>> without using any of them of as well. >>> >>> >>>> >>>> So we have a 10G aggregation switch to aggregate the chassis uplink >>>> connections, and a 10G router BGP capable router. >>>> >>>> I really liked your article on DHCP vs PPP for DSL networks. We >>>> definitely agree the way to go is with a DHCP server. A couple of items >>>> your article left as big questions: >>>> >>> >>> >>>> 1. The article mentioned DHCP doesn't do the other part of what PPPoE >>>> or PPPoA does, which is generate RADIUS accounting records that give us the >>>> bandwidth information. So that’s one of the main challenges in switching to >>>> a DHCP based system. So, how do you handle bandwidth tracking in an all >>>> DHCP environment then? If I want to track how many GB a customer used last >>>> month, or the average Mbps used how do you do so? >>>> >>> >>> There are a few ways to get at that problem. You can use Netflow/IPFIX >>> collection to gather the usage from your router, accepting that you're only >>> going to get information on layer 3 traffic, which generally isn't a >>> problem. You will need to match the IPs up against your Option 82 parsing >>> which will give you the circuit ID, IP address, and WAN MAC of the ONT. >>> You can also poll your shelves via SNMP, CLI, TL-1, and/or Netconf to >>> collect the data and put it into a database in much the same way you can >>> use RADIUS accounting data. >>> >>> >>>> >>>> 2. I liked your option 82 example, and that works well for DSL networks >>>> where one port is tied to one customer. But how does option 82 work when >>>> you have multiple customers hanging off a GPON port? What does GPON use a >>>> subport identifier? >>>> >>> >>> Yep, the different vendors implement it slightly differently, usually >>> the ONT MAC/serial will be included or the ONT ID will be included. Talk >>> with your vendor, all the major OLT vendors are very familiar with Option >>> 82 and in many cases they can tailor what their boxes send to make it >>> easier for you. >>> >>> >>> >>>> 3. You mentioned, DHCP is again, not a authentication protocol. So what >>>> handles authentication then if only DHCP is used, and there are no >>>> usernames and passwords? I guess for DSL networks you can enable or disable >>>> the port to allow or disallow access, and Option 82 for identification? I >>>> assume you wouldn't want to shut off the GPON OLT port if one customer >>>> wasn't paying their bill as it would affect the other customers on that >>>> port. I assume access vendors allow you to shut down the sub port or ONT in >>>> this situation for GPON? Still that seems messy having to login to a shelf >>>> or EMS system or API to an EMS system especially if you have multiple >>>> access vendors in a network. Is there a way to do authentication with DHCP? >>>> What about open networks like wifi where anyone can connect, so you don't >>>> have the ability to turn of the port or disable the end device? >>>> 4. I don't think anyone is buying a BRAS anymore, but looks like Cisco, >>>> Juniper, and ALU have what they call BGN, Broadband Subscriber Management, >>>> and other similar software. How are these different from BRAS >>>> functionality? >>>> >>> >>> First, if you can manage it turn on DOCSIS provisioning of your GPON >>> network. AFAIK only Calix has announced this functionality, but I expect >>> the others to follow suit now that there is an official effort at CableLabs >>> to allow that. >>> >>> >>> http://www.lightreading.com/cable-video/docsis/calix-launches-docsis-provisioning-of-gpon/d/d-id/709859 >>> >>> The notion of managing ports and profiles via (an ever changing) shelf >>> API is one of the main reasons that telco billing systems cost so much >>> compared to cable billing systems. If you can't swing DPoG then you're >>> kind of stuck, either you can implement the API your vendor supplies with >>> your billing system, manage the profile assignment manually (yuck), or just >>> provision everyone with the same speed (only works for data only >>> deployments), or go down the older route of putting in a BRAS and making >>> sure the ONTs you're deploying have a PPPoE client embedded in them. >>> Having to deploy an external router for each customer, which I've seen >>> some operators do, makes your install costs higher and makes >>> troubleshooting harder. >>> >>> >>>> So it looks like there are open source and commercial solutions for >>>> DHCP and DNS. Some providers like Infloblox seems to integrate all these >>>> into one. >>>> >>>> So if we have a core router that speaks BGP, a 10G aggregation switch >>>> to aggregate the the chassis, and a device like Infloblox or the other >>>> commercial solutions you mentioned that do DHCP/DNS, is there anything else >>>> that is needed besides the access gear already mentioned in the >>>> assumptions? Are these large and expensive commercial BGN/Broadband >>>> Subscriber management products a thing of the past or still very relevant >>>> in todays environment? >>>> >>> >>> They're not very relevant, once the the OLT vendors realized they could >>> snoop the DHCP session and enforce what the server provided the need for >>> subscriber management pieces really dropped. You've listed the bare >>> essentials for a functional network, there are lots of things that are >>> helpful or useful, but what you have is functional. Having said all of >>> that, there are some relatively unobtrusive ways to have some level of >>> authentication, I just don't think they're very valuable. >>> >>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> On Thu, Jul 31, 2014 at 8:54 AM, Scott Helms <khe...@zcorum.com> wrote: >>>> >>>>> "What is the ideal way to aggregate the 40 10G connections from the >>>>> uplinks >>>>> of the chassis? I would guess a 10G switch since 10G ports on a router >>>>> would be much more expensive?" >>>>> >>>>> Definitely aggregate into a switch first unless you want to run a >>>>> Layer 3 switch as your router, which I don't recommend. >>>>> >>>>> >>>>> "Which router is recommended to handle 4 10G internet connections with >>>>> full >>>>> tables, and then at least 4 10G ports going back to the 10G aggregation >>>>> switch?" >>>>> >>>>> Your math is a little backwards, its very unlikely that you're going >>>>> to have 40 Gbps of Internet (or other interconnection) for the router to >>>>> actually have to process. What is the average provisioned speed for each >>>>> of the 10k PON ports? What over subscription rate are you planning for? >>>>> What, if anything, will you be carrying on net, ie bandwidth consumption >>>>> that won't come from or go to the public Internet? Your own video, voice, >>>>> or other service are examples of things that are often on net. In any >>>>> case >>>>> you're probably in the ASR family with Cisco and I can't remember >>>>> the equivalent from Juniper. >>>>> >>>>> >>>>> How do you handle IP address management? a /20 is only 4096 IP >>>>> addresses, >>>>> but the network would have potentially 10,000 customers. Assume that >>>>> getting more space from ARIN is not an option. Is CGN an option? >>>>> >>>>> CGN is the option of last resort IMO, but you may have to consider it. >>>>> A better approach is to see if your backbone providers will agree to give >>>>> some blocks that you can announce and use those blocks for dynamic >>>>> customers only. Your static IP customers should come from your direct >>>>> ARIN >>>>> allotment in case you need to choose a new backbone provider, which is >>>>> extremely common over time. >>>>> >>>>> >>>>> "Dynamic IP >>>>> addresses? DHCP?" >>>>> >>>>> DHCP with enforcement from the shelves. All the major OLT vendors >>>>> support doing this so that a customer can only use the address assigned to >>>>> him by DHCP and nothing else, except for those customers that you choose >>>>> to >>>>> hard code. Make most of your "static" customers actually DHCP >>>>> reservations >>>>> and only hard code those that you must. >>>>> >>>>> "How do you separate users and traffic? VLANs, Service VLANs, Per >>>>> Customer >>>>> VLANs, Usernames? Passwords? PPPoE? MAC Separation? >>>>> Is a BRAS or BGN functionally really needed or are these older >>>>> concepts?" >>>>> >>>>> DHCP, with Option 82 logging for the circuit ID is the better path >>>>> than a BRAS (PPPoE) these days. Here's a paper we put together on that >>>>> topic a while back: >>>>> >>>>> >>>>> http://www.zcorum.com/wp-content/uploads/Why-Should-I-Move-from-PPPoA-or-PPPoE-to-DHCP.pdf >>>>> >>>>> Depending on your OLT vendor you can either use their built in port >>>>> isolation or QinQ tagging, both are reliable and scalable, just ask your >>>>> vendor which is the best option for your specific gear. >>>>> >>>>> >>>>> >>>>> "If CGNAT or DHCP is needed, what will host the CGNAT or DHCP service? >>>>> The >>>>> core router, a linux box, or something else?" >>>>> >>>>> I wouldn't have those two services connected personally, though there >>>>> are hooks for some of the CGN boxes to talk to DHCP servers. I would hope >>>>> you can get another 6k addresses and avoid the need for CGN altogether. >>>>> Having said that, have you tested your OLTs and ONTs for IPv6 >>>>> interoperability? If they don't handle it well then you're going to have >>>>> to think about alternatives like 6RD ( >>>>> http://en.wikipedia.org/wiki/IPv6_rapid_deployment) >>>>> >>>>> For DHCP at your scale you can run ISC DHCP ( >>>>> http://www.isc.org/downloads/dhcp/) which is the most common open >>>>> source DHCP daemon if you someone who can take care of a Linux server, >>>>> parse the Option 82 information for logging, and handle the configuration >>>>> of the DHCP daemon itself. Otherwise you might want to look at commercial >>>>> products designed for the service provider market like Incongito's BCC and >>>>> Cisco's BAC (CNR replacement) >>>>> >>>>> http://www.incognito.com/products/broadband-command-center/ >>>>> >>>>> http://www.cisco.com/c/en/us/products/cloud-systems-management/broadband-access-center/index.html >>>>> >>>>> >>>>> "What about DNS? >>>>> Is a firewall needed in the core? >>>>> What else is needed?" >>>>> >>>>> There are two kinds of DNS, caching (recursive) and authoritative. >>>>> The first is what your customers will use to resolve things on the >>>>> Internet and the second is used to provide caching name servers on the >>>>> Internet with information about domains you control (are authoritative >>>>> for). The first needs good performance, availability, and >>>>> scalability since your customers will use your caching name servers >>>>> constantly. Most people can run BIND at your scale, again if you have >>>>> someone with Linux experience, but there are other alternatives. PowerDNS >>>>> has both caching and authoritative modules and there are some commercial >>>>> offerings out there both as cloud hosting and local deployments. Your >>>>> backbone provider will also often have caching name servers your customers >>>>> can use, but the quality varies quite a bit. You can also, especially at >>>>> first, leverage some of the free offerings like Google's DNS. I >>>>> don't recommend firewalls for service provider networks, but you should >>>>> make sure your gear can run (and is configured to do so) BCP 38. >>>>> >>>>> >>>>> Scott Helms >>>>> Vice President of Technology >>>>> ZCorum >>>>> (678) 507-5000 >>>>> -------------------------------- >>>>> http://twitter.com/kscotthelms >>>>> -------------------------------- >>>>> >>>>> >>>>> On Thu, Jul 31, 2014 at 9:23 AM, Colton Conor <colton.co...@gmail.com> >>>>> wrote: >>>>> >>>>>> If a new operator or city is building a greenfield access network >>>>>> from the >>>>>> ground up, what software and hardware is needed in the core network to >>>>>> provide and manage residential and business internet services similar >>>>>> to >>>>>> the likes of AT&T, Comcast, and Google Fiber? Television and Telephone >>>>>> services are not to be considered only internet. >>>>>> >>>>>> Assume hypothetically the operator already has the following in place: >>>>>> 10 GPON OLTs Chassis from an access vendor in 10 POPs around town >>>>>> (each POP >>>>>> has 1 Chassis). Each OLT Chassis has 4 10G Uplinks back to the core. >>>>>> Dark fiber going from the POP locations back to the core location >>>>>> Assume a 32:1 way split, and each OLT chassis has enough ports >>>>>> populated to >>>>>> serve the area. >>>>>> 10,000 GPON ONTs. The ONTs can be put in routed gateway or bridged >>>>>> mode. >>>>>> Assume you are building a network designed to serve 10,000 subs >>>>>> All the fiber splitters, ducts, fiber, etc connecting the OLTs to the >>>>>> ONTs >>>>>> is already in place >>>>>> ASN from ARIN >>>>>> /20 of IPv4 space and /32 of IPv6 space from ARIN >>>>>> 4 burstable 10G internet connections from 4 tier 1 internet providers >>>>>> >>>>>> Questions are: >>>>>> >>>>>> What is the ideal way to aggregate the 40 10G connections from the >>>>>> uplinks >>>>>> of the chassis? I would guess a 10G switch since 10G ports on a router >>>>>> would be much more expensive? >>>>>> Which router is recommended to handle 4 10G internet connections with >>>>>> full >>>>>> tables, and then at least 4 10G ports going back to the 10G >>>>>> aggregation >>>>>> switch? >>>>>> How do you handle IP address management? a /20 is only 4096 IP >>>>>> addresses, >>>>>> but the network would have potentially 10,000 customers. Assume that >>>>>> getting more space from ARIN is not an option. Is CGN an option? >>>>>> Dynamic IP >>>>>> addresses? DHCP? >>>>>> How do you separate users and traffic? VLANs, Service VLANs, Per >>>>>> Customer >>>>>> VLANs, Usernames? Passwords? PPPoE? MAC Separation? >>>>>> Is a BRAS or BGN functionally really needed or are these older >>>>>> concepts? >>>>>> If CGNAT or DHCP is needed, what will host the CGNAT or DHCP service? >>>>>> The >>>>>> core router, a linux box, or something else? >>>>>> What about DNS? >>>>>> Is a firewall needed in the core? >>>>>> What else is needed? >>>>>> >>>>>> Is there a guide out there somewhere? I know many cities are looking >>>>>> at >>>>>> building their own network, and have similar questions. Access >>>>>> vendors are >>>>>> willing to sell gear all day long, but then they leave it up to the >>>>>> operator/city to answer these harder questions. >>>>>> >>>>>> How would you build a access network from the ground up if you had the >>>>>> resources and time to do so? Would you even use GPON? Even if GPON >>>>>> was not >>>>>> used and another access technology like AE, VDSL2, or wireless was >>>>>> used I >>>>>> think many of these questions would be the same. >>>>>> >>>>> >>>>> >>>> >>> >> >