I am a bit worried about phrases like "If Akamai was doing these updates more frequently”. Akamai does not decide these things. You may as well say “if the fiber carriers sent the bits over several hours instead of all at once.” And please do not say you were just using shorthand. You have blamed the CDNs and Akamai by name several times in this thread.
I know first hand that Akamai has explained to large customers the possible problems with multi-GB updates to millions of users simultaneously. If the game company does not care, then I do not see what you expect the CDN to do about it. Most CDNs do their best to deliver traffic optimally. It is in their own best interest. They want to avoid dropped packets even more than you do. If you do not like the way a CDN will deliver the traffic, talk to them. Perhaps there is a compromise, perhaps not. But most of them will at least kick ideas around to see what can be done. And after all that, I still do not see what we are arguing about? You want the game companies to change their business model, but you do not want to change yours. Please do not say something like “but if they just ….” Unless you want the game companies to say “but if the ISPs just ….” Either way, stop trying to say someone else - the game provider, the CDN, the user, whoever - should change their model or spend their money to keep your business above water. -- TTFN, patrick P.S. It is not 1995. “The Internet” is a bit more mature, and users expect a bit more. > On Apr 1, 2021, at 6:27 PM, Matt Erculiani <merculi...@gmail.com> wrote: > > Patrick, > > > Matt: Are you arguing the CDNs are at fault because the game companies tell > > everyone to download simultaneously, and > > the ISPs sold the users connectivity to do that download? > > While a gross oversimplification, yes, that's basically what I'm saying; I > know it may not be a popular opinion, but I stand by it. There aren't any > villains here though, just lots of good suggestions in this thread to make > the internet work better for everyone, without spending large swaths of money > to cover the demand of an infrequent, large, update for a single game. > > CNDs do, however, have a responsibility to be good netizens and get this data > out in a manner that doesn't cause disruption. They know the technical > challenges of distributing that much data to the masses, the game company > does not, that's why they outsourced it to a CDN. If the CDN knows what the > gaming company is asking for is pushing the limits of our current > infrastructure, they have a responsibility to relay those limitations that > are outside of their control to their customer, as any responsible vendor > should. Instead, there may be an element of "oh yeah sure, we can do that" or > "the customer is always right" going on here and modern limitations are being > disregarded. > > The idea behind the internet is not that every user can always have their > entire capacity available for a single destination regardless of what > everyone else is doing (and especially if they're all going to the same place > too), the user has purchased that capacity into their provider's network as a > whole, gaining access to all of their connections to all of the various > endpoints on the internet at a backbone and peering capacity that is > economically viable given normal peak demand with some cushion built-in for > redundancy. If that's the desire to have full capacity available to Akamai > available at all times, then everyone needs dedicated P2P circuits direct to > Akamai, but that's not practical. > > If you own an ISP and you're not oversubscribing, you're not making money, > period. To use your analogy, if you've ever been to a gym in January, you've > seen a similar phenomenon first-hand. There aren't enough machines for > everyone, and the gym isn't going to add them because this is a once-a-year > thing and it goes away after a few weeks when many people get tired of > fulfilling their new year's resolutions. Why should the gym limit its sales > to exactly the capacity it has available (or add a lot more machines), when > it knows that for the overwhelming majority of the year, there will always be > dozens of empty machines across the floor? > > If Akamai was doing these updates more frequently (weekly for example) then > sure, it is on the ISP to augment, because this has become the "new normal". > But these updates for a single game that happen once per quarter are hardly > able to be considered normal. Sure, some day 50GB updates will be the norm, > but that's not today, and when it is, somebody else will be pushing out 250GB > updates quarterly. This problem isn't going away soon, and it can't be fixed > permanently by just adding more capacity, it's a complex technical challenge > that CDN's ought to give some more thought, and game publishers should start > considering too. > > -Matt > > On Thu, Apr 1, 2021 at 3:30 PM Patrick W. Gilmore <patr...@ianai.net > <mailto:patr...@ianai.net>> wrote: > I am sorry, maybe I misunderstand. > > Matt: Are you arguing the CDNs are at fault because the game companies tell > everyone to download simultaneously, and the ISPs sold the users connectivity > to do that download? > > If so, are you really arguing “I sold my users XXX Mbps, but if they try to > use it, I want *YOU* to tell them no”? Because that is what it sounds like to > me. > > Imagine a gym sold 10,000 memberships with 10 machines because they figured > everyone would sit on their ass. They would be right most of the time - and > rake in that sweet, sweet monthly cash for zero effort after the initial > sale. But if Oprah or Cher or Biden or some other person famous enough to go > by one name tweets “get your ass to the gym!!", does the gym really think > getting mad at Oprah is the solution? Or do they expect Oprah to pay for the > extra machines they have to buy now? > > Selling a service you know will not work if everyone uses it simultaneously > can be profitable, but there is risk. Do not blame third parties when you > lose that bet. > > -- > TTFN, > patrick > >> On Apr 1, 2021, at 5:04 PM, Tom Beecher <beec...@beecher.cc >> <mailto:beec...@beecher.cc>> wrote: >> >> No disrespect taken, or intended back in your direction, but again, I >> disagree. >> >> If thousands of users are downloading 50G files at the same time, it really >> doesn't matter if they are pulling from a CDN or the origin directly. The >> volume of traffic still has to be handled. Yes, it's a burden on the ISP, >> but it's a burden created by the usage created by their subscribers. >> >> >> On Thu, Apr 1, 2021 at 4:57 PM Matt Erculiani <merculi...@gmail.com >> <mailto:merculi...@gmail.com>> wrote: >> Tom, >> >> All due respect, but there is a massive difference between one user >> downloading 50G and thousands of users each downloading 50G when they all go >> to play their videogame of choice at around the same time. >> >> -Matt >> >> >> >> On Thu, Apr 1, 2021 at 2:46 PM Tom Beecher <beec...@beecher.cc >> <mailto:beec...@beecher.cc>> wrote: >> A user sends a few megabytes of request and receives 50 gigs of reply. They >> aren't DDoSing the network, but they're amplifying a single 50 gig copy they >> receive from the mothership and turning it into likely tens of terabytes of >> traffic. >> Yes, that's a CDN's job, but that volume of legitimate traffic and the very >> tiny window with which it is transmitted is likely to be a burden for even >> the largest residential ISPs. >> >> I'm sitting at home, and I could send a 50k request for a 50G file right now >> from a source not fronted by a CDN. What do? My ISP is still has to deliver >> it to me. The fact that the 50G file does or does not come from a CDN is >> irrelevant. The CDN just happens to be a point source that a lot of users >> happen to connect to. >> >> CDNs want to have the best performance to users because that's what brings >> them business. A poorly performing CDN will lose customers to a better >> performing one. The trend for years has been instead of ISPs investing in >> infrastructure to effectively handle the traffic that their users request, >> they turf that to CDNs. In many cases, a CDN will put a cache box in or >> extend a circuit at a loss to them, because they know if the performance >> metrics get bad, business will be taken elsewhere, even if the CAUSE of the >> poor performance is actually at the edge of, or inside , the ISPs network. >> >> ISPs in the US can get away with this because their users are captive and >> rarely have an alternative choice of provider. >> >> >> On Thu, Apr 1, 2021 at 4:33 PM Matt Erculiani <merculi...@gmail.com >> <mailto:merculi...@gmail.com>> wrote: >> Patrick, >> >> > First, to be blunt, if you really think Akamai nodes are “sitting idle for >> > weeks” before CoD comes out with a new game, >> > you are clearly confused. >> >> "Idle" in the sense that when you look at a graph of traffic before and >> after a large push such as this makes the rest of the week's traffic look >> like a horizontal line at the bottom, admittedly poor word choice, yes, but >> far from "confused" as to what CDNs do under relatively normal >> circumstances. Otherwise very valid points you've raised. >> >> Tom, >> >> > Akamai, and other CDNs, do not **generate** traffic ; they serve the >> > requests generated by users. >> >> A user sends a few megabytes of request and receives 50 gigs of reply. They >> aren't DDoSing the network, but they're amplifying a single 50 gig copy they >> receive from the mothership and turning it into likely tens of terabytes of >> traffic. >> Yes, that's a CDN's job, but that volume of legitimate traffic and the very >> tiny window with which it is transmitted is likely to be a burden for even >> the largest residential ISPs. >> >> -Matt >> >> On Thu, Apr 1, 2021 at 2:09 PM Patrick W. Gilmore <patr...@ianai.net >> <mailto:patr...@ianai.net>> wrote: >> Matt: >> >> I am going to disagree with your characterization of how Akamai - and many >> other CDNs - manage things. First, to be blunt, if you really think Akamai >> nodes are “sitting idle for weeks” before CoD comes out with a new game, you >> are clearly confused. >> >> More importantly, I know for a fact Akamai has spent ungodly amounts of >> money & resources putting content precisely where the ISPs ask them to put >> it, deliver it over the pipes the ISPs ask them to deliver it, at precisely >> the capacity the ISPs tell them. >> >> On the other hand, I agree with your characterization of residential >> broadband. It is ridiculous to expect a neighborhood with 1,000 homes each >> with 1 Gbps links to have a terabit of uplink capacity. But it also should >> have a lot more than 10 Gbps, IMHO. Unfortunately, most neighborhoods I have >> seen are closer to the latter than the former. >> >> Finally, this could quickly devolve into finger pointing. You say the CDNs >> bear some responsibility? They may well respond that the large broadband >> providers ask for cash to interconnect - but still require the CDNs to do >> all the work. The CDNs did not create the content, or tell the users which >> content to pull. When I pay $NATIONAL_PROVIDER, I expect them to provide me >> with access to the Internet. Not just to the content that pays that provider. >> >> Personally, I have zero problems with the ISPs saying “give me a cache to >> put here with this sized uplink” or “please deliver to these users over this >> xconn / IX / whatever”. I have a huge problem with the ISPs blaming the ISPs >> for delivering what the ISP’s users request. >> >> Of course, this could all be solved if there were more competition in >> broadband in the US (and many other countries). But that is a totally >> different 10,000 post thread (that we have had many dozens of times). >> >> -- >> TTFN, >> patrick >> >>> On Apr 1, 2021, at 3:53 PM, Matt Erculiani <merculi...@gmail.com >>> <mailto:merculi...@gmail.com>> wrote: >>> >>> Niels, >>> >>> I think to clarify Jean's point, when you buy a 300mbps circuit, you're >>> paying for 300mbps of internet access. >>> >>> That does not mean that a network should (and in this case small-medium >>> ones simply can't) build all of their capacity to service a large number of >>> customer circuits at line rate at the same time for an extended period, >>> ESPECIALLY to the exact same endpoint. It's just not economically >>> reasonable to expect that. Remember we're talking about residential service >>> here, not enterprise circuits. >>> >>> Therefore, how do you prevent this spike of [insert large number here] >>> gigabits traversing the network at the same time from causing issues? Build >>> more network? That sounds easy, but there are plenty of legitimate reasons >>> why ISPs can't or don't want to do that, particularly for an event that >>> only occurs once per quarter or so. >>> >>> Does Akamai bear some burden here to make these rollouts less troublesome >>> for the ISPs they traverse through the last mile(s)? IMO yes, yes they do. >>> When you're doing something new and unprecedented, as Akamai frequently >>> brags about on Twitter, like having rapid, bursty growth of traffic, you >>> need to consider that just because you can generate it, doesn't mean it can >>> be delivered. They've gotta be more sophisticated than a bunch of servers >>> with SSD arrays, ramdisks, and 100 gig interfaces, so there's no excuse for >>> them here to just blindly fill every link they have after sitting idle for >>> weeks/months at a time and expect everything to come out alright and nobody >>> to complain about it. >>> >>> On Thu, Apr 1, 2021 at 1:21 PM Niels Bakker <niels=na...@bakker.net >>> <mailto:na...@bakker.net>> wrote: >>> * nanog@nanog.org <mailto:nanog@nanog.org> (Jean St-Laurent via NANOG) [Thu >>> 01 Apr 2021, 21:03 CEST]: >>> >An artificial roll out penalty somehow? Probably not at the ISP >>> >level, but more at the game level. Well, ISP could also have some >>> >mechanisms to reduce the impact or even Akamai could force a >>> >progressive roll out. >>> >>> It's an online game. You can't play the game with outdated assets. >>> You'd not see walls where other players would, for example. >>> >>> What you're suggesting is the ability of ISPs to market Internet access >>> at a certain speed but not have to deliver it based on conditions they >>> create. >>> >>> >>> -- Niels. >>> >>> >>> -- >>> Matt Erculiani >>> ERCUL-ARIN >> >> >> >> -- >> Matt Erculiani >> ERCUL-ARIN >> >> >> -- >> Matt Erculiani >> ERCUL-ARIN > > > > -- > Matt Erculiani > ERCUL-ARIN
