subject:"Re\: Observations of an Internet Middleman \(Level3\) \(was\: RIP"

On Friday, May 16, 2014 08:47:53 PM Blake Hudson wrote:

 How residential ISPs recoup costs (or simply increase
 revenue/profit) is another question entirely. I think
 the most insightful comment in this discussion was made
 by Mr. Rick Astley (I assume a pseudonym), when he
 states that ISPs have several options to increase
 revenue A) Increase price of their product, B) Implement
 usage restrictions, or C) Charge someone else/Make
 someone else your customer. I think he successfully
 argues that option C may be the best. As we've seen, the
 wireless market in the US went for option B. We've yet
 to see where the wireline market will go.

Some of the operators, here in Africa, who are venturing 
into FTTH, are continuing on with their data caps.

I suppose if you're primarily a mobile carrier who uses data 
caps to charge for access (and makes lots of money in the 
process for, pretty much, selling nothing), the model 
becomes attractive regardless of the medium.

New providers who are not encumbered by this type of 
thinking, obviously, have a more flexible and forward-
thinking business model.

Mark.


signature.asc
Description: This is a digitally signed message part.

Re: Observations of an Internet Middleman (Level3) (was: RIP

On Friday, May 16, 2014 08:52:31 PM Christopher Morrow 
wrote:

 is 'symmetric traffic ratios' even relevant though?
 Peering is about offsetting costs, right? it might not
 be important that the ratio be 1:1 or 2:1... or even
 10:1, if it's going to cost you 20x to get the traffic
 over longer/transit/etc paths... or if you have to build
 into some horrific location(s) to access the content in
 question.
 
 Harping on symmetric ratios seems very 1990... and not
 particularly germaine to the conversation at hand.

Agree.

We don't have a ratio requirement, for example. We have a 
if it makes sense requirement.

I'm forced to peer with certain African providers in London 
and Amsterdam because they don't want to peer in Africa, 
where we are literally are an x-connect away from each 
other. And the reasons are not even because either of us is 
larger or smaller than the other... it's just legacy 
thinking and we're the new guy that has grown rapidly.

Now we both have to pay for traffic to get sent to Europe 
and back. How nice...

Mark.


signature.asc
Description: This is a digitally signed message part.

Re: Observations of an Internet Middleman (Level3) (was: RIP

On Friday, May 16, 2014 09:11:56 PM Blake Hudson wrote:

 But hey, why peer at little or no cost if they
 can instead hold out and possibly peer at a negative
 cost?

Because they hope that, one day, you'll cave and become a 
customer. Isn't that more prestigious :-)?

Mark.


signature.asc
Description: This is a digitally signed message part.

Re: Observations of an Internet Middleman (Level3) (was: RIP

On Friday, May 16, 2014 09:44:55 PM Scott Helms wrote:

  I don't think that anyone disputes that when you improve
 the upstream you do get an uptick in usage in that
 direction.  What I take issue with is the notion that
 the upstream is anything like downstream even when the
 capacity is there.

Certainly not - what I'm saying is that there can be a lot 
more upstream utilization than we are typically seeing 
today, if that stopper is unblocked.

We can, then, take it from there...

Mark.


signature.asc
Description: This is a digitally signed message part.

Re: Observations of an Internet Middleman (Level3) (was: RIP

2014-05-16 Thread Jean-Francois Mezei

On 14-05-15 16:17, Keenan Tims wrote:

 As primarily an eyeball network with a token (8000 quoted) number of transit 
 customers it does not seem reasonable for them to expect balanced ratios on 
 peering links. 


Pardon my ignorance here, but isn't there a massive difference between
settlement-free peering between large transit providers at the core
which happens with balanced traffic,

 and some free peering at local exchanges at the edge where there is no
expectation of balanced traffic, just an oppportunity to exchange
traffic without using transit capacity. (isn't that how CDN nodes in a
exchange works ? Lets ISPs connect to it and bypass transit links to
save money ?

Seems to me like the word peering shouldn't have been used to denote
relationships at the core between the big guys if it is also used at the
edge for a fairly different purpose.

Re: Observations of an Internet Middleman (Level3) (was: RIP

On Thursday, May 15, 2014 09:05:57 PM Joe Greco wrote:

 Hi I'm an Internet company.  I don't actually know what
 the next big thing next year will be but I promise that
 I won't host it on my network and cause our traffic to
 become lopsided.

You mean like almost every other mobile carrier the world 
over, and their data-capped services?

Want to guess how many mobile carrier executives converge 
around a table on a daily basis to discuss how to stem 
growth in demand for traffic by their customers?

Mark.


signature.asc
Description: This is a digitally signed message part.

Re: Observations of an Internet Middleman (Level3) (was: RIP Network Neutrality (was: Wow its been quiet here...

2014-05-16 Thread Rick Astley

What you're missing is that the transit provider is
selling full routes.  The access network is selling
paid peering, which is a tiny fraction of the routes.

Considering they charge on a $per/mb basis I don't think its just routes
they are selling. It looks a lot like they are selling bits. From the
perspective of a content provider it looks like they swivel chair most
those bits to access networks for delivery. That tiny fraction of routes
on the access networks make up most delivered content. In total network
size the access networks are larger although less spread out globally.
Being globally connected is useful but it doesn't make a legitimate case
for having exclusive rights to charge for content being delivered in North
America. If you are planning to serve large scale data over oceanic fiber
it's a strong selling point but that's not the case here. If instead of
$per/mb traffic delivery you want to get into arbitrary justifications
access networks have more directly assigned IP addresses than transit
networks. I'm not making the case that a middle man should never be used,
I'm making the case that they shouldn't be used where there isn't a
requirement for one. Bypassing the middleman is generally better for
everyone but the middle man.


So, at the end of the week, I *had* been paying $10/mb to
send traffic through transit to reach the whole rest of the
internet.  Now, I'm paying $5+$4+$4+$5+$2, or $30, and
I don't have a full set of routes, so I've still got to keep
paying the transit provider as well at $10.

If this is the math you are using to justify your stance it's probably
worth reconsidering. You ignore that each of those if sent through transit
would have been $10 so the cost of $5, $4, and $2/Mb represent a savings of
$5, $6,and $8. Why would you add them? Sure there are factors you have to
evaluate like putting yourself under a minimum commit with $transit or if
the amount of traffic is worth peering over but you would generally have to
make those evaluations for peering anyway. The real difference is the
volume of traffic needed before a $2/mb savings is worth peering directly
for is higher than if the savings were the full $10 but that doesn't mean
its never worth it. There is a difference between saying I did the math
and transit remains the cheaper option and saying Paid peering would save
us both money and improve performance at the same time but we refuse to do
it anyway on principal.

The concept of fair gets brought up a lot when talking about the ability of
a startup to come in to compete against bigger players in the content space
but really what do you think the impact is if the largest established
content providers peer freely and smaller newcomers only have paid options
available for traffic?



Some other things I also want to get to:
On Vi's analogy vs Amazon prime

One major different I think people overlook is overusing Amazon prime would
mean buying too many things from Amazon. Even when you purchase through
companies selling through Amazon they get a cut of the sale and some of
that I assume gets applied to covering any additional shipping costs not
covered by Prime. If Internet traffic used the same model would ISP's
receive a portion of proceeds for ad revenue on places like Youtube or a
percentage of Netflix subscription fees? I'm not making the case that thats
the model that should be used I'm only pointing out that analogies are best
to break things down into simple terms for people but have diminishing
returns in usefulness when getting into details.

The other problem with Vi's analogy is the shipping company delivers to the
driveway of the customer where a more real life scenario would be something
closer to Amazon having a distribution center in that city, and both
Comcast and FedEx are already both sitting idle in the parking lot. Amazon
pays FedEx to give the package to Comcast in the next parking space, who
then drives it to the customers house. Comcast says to Netflix, since I am
the one driving this from the parking lot to the customers house, why not
just pay me instead of paying FedEx more money to just put it on my truck?
Amazon says, but FedEx will deliver the package to France if I tell them
to. Comcast says, but you don't even serve france out of this distribution
center, and I am not asking to be charged for all packages, only the ones I
deliver instead of FedEx. Amazon says, you are right, we have technology to
give your packages directly to you and stuff going to France to Fedex, and
it would be best for both of us to do it, but unless you'll deliver my
packages for free I'm going to keep paying FedEx to just keep loading them
on your truck. Comcast says have at it, there are 5 trucks for FedEx to
load freely now but if you need more you have to compromise with us on a
deal that works better for both of us. Amazon says, when we are done with
you in the media we won't need to compromise.


Government regulation of interconnects

I agree with

Re: Observations of an Internet Middleman (Level3) (was: RIP

2014-05-16 Thread Owen DeLong

All the talk about ratios is a red herring… The real issue boils down to this:

1.  The access (eyeball) networks don’t want to bear the cost of delivering 
what they promised to their customers.
2.  This is because when they built their business models, they didn’t 
expect their customers to use nearly as much
of their promised bandwidth as they are now using. Most of the models 
were constructed around the idea that
a customer receiving, say 27mbps down/7mbps up would use all of that 
bandwidth in short bursts and mostly
use less than a megabit.
3.  New services have been developed (streaming video, et al.) which have 
created an increasing demand from
customers for more of the bandwidth they were sold.
4.  Instead of raising the prices to the access network customers or 
accepting that the lavish profits that they eyeball
networks had been pocketing were no more, the access networks are 
trying to slough off the costs of delivering
that higher fraction of what they sold onto someone else.
5.  The content providers looked like an easy target with the advantage 
that:
A.  Some of them appear to have deep pockets.
B.  They are the competition for many of the access network’s other 
lines of business, so increasing their costs
helps make them less competitive.
C.  Consumers are emotional about price increases. Content 
providers look at it as a business problem and
perform a mathematical analysis. If their customer satisfaction 
impact costs more than paying the extortion
from the access networks, they’ll pay it.

In reality, if the $ACCESS_PROVIDERS wanted to satisfy their customers, they’d 
be aggressively seeking to peer with content providers in as many locations as 
possible. They might (reasonably) require content providers to build out to 
additional locations to keep their long-haul costs down (It’s reasonable, IMHO, 
for a content provider not to want to carry multiple gigabits of traffic from a 
content provider clear across the country for free. If $CONTENT_PROVIDER wants 
to access California customers of $ACCESS_PROVIDER, then it’s reasonable for 
$ACCESS_PROVIDER to insist that $CONTENT_PROVIDER peer in California for 
delivering those bits.)

Neither side of this issue has completely clean hands. Both have been trying to 
take as much of the money on the table for themselves with limited regard for 
serving the consumer. The Access Networks have done a far worse job of serving 
the consumer than the content providers and that’s a big part of what is 
driving the current backlash. As a general rule, access customers don’t select 
the provider they love the most, they select the one they think sucks the least.

I think the recent FCC NPRM is a bit optimistic in that it expects the 
$ACCESS_PROVIDERS to act in good faith. If they do, it will likely turn out to 
be a limited victory for the $ACCESS_PROVIDERS. However, I don’t expect the 
$ACCESS_PROVIDERS to live within that limited victory. Assuming the NRPM 
becomes rule and then withstands the likely legal challenges, I expect they 
will, as usual, play in the gray areas of the ruling as much as they think they 
legally can and push the edges as far as possible to try and extort every 
dollar they can from $CONTENT_PROVIDERS with this so-called fast-lane (which we 
all know is just preferential peering and/or QoS[1] tuning).  I suspect they 
will likely push this far enough that over the next several years, things will 
get progressively worse until the FCC finally decides that they have to move 
from section 706 to Title II.

OTOH, if I’m wrong and the $ACCESS_PROVIDERS suddenly start behaving like 
civilized companies, develop a sudden concern for their customers’ experiences, 
and start unimaginably acting in good faith, the proposed rule wouldn’t be so 
bad for $CONTENT_PROVIDERS, $CONSUMERS, or $ACCESS_PROVIDERS. 

Of course, you can already see the $ACCESS_PROVIDERS laying the groundwork to 
try and mount a legal challenge against the FCC’s authority to use rule 706. 
Sadly, some of this groundwork is being laid by FCC commissioners. Said 
commissioners clearly have no interest in representing the people’s interest 
and are strictly there as mouth-pieces for some of the big players in the 
industry.

Owen

[1] QoS — A deceptive name if ever there was one. QoS is not about Quality of 
Service, it’s about screwing over network users by choice rather than by chance 
when you haven’t built an adequate network.

Re: Observations of an Internet Middleman (Level3) (was: RIP Network Neutrality (was: Wow its been quiet here...

2014-05-16 Thread Owen DeLong


On May 16, 2014, at 3:25 AM, Rick Astley jna...@gmail.com wrote:

 Broadband is too expensive in the US compared to other places
 
 I have seen this repeated so many times that I assume it's true but I have
 never seen anything objective as to why. I can tell you if you look at
 population density by country the US is 182nd in the world and the average
 broadband speed (based on OOKLA:
 http://www.netindex.com/download/allcountries/) is 30th in the world. South
 Korea that is well known for its fast broadband speeds has a density of
 505/km vs the US at 32/km. We have about 1/15 of the population density and
 about 1/2 the average broadband speed. Hong Kong, Singapore, Netherlands,
 Japan, Macau etc. all have more than 10x the population density in the US
 so definitely not all countries with fast broadband make for a fair
 comparison and there are likely fewer that do. The UK is only beating the
 US by 2Mbps but has a population density of 262/km.
 
 So while its a fair assessment that broadband in the US is very bias to
 ignore some of the other factors involved. Another mistake I see people
 keep making is in comparing the cost of broadband in the US in $USD to
 other countries around the world. The cost of broadband in Estonia is only
 about $30/month. OMG, I can't believe broadband is cheaper in Estonia! What
 people ignore is everything is cheaper in Estonia, the average household
 income in Estonia is $14k vs $55k here. By that measure broadband is more
 expensive for families there than it is in the US. This is another point
 people repeat without bothering to qualify. This would be like my
 grandfather comparing the costs of a candy bar from back when he was a kid
 to today but ignoring inflation.

I might be willing to accept this argument if it weren’t for the fact that rural
locations in the US are far more likely to have FTTH than higher density areas
because the whole USF thing has inverted the priorities.

I live in the largest city in the bay area, yet there is only one facilities
based provider in my area that can deliver 2mbps or more and that’s over
HFC. Twisted pair is abysmal and there is no fiber.

The situation is not significantly better in the densest city in the bay area, 
either.

South Korea averages 4x US Speed for an average $28.50/month.
US averages 1x US Speed for an average $45.50/month.
(http://edition.cnn.com/2010/TECH/03/31/broadband.south.korea/)

Korean average annual wage: $36,757 @ 21% tax = $29,038 take-home.
US Average annual wage: $55,048 @ 29.6% tax = $38,753 take-home.
(http://en.wikipedia.org/wiki/List_of_countries_by_average_wage)

So that says KR take-home wage = ~75% of US wage.
75% of $45.50 is $34.125

So 4x speed is still approximately $5 cheaper per month in KR than in the US.

Owen

Re: Observations of an Internet Middleman (Level3) (was: RIP

On Friday, May 16, 2014 03:54:33 PM Owen DeLong wrote:

 customers. 2. This is because when they built their
 business models, they didn’t expect their customers to
 use nearly as much of their promised bandwidth as they
 are now using. Most of the models were constructed
 around the idea that a customer receiving, say 27mbps
 down/7mbps up would use all of that bandwidth in short
 bursts and mostly use less than a megabit.

And in general, models have assumed, for a long time, that 
customer demand patterns are largely asymmetric.

While that is true a lot of the time (especially for eyeball 
networks), it is less so now due to social media. Social 
media forces the use of symmetric bandwidth (like FTTH), 
putting even more demand on the network, and making the gist 
of this thread an even bigger issue, if you discount the 
fact, of course, that Broadband in the U.S. currently sucks 
for a developed market.

Mark.


signature.asc
Description: This is a digitally signed message part.

Re: Observations of an Internet Middleman (Level3) (was: RIP

Social media is not a big driver of symmetrical traffic here in the US or
internationally.  Broadband suffers here for a number of reasons, mainly
topological and population density, in comparison to places like Japan,
parts (but certainly not all) of Europe, and South Korea.


Scott Helms
Vice President of Technology
ZCorum
(678) 507-5000

http://twitter.com/kscotthelms



On Fri, May 16, 2014 at 11:02 AM, Mark Tinka mark.ti...@seacom.mu wrote:

 On Friday, May 16, 2014 03:54:33 PM Owen DeLong wrote:

  customers. 2. This is because when they built their
  business models, they didn’t expect their customers to
  use nearly as much of their promised bandwidth as they
  are now using. Most of the models were constructed
  around the idea that a customer receiving, say 27mbps
  down/7mbps up would use all of that bandwidth in short
  bursts and mostly use less than a megabit.

 And in general, models have assumed, for a long time, that
 customer demand patterns are largely asymmetric.

 While that is true a lot of the time (especially for eyeball
 networks), it is less so now due to social media. Social
 media forces the use of symmetric bandwidth (like FTTH),
 putting even more demand on the network, and making the gist
 of this thread an even bigger issue, if you discount the
 fact, of course, that Broadband in the U.S. currently sucks
 for a developed market.

 Mark.

Re: Observations of an Internet Middleman (Level3) (was: RIP

On Friday, May 16, 2014 05:08:33 PM Scott Helms wrote:

 Social media is not a big driver of symmetrical traffic
 here in the US or internationally.  Broadband suffers
 here for a number of reasons, mainly topological and
 population density, in comparison to places like Japan,
 parts (but certainly not all) of Europe, and South
 Korea.

It might not be (now), but if symmetrical bandwidth will go 
in on the back of teenagers wanting to upload videos about 
their lives, the meer fact that the bandwidth is there means 
someone will find bigger and better use for it, than social 
media.

We saw this when we deployed FTTH in Malaysia, back in '09.

Mark.


signature.asc
Description: This is a digitally signed message part.

Re: Observations of an Internet Middleman (Level3) (was: RIP

2014-05-16 Thread Jay Ashworth

- Original Message -
 From: Mark Tinka mark.ti...@seacom.mu

 While that is true a lot of the time (especially for eyeball
 networks), it is less so now due to social media. Social
 media forces the use of symmetric bandwidth (like FTTH),
 putting even more demand on the network, 

Oh yes; clearly, Twitter will be the end of L3.

:-)

Could you expand a bit, Mark on Social media forces the use of symmetric
bandwidth?  Which social media platform is it that you think has a)
symmetrical flows that b) are big enough to figure into transit symmetry?

Cheers,
-- jra
-- 
Jay R. Ashworth  Baylink   j...@baylink.com
Designer The Things I Think   RFC 2100
Ashworth  Associates   http://www.bcp38.info  2000 Land Rover DII
St Petersburg FL USA  BCP38: Ask For It By Name!   +1 727 647 1274

Re: Observations of an Internet Middleman (Level3) (was: RIP

Mark,

Bandwidth use trends are actually increasingly asymmetical because of the
popularity of OTT video.

Social media, even with video uploading, simply doesn't generate that much
traffic per session.

During peak period, Real-Time Entertainment traffic is by far the most
dominant traffic category, accounting for almost
half of the downstream bytes on the network. As observed in past reports,
Social Networking applications continue to
be very well represented on the mobile network. This speaks to their
popularity with subscribers as these applications
typically generate far less traffic than those that stream audio and video.

https://www.sandvine.com/downloads/general/global-internet-phenomena/2013/sandvine-global-internet-phenomena-report-1h-2013.pdf

Scott Helms
Vice President of Technology
ZCorum
(678) 507-5000

http://twitter.com/kscotthelms

On Fri, May 16, 2014 at 11:26 AM, Mark Tinka mark.ti...@seacom.mu wrote:

On Friday, May 16, 2014 05:08:33 PM Scott Helms wrote:

Social media is not a big driver of symmetrical traffic
here in the US or internationally. Broadband suffers
here for a number of reasons, mainly topological and
population density, in comparison to places like Japan,
parts (but certainly not all) of Europe, and South
Korea.

It might not be (now), but if symmetrical bandwidth will go
in on the back of teenagers wanting to upload videos about
their lives, the meer fact that the bandwidth is there means
someone will find bigger and better use for it, than social
media.

We saw this when we deployed FTTH in Malaysia, back in '09.

Mark.

Re: Observations of an Internet Middleman (Level3) (was: RIP



Jay Ashworth wrote the following on 5/16/2014 10:35 AM:

- Original Message -

From: Mark Tinka mark.ti...@seacom.mu
While that is true a lot of the time (especially for eyeball
networks), it is less so now due to social media. Social
media forces the use of symmetric bandwidth (like FTTH),
putting even more demand on the network,

Oh yes; clearly, Twitter will be the end of L3.

:-)

Could you expand a bit, Mark on Social media forces the use of symmetric
bandwidth?  Which social media platform is it that you think has a)
symmetrical flows that b) are big enough to figure into transit symmetry?

Cheers,
-- jra
Applications like Skype and Facetime (especially conference calls) would 
be one example where an application benefits from symmetric (or 
asymmetric in favor of higher upload speed) connectivity. Cloud office 
applications like storage of documents, email, and IVR telephony also 
benefit from symmetrical connectivity. Off-site backup software is 
another great example. Most residential connections are ill suited for 
this. I believe these applications (and derivatives) would be more 
popular today if the connectivity was available.


--Blake

Re: Observations of an Internet Middleman (Level3) (was: RIP

Blake,

None of those applications come close to causing symmetrical traffic
patterns and for many/most networks the upstream connectivity has greatly
improved.  Anything related to voice is no more than 80 kbps per line, even
if the SIP traffic isn't trunked (less if it is because the signaling data
is shared).  Document sharing is not being impinged, on my residential
account right now I've uploaded about 30 documents this morning including
large PDFs and Power Point presentations.

Off site back up is one use that could drive traffic, but I don't believe
that the limiting factor is bandwidth.  We looked at getting into that
business and from what we saw the limiting factor was that most residential
and SOHO accounts didn't want to pay enough to cover your storage 
management costs.  In our analysis the impact of bandwidth on the consumer
side adoption was basically zero.  There is no expectation that back ups
run instantly.  Having said all of that, even if hosted back up became
wildly popular would not change the balance of power because OTT video is
both larger, especially for HD streams, and used much more frequently.


Scott Helms
Vice President of Technology
ZCorum
(678) 507-5000

http://twitter.com/kscotthelms



On Fri, May 16, 2014 at 11:53 AM, Blake Hudson bl...@ispn.net wrote:


 Jay Ashworth wrote the following on 5/16/2014 10:35 AM:

 - Original Message -

 From: Mark Tinka mark.ti...@seacom.mu
 While that is true a lot of the time (especially for eyeball
 networks), it is less so now due to social media. Social
 media forces the use of symmetric bandwidth (like FTTH),
 putting even more demand on the network,

 Oh yes; clearly, Twitter will be the end of L3.

 :-)

 Could you expand a bit, Mark on Social media forces the use of symmetric
 bandwidth?  Which social media platform is it that you think has a)
 symmetrical flows that b) are big enough to figure into transit symmetry?

 Cheers,
 -- jra

 Applications like Skype and Facetime (especially conference calls) would
 be one example where an application benefits from symmetric (or asymmetric
 in favor of higher upload speed) connectivity. Cloud office applications
 like storage of documents, email, and IVR telephony also benefit from
 symmetrical connectivity. Off-site backup software is another great
 example. Most residential connections are ill suited for this. I believe
 these applications (and derivatives) would be more popular today if the
 connectivity was available.

 --Blake

Re: Observations of an Internet Middleman (Level3) (was: RIP

Certainly video is one of the most bandwidth intensive applications. I 
don't deny that a  1 Mbps video call is both less common and consumes 
less bandwidth than an 8Mbps HD stream. However, if Americans had access 
to symmetric connections capable of reliably making HD video calls (they 
don't, in my experience), we might be seeing video calls as a common 
occurrence and not a novelty. I think the state of usage is a reflection 
on the technology available.


If the capability was available at an affordable price to residential 
consumers, we might see those consumers stream movies or send videos 
from their home or mobile devices via their internet connection directly 
to the recipient rather than through a centralized source like Disney, 
NetFlix, Youtube, etc. Video sharing sites (like youtube, vimeo, etc) 
primary reason for existence is due to the inability of the site's users 
to distribute content themselves. One of the hurdles to overcome in 
video sharing is the lack of availability in affordable internet 
connectivity that is capable of sending video at reasonable (greater 
than real time) speeds.


--Blake

Scott Helms wrote the following on 5/16/2014 11:02 AM:

Blake,

None of those applications come close to causing symmetrical traffic 
patterns and for many/most networks the upstream connectivity has 
greatly improved.  Anything related to voice is no more than 80 kbps 
per line, even if the SIP traffic isn't trunked (less if it is because 
the signaling data is shared).  Document sharing is not being 
impinged, on my residential account right now I've uploaded about 30 
documents this morning including large PDFs and Power Point presentations.


Off site back up is one use that could drive traffic, but I don't 
believe that the limiting factor is bandwidth.  We looked at getting 
into that business and from what we saw the limiting factor was that 
most residential and SOHO accounts didn't want to pay enough to cover 
your storage  management costs.  In our analysis the impact of 
bandwidth on the consumer side adoption was basically zero.  There is 
no expectation that back ups run instantly.  Having said all of that, 
even if hosted back up became wildly popular would not change the 
balance of power because OTT video is both larger, especially for HD 
streams, and used much more frequently.



Scott Helms
Vice President of Technology
ZCorum
(678) 507-5000

http://twitter.com/kscotthelms



On Fri, May 16, 2014 at 11:53 AM, Blake Hudson bl...@ispn.net 
mailto:bl...@ispn.net wrote:



Jay Ashworth wrote the following on 5/16/2014 10:35 AM:

- Original Message -

From: Mark Tinka mark.ti...@seacom.mu
mailto:mark.ti...@seacom.mu
While that is true a lot of the time (especially for eyeball
networks), it is less so now due to social media. Social
media forces the use of symmetric bandwidth (like FTTH),
putting even more demand on the network,

Oh yes; clearly, Twitter will be the end of L3.

:-)

Could you expand a bit, Mark on Social media forces the use
of symmetric
bandwidth?  Which social media platform is it that you think
has a)
symmetrical flows that b) are big enough to figure into
transit symmetry?

Cheers,
-- jra

Applications like Skype and Facetime (especially conference calls)
would be one example where an application benefits from symmetric
(or asymmetric in favor of higher upload speed) connectivity.
Cloud office applications like storage of documents, email, and
IVR telephony also benefit from symmetrical connectivity. Off-site
backup software is another great example. Most residential
connections are ill suited for this. I believe these applications
(and derivatives) would be more popular today if the connectivity
was available.

--Blake

Re: Observations of an Internet Middleman (Level3) (was: RIP

Blake,

I might agree with your premise if weren't for a couple of items.

1)  Very few consumers are walking around with a HD or 4K camera today.

2)  Most consumers who want to share video wouldn't know how to host it
themselves, which isn't an insurmountable issue but is a big barrier to
entry especially given the number of NAT'ed connections.  I think this is
much more of a problem than available bandwidth.

3)  Most consumers who want to share videos seem to be satisfied with
sharing via one of the cloud services whether that be YouTube (which was
created originally for that use), Vimeo, or one of the other legions of
services like DropBox.

4)  Finally, upstream bandwidth has increased on many/most operators.  I
just ran the FCC's speedtest (mLab not Ookla) and got 22 mbps on my
residential cable internet service.  I subscribe to one of the major MSOs
for a normal residential package.


Scott Helms
Vice President of Technology
ZCorum
(678) 507-5000

http://twitter.com/kscotthelms



On Fri, May 16, 2014 at 12:38 PM, Blake Hudson bl...@ispn.net wrote:

 Certainly video is one of the most bandwidth intensive applications. I
 don't deny that a  1 Mbps video call is both less common and consumes less
 bandwidth than an 8Mbps HD stream. However, if Americans had access to
 symmetric connections capable of reliably making HD video calls (they
 don't, in my experience), we might be seeing video calls as a common
 occurrence and not a novelty. I think the state of usage is a reflection on
 the technology available.

 If the capability was available at an affordable price to residential
 consumers, we might see those consumers stream movies or send videos from
 their home or mobile devices via their internet connection directly to the
 recipient rather than through a centralized source like Disney, NetFlix,
 Youtube, etc. Video sharing sites (like youtube, vimeo, etc) primary reason
 for existence is due to the inability of the site's users to distribute
 content themselves. One of the hurdles to overcome in video sharing is the
 lack of availability in affordable internet connectivity that is capable of
 sending video at reasonable (greater than real time) speeds.

 --Blake

 Scott Helms wrote the following on 5/16/2014 11:02 AM:

 Blake,

 None of those applications come close to causing symmetrical traffic
 patterns and for many/most networks the upstream connectivity has greatly
 improved.  Anything related to voice is no more than 80 kbps per line, even
 if the SIP traffic isn't trunked (less if it is because the signaling data
 is shared).  Document sharing is not being impinged, on my residential
 account right now I've uploaded about 30 documents this morning including
 large PDFs and Power Point presentations.

 Off site back up is one use that could drive traffic, but I don't believe
 that the limiting factor is bandwidth.  We looked at getting into that
 business and from what we saw the limiting factor was that most residential
 and SOHO accounts didn't want to pay enough to cover your storage 
 management costs.  In our analysis the impact of bandwidth on the consumer
 side adoption was basically zero.  There is no expectation that back ups
 run instantly.  Having said all of that, even if hosted back up became
 wildly popular would not change the balance of power because OTT video is
 both larger, especially for HD streams, and used much more frequently.


 Scott Helms
 Vice President of Technology
 ZCorum
 (678) 507-5000
 
 http://twitter.com/kscotthelms
 


 On Fri, May 16, 2014 at 11:53 AM, Blake Hudson bl...@ispn.net mailto:
 bl...@ispn.net wrote:


 Jay Ashworth wrote the following on 5/16/2014 10:35 AM:

 - Original Message -

 From: Mark Tinka mark.ti...@seacom.mu
 mailto:mark.ti...@seacom.mu
 While that is true a lot of the time (especially for eyeball
 networks), it is less so now due to social media. Social
 media forces the use of symmetric bandwidth (like FTTH),
 putting even more demand on the network,

 Oh yes; clearly, Twitter will be the end of L3.

 :-)

 Could you expand a bit, Mark on Social media forces the use
 of symmetric
 bandwidth?  Which social media platform is it that you think
 has a)
 symmetrical flows that b) are big enough to figure into
 transit symmetry?

 Cheers,
 -- jra

 Applications like Skype and Facetime (especially conference calls)
 would be one example where an application benefits from symmetric
 (or asymmetric in favor of higher upload speed) connectivity.
 Cloud office applications like storage of documents, email, and
 IVR telephony also benefit from symmetrical connectivity. Off-site
 backup software is another great example.

Re: Observations of an Internet Middleman (Level3) (was: RIP

Thanks for the insight Scott. I appreciate the experience and point of 
view you're adding to this discussion (not just the responses to me). 
While I might be playing the devil's advocate here a bit, I think one 
could argue each of the points you've made below.


I do feel that general usage patterns are a reflection of the 
technologies that have traditionally been available to consumers. New 
uses and applications would be available to overcome hurdles if the 
technologies had developed to be symmetrical. I'm not saying that the 
asymmetrical choice was a bad one, but it was not without consequences. 
If residential ISPs sell asymmetric connections for decades, how can the 
ISP expect that application developers would not take this into account 
when developing applications? I don't think my application would be very 
successful if it required X Mbps and half of my market did not meet this 
requirement. Of course content/service providers are going to tailor 
their services based around their market.


--Blake

Scott Helms wrote the following on 5/16/2014 12:06 PM:

Blake,

I might agree with your premise if weren't for a couple of items.

1)  Very few consumers are walking around with a HD or 4K camera today.

2)  Most consumers who want to share video wouldn't know how to host 
it themselves, which isn't an insurmountable issue but is a big 
barrier to entry especially given the number of NAT'ed connections.  I 
think this is much more of a problem than available bandwidth.


3)  Most consumers who want to share videos seem to be satisfied with 
sharing via one of the cloud services whether that be YouTube (which 
was created originally for that use), Vimeo, or one of the other 
legions of services like DropBox.


4)  Finally, upstream bandwidth has increased on many/most operators. 
 I just ran the FCC's speedtest (mLab not Ookla) and got 22 mbps on my 
residential cable internet service.  I subscribe to one of the major 
MSOs for a normal residential package.



Scott Helms
Vice President of Technology
ZCorum
(678) 507-5000

http://twitter.com/kscotthelms



On Fri, May 16, 2014 at 12:38 PM, Blake Hudson bl...@ispn.net 
mailto:bl...@ispn.net wrote:


Certainly video is one of the most bandwidth intensive
applications. I don't deny that a  1 Mbps video call is both less
common and consumes less bandwidth than an 8Mbps HD stream.
However, if Americans had access to symmetric connections capable
of reliably making HD video calls (they don't, in my experience),
we might be seeing video calls as a common occurrence and not a
novelty. I think the state of usage is a reflection on the
technology available.

If the capability was available at an affordable price to
residential consumers, we might see those consumers stream movies
or send videos from their home or mobile devices via their
internet connection directly to the recipient rather than through
a centralized source like Disney, NetFlix, Youtube, etc. Video
sharing sites (like youtube, vimeo, etc) primary reason for
existence is due to the inability of the site's users to
distribute content themselves. One of the hurdles to overcome in
video sharing is the lack of availability in affordable internet
connectivity that is capable of sending video at reasonable
(greater than real time) speeds.

--Blake

Scott Helms wrote the following on 5/16/2014 11:02 AM:

Blake,

None of those applications come close to causing symmetrical
traffic patterns and for many/most networks the upstream
connectivity has greatly improved.  Anything related to voice
is no more than 80 kbps per line, even if the SIP traffic
isn't trunked (less if it is because the signaling data is
shared).  Document sharing is not being impinged, on my
residential account right now I've uploaded about 30 documents
this morning including large PDFs and Power Point presentations.

Off site back up is one use that could drive traffic, but I
don't believe that the limiting factor is bandwidth.  We
looked at getting into that business and from what we saw the
limiting factor was that most residential and SOHO accounts
didn't want to pay enough to cover your storage  management
costs.  In our analysis the impact of bandwidth on the
consumer side adoption was basically zero.  There is no
expectation that back ups run instantly.  Having said all of
that, even if hosted back up became wildly popular would not
change the balance of power because OTT video is both larger,
especially for HD streams, and used much more frequently.


Scott Helms
Vice President of Technology
ZCorum
(678) 507-5000 tel:%28678%29%20507-5000

Re: Observations of an Internet Middleman (Level3) (was: RIP

Blake,

You're absolutely correct.  The world adapts to the reality that we find
ourselves in via normal market mechanics.  The problem with proposing that
connectivity for residential customers should be more symmetrical is that
its expensive, which is why we as operators didn't roll it out that way to
start.  We also don't see consumer demand for symmetrical connections and
with the decline in peer to peer file sharing we've actually seen a
decrease the ratio of used upstream bandwidth (though not a decrease in
absolute terms).

I would like to deliver symmetrical bandwidth to all consumers just so
those few customers who need it today would have lower bills but trying to
justify that to our CFO without being able to point to an increase in
revenue either because of more revenue per sub or more subs is a very tough
task.  I don't believe my situation is uncommon.


Scott Helms
Vice President of Technology
ZCorum
(678) 507-5000

http://twitter.com/kscotthelms



On Fri, May 16, 2014 at 1:20 PM, Blake Hudson bl...@ispn.net wrote:

 Thanks for the insight Scott. I appreciate the experience and point of
 view you're adding to this discussion (not just the responses to me). While
 I might be playing the devil's advocate here a bit, I think one could argue
 each of the points you've made below.

 I do feel that general usage patterns are a reflection of the technologies
 that have traditionally been available to consumers. New uses and
 applications would be available to overcome hurdles if the technologies had
 developed to be symmetrical. I'm not saying that the asymmetrical choice
 was a bad one, but it was not without consequences. If residential ISPs
 sell asymmetric connections for decades, how can the ISP expect that
 application developers would not take this into account when developing
 applications? I don't think my application would be very successful if it
 required X Mbps and half of my market did not meet this requirement. Of
 course content/service providers are going to tailor their services based
 around their market.

 --Blake

 Scott Helms wrote the following on 5/16/2014 12:06 PM:

 Blake,

 I might agree with your premise if weren't for a couple of items.

 1)  Very few consumers are walking around with a HD or 4K camera today.

 2)  Most consumers who want to share video wouldn't know how to host it
 themselves, which isn't an insurmountable issue but is a big barrier to
 entry especially given the number of NAT'ed connections.  I think this is
 much more of a problem than available bandwidth.

 3)  Most consumers who want to share videos seem to be satisfied with
 sharing via one of the cloud services whether that be YouTube (which was
 created originally for that use), Vimeo, or one of the other legions of
 services like DropBox.

 4)  Finally, upstream bandwidth has increased on many/most operators.  I
 just ran the FCC's speedtest (mLab not Ookla) and got 22 mbps on my
 residential cable internet service.  I subscribe to one of the major MSOs
 for a normal residential package.


 Scott Helms
 Vice President of Technology
 ZCorum
 (678) 507-5000
 
 http://twitter.com/kscotthelms
 


 On Fri, May 16, 2014 at 12:38 PM, Blake Hudson bl...@ispn.net mailto:
 bl...@ispn.net wrote:

 Certainly video is one of the most bandwidth intensive
 applications. I don't deny that a  1 Mbps video call is both less
 common and consumes less bandwidth than an 8Mbps HD stream.
 However, if Americans had access to symmetric connections capable
 of reliably making HD video calls (they don't, in my experience),
 we might be seeing video calls as a common occurrence and not a
 novelty. I think the state of usage is a reflection on the
 technology available.

 If the capability was available at an affordable price to
 residential consumers, we might see those consumers stream movies
 or send videos from their home or mobile devices via their
 internet connection directly to the recipient rather than through
 a centralized source like Disney, NetFlix, Youtube, etc. Video
 sharing sites (like youtube, vimeo, etc) primary reason for
 existence is due to the inability of the site's users to
 distribute content themselves. One of the hurdles to overcome in
 video sharing is the lack of availability in affordable internet
 connectivity that is capable of sending video at reasonable
 (greater than real time) speeds.

 --Blake

 Scott Helms wrote the following on 5/16/2014 11:02 AM:

 Blake,

 None of those applications come close to causing symmetrical
 traffic patterns and for many/most networks the upstream
 connectivity has greatly improved.  Anything related to voice
 is no more than 80 kbps per line, even if the SIP traffic
 isn't trunked (less if it is

Re: Observations of an Internet Middleman (Level3) (was: RIP

Oh, I'm not proposing symmetrical connectivity at all. I'm just 
supporting the argument that in the context of this discussion I think 
it's silly for a residential ISP to purport themselves to be a neutral 
carrier of traffic and expect peering ratios to be symmetric when the 
overwhelming majority of what they're selling (and have been selling for 
over a decade) is asymmetric connectivity. Their traffic imbalance is, 
arguably, their own doing.


How residential ISPs recoup costs (or simply increase revenue/profit) is 
another question entirely. I think the most insightful comment in this 
discussion was made by Mr. Rick Astley (I assume a pseudonym), when he 
states that ISPs have several options to increase revenue A) Increase 
price of their product, B) Implement usage restrictions, or C) Charge 
someone else/Make someone else your customer. I think he successfully 
argues that option C may be the best. As we've seen, the wireless market 
in the US went for option B. We've yet to see where the wireline market 
will go.


Of course, the market would ideally keep ISPs' demands for 
revenue/profit in check and we'd all reach a satisfactory solution. One 
of the arguments, one I happen to support, in this thread is that there 
is not a free market for internet connectivity in many parts of the US. 
If there was, I believe Comcast would be focusing on how to provide a 
balance between the best product at the lowest cost and not on how they 
can monetize their paying customers in order to increase profits. I 
appreciate honesty; When a service provider advertises X Mbps Internet 
speeds, I expect they can deliver on their claims (to the whole 
Internet, and not just the portions of it they've decided). I understand 
congestion, overselling, etc. But choosing which portions of the 
internet work well and which don't is a lot more like censorship than 
service.


--Blake

Scott Helms wrote the following on 5/16/2014 12:39 PM:

Blake,

You're absolutely correct.  The world adapts to the reality that we 
find ourselves in via normal market mechanics.  The problem with 
proposing that connectivity for residential customers should be more 
symmetrical is that its expensive, which is why we as operators didn't 
roll it out that way to start.  We also don't see consumer demand for 
symmetrical connections and with the decline in peer to peer file 
sharing we've actually seen a decrease the ratio of used upstream 
bandwidth (though not a decrease in absolute terms).


I would like to deliver symmetrical bandwidth to all consumers just so 
those few customers who need it today would have lower bills but 
trying to justify that to our CFO without being able to point to an 
increase in revenue either because of more revenue per sub or more 
subs is a very tough task.  I don't believe my situation is uncommon.



Scott Helms
Vice President of Technology
ZCorum
(678) 507-5000

http://twitter.com/kscotthelms



On Fri, May 16, 2014 at 1:20 PM, Blake Hudson bl...@ispn.net 
mailto:bl...@ispn.net wrote:


Thanks for the insight Scott. I appreciate the experience and
point of view you're adding to this discussion (not just the
responses to me). While I might be playing the devil's advocate
here a bit, I think one could argue each of the points you've made
below.

I do feel that general usage patterns are a reflection of the
technologies that have traditionally been available to consumers.
New uses and applications would be available to overcome hurdles
if the technologies had developed to be symmetrical. I'm not
saying that the asymmetrical choice was a bad one, but it was not
without consequences. If residential ISPs sell asymmetric
connections for decades, how can the ISP expect that application
developers would not take this into account when developing
applications? I don't think my application would be very
successful if it required X Mbps and half of my market did not
meet this requirement. Of course content/service providers are
going to tailor their services based around their market.

--Blake

Scott Helms wrote the following on 5/16/2014 12:06 PM:

Blake,

I might agree with your premise if weren't for a couple of items.

1)  Very few consumers are walking around with a HD or 4K
camera today.

2)  Most consumers who want to share video wouldn't know how
to host it themselves, which isn't an insurmountable issue but
is a big barrier to entry especially given the number of
NAT'ed connections.  I think this is much more of a problem
than available bandwidth.

3)  Most consumers who want to share videos seem to be
satisfied with sharing via one of the cloud services whether
that be YouTube (which was created originally for that use),
Vimeo, or one of the other legions of

Re: Observations of an Internet Middleman (Level3) (was: RIP

2014-05-16 Thread Christopher Morrow

On Fri, May 16, 2014 at 2:47 PM, Blake Hudson bl...@ispn.net wrote:
 in the context of this discussion I think it's silly for a residential ISP
 to purport themselves to be a neutral carrier of traffic and expect peering
 ratios to be symmetric

is 'symmetric traffic ratios' even relevant though? Peering is about
offsetting costs, right? it might not be important that the ratio be
1:1 or 2:1... or even 10:1, if it's going to cost you 20x to get the
traffic over longer/transit/etc paths... or if you have to build into
some horrific location(s) to access the content in question.

Harping on symmetric ratios seems very 1990... and not particularly
germaine to the conversation at hand.

Re: Observations of an Internet Middleman (Level3) (was: RIP

Blake,

I'm not sure what the relationship between what an access network sells has
to do with how their peering is done.  I realize that everyone's favorite
target is Comcast right now, but would anyone bat an eye over ATT making
the same requirement since they have much more in the way of transit
traffic?  I don't think anyone forced Level 3 into their peering agreement
with Comcast and it was (roughly) symmetrical for years before Level 3 was
contracted by Netflix.  Shouldn't Level 3 gone to Comcast and told them
they needed to change their peering or get a different contract?  Why was
Cogent able to maintain (roughly) symmetrical traffic with Comcast when
they were the primary path for Netflix to Comcast users?

Scott Helms
Vice President of Technology
ZCorum
(678) 507-5000

http://twitter.com/kscotthelms



On Fri, May 16, 2014 at 2:47 PM, Blake Hudson bl...@ispn.net wrote:

 Oh, I'm not proposing symmetrical connectivity at all. I'm just supporting
 the argument that in the context of this discussion I think it's silly for
 a residential ISP to purport themselves to be a neutral carrier of traffic
 and expect peering ratios to be symmetric when the overwhelming majority of
 what they're selling (and have been selling for over a decade) is
 asymmetric connectivity. Their traffic imbalance is, arguably, their own
 doing.

 How residential ISPs recoup costs (or simply increase revenue/profit) is
 another question entirely. I think the most insightful comment in this
 discussion was made by Mr. Rick Astley (I assume a pseudonym), when he
 states that ISPs have several options to increase revenue A) Increase price
 of their product, B) Implement usage restrictions, or C) Charge someone
 else/Make someone else your customer. I think he successfully argues that
 option C may be the best. As we've seen, the wireless market in the US went
 for option B. We've yet to see where the wireline market will go.

 Of course, the market would ideally keep ISPs' demands for revenue/profit
 in check and we'd all reach a satisfactory solution. One of the arguments,
 one I happen to support, in this thread is that there is not a free market
 for internet connectivity in many parts of the US. If there was, I believe
 Comcast would be focusing on how to provide a balance between the best
 product at the lowest cost and not on how they can monetize their paying
 customers in order to increase profits. I appreciate honesty; When a
 service provider advertises X Mbps Internet speeds, I expect they can
 deliver on their claims (to the whole Internet, and not just the portions
 of it they've decided). I understand congestion, overselling, etc. But
 choosing which portions of the internet work well and which don't is a lot
 more like censorship than service.

 --Blake

Re: Observations of an Internet Middleman (Level3) (was: RIP



Christopher Morrow wrote the following on 5/16/2014 1:52 PM:

On Fri, May 16, 2014 at 2:47 PM, Blake Hudson bl...@ispn.net wrote:

in the context of this discussion I think it's silly for a residential ISP
to purport themselves to be a neutral carrier of traffic and expect peering
ratios to be symmetric

is 'symmetric traffic ratios' even relevant though? Peering is about
offsetting costs, right? it might not be important that the ratio be
1:1 or 2:1... or even 10:1, if it's going to cost you 20x to get the
traffic over longer/transit/etc paths... or if you have to build into
some horrific location(s) to access the content in question.

Harping on symmetric ratios seems very 1990... and not particularly
germaine to the conversation at hand.
I agree about the term being passe ...and that it never applied to ISPs 
...and that peering is about cost reduction, reliability, and 
performance. It seems to me that many CDNs or content providers want to 
setup peering relationships and are willing to do so at a cost to them 
in order to bypass the internet middle men. But I mention traffic 
ratios because some folks in this discussion seem to be using it as 
justification for not peering. But hey, why peer at little or no cost if 
they can instead hold out and possibly peer at a negative cost?


--Blake

Re: Observations of an Internet Middleman (Level3) (was: RIP

2014-05-16 Thread James R Cutler


All this talk about symmetry and asymmetry is interesting.  

Has anyone actually quantified how much congestion is due to buffer bloat which 
is, in turn, exacerbated by asymmetric connections?


James R. Cutler
james.cut...@consultant.com
PGP keys at http://pgp.mit.edu





signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Observations of an Internet Middleman (Level3) (was: RIP

2014-05-16 Thread Matthew Petach

On Fri, May 16, 2014 at 11:52 AM, Christopher Morrow 
morrowc.li...@gmail.com wrote:

 On Fri, May 16, 2014 at 2:47 PM, Blake Hudson bl...@ispn.net wrote:
  in the context of this discussion I think it's silly for a residential
 ISP
  to purport themselves to be a neutral carrier of traffic and expect
 peering
  ratios to be symmetric

 is 'symmetric traffic ratios' even relevant though? Peering is about
 offsetting costs, right? it might not be important that the ratio be
 1:1 or 2:1... or even 10:1, if it's going to cost you 20x to get the
 traffic over longer/transit/etc paths... or if you have to build into
 some horrific location(s) to access the content in question.

 Harping on symmetric ratios seems very 1990... and not particularly
 germaine to the conversation at hand.


Traffic asymmetry across peering connections
was what lit the fuse on this whole powder keg,
if I understand correctly; at the point the traffic
went asymmetric, the refusals to augment
capacity kicked in, and congestion became
a problem.

I've seen the same thing; pretty much every
rejection is based on ratio issues, even when
offering to cold-potato haul the traffic to the
home market for the users.

If the refusals hinged on any other clause
of the peering requirements, you'd be right;
but at the moment, that's the flag networks
are waving around as their speedbump-du-jour.
So, it may be very 1990, but unfortunately
that seems to be the year many people in
the industry are mentally stuck in.  :(

Matt

Re: Observations of an Internet Middleman (Level3) (was: RIP

2014-05-16 Thread Matthew Petach

On Fri, May 16, 2014 at 12:14 PM, James R Cutler 
james.cut...@consultant.com wrote:


 All this talk about symmetry and asymmetry is interesting.

 Has anyone actually quantified how much congestion is due to buffer bloat
 which is, in turn, exacerbated by asymmetric connections?


 James R. Cutler
 james.cut...@consultant.com
 PGP keys at http://pgp.mit.edu


I think you might have the cart before the horse.

If there's no congestion on a peering link,
buffering doesn't come into play, at least
not within the transport infrastructure.

We're not talking congestion on the last mile
side, we're looking at congestion on the
interconnect links between networks,
typically 10G or 100G ports.  Unless
you're running those links near or at
capacity, buffering should be a complete
non-issue.  And if you're running those
links at capacity, then the congestion
is due to too much traffic, period, not
to the size of buffers involved on either
side of the link.  ^_^;

Thanks!

Matt

Re: Observations of an Internet Middleman (Level3) (was: RIP

Matthew,

There is a difference between what should be philosophically and what
happened with Level 3 which is a contractual issue.


Scott Helms
Vice President of Technology
ZCorum
(678) 507-5000

http://twitter.com/kscotthelms



On Fri, May 16, 2014 at 3:15 PM, Matthew Petach mpet...@netflight.comwrote:

 On Fri, May 16, 2014 at 11:52 AM, Christopher Morrow 
 morrowc.li...@gmail.com wrote:

  On Fri, May 16, 2014 at 2:47 PM, Blake Hudson bl...@ispn.net wrote:
   in the context of this discussion I think it's silly for a residential
  ISP
   to purport themselves to be a neutral carrier of traffic and expect
  peering
   ratios to be symmetric
 
  is 'symmetric traffic ratios' even relevant though? Peering is about
  offsetting costs, right? it might not be important that the ratio be
  1:1 or 2:1... or even 10:1, if it's going to cost you 20x to get the
  traffic over longer/transit/etc paths... or if you have to build into
  some horrific location(s) to access the content in question.
 
  Harping on symmetric ratios seems very 1990... and not particularly
  germaine to the conversation at hand.
 
 
 Traffic asymmetry across peering connections
 was what lit the fuse on this whole powder keg,
 if I understand correctly; at the point the traffic
 went asymmetric, the refusals to augment
 capacity kicked in, and congestion became
 a problem.

 I've seen the same thing; pretty much every
 rejection is based on ratio issues, even when
 offering to cold-potato haul the traffic to the
 home market for the users.

 If the refusals hinged on any other clause
 of the peering requirements, you'd be right;
 but at the moment, that's the flag networks
 are waving around as their speedbump-du-jour.
 So, it may be very 1990, but unfortunately
 that seems to be the year many people in
 the industry are mentally stuck in.  :(

 Matt

Re: Observations of an Internet Middleman (Level3) (was: RIP

On Friday, May 16, 2014 05:35:39 PM Jay Ashworth wrote:

 Could you expand a bit, Mark on Social media forces the
 use of symmetric bandwidth?  Which social media
 platform is it that you think has a) symmetrical flows
 that b) are big enough to figure into transit symmetry?

What we saw with FTTH deployments is that customers uploaded 
more videos and photos to Youtube, Facebook, MySpace, e.t.c. 
They didn't do this on ADSL as much (it's too frustrating).

When that caught on, customers started buying online backup 
services - synchronizing backups of their home or office 
computers to remote backup infrastructure. Again, they never 
did this with ADSL.

What we learned: don't take it for granted that you will 
always know what your customers (or the content providers 
who serve them) will do with the bandwidth. If they have it, 
expect the worst, and plan for it as best you can.

Mark.


signature.asc
Description: This is a digitally signed message part.

Re: Observations of an Internet Middleman (Level3) (was: RIP

2014-05-16 Thread Christopher Morrow

On Fri, May 16, 2014 at 3:15 PM, Matthew Petach mpet...@netflight.com wrote:



 On Fri, May 16, 2014 at 11:52 AM, Christopher Morrow
 morrowc.li...@gmail.com wrote:

 On Fri, May 16, 2014 at 2:47 PM, Blake Hudson bl...@ispn.net wrote:
  in the context of this discussion I think it's silly for a residential
  ISP
  to purport themselves to be a neutral carrier of traffic and expect
  peering
  ratios to be symmetric

 is 'symmetric traffic ratios' even relevant though? Peering is about
 offsetting costs, right? it might not be important that the ratio be
 1:1 or 2:1... or even 10:1, if it's going to cost you 20x to get the
 traffic over longer/transit/etc paths... or if you have to build into
 some horrific location(s) to access the content in question.

 Harping on symmetric ratios seems very 1990... and not particularly
 germaine to the conversation at hand.


 Traffic asymmetry across peering connections
 was what lit the fuse on this whole powder keg,
 if I understand correctly; at the point the traffic
 went asymmetric, the refusals to augment
 capacity kicked in, and congestion became
 a problem.

Is it that? or is it that planning at some ISP pair had a '6 months to
upgrade' regularly penciled in, then 'all of a sudden' their links
were filling up faster than every 6months and... now they are 1x or 2x
upgrade cycles behind?

I imagine that up to a point upgrading a router that does only
'peering' (SFP) is 'easy', but at some step function of upgrades on
the edge ports you need to provision more backhaul and more core and
probably upgrade the link types and the chassis and ...

At some ISPs this process involves more than 1 dude/group. So
coordination and budget issues and scheduling ... become a bit harder.
Adjusting to the new reality of 'you need to plan for pipe filling
more often, increase upgrade cycle crank speed!' seems like at least
one problem, to me at least.

It's really hard to tell what's upsetting people about this whole topic :(
There's a mix of 'my access link blows' to 'isps should peer better
and for freer' and a bunch of other stuff all mixed in the middle :(

 I've seen the same thing; pretty much every
 rejection is based on ratio issues, even when
 offering to cold-potato haul the traffic to the
 home market for the users.

yes, welp... it's often rough to get folk who want to think in terms
of apples to suddenly thing in terms of the new best fruit 'acai
berry'. Especially at large and entrenched organizations.

 If the refusals hinged on any other clause
 of the peering requirements, you'd be right;
 but at the moment, that's the flag networks
 are waving around as their speedbump-du-jour.

sure, it's also super easy for them to do this, see entrenched org
comment above.
it seems to me that the point of peering is not stalin voice'ratio
or bust'/stalin voice but 'mutual benefit'. If a skewed ratio of
100:1 in a local market still is cheaper than 'backhaul that traffic
from LHR to SFO' there's mutual benefit and a reason to peer.

I understand that this is a bit of a rosy landscape I'm painting, but...

 So, it may be very 1990, but unfortunately
 that seems to be the year many people in
 the industry are mentally stuck in.  :(

oh, entrenched. I see.

thanks!
-chris

Re: Observations of an Internet Middleman (Level3) (was: RIP

2014-05-16 Thread Christopher Morrow

On Fri, May 16, 2014 at 3:11 PM, Blake Hudson bl...@ispn.net wrote:

 Christopher Morrow wrote the following on 5/16/2014 1:52 PM:

 On Fri, May 16, 2014 at 2:47 PM, Blake Hudson bl...@ispn.net wrote:

 in the context of this discussion I think it's silly for a residential
 ISP
 to purport themselves to be a neutral carrier of traffic and expect
 peering
 ratios to be symmetric

 is 'symmetric traffic ratios' even relevant though? Peering is about
 offsetting costs, right? it might not be important that the ratio be
 1:1 or 2:1... or even 10:1, if it's going to cost you 20x to get the
 traffic over longer/transit/etc paths... or if you have to build into
 some horrific location(s) to access the content in question.

 Harping on symmetric ratios seems very 1990... and not particularly
 germaine to the conversation at hand.

 I agree about the term being passe ...and that it never applied to ISPs
 ...and that peering is about cost reduction, reliability, and performance.

ok.

 It seems to me that many CDNs or content providers want to setup peering
 relationships and are willing to do so at a cost to them in order to bypass
 the internet middle men. But I mention traffic ratios because some folks

'the internet middle men' - is really, it seems to me, 'people I have
no business relationship with'. There's also no way to control the
capacity planning process with these middle-men, right? Some AS in the
middle of my 3-AS-way conversation isn't someone I can capacity plan
with :(

-chris

 in this discussion seem to be using it as justification for not peering. But
 hey, why peer at little or no cost if they can instead hold out and possibly
 peer at a negative cost?

 --Blake

Re: Observations of an Internet Middleman (Level3) (was: RIP

On Friday, May 16, 2014 05:45:06 PM Scott Helms wrote:

 Bandwidth use trends are actually increasingly
 asymmetical because of the popularity of OTT video.
 
 Social media, even with video uploading, simply doesn't
 generate that much traffic per session.

Our experience showed that there is a direct co-relation 
between the lack of traffic in the upstream direction and 
poor upload bandwidth (primarily, due to asymmetric tech. 
such as ADSL), e.g., because of the ADSL I have at home 
(512Kbps up, 4Mbps down), I generally do not send very large 
e-mails when working from home; nor do I use my laptop for 
remote router/switch updates as the software images are a 
nightmare to upload.

And yes, there is a larger proportion of downstream traffic 
than there is upstream traffic pretty much most of the time 
(even with symmetric links). However, with symmetry, 
upstream traffic will increase significantly as customers 
realize it is now available.

One of the use-cases we thought about when deploying an FTTH 
backbone was having remote PVR's. So rather than record and 
save linear Tv programming on the STB, record and save it in 
the network. This could only be done with symmetric 
bandwidth.

Mark.


signature.asc
Description: This is a digitally signed message part.

Re: Observations of an Internet Middleman (Level3) (was: RIP

Mark,

 I don't think that anyone disputes that when you improve the upstream you
do get an uptick in usage in that direction.  What I take issue with is the
notion that the upstream is anything like downstream even when the capacity
is there.  Upstream on ADSL is horribad, especially the first generations
(g.lite and g.dmt).


Scott Helms
Vice President of Technology
ZCorum
(678) 507-5000

http://twitter.com/kscotthelms



On Fri, May 16, 2014 at 3:25 PM, Mark Tinka mark.ti...@seacom.mu wrote:

 On Friday, May 16, 2014 05:35:39 PM Jay Ashworth wrote:

  Could you expand a bit, Mark on Social media forces the
  use of symmetric bandwidth?  Which social media
  platform is it that you think has a) symmetrical flows
  that b) are big enough to figure into transit symmetry?

 What we saw with FTTH deployments is that customers uploaded
 more videos and photos to Youtube, Facebook, MySpace, e.t.c.
 They didn't do this on ADSL as much (it's too frustrating).

 When that caught on, customers started buying online backup
 services - synchronizing backups of their home or office
 computers to remote backup infrastructure. Again, they never
 did this with ADSL.

 What we learned: don't take it for granted that you will
 always know what your customers (or the content providers
 who serve them) will do with the bandwidth. If they have it,
 expect the worst, and plan for it as best you can.

 Mark.

Re: Observations of an Internet Middleman (Level3) (was: RIP

2014-05-16 Thread Ca By

On May 16, 2014 12:21 PM, Matthew Petach mpet...@netflight.com wrote:

 On Fri, May 16, 2014 at 11:52 AM, Christopher Morrow 
 morrowc.li...@gmail.com wrote:

  On Fri, May 16, 2014 at 2:47 PM, Blake Hudson bl...@ispn.net wrote:
   in the context of this discussion I think it's silly for a residential
  ISP
   to purport themselves to be a neutral carrier of traffic and expect
  peering
   ratios to be symmetric
 
  is 'symmetric traffic ratios' even relevant though? Peering is about
  offsetting costs, right? it might not be important that the ratio be
  1:1 or 2:1... or even 10:1, if it's going to cost you 20x to get the
  traffic over longer/transit/etc paths... or if you have to build into
  some horrific location(s) to access the content in question.
 
  Harping on symmetric ratios seems very 1990... and not particularly
  germaine to the conversation at hand.
 
 
 Traffic asymmetry across peering connections
 was what lit the fuse on this whole powder keg,
 if I understand correctly; at the point the traffic
 went asymmetric, the refusals to augment
 capacity kicked in, and congestion became
 a problem.


What lit this powder keg?:

conspiracy theory

Netflix bought transit from cogent and expected it to work. C'mon.  This
happens every month on this list and every month people tell others not to
rely on cogent. Right? Netflix is smart, they know cogent is willing to
burn down their network and blow up their customers for 15 minutes of fame
$0.03 a meg.

This makes me think the whole thing is a net neutrality strawman.

They set the stage and all the players played their part.

Now, what will be the result?  I expect some concession from the
comcast/twc deal.  They made a big deal about net neutrality  / netflix /
strawman so they can trump up a meaningful concession to allow the
merger.

/conspiracy

 I've seen the same thing; pretty much every
 rejection is based on ratio issues, even when
 offering to cold-potato haul the traffic to the
 home market for the users.

 If the refusals hinged on any other clause
 of the peering requirements, you'd be right;
 but at the moment, that's the flag networks
 are waving around as their speedbump-du-jour.
 So, it may be very 1990, but unfortunately
 that seems to be the year many people in
 the industry are mentally stuck in.  :(

 Matt

Re: Observations of an Internet Middleman (Level3) (was: RIP Network Neutrality

Oh, please do explicate on how this is inaccurate…

Owen

On May 14, 2014, at 2:14 PM, McElearney, Kevin 
kevin_mcelear...@cable.comcast.com wrote:

 Respectfully, this is a highly inaccurate sound bite
 
 - Kevin
 
 215-313-1083
 
 On May 14, 2014, at 3:05 PM, Owen DeLong o...@delong.com wrote:
 
 Yes, the more accurate statement would be aggressively seeking new
 ways to monetize the existing infrastructure without investing in upgrades
 or additional buildout any more than absolutely necessary.
 
 Owen
 
 On May 14, 2014, at 8:02 AM, Hugo Slabbert h...@slabnet.com wrote:
 
 
 So they seek new sources of revenues, and/or attempt to thwart
 competition any way they can.
 No to the first. Yes to the second. If they were seeking new sources of
 revenue, they'd be massively expanding into un/der served markets and
 aggressively growing over the top services (which are fat margin).
 
 Sure they are (seeking new sources of revenue).  They're not necessarily
 creating new products or services, i.e. actually adding any value, but they
 are finding ways to extract additional revenue from the same pipes, e.g.
 through paid peering with content providers.
 
 I'm not endorsing this; just pointing out that you two are actually in
 agreement here.
 
 --
 Hugo
 
 
 On Wed, May 14, 2014 at 7:23 AM, char...@thefnf.org wrote:
 
 On 2014-05-14 02:04, Jean-Francois Mezei wrote:
 
 On 14-05-13 22:50, Daniel Staal wrote:
 
 They have the money.  They have the ability to get more money.  *They see
 no reason to spend money making customers happy.*  They can make more
 profit without it.
 
 There is the issue of control over the market. But also the pressure
 from shareholders for continued growth.
 
 
 Yes. That is true. Except that it's not.
 
 How do service providers grow? Let's explore that:
 
 What is growth for a transit provider?
 
 More (new) access network(s) (connections).
 More bandwidth across backbone pipes.
 
 
 What is growth for access network?
 More subscribers.
 
 Except that the incumbent carriers have shown they have no interest in
 providing decent bandwidth to anywhere but the most profitable rate
 centers. I'd say about 2/3 of the USA is served with quite terrible access.
 
 
 
 
 The problem with the internet is that while it had promises of wild
 growth in the 90s and 00s, once penetration reaches a certain level,
 growth stabilizes.
 
 Penetration is ABYSMAL sir. Huge swaths of underserved americans exist.
 
 
 
 When you combine this with threath to large incumbents's media and media
 distribution endeavours by the likes of Netflix (and cat videos on
 Youtube), large incumbents start thinking about how they will be able to
 continue to grow revenus/profits when customers will shift spending to
 vspecialty channels/cableTV to Netflix and customer growth will not
 compensate.
 
 Except they aren't. Even in the most profitable rate centers, they've
 declined to really invest in the networks. They aren't a real business. You
 have to remember that. They have regulatory capture, natural/defacto
 monopoly etc etc. They don't operate in the real world of
 risk/reward/profit/loss/uncertainty like any other real business has to.
 
 
 
 So they seek new sources of revenues, and/or attempt to thwart
 competition any way they can.
 
 No to the first. Yes to the second. If they were seeking new sources of
 revenue, they'd be massively expanding into un/der served markets and
 aggressively growing over the top services (which are fat margin). They did
 a bit of an advertising campaign of smart home offerings, but that seems
 to have never grown beyond a pilot.
 
 
 
 The current trend is to if you can't fight them, jon them where
 cablecos start to include the Netflix app into their proprietary set-top
 boxes. The idea is that you at least make the customer continue to use
 your box and your remote control which makes it easier for them to
 switch between netflix and legacy TV.
 True. I don't know why one of the cablecos hasn't licensed roku, added
 cable card and made that available as a hip/cool set top box offering and
 charge another 10.00 a month on top of the standard dvr rental.
 
 
 
 Would be interesting to see if those cable companies that are agreeing
 to add the Netflix app onto their proprietary STBs also  play peering
 capacity games to degrade the service or not.
 
 So how is the content delivered? Is it over the internet? Or is it over
 the cable plant, from cable headends?

Re: Observations of an Internet Middleman (Level3) (was: RIP Network Neutrality

I don’t disagree.  However, given the choice between Comcast and broadband 
services in NL, Chatanooga, or Seoul, just to name a few, Comcast loses badly.

Choosing between Comcast and a legacy Telco is like choosing between 
legionnaire’s disease and SARS.

Owen

On May 14, 2014, at 5:15 PM, Jared Mauch ja...@puck.nether.net wrote:

 Owen,
 
 I've seen a vast difference between Comcast and others in the marketplace.  
 Right now, if I had the choice between Comcast and a legacy telco, I would 
 pick Comcast hands-down for:
 
 a) performance
 b) IPv6 support
 c) willingness to work on issues
 
 - Jared
 
 On May 14, 2014, at 5:14 PM, McElearney, Kevin 
 kevin_mcelear...@cable.comcast.com wrote:
 
 Respectfully, this is a highly inaccurate sound bite
 
- Kevin
 
 215-313-1083
 
 On May 14, 2014, at 3:05 PM, Owen DeLong o...@delong.com wrote:
 
 Yes, the more accurate statement would be aggressively seeking new
 ways to monetize the existing infrastructure without investing in upgrades
 or additional buildout any more than absolutely necessary.
 
 Owen
 
 On May 14, 2014, at 8:02 AM, Hugo Slabbert h...@slabnet.com wrote:
 
 
 So they seek new sources of revenues, and/or attempt to thwart
 competition any way they can.
 No to the first. Yes to the second. If they were seeking new sources of
 revenue, they'd be massively expanding into un/der served markets and
 aggressively growing over the top services (which are fat margin).
 
 Sure they are (seeking new sources of revenue).  They're not necessarily
 creating new products or services, i.e. actually adding any value, but they
 are finding ways to extract additional revenue from the same pipes, e.g.
 through paid peering with content providers.
 
 I'm not endorsing this; just pointing out that you two are actually in
 agreement here.
 
 --
 Hugo
 
 
 On Wed, May 14, 2014 at 7:23 AM, char...@thefnf.org wrote:
 
 On 2014-05-14 02:04, Jean-Francois Mezei wrote:
 
 On 14-05-13 22:50, Daniel Staal wrote:
 
 They have the money.  They have the ability to get more money.  *They see
 no reason to spend money making customers happy.*  They can make more
 profit without it.
 
 There is the issue of control over the market. But also the pressure
 from shareholders for continued growth.
 
 
 Yes. That is true. Except that it's not.
 
 How do service providers grow? Let's explore that:
 
 What is growth for a transit provider?
 
 More (new) access network(s) (connections).
 More bandwidth across backbone pipes.
 
 
 What is growth for access network?
 More subscribers.
 
 Except that the incumbent carriers have shown they have no interest in
 providing decent bandwidth to anywhere but the most profitable rate
 centers. I'd say about 2/3 of the USA is served with quite terrible 
 access.
 
 
 
 
 The problem with the internet is that while it had promises of wild
 growth in the 90s and 00s, once penetration reaches a certain level,
 growth stabilizes.
 
 Penetration is ABYSMAL sir. Huge swaths of underserved americans exist.
 
 
 
 When you combine this with threath to large incumbents's media and media
 distribution endeavours by the likes of Netflix (and cat videos on
 Youtube), large incumbents start thinking about how they will be able to
 continue to grow revenus/profits when customers will shift spending to
 vspecialty channels/cableTV to Netflix and customer growth will not
 compensate.
 
 Except they aren't. Even in the most profitable rate centers, they've
 declined to really invest in the networks. They aren't a real business. 
 You
 have to remember that. They have regulatory capture, natural/defacto
 monopoly etc etc. They don't operate in the real world of
 risk/reward/profit/loss/uncertainty like any other real business has to.
 
 
 
 So they seek new sources of revenues, and/or attempt to thwart
 competition any way they can.
 
 No to the first. Yes to the second. If they were seeking new sources of
 revenue, they'd be massively expanding into un/der served markets and
 aggressively growing over the top services (which are fat margin). They 
 did
 a bit of an advertising campaign of smart home offerings, but that seems
 to have never grown beyond a pilot.
 
 
 
 The current trend is to if you can't fight them, jon them where
 cablecos start to include the Netflix app into their proprietary set-top
 boxes. The idea is that you at least make the customer continue to use
 your box and your remote control which makes it easier for them to
 switch between netflix and legacy TV.
 True. I don't know why one of the cablecos hasn't licensed roku, added
 cable card and made that available as a hip/cool set top box offering 
 and
 charge another 10.00 a month on top of the standard dvr rental.
 
 
 
 Would be interesting to see if those cable companies that are agreeing
 to add the Netflix app onto their proprietary STBs also  play peering
 capacity games to degrade the service or not.
 
 So how is the content delivered? Is it over the internet? Or is it over
 the

Re: Observations of an Internet Middleman (Level3) (was: RIP Network Neutrality

Having an actual free market would require having competition. So long as we 
have monopoly layer 1 providers being allowed to use that monopoly as leverage 
for higher layer service monopolies, (or oligopolies), an actual free market is 
virtually impossible.

The result of deregulating the current environment would only be more pain and 
cost to the consumer than we currently have with no improvement in speeds or 
capabilities and no additional innovation.

Owen

On May 14, 2014, at 6:05 PM, Matt Palmer mpal...@hezmatt.org wrote:

 On Wed, May 14, 2014 at 07:01:36PM -0500, Larry Sheldon wrote:
 Maybe it is time to try a free market.
 
 Can't do that, it would be UnAmerican!
 
 - Matt
 
 -- 
 I can only guess that the designer of the things had a major Toilet Duck
 habit and had managed to score a couple of industrial-sized bottles of the
 stuff the night before.
   -- Tanuki

Re: Observations of an Internet Middleman (Level3) (was: RIP Network Neutrality

2014-05-15 Thread McElearney, Kevin

Upgrades/buildout are happening every day.  They are continuous to keep ahead 
of demand and publicly measured by SamKnows (FCC measuring broadband), Akamai, 
Ookla, etc

What is not well known is that Comcast has been an existing commercial transit 
business for 15+ years (with over 8000 commercial fiber customers).  Comcast 
also has over 40 balanced peers with plenty of capacity, and some of the 
largest Internet companies as customers.

  - Kevin

215-313-1083

 On May 15, 2014, at 10:19 AM, Owen DeLong o...@delong.com wrote:
 
 Oh, please do explicate on how this is inaccurate…
 
 Owen
 
 On May 14, 2014, at 2:14 PM, McElearney, Kevin 
 kevin_mcelear...@cable.comcast.com wrote:
 
 Respectfully, this is a highly inaccurate sound bite
 
- Kevin
 
 215-313-1083
 
 On May 14, 2014, at 3:05 PM, Owen DeLong o...@delong.com wrote:
 
 Yes, the more accurate statement would be aggressively seeking new
 ways to monetize the existing infrastructure without investing in upgrades
 or additional buildout any more than absolutely necessary.
 
 Owen
 
 On May 14, 2014, at 8:02 AM, Hugo Slabbert h...@slabnet.com wrote:
 
 
 So they seek new sources of revenues, and/or attempt to thwart
 competition any way they can.
 No to the first. Yes to the second. If they were seeking new sources of
 revenue, they'd be massively expanding into un/der served markets and
 aggressively growing over the top services (which are fat margin).
 
 Sure they are (seeking new sources of revenue).  They're not necessarily
 creating new products or services, i.e. actually adding any value, but they
 are finding ways to extract additional revenue from the same pipes, e.g.
 through paid peering with content providers.
 
 I'm not endorsing this; just pointing out that you two are actually in
 agreement here.
 
 --
 Hugo
 
 
 On Wed, May 14, 2014 at 7:23 AM, char...@thefnf.org wrote:
 
 On 2014-05-14 02:04, Jean-Francois Mezei wrote:
 
 On 14-05-13 22:50, Daniel Staal wrote:
 
 They have the money.  They have the ability to get more money.  *They see
 no reason to spend money making customers happy.*  They can make more
 profit without it.
 
 There is the issue of control over the market. But also the pressure
 from shareholders for continued growth.
 
 
 Yes. That is true. Except that it's not.
 
 How do service providers grow? Let's explore that:
 
 What is growth for a transit provider?
 
 More (new) access network(s) (connections).
 More bandwidth across backbone pipes.
 
 
 What is growth for access network?
 More subscribers.
 
 Except that the incumbent carriers have shown they have no interest in
 providing decent bandwidth to anywhere but the most profitable rate
 centers. I'd say about 2/3 of the USA is served with quite terrible 
 access.
 
 
 
 
 The problem with the internet is that while it had promises of wild
 growth in the 90s and 00s, once penetration reaches a certain level,
 growth stabilizes.
 
 Penetration is ABYSMAL sir. Huge swaths of underserved americans exist.
 
 
 
 When you combine this with threath to large incumbents's media and media
 distribution endeavours by the likes of Netflix (and cat videos on
 Youtube), large incumbents start thinking about how they will be able to
 continue to grow revenus/profits when customers will shift spending to
 vspecialty channels/cableTV to Netflix and customer growth will not
 compensate.
 
 Except they aren't. Even in the most profitable rate centers, they've
 declined to really invest in the networks. They aren't a real business. 
 You
 have to remember that. They have regulatory capture, natural/defacto
 monopoly etc etc. They don't operate in the real world of
 risk/reward/profit/loss/uncertainty like any other real business has to.
 
 
 
 So they seek new sources of revenues, and/or attempt to thwart
 competition any way they can.
 
 No to the first. Yes to the second. If they were seeking new sources of
 revenue, they'd be massively expanding into un/der served markets and
 aggressively growing over the top services (which are fat margin). They 
 did
 a bit of an advertising campaign of smart home offerings, but that seems
 to have never grown beyond a pilot.
 
 
 
 The current trend is to if you can't fight them, jon them where
 cablecos start to include the Netflix app into their proprietary set-top
 boxes. The idea is that you at least make the customer continue to use
 your box and your remote control which makes it easier for them to
 switch between netflix and legacy TV.
 True. I don't know why one of the cablecos hasn't licensed roku, added
 cable card and made that available as a hip/cool set top box offering 
 and
 charge another 10.00 a month on top of the standard dvr rental.
 
 
 
 Would be interesting to see if those cable companies that are agreeing
 to add the Netflix app onto their proprietary STBs also  play peering
 capacity games to degrade the service or not.
 
 So how is the content delivered? Is it over the internet? Or is it over
 the cable

Re: Observations of an Internet Middleman (Level3) (was: RIP Network Neutrality

Yes, you've got some of the largest Internet companies as customers.
Because you told them if you don't pay us, we'll throttle you.  Then you
throttled them.  I'm sorry, not a winning argument.
Nick


On Thu, May 15, 2014 at 10:57 AM, McElearney, Kevin 
kevin_mcelear...@cable.comcast.com wrote:

 Upgrades/buildout are happening every day.  They are continuous to keep
 ahead of demand and publicly measured by SamKnows (FCC measuring
 broadband), Akamai, Ookla, etc

 What is not well known is that Comcast has been an existing commercial
 transit business for 15+ years (with over 8000 commercial fiber customers).
  Comcast also has over 40 balanced peers with plenty of capacity, and some
 of the largest Internet companies as customers.

   - Kevin

 215-313-1083

  On May 15, 2014, at 10:19 AM, Owen DeLong o...@delong.com wrote:
 
  Oh, please do explicate on how this is inaccurate…
 
  Owen
 
  On May 14, 2014, at 2:14 PM, McElearney, Kevin 
 kevin_mcelear...@cable.comcast.com wrote:
 
  Respectfully, this is a highly inaccurate sound bite
 
 - Kevin
 
  215-313-1083
 
  On May 14, 2014, at 3:05 PM, Owen DeLong o...@delong.com wrote:
 
  Yes, the more accurate statement would be aggressively seeking new
  ways to monetize the existing infrastructure without investing in
 upgrades
  or additional buildout any more than absolutely necessary.
 
  Owen
 
  On May 14, 2014, at 8:02 AM, Hugo Slabbert h...@slabnet.com wrote:
 
 
  So they seek new sources of revenues, and/or attempt to thwart
  competition any way they can.
  No to the first. Yes to the second. If they were seeking new sources
 of
  revenue, they'd be massively expanding into un/der served markets and
  aggressively growing over the top services (which are fat margin).
 
  Sure they are (seeking new sources of revenue).  They're not
 necessarily
  creating new products or services, i.e. actually adding any value,
 but they
  are finding ways to extract additional revenue from the same pipes,
 e.g.
  through paid peering with content providers.
 
  I'm not endorsing this; just pointing out that you two are actually in
  agreement here.
 
  --
  Hugo
 
 
  On Wed, May 14, 2014 at 7:23 AM, char...@thefnf.org wrote:
 
  On 2014-05-14 02:04, Jean-Francois Mezei wrote:
 
  On 14-05-13 22:50, Daniel Staal wrote:
 
  They have the money.  They have the ability to get more money.
  *They see
  no reason to spend money making customers happy.*  They can make
 more
  profit without it.
 
  There is the issue of control over the market. But also the pressure
  from shareholders for continued growth.
 
 
  Yes. That is true. Except that it's not.
 
  How do service providers grow? Let's explore that:
 
  What is growth for a transit provider?
 
  More (new) access network(s) (connections).
  More bandwidth across backbone pipes.
 
 
  What is growth for access network?
  More subscribers.
 
  Except that the incumbent carriers have shown they have no interest
 in
  providing decent bandwidth to anywhere but the most profitable rate
  centers. I'd say about 2/3 of the USA is served with quite terrible
 access.
 
 
 
 
  The problem with the internet is that while it had promises of wild
  growth in the 90s and 00s, once penetration reaches a certain level,
  growth stabilizes.
 
  Penetration is ABYSMAL sir. Huge swaths of underserved americans
 exist.
 
 
 
  When you combine this with threath to large incumbents's media and
 media
  distribution endeavours by the likes of Netflix (and cat videos on
  Youtube), large incumbents start thinking about how they will be
 able to
  continue to grow revenus/profits when customers will shift spending
 to
  vspecialty channels/cableTV to Netflix and customer growth will not
  compensate.
 
  Except they aren't. Even in the most profitable rate centers, they've
  declined to really invest in the networks. They aren't a real
 business. You
  have to remember that. They have regulatory capture, natural/defacto
  monopoly etc etc. They don't operate in the real world of
  risk/reward/profit/loss/uncertainty like any other real business has
 to.
 
 
 
  So they seek new sources of revenues, and/or attempt to thwart
  competition any way they can.
 
  No to the first. Yes to the second. If they were seeking new sources
 of
  revenue, they'd be massively expanding into un/der served markets and
  aggressively growing over the top services (which are fat margin).
 They did
  a bit of an advertising campaign of smart home offerings, but that
 seems
  to have never grown beyond a pilot.
 
 
 
  The current trend is to if you can't fight them, jon them where
  cablecos start to include the Netflix app into their proprietary
 set-top
  boxes. The idea is that you at least make the customer continue to
 use
  your box and your remote control which makes it easier for them to
  switch between netflix and legacy TV.
  True. I don't know why one of the cablecos hasn't licensed roku,
 added
  cable card and made that available

Re: Observations of an Internet Middleman (Level3) (was: RIP Network Neutrality

2014-05-15 Thread McElearney, Kevin

Guys, I'm already pretty far off the reservation and will not respond to 
trolling. I think most ISPs are starting to avoid participation here for the 
same reason. I'm going to stop for a while.

  - Kevin


On May 15, 2014, at 12:42 PM, Nick B 
n...@pelagiris.orgmailto:n...@pelagiris.org wrote:

Yes, you've got some of the largest Internet companies as customers.  Because 
you told them if you don't pay us, we'll throttle you.  Then you throttled 
them.  I'm sorry, not a winning argument.
Nick


On Thu, May 15, 2014 at 10:57 AM, McElearney, Kevin 
kevin_mcelear...@cable.comcast.commailto:kevin_mcelear...@cable.comcast.com 
wrote:
Upgrades/buildout are happening every day.  They are continuous to keep ahead 
of demand and publicly measured by SamKnows (FCC measuring broadband), Akamai, 
Ookla, etc

What is not well known is that Comcast has been an existing commercial transit 
business for 15+ years (with over 8000 commercial fiber customers).  Comcast 
also has over 40 balanced peers with plenty of capacity, and some of the 
largest Internet companies as customers.

  - Kevin

215-313-1083tel:215-313-1083

 On May 15, 2014, at 10:19 AM, Owen DeLong 
 o...@delong.commailto:o...@delong.com wrote:

 Oh, please do explicate on how this is inaccurate…

 Owen

 On May 14, 2014, at 2:14 PM, McElearney, Kevin 
 kevin_mcelear...@cable.comcast.commailto:kevin_mcelear...@cable.comcast.com
  wrote:

 Respectfully, this is a highly inaccurate sound bite

- Kevin

 215-313-1083tel:215-313-1083

 On May 14, 2014, at 3:05 PM, Owen DeLong 
 o...@delong.commailto:o...@delong.com wrote:

 Yes, the more accurate statement would be aggressively seeking new
 ways to monetize the existing infrastructure without investing in upgrades
 or additional buildout any more than absolutely necessary.

 Owen

 On May 14, 2014, at 8:02 AM, Hugo Slabbert 
 h...@slabnet.commailto:h...@slabnet.com wrote:


 So they seek new sources of revenues, and/or attempt to thwart
 competition any way they can.
 No to the first. Yes to the second. If they were seeking new sources of
 revenue, they'd be massively expanding into un/der served markets and
 aggressively growing over the top services (which are fat margin).

 Sure they are (seeking new sources of revenue).  They're not necessarily
 creating new products or services, i.e. actually adding any value, but they
 are finding ways to extract additional revenue from the same pipes, e.g.
 through paid peering with content providers.

 I'm not endorsing this; just pointing out that you two are actually in
 agreement here.

 --
 Hugo


 On Wed, May 14, 2014 at 7:23 AM, 
 char...@thefnf.orgmailto:char...@thefnf.org wrote:

 On 2014-05-14 02:04, Jean-Francois Mezei wrote:

 On 14-05-13 22:50, Daniel Staal wrote:

 They have the money.  They have the ability to get more money.  *They see
 no reason to spend money making customers happy.*  They can make more
 profit without it.

 There is the issue of control over the market. But also the pressure
 from shareholders for continued growth.


 Yes. That is true. Except that it's not.

 How do service providers grow? Let's explore that:

 What is growth for a transit provider?

 More (new) access network(s) (connections).
 More bandwidth across backbone pipes.


 What is growth for access network?
 More subscribers.

 Except that the incumbent carriers have shown they have no interest in
 providing decent bandwidth to anywhere but the most profitable rate
 centers. I'd say about 2/3 of the USA is served with quite terrible 
 access.




 The problem with the internet is that while it had promises of wild
 growth in the 90s and 00s, once penetration reaches a certain level,
 growth stabilizes.

 Penetration is ABYSMAL sir. Huge swaths of underserved americans exist.



 When you combine this with threath to large incumbents's media and media
 distribution endeavours by the likes of Netflix (and cat videos on
 Youtube), large incumbents start thinking about how they will be able to
 continue to grow revenus/profits when customers will shift spending to
 vspecialty channels/cableTV to Netflix and customer growth will not
 compensate.

 Except they aren't. Even in the most profitable rate centers, they've
 declined to really invest in the networks. They aren't a real business. 
 You
 have to remember that. They have regulatory capture, natural/defacto
 monopoly etc etc. They don't operate in the real world of
 risk/reward/profit/loss/uncertainty like any other real business has to.



 So they seek new sources of revenues, and/or attempt to thwart
 competition any way they can.

 No to the first. Yes to the second. If they were seeking new sources of
 revenue, they'd be massively expanding into un/der served markets and
 aggressively growing over the top services (which are fat margin). They 
 did
 a bit of an advertising campaign of smart home offerings, but that seems
 to have never grown beyond a pilot.



 The current trend is to if you can't

Re: Observations of an Internet Middleman (Level3) (was: RIP Network Neutrality

To be fair, I have no evidence that Comcast demanded money in advance.  As
far as I can tell, Level 3, Cogent and Comcast all agree on the rest
though, Comcast's peering filled up.  Both Level 3 and Cogent
offered/requested to upgrade.  Then at least Cogent (IIRC?) offered to
upgrade *and pay Comcast to upgrade*.  All of these offers were ignored or
rejected, and at this point I think all parties agree that ransom was
demanded by Comcast.  Then, just weeks after Verizon's successful
litigation against Net Neutrality, Netflix agreed to pay Comcast directly.
I'm not sure if the removal of Netflix is enough to resolve the congestion
on the Level 3 and Cogent links, but I'm not sure how one could explain the
above situation other than deliberate throttling in an attempt to extort
money.
Nick


On Thu, May 15, 2014 at 12:51 PM, arvindersi...@mail2tor.com wrote:

 Yes Kevin, this is understood - but valid observation from Nick.

 Can you pls answer my question first?  Very curious.

 Arvinder

  Guys, I'm already pretty far off the reservation and will not respond to
  trolling. I think most ISPs are starting to avoid participation here for
  the same reason. I'm going to stop for a while.
 
- Kevin
 
 
  On May 15, 2014, at 12:42 PM, Nick B
  n...@pelagiris.orgmailto:n...@pelagiris.org wrote:
 
  Yes, you've got some of the largest Internet companies as customers.
  Because you told them if you don't pay us, we'll throttle you.  Then
 you
  throttled them.  I'm sorry, not a winning argument.
  Nick
 
 
  On Thu, May 15, 2014 at 10:57 AM, McElearney, Kevin
  kevin_mcelear...@cable.comcast.commailto:
 kevin_mcelear...@cable.comcast.com
  wrote:
  Upgrades/buildout are happening every day.  They are continuous to keep
  ahead of demand and publicly measured by SamKnows (FCC measuring
  broadband), Akamai, Ookla, etc
 
  What is not well known is that Comcast has been an existing commercial
  transit business for 15+ years (with over 8000 commercial fiber
  customers).  Comcast also has over 40 balanced peers with plenty of
  capacity, and some of the largest Internet companies as customers.
 
- Kevin
 
  215-313-1083tel:215-313-1083
 
  On May 15, 2014, at 10:19 AM, Owen DeLong
  o...@delong.commailto:o...@delong.com wrote:
 
  Oh, please do explicate on how this is inaccurate…
 
  Owen
 
  On May 14, 2014, at 2:14 PM, McElearney, Kevin
  kevin_mcelear...@cable.comcast.commailto:
 kevin_mcelear...@cable.comcast.com
  wrote:
 
  Respectfully, this is a highly inaccurate sound bite
 
 - Kevin
 
  215-313-1083tel:215-313-1083
 
  On May 14, 2014, at 3:05 PM, Owen DeLong
  o...@delong.commailto:o...@delong.com wrote:
 
  Yes, the more accurate statement would be aggressively seeking new
  ways to monetize the existing infrastructure without investing in
  upgrades
  or additional buildout any more than absolutely necessary.
 
  Owen
 
  On May 14, 2014, at 8:02 AM, Hugo Slabbert
  h...@slabnet.commailto:h...@slabnet.com wrote:
 
 
  So they seek new sources of revenues, and/or attempt to thwart
  competition any way they can.
  No to the first. Yes to the second. If they were seeking new sources
  of
  revenue, they'd be massively expanding into un/der served markets
  and
  aggressively growing over the top services (which are fat margin).
 
  Sure they are (seeking new sources of revenue).  They're not
  necessarily
  creating new products or services, i.e. actually adding any value,
  but they
  are finding ways to extract additional revenue from the same pipes,
  e.g.
  through paid peering with content providers.
 
  I'm not endorsing this; just pointing out that you two are actually
  in
  agreement here.
 
  --
  Hugo
 
 
  On Wed, May 14, 2014 at 7:23 AM,
  char...@thefnf.orgmailto:char...@thefnf.org wrote:
 
  On 2014-05-14 02:04, Jean-Francois Mezei wrote:
 
  On 14-05-13 22:50, Daniel Staal wrote:
 
  They have the money.  They have the ability to get more money.
  *They see
  no reason to spend money making customers happy.*  They can make
  more
  profit without it.
 
  There is the issue of control over the market. But also the
  pressure
  from shareholders for continued growth.
 
 
  Yes. That is true. Except that it's not.
 
  How do service providers grow? Let's explore that:
 
  What is growth for a transit provider?
 
  More (new) access network(s) (connections).
  More bandwidth across backbone pipes.
 
 
  What is growth for access network?
  More subscribers.
 
  Except that the incumbent carriers have shown they have no interest
  in
  providing decent bandwidth to anywhere but the most profitable rate
  centers. I'd say about 2/3 of the USA is served with quite terrible
  access.
 
 
 
 
  The problem with the internet is that while it had promises of wild
  growth in the 90s and 00s, once penetration reaches a certain
  level,
  growth stabilizes.
 
  Penetration is ABYSMAL sir. Huge swaths of underserved americans
  exist.
 
 
 
  When you combine this with threath to

Re: Observations of an Internet Middleman (Level3) (was: RIP Network Neutrality

2014-05-15 Thread Jerry Dent

 What is not well known is that Comcast has been an existing commercial
transit business for 15+ years (with over 8000 commercial fiber customers).
 Comcast also has over 40 balanced peers with plenty of capacity, and some
of the largest Internet companies as customers.


Peers that are balanced only due strong arm tactics used against smaller
hosting and content providers.


On Thu, May 15, 2014 at 11:46 AM, McElearney, Kevin 
kevin_mcelear...@cable.comcast.com wrote:

 Guys, I'm already pretty far off the reservation and will not respond to
 trolling. I think most ISPs are starting to avoid participation here for
 the same reason. I'm going to stop for a while.

   - Kevin


 On May 15, 2014, at 12:42 PM, Nick B n...@pelagiris.orgmailto:
 n...@pelagiris.org wrote:

 Yes, you've got some of the largest Internet companies as customers.
  Because you told them if you don't pay us, we'll throttle you.  Then you
 throttled them.  I'm sorry, not a winning argument.
 Nick


 On Thu, May 15, 2014 at 10:57 AM, McElearney, Kevin 
 kevin_mcelear...@cable.comcast.commailto:
 kevin_mcelear...@cable.comcast.com wrote:
 Upgrades/buildout are happening every day.  They are continuous to keep
 ahead of demand and publicly measured by SamKnows (FCC measuring
 broadband), Akamai, Ookla, etc

 What is not well known is that Comcast has been an existing commercial
 transit business for 15+ years (with over 8000 commercial fiber customers).
  Comcast also has over 40 balanced peers with plenty of capacity, and some
 of the largest Internet companies as customers.

   - Kevin

 215-313-1083tel:215-313-1083

  On May 15, 2014, at 10:19 AM, Owen DeLong o...@delong.commailto:
 o...@delong.com wrote:
 
  Oh, please do explicate on how this is inaccurate…
 
  Owen
 
  On May 14, 2014, at 2:14 PM, McElearney, Kevin 
 kevin_mcelear...@cable.comcast.commailto:
 kevin_mcelear...@cable.comcast.com wrote:
 
  Respectfully, this is a highly inaccurate sound bite
 
 - Kevin
 
  215-313-1083tel:215-313-1083
 
  On May 14, 2014, at 3:05 PM, Owen DeLong o...@delong.commailto:
 o...@delong.com wrote:
 
  Yes, the more accurate statement would be aggressively seeking new
  ways to monetize the existing infrastructure without investing in
 upgrades
  or additional buildout any more than absolutely necessary.
 
  Owen
 
  On May 14, 2014, at 8:02 AM, Hugo Slabbert h...@slabnet.commailto:
 h...@slabnet.com wrote:
 
 
  So they seek new sources of revenues, and/or attempt to thwart
  competition any way they can.
  No to the first. Yes to the second. If they were seeking new sources
 of
  revenue, they'd be massively expanding into un/der served markets and
  aggressively growing over the top services (which are fat margin).
 
  Sure they are (seeking new sources of revenue).  They're not
 necessarily
  creating new products or services, i.e. actually adding any value,
 but they
  are finding ways to extract additional revenue from the same pipes,
 e.g.
  through paid peering with content providers.
 
  I'm not endorsing this; just pointing out that you two are actually in
  agreement here.
 
  --
  Hugo
 
 
  On Wed, May 14, 2014 at 7:23 AM, char...@thefnf.orgmailto:
 char...@thefnf.org wrote:
 
  On 2014-05-14 02:04, Jean-Francois Mezei wrote:
 
  On 14-05-13 22:50, Daniel Staal wrote:
 
  They have the money.  They have the ability to get more money.
  *They see
  no reason to spend money making customers happy.*  They can make
 more
  profit without it.
 
  There is the issue of control over the market. But also the pressure
  from shareholders for continued growth.
 
 
  Yes. That is true. Except that it's not.
 
  How do service providers grow? Let's explore that:
 
  What is growth for a transit provider?
 
  More (new) access network(s) (connections).
  More bandwidth across backbone pipes.
 
 
  What is growth for access network?
  More subscribers.
 
  Except that the incumbent carriers have shown they have no interest
 in
  providing decent bandwidth to anywhere but the most profitable rate
  centers. I'd say about 2/3 of the USA is served with quite terrible
 access.
 
 
 
 
  The problem with the internet is that while it had promises of wild
  growth in the 90s and 00s, once penetration reaches a certain level,
  growth stabilizes.
 
  Penetration is ABYSMAL sir. Huge swaths of underserved americans
 exist.
 
 
 
  When you combine this with threath to large incumbents's media and
 media
  distribution endeavours by the likes of Netflix (and cat videos on
  Youtube), large incumbents start thinking about how they will be
 able to
  continue to grow revenus/profits when customers will shift spending
 to
  vspecialty channels/cableTV to Netflix and customer growth will not
  compensate.
 
  Except they aren't. Even in the most profitable rate centers, they've
  declined to really invest in the networks. They aren't a real
 business. You
  have to remember that. They have regulatory capture, natural/defacto
  monopoly etc

Re: Observations of an Internet Middleman (Level3) (was: RIP Network Neutrality

2014-05-15 Thread Jean-Francois Mezei

On 14-05-15 10:26, Owen DeLong wrote:
 Choosing between Comcast and a legacy Telco is like choosing between 
 legionnaire’s disease and SARS.

Twisted pair is certantly legacy.

Is there a feeling that coax cable/DOSCIS is also legacy in terms of
current capacity/speeds ? Or is that technology still considered viable
against FTTH ?

I realise that business practices make north american incumbents
undesirable compared to the rest of the world, especially Comcast's
dirty tricks with Netflix as an example.

But in terms of the last mile technology and wiring (for instance, homes
per HFC node) sre north american cavlecos up to par with the rest of the
world ?

Re: Observations of an Internet Middleman (Level3) (was: RIP Network Neutrality


On May 15, 2014, at 7:57 AM, McElearney, Kevin 
kevin_mcelear...@cable.comcast.com wrote:

 Upgrades/buildout are happening every day.  They are continuous to keep ahead 
 of demand and publicly measured by SamKnows (FCC measuring broadband), 
 Akamai, Ookla, etc

I didn’t say they weren’t doing any upgrades/buildouts.

I will say that the copper capabilities in my neighborhood are so far behind 
demand(s) that it is abysmal. There hasn’t been significant maintenance to the 
$TELCO copper plant in my neighborhood since it was installed in 1960.

 What is not well known is that Comcast has been an existing commercial 
 transit business for 15+ years (with over 8000 commercial fiber customers).  
 Comcast also has over 40 balanced peers with plenty of capacity, and some of 
 the largest Internet companies as customers.

I’ve been asked by my employer to stop picking on specific large ISPs. However, 
my experiences with $CABLECO have been as described. The infrastructure in my 
neighborhood was horrible and did not improve at all until I ordered business 
class service from them. I’ve seen nothing to indicate that there is any 
significant effort to improve customer satisfaction, but lots of things to 
indicate that they are trying to leverage as much revenue out of as little 
investment as possible.

Owen

 
  - Kevin
 
 215-313-1083
 
 On May 15, 2014, at 10:19 AM, Owen DeLong o...@delong.com wrote:
 
 Oh, please do explicate on how this is inaccurate…
 
 Owen
 
 On May 14, 2014, at 2:14 PM, McElearney, Kevin 
 kevin_mcelear...@cable.comcast.com wrote:
 
 Respectfully, this is a highly inaccurate sound bite
 
   - Kevin
 
 215-313-1083
 
 On May 14, 2014, at 3:05 PM, Owen DeLong o...@delong.com wrote:
 
 Yes, the more accurate statement would be aggressively seeking new
 ways to monetize the existing infrastructure without investing in upgrades
 or additional buildout any more than absolutely necessary.
 
 Owen
 
 On May 14, 2014, at 8:02 AM, Hugo Slabbert h...@slabnet.com wrote:
 
 
 So they seek new sources of revenues, and/or attempt to thwart
 competition any way they can.
 No to the first. Yes to the second. If they were seeking new sources of
 revenue, they'd be massively expanding into un/der served markets and
 aggressively growing over the top services (which are fat margin).
 
 Sure they are (seeking new sources of revenue).  They're not necessarily
 creating new products or services, i.e. actually adding any value, but 
 they
 are finding ways to extract additional revenue from the same pipes, e.g.
 through paid peering with content providers.
 
 I'm not endorsing this; just pointing out that you two are actually in
 agreement here.
 
 --
 Hugo
 
 
 On Wed, May 14, 2014 at 7:23 AM, char...@thefnf.org wrote:
 
 On 2014-05-14 02:04, Jean-Francois Mezei wrote:
 
 On 14-05-13 22:50, Daniel Staal wrote:
 
 They have the money.  They have the ability to get more money.  *They 
 see
 no reason to spend money making customers happy.*  They can make more
 profit without it.
 
 There is the issue of control over the market. But also the pressure
 from shareholders for continued growth.
 
 
 Yes. That is true. Except that it's not.
 
 How do service providers grow? Let's explore that:
 
 What is growth for a transit provider?
 
 More (new) access network(s) (connections).
 More bandwidth across backbone pipes.
 
 
 What is growth for access network?
 More subscribers.
 
 Except that the incumbent carriers have shown they have no interest in
 providing decent bandwidth to anywhere but the most profitable rate
 centers. I'd say about 2/3 of the USA is served with quite terrible 
 access.
 
 
 
 
 The problem with the internet is that while it had promises of wild
 growth in the 90s and 00s, once penetration reaches a certain level,
 growth stabilizes.
 
 Penetration is ABYSMAL sir. Huge swaths of underserved americans exist.
 
 
 
 When you combine this with threath to large incumbents's media and media
 distribution endeavours by the likes of Netflix (and cat videos on
 Youtube), large incumbents start thinking about how they will be able to
 continue to grow revenus/profits when customers will shift spending to
 vspecialty channels/cableTV to Netflix and customer growth will not
 compensate.
 
 Except they aren't. Even in the most profitable rate centers, they've
 declined to really invest in the networks. They aren't a real business. 
 You
 have to remember that. They have regulatory capture, natural/defacto
 monopoly etc etc. They don't operate in the real world of
 risk/reward/profit/loss/uncertainty like any other real business has to.
 
 
 
 So they seek new sources of revenues, and/or attempt to thwart
 competition any way they can.
 
 No to the first. Yes to the second. If they were seeking new sources of
 revenue, they'd be massively expanding into un/der served markets and
 aggressively growing over the top services (which are fat margin). They 
 did
 a bit of an advertising campaign of smart

Re: Observations of an Internet Middleman (Level3) (was: RIP Network Neutrality

On 5/15/14, 12:43 PM, Nick B n...@pelagiris.org wrote:


Yes, you've got some of the largest Internet companies as customers².
Because you told them if you don't pay us, we'll throttle you.  Then
you throttled them.  I'm sorry, not a winning argument.
Nick

That is categorically untrue, however nice a soundbite it may be. If you
or anyone else truly believes we are throttling someone then I encourage
you to file a formal complaint with the FCC. According to their Open
Internet rules that we are bound to through at least 2018 (IIRC) we may
not discriminate on traffic in that way, so there is a clear rule and a
clear process for complaints.

Jason

Re: Observations of an Internet Middleman (Level3) (was: RIP Network Neutrality

By categorically untrue do you mean FCC's open internet rules allow us
to refuse to upgrade full peers?
Nick


On Thu, May 15, 2014 at 1:26 PM, Livingood, Jason 
jason_living...@cable.comcast.com wrote:

 On 5/15/14, 12:43 PM, Nick B n...@pelagiris.org wrote:


 Yes, you've got some of the largest Internet companies as customers².
 Because you told them if you don't pay us, we'll throttle you.  Then
 you throttled them.  I'm sorry, not a winning argument.
 Nick

 That is categorically untrue, however nice a soundbite it may be. If you
 or anyone else truly believes we are throttling someone then I encourage
 you to file a formal complaint with the FCC. According to their Open
 Internet rules that we are bound to through at least 2018 (IIRC) we may
 not discriminate on traffic in that way, so there is a clear rule and a
 clear process for complaints.

 Jason

Re: Observations of an Internet Middleman (Level3) (was: RIP Network Neutrality

On 5/15/14, 1:28 PM, Nick B n...@pelagiris.orgmailto:n...@pelagiris.org 
wrote:

By categorically untrue do you mean FCC's open internet rules allow us to 
refuse to upgrade full peers?

Throttling is taking, say, a link from 10G and applying policy to constrain it 
to 1G, for example. What if a peer wants to go from a balanced relationship to 
10,000:1, well outside of the policy binding the relationship? Should we just 
unquestionably toss out our published policy – which is consistent with other 
networks – and ignore expectations for other peers?

Jason

Re: Observations of an Internet Middleman (Level3) (was: RIP Network Neutrality

Yes, throttling an entire ISP by refusing to upgrade peering is clearly a
way to avoid technically throttling.  Interestingly enough only Comcast and
Verizon are having this problem, though I'm sure now that you have set an
example others will follow.
Nick


On Thu, May 15, 2014 at 1:34 PM, Livingood, Jason 
jason_living...@cable.comcast.com wrote:

   On 5/15/14, 1:28 PM, Nick B n...@pelagiris.org wrote:

   By categorically untrue do you mean FCC's open internet rules allow
 us to refuse to upgrade full peers?


  Throttling is taking, say, a link from 10G and applying policy to
 constrain it to 1G, for example. What if a peer wants to go from a balanced
 relationship to 10,000:1, well outside of the policy binding the
 relationship? Should we just unquestionably toss out our published policy –
 which is consistent with other networks – and ignore expectations for other
 peers?

  Jason

Re: Observations of an Internet Middleman (Level3) (was: RIP Network Neutrality

So by extension, if you enter an agreement and promise to remain balanced you 
can just willfully throw that out and abuse the heck out of it? Where does it 
end? Why even bother having peering policies at all then?

To use an analogy, if you and I agree to buy a car together and agree to switch 
off who uses it every other day, can I just say forget our agreement – I’m 
just going to drive the car myself every single day – its all mine”?

And as you say, “interestingly enough only Comcast and Verizon are having this 
problem” someone else might say “interestingly enough one content distributor 
is at the center of all of these issues.” I’m frankly surprised that no one is 
stepping back to try to understand what was and is driving those changes.

Jason

On 5/15/14, 1:43 PM, Nick B n...@pelagiris.orgmailto:n...@pelagiris.org 
wrote:

Yes, throttling an entire ISP by refusing to upgrade peering is clearly a way 
to avoid technically throttling.  Interestingly enough only Comcast and Verizon 
are having this problem, though I'm sure now that you have set an example 
others will follow.
Nick

Re: Observations of an Internet Middleman (Level3) (was: RIP Network Neutrality

2014-05-15 Thread McElearney, Kevin

I said I would step away, but trying to keep some level of emotion out of 
this...  We all need rational actor behavior in the ecosystem. We need our 
policies and agree to live up to those policies between players. Random and 
inconsistent behavior does not build a well functioning market and is the root 
of most disputes

We can argue about what the policy should be, the impacts, etc and that is a 
fair discussion.

  - Kevin

215-313-1083

On May 15, 2014, at 2:11 PM, Livingood, Jason 
jason_living...@cable.comcast.commailto:jason_living...@cable.comcast.com 
wrote:

So by extension, if you enter an agreement and promise to remain balanced you 
can just willfully throw that out and abuse the heck out of it? Where does it 
end? Why even bother having peering policies at all then?

To use an analogy, if you and I agree to buy a car together and agree to switch 
off who uses it every other day, can I just say forget our agreement – I’m 
just going to drive the car myself every single day – its all mine”?

And as you say, “interestingly enough only Comcast and Verizon are having this 
problem” someone else might say “interestingly enough one content distributor 
is at the center of all of these issues.” I’m frankly surprised that no one is 
stepping back to try to understand what was and is driving those changes.

Jason

On 5/15/14, 1:43 PM, Nick B n...@pelagiris.orgmailto:n...@pelagiris.org 
wrote:

Yes, throttling an entire ISP by refusing to upgrade peering is clearly a way 
to avoid technically throttling.  Interestingly enough only Comcast and Verizon 
are having this problem, though I'm sure now that you have set an example 
others will follow.
Nick

Re: Observations of an Internet Middleman (Level3) (was: RIP Network Neutrality

2014-05-15 Thread Blake Dunlap

I agree, and those peers should be then paid for the bits that your
customers are requesting that they send through you if you cannot
maintain a balanced peer relationship with them. It's shameful that
access networks are attempting to not pay for their leeching of mass
amounts of data in clear violation of standard expectations for
balanced peering agreements.

Oh... you meant something else?

-Blake

On Thu, May 15, 2014 at 12:34 PM, Livingood, Jason
jason_living...@cable.comcast.com wrote:
 On 5/15/14, 1:28 PM, Nick B n...@pelagiris.orgmailto:n...@pelagiris.org 
 wrote:

 By categorically untrue do you mean FCC's open internet rules allow us to 
 refuse to upgrade full peers?

 Throttling is taking, say, a link from 10G and applying policy to constrain 
 it to 1G, for example. What if a peer wants to go from a balanced 
 relationship to 10,000:1, well outside of the policy binding the 
 relationship? Should we just unquestionably toss out our published policy – 
 which is consistent with other networks – and ignore expectations for other 
 peers?

 Jason

Re: Observations of an Internet Middleman (Level3) (was: RIP Network Neutrality


On May 15, 2014, at 10:18 AM, Jean-Francois Mezei jfmezei_na...@vaxination.ca 
wrote:

 On 14-05-15 10:26, Owen DeLong wrote:
 Choosing between Comcast and a legacy Telco is like choosing between 
 legionnaire’s disease and SARS.
 
 Twisted pair is certantly legacy.
 
 Is there a feeling that coax cable/DOSCIS is also legacy in terms of
 current capacity/speeds ? Or is that technology still considered viable
 against FTTH ?
 
 I realise that business practices make north american incumbents
 undesirable compared to the rest of the world, especially Comcast's
 dirty tricks with Netflix as an example.
 
 But in terms of the last mile technology and wiring (for instance, homes
 per HFC node) sre north american cavlecos up to par with the rest of the
 world ?

I am not speaking specifically about any one company here.

In North America, very few places have any level of FTTH. If you are in a rural 
area with USF subsidies, you are more likely to have FTTH than many urban 
areas. Co-ax, or if you’re somewhat lucky, HFC is about the best last mile 
technology available to most US subscribers.

In states where some city invested in municipal FTTH on an open-access basis, 
the incumbent $CABLECOS and $TELCOS have fought hard to push legislation making 
it illegal for other cities in the state to do the same.

The state of broadband networks in the US in general can best be described as 
pathetic and/or apathetic when it comes to the consumer’s interest. Lilly 
Tomlin summed this up very well in a number of her early comedy sketches where 
she pretended to be a telephone company operator. Her catch phrase was “We 
don’t care. We don’t have to. We’re the phone company!”

Further, it appears that several of the $CABLECOS and $TELCOS will actually 
attempt to quash their more vocal opponents by discussing public comments they 
make on a personal basis with said opponents employer and using them as a 
“negotiating tactic”. Personally, I think this is one of the most underhanded 
and lowest forms of an act of desperation to try and squash public debate. To 
be very clear… This statement is absolutely not targeted at any one company. 
There were several.

Owen

Re: Observations of an Internet Middleman (Level3) (was: RIP

2014-05-15 Thread Joe Greco

 Throttling is taking, say, a link from 10G and applying policy to constrain=
  it to 1G, for example.

Throttling is also trying to cram 20G of traffic through that same 10G
link.

 What if a peer wants to go from a balanced relation=
 ship to 10,000:1, well outside of the policy binding the relationship?

What if you're running a 10G port at saturation in both directions and
you decide to stop accepting announcements from the peer on that port?
Now you have a 10,000:0 ratio.  Then what?

 Should we just unquestionably toss out our published policy =96 which is 
 consis=
 tent with other networks =96 and ignore expectations for other peers?

What's your goal at the end of the day?

You have customers who are paying you for connectivity to Teh Interwebz.

Do you have an obligation to run a dedicated 100GbE to each and every
host on the planet?  No.

Do you have an obligation to make a reasonable effort to move the 
traffic that your customer is paying you for?  Yes.

At the end of the day, if I'm your customer and I'm trying to pull
50Mbps of data on my 50Mbps connection that I am buying from you, 
then it seems like a reasonable thing to expect that you'll have
the 50Mbps of capacity to actually fulfill the demand.  That does
not mean that I will actually GET 50Mbps - it just means that you
should be making a reasonable effort and especially that you are 
not actively sabotaging it, by aggregating it through a congested 
10Gbps port, or forcing the packets through a congested peer, or
any of a number of other underhanded things.  

If you cannot figure out how to arrange your transit and peering
affairs in a manner that allows you to deliver on what you've sold
to customers in the current unregulated model, I think you'll find
that the alternative of regulation is very much less palatable.

So, to answer your question, yes, if you're unable to figure out 
that Netflix is always going to generate tons more traffic than it
receives, and that your customers desperately want to get good
connectivity to there, then that's dumb.  Perhaps you should 
figure out how to arrange peering with sites where there's 
obviously going to be an unrectifiable traffic imbalance.

You're a service provider.  What should your goal be?  I would
have thought it obvious:  Provide the service.

... JG
-- 
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
We call it the 'one bite at the apple' rule. Give me one chance [and] then I
won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN)
With 24 million small businesses in the US alone, that's way too many apples.

Re: Observations of an Internet Middleman (Level3) (was: RIP Network Neutrality

2014-05-15 Thread Scott Helms

AFAIK Comcast wasn't consuming, mass amounts of data from Level 3
(Netflix's transit to them).  Are you implying that a retail customer has a
similar expectation (or should) as a tier 1 ISP has for peering?  I hope
not, that would be hyperbole verging on the silly.  Retail customer
agreement spell out, in every example I've seen, realistic terms and
expectations for service and those are very different from peering
arrangements.


Scott Helms
Vice President of Technology
ZCorum
(678) 507-5000

http://twitter.com/kscotthelms



On Thu, May 15, 2014 at 2:28 PM, Blake Dunlap iki...@gmail.com wrote:

 I agree, and those peers should be then paid for the bits that your
 customers are requesting that they send through you if you cannot
 maintain a balanced peer relationship with them. It's shameful that
 access networks are attempting to not pay for their leeching of mass
 amounts of data in clear violation of standard expectations for
 balanced peering agreements.

 Oh... you meant something else?

 -Blake

 On Thu, May 15, 2014 at 12:34 PM, Livingood, Jason
 jason_living...@cable.comcast.com wrote:
  On 5/15/14, 1:28 PM, Nick B n...@pelagiris.orgmailto:
 n...@pelagiris.org wrote:
 
  By categorically untrue do you mean FCC's open internet rules allow
 us to refuse to upgrade full peers?
 
  Throttling is taking, say, a link from 10G and applying policy to
 constrain it to 1G, for example. What if a peer wants to go from a balanced
 relationship to 10,000:1, well outside of the policy binding the
 relationship? Should we just unquestionably toss out our published policy –
 which is consistent with other networks – and ignore expectations for other
 peers?
 
  Jason

Re: Observations of an Internet Middleman (Level3) (was: RIP Network Neutrality

2014-05-15 Thread Jerry Dent

If traffic is unbalanced, what determines who is the payer and who is the
payee? Apparently whoever can hold on to their customers better while
performance is shit.


On Thu, May 15, 2014 at 1:28 PM, Blake Dunlap iki...@gmail.com wrote:

 I agree, and those peers should be then paid for the bits that your
 customers are requesting that they send through you if you cannot
 maintain a balanced peer relationship with them. It's shameful that
 access networks are attempting to not pay for their leeching of mass
 amounts of data in clear violation of standard expectations for
 balanced peering agreements.

 Oh... you meant something else?

 -Blake

 On Thu, May 15, 2014 at 12:34 PM, Livingood, Jason
 jason_living...@cable.comcast.com wrote:
  On 5/15/14, 1:28 PM, Nick B n...@pelagiris.orgmailto:
 n...@pelagiris.org wrote:
 
  By categorically untrue do you mean FCC's open internet rules allow
 us to refuse to upgrade full peers?
 
  Throttling is taking, say, a link from 10G and applying policy to
 constrain it to 1G, for example. What if a peer wants to go from a balanced
 relationship to 10,000:1, well outside of the policy binding the
 relationship? Should we just unquestionably toss out our published policy –
 which is consistent with other networks – and ignore expectations for other
 peers?
 
  Jason

Re: Observations of an Internet Middleman (Level3) (was: RIP

2014-05-15 Thread Joe Greco

 So by extension, if you enter an agreement and promise to remain balanced y=
 ou can just willfully throw that out and abuse the heck out of it? Where do=
 es it end? Why even bother having peering policies at all then?

It doesn't strike you as a ridiculous promise to extract from someone?

Hi I'm an Internet company.  I don't actually know what the next big
thing next year will be but I promise that I won't host it on my network 
and cause our traffic to become lopsided.

Wow.  Is that what you're saying?

 To use an analogy, if you and I agree to buy a car together and agree to sw=
 itch off who uses it every other day, can I just say forget our agreement =
 =96 I=92m just going to drive the car myself every single day =96 its all m=
 ine=94?

Seems like a poor analogy since I'm pretty sure both parties on a peering
can use the port at the same time.

... JG
-- 
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
We call it the 'one bite at the apple' rule. Give me one chance [and] then I
won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN)
With 24 million small businesses in the US alone, that's way too many apples.

Re: Observations of an Internet Middleman (Level3) (was: RIP