Re: Strange Problem with 16 byte packets

[Glen Kent]

Thanks i will. However, the doubt is that what does introducing a 16 byte
data into the steam does that causes the session to time out. I added
instrumentation to push some dummy data so that instead of 16 bytes, we
push 1 MB of data. In that case i saw no issues. Any idea if there is a
firewall setting that could be coming into play here?

On Thu, Jun 16, 2016 at 2:17 PM, Ruairi Carroll 
wrote:

> Follow the TCP stream - which side times out the link, and for what
> sequences of data do you get ACKs for?
>
> /Ruairi
>
> On 16 June 2016 at 10:43, Glen Kent  wrote:
>
>> Hi,
>>
>> I am using a proprietary protocol and sending a bunch of bytes to a
>> Draytek
>> router at an enterprise site. When i send the data in TCP batches of 1 MB
>> i
>> see no problem. However,  when i first send 16 bytes followed by 1 MB of
>> data, and then repeat this till the entire data has beeen sent out. During
>> this process I see that my TCP session times out. Unable to understand why
>> this could be happening? How can sending 16 bytes of data followed by 1MB
>> of data affect the transfer.
>>
>> Thanks !
>>
>
>

Strange Problem with 16 byte packets

[Glen Kent]

Hi,

I am using a proprietary protocol and sending a bunch of bytes to a Draytek
router at an enterprise site. When i send the data in TCP batches of 1 MB i
see no problem. However,  when i first send 16 bytes followed by 1 MB of
data, and then repeat this till the entire data has beeen sent out. During
this process I see that my TCP session times out. Unable to understand why
this could be happening? How can sending 16 bytes of data followed by 1MB
of data affect the transfer.

Thanks !

Re: NANOG67 - Tipping point of community and sponsor bashing?

[Owen DeLong]

> Cough cough ARIN cough. I don't know why they need to meet face to face 2
> or 3 times a year. But, i am sure ppml will tell you it is a ground up
> process and these people on ppml like traveling and talking about
> policy And they do what members want.

I don’t speak for the organization or even for the Advisory Council, but I can
tell you that we get more focused community input and participation during those
face to face meetings than we do during the rest of the year.

Like it or not, when trying to get interactive participation from more than 5 
or so people,
there’s really no useful substitute for the face-to-face meeting. Every existing
alternative is less useful and comes with significant drawbacks.

So, yes, I believe ARIN does what the members want, but more importantly, I
believe that the face to face meetings are a useful mechanism to preserve the
bottom up nature of the policy development process.

> Much of our industry can be gleened by googling pictures of "peering
> cruise". At my office, we joke about that a lot. Peering cruise …jeeshhh.

I suppose you can joke about anything you want. Personally, I’ve never been on a
peering cruise, but I will say that it’s a pretty classic fallacy to discount 
the
value of the social times at conferences. In fact, I find those times to often
be when most of the real work gets done and when most of the benefit of getting
everyone together in the same place is realized.

While NANOG puts on a good technical program, my company gets far more benefit 
from
the time I spend meeting with network partners, suppliers, and potential 
customers
during the conference than they will ever see from my time in the sessions. So 
much
so that I generally attend the sessions on an as-available basis  when I’m not 
able
to schedule a more useful meeting. There are, of course exceptions. Some of the
sessions (maybe 3-4 per conference) are worth holding my time open to attend, 
but
most are not.

This does not mean that I don’t consider NANOG valuable, just that the primary 
value
is in the ability to meet with other attendees rather than directly in the 
technical
program itself.

I don’t mean this to be insulting to NANOG. I think the PC generally does a 
fine job
of putting on a good conference with great content. Most importantly, it’s good 
enough
that it draws in a large selection of people I need/want to meet with in a 
concentrated
time frame in a single location.

Peering fora, peering cruises, and the like have a similar effect.

So scoff all you want, but if you imagine that these events are silly junkets 
where
nothing gets accomplished, then you are seriously underestimating this 
community, IMHO.

Owen

Re: Strange Problem with 16 byte packets

[Ruairi Carroll]

Follow the TCP stream - which side times out the link, and for what
sequences of data do you get ACKs for?

/Ruairi

On 16 June 2016 at 10:43, Glen Kent  wrote:

> Hi,
>
> I am using a proprietary protocol and sending a bunch of bytes to a Draytek
> router at an enterprise site. When i send the data in TCP batches of 1 MB i
> see no problem. However,  when i first send 16 bytes followed by 1 MB of
> data, and then repeat this till the entire data has beeen sent out. During
> this process I see that my TCP session times out. Unable to understand why
> this could be happening? How can sending 16 bytes of data followed by 1MB
> of data affect the transfer.
>
> Thanks !
>

Re: Barefoot "Tofino": 6.4 Tbps whitebox switch silicon?

[Saku Ytti]

On 16 June 2016 at 06:21, Eric Kuhnke  wrote:
> Based on their investors, could have interesting results for much lower
> cost 100GbE whitebox switches.

Why lower cost? The BOM isn't the expensive part, the code is the
expensive part. Only way I see this happening, is if we get open
source routing suite for the box, i.e. 0 cost software.

If you're thinking of writing your own routing suite, even if your
requirements are trivial, it's still probably take 2-3 years and
+2MUSD in salaries, and then maintenance +300kUSD/year in salaries.
Need quite significant annual unit number scale to make it cheap.

I'm quite fascinated by the idea of doing something really novel in
routing suite space, but I don't see how it could possibly work
commercially. How many customers would there be for licensing COTS
routing-suite when costs are millions annually to develop it for
general use-case.

-- 
  ++ytti

Re: Strange Problem with 16 byte packets

[Ruairi Carroll]

It's hard to tell based on no data. Anything from here would be an
assumption and hear-say, since you're debugging a black box and trying to
infer inner workings based on external observations.

You _need_ to collect more data and observe the data at the source and
destination devices, and probably in transit if you're
tunnelling/fragmenting.

Basica,


On 16 June 2016 at 11:36, Glen Kent  wrote:

> Thanks i will. However, the doubt is that what does introducing a 16 byte
> data into the steam does that causes the session to time out. I added
> instrumentation to push some dummy data so that instead of 16 bytes, we
> push 1 MB of data. In that case i saw no issues. Any idea if there is a
> firewall setting that could be coming into play here?
>
> On Thu, Jun 16, 2016 at 2:17 PM, Ruairi Carroll 
> wrote:
>
>> Follow the TCP stream - which side times out the link, and for what
>> sequences of data do you get ACKs for?
>>
>> /Ruairi
>>
>> On 16 June 2016 at 10:43, Glen Kent  wrote:
>>
>>> Hi,
>>>
>>> I am using a proprietary protocol and sending a bunch of bytes to a
>>> Draytek
>>> router at an enterprise site. When i send the data in TCP batches of 1
>>> MB i
>>> see no problem. However,  when i first send 16 bytes followed by 1 MB of
>>> data, and then repeat this till the entire data has beeen sent out.
>>> During
>>> this process I see that my TCP session times out. Unable to understand
>>> why
>>> this could be happening? How can sending 16 bytes of data followed by 1MB
>>> of data affect the transfer.
>>>
>>> Thanks !
>>>
>>
>>
>

Re: 1GE L3 aggregation

[Pierre Emeriaud]

Hello Saku,


> I've casually talked with other people, and it seems I'm not really
> alone here. My dream box would be 96xSFP + 2xQSFP28, with pretty much
> full edge features (BGP, LDP, ISIS, +1M FIB, +5M RIB, per-interface
> VLANs, ipfix or sflow, at least per-port QoS with shaper, martini
> pseudowires).

I'd go with a Nokia/ALu 7750. Two 48x1G IMM, or 5x m20-1gb-xp-sfp MDA
+ 3 IOM3-XP, and 100G IMMs (1 or 2 CFP ports - no idea about qsfp28
options though) with SF/CPM5. This would fit in a SR7. Not sure about
the smaller models (-c or -a series). The 7450 might be an option too,
but I don't know this hardware.

RIB size is 20M iirc (shared between all af) and fib size 3 or 5M
depending of the card. SR-OS is weird when you're used to ios or
junos, but in the end the cli is very efficient and quite enjoyable
actually.


HTH,
pierre

Re: Strange Problem with 16 byte packets

[Eygene Ryabinkin]

Thu, Jun 16, 2016 at 03:06:24PM +0530, Glen Kent wrote:
> Thanks i will. However, the doubt is that what does introducing a 16 byte
> data into the steam does that causes the session to time out. I added
> instrumentation to push some dummy data so that instead of 16 bytes, we
> push 1 MB of data. In that case i saw no issues. Any idea if there is a
> firewall setting that could be coming into play here?

Collect TCP dumps from both sides, come back and show that
(or analyze yourself ;)
-- 
Eygene Ryabinkin, National Research Centre "Kurchatov Institute"

Always code as if the guy who ends up maintaining your code will be
a violent psychopath who knows where you live.

Re: Strange Problem with 16 byte packets

[Randy Bush]

tcpdump is your friend

Re: RPKI implementation

[Jakob Heitz (jheitz)]

That is also configurable.

Thanks,
Jakob.


On Jun 16, 2016, at 4:39 AM, Randy Bush  wrote:

>> When a cache loses connectivity, the entries from that cache
>> are purged after a time interval. Default is 60 seconds
> 
> why not the poll interval for that cache server?
> 
> randy

Re: Barefoot "Tofino": 6.4 Tbps whitebox switch silicon?

[Jennifer Rexford]

> I do think p4 is very interesting, but is it really much different from
> openflow? Which...umm ... Did not succeed in the market.

P4 is quite different from OpenFlow.  The various generations of the OpenFlow 
specification assume a fixed-format packet header with an ever increasing 
number of header fields (from 12 in OF 1.0 to 41 in OF 1.4), a fixed set of 
packet-processing actions, and so on. P4 allows programmable parsing and 
flexible match-action processing.  In P4, the program tells the data plane how 
to act; in OpenFlow, the way the data plane can act is “baked” in advance.  So, 
OpenFlow 1.x can be one of many *programs* you can write in P4, whereas many 
other P4 programs can exist.  For some discussion of the differences, see

  Clarifying the differences between P4 and OpenFlow
  http://p4.org/p4/clarifying-the-differences-between-p4-and-openflow/ 


— Jen

Re: Barefoot "Tofino": 6.4 Tbps whitebox switch silicon?

[Prem Jonnalagadda]

On Wed, Jun 15, 2016 at 8:31 PM, Ca By  wrote:

> On Wednesday, June 15, 2016, Eric Kuhnke  wrote:
>
> > a lot of PR fluff, but this may be of interest:
> >
> >
> >
> http://www.wired.com/2016/06/barefoot-networks-new-chips-will-transform-tech-industry/
> >
> >
> >
> https://barefootnetworks.com/media/white_papers/Barefoot-Worlds-Fastest-Most-Programmable-Networks.pdf
> >
> >
> > Based on their investors, could have interesting results for much lower
> > cost 100GbE whitebox switches.
> >
>
> Where is the price tag? Why would you think it is inexpensive?
>
> I do think p4 is very interesting, but is it really much different from
> openflow?


Great question!

OpenFlow enabled decoupling of the control plane from the data plane,
whereas P4 allows you to define the data plane. Check out this popular P4
program that defines the data plane of a L2/L3 switch -
https://github.com/p4lang/switch/tree/master/p4src

OpenFlow is a protocol, whereas P4 is a programming language. You can use
OpenFlow controller to control a P4 data plane - Check out this OpenFlow
agent on top of a P4 data plane - https://github.com/p4lang/p4ofagent

Check out this blog post which explains the differences between P4 and
OpenFlow quite well -
http://p4.org/p4/clarifying-the-differences-between-p4-and-openflow/

More questions? Shoot me an email :-)

Which...umm ... Did not succeed in the market.
>

Re: Barefoot "Tofino": 6.4 Tbps whitebox switch silicon?

[Colton Conor]

Saku,

I agree completely. Isn't this what Arista did? They coded from like 2004
to 2008 before launching EOS using commercial  chipsets. You seem to really
understand routing software, so I would love to hear your take on Arista
EOS.

On Thu, Jun 16, 2016 at 3:19 AM, Saku Ytti  wrote:

> On 16 June 2016 at 06:21, Eric Kuhnke  wrote:
> > Based on their investors, could have interesting results for much lower
> > cost 100GbE whitebox switches.
>
> Why lower cost? The BOM isn't the expensive part, the code is the
> expensive part. Only way I see this happening, is if we get open
> source routing suite for the box, i.e. 0 cost software.
>
> If you're thinking of writing your own routing suite, even if your
> requirements are trivial, it's still probably take 2-3 years and
> +2MUSD in salaries, and then maintenance +300kUSD/year in salaries.
> Need quite significant annual unit number scale to make it cheap.
>
> I'm quite fascinated by the idea of doing something really novel in
> routing suite space, but I don't see how it could possibly work
> commercially. How many customers would there be for licensing COTS
> routing-suite when costs are millions annually to develop it for
> general use-case.
>
> --
>   ++ytti
>

Re: NANOG67 - Tipping point of community and sponsor bashing?

[Ca By]

On Thursday, June 16, 2016, Owen DeLong  wrote:

> > Cough cough ARIN cough. I don't know why they need to meet face to face 2
> > or 3 times a year. But, i am sure ppml will tell you it is a ground up
> > process and these people on ppml like traveling and talking about
> > policy And they do what members want.
>
> I don’t speak for the organization or even for the Advisory Council, but I
> can
> tell you that we get more focused community input and participation during
> those
> face to face meetings than we do during the rest of the year.
>
> Like it or not, when trying to get interactive participation from more
> than 5 or so people,
> there’s really no useful substitute for the face-to-face meeting. Every
> existing
> alternative is less useful and comes with significant drawbacks.
>
> So, yes, I believe ARIN does what the members want, but more importantly, I
> believe that the face to face meetings are a useful mechanism to preserve
> the
> bottom up nature of the policy development process.
>
> > Much of our industry can be gleened by googling pictures of "peering
> > cruise". At my office, we joke about that a lot. Peering cruise …jeeshhh.
>
> I suppose you can joke about anything you want. Personally, I’ve never
> been on a
> peering cruise, but I will say that it’s a pretty classic fallacy to
> discount the
> value of the social times at conferences. In fact, I find those times to
> often
> be when most of the real work gets done and when most of the benefit of
> getting
> everyone together in the same place is realized.
>
> While NANOG puts on a good technical program, my company gets far more
> benefit from
> the time I spend meeting with network partners, suppliers, and potential
> customers
> during the conference than they will ever see from my time in the
> sessions. So much
> so that I generally attend the sessions on an as-available basis  when I’m
> not able
> to schedule a more useful meeting. There are, of course exceptions. Some
> of the
> sessions (maybe 3-4 per conference) are worth holding my time open to
> attend, but
> most are not.
>
> This does not mean that I don’t consider NANOG valuable, just that the
> primary value
> is in the ability to meet with other attendees rather than directly in the
> technical
> program itself.
>
> I don’t mean this to be insulting to NANOG. I think the PC generally does
> a fine job
> of putting on a good conference with great content. Most importantly, it’s
> good enough
> that it draws in a large selection of people I need/want to meet with in a
> concentrated
> time frame in a single location.
>
> Peering fora, peering cruises, and the like have a similar effect.
>
> So scoff all you want, but if you imagine that these events are silly
> junkets where
> nothing gets accomplished, then you are seriously underestimating this
> community, IMHO.
>
> Owen
>
>
>
Owen,

I agree with most of what you are saying. I'll digress on if arin needs to
meet or exist.

Perhaps it is me and my sensibilities, perhaps it  is my miser corp
culture, but i could not even dream of asking to go to Jamaica (arin
area) for the last ARIN meeting.

I am not alone. Have a look at Ren's comments from

http://research.dyn.com/2006/02/lovely-peering-cruise-on-lake/

Posted here:
"
The Peering Forum is more for peer & IX information distribution and
contact refresh across a multi-continent body of participants than it is
for initial trial concerns. The invite only nature implies the attendees
are actively peering at one or more of the IXs sponsoring portions of the
Forum. Many of us, hand raised here, are taking vacation time, covering
flights, upgraded cabins, etc. to help remove the conflict of interest
concerns. It is unfortunate there is not a link to the agenda as it is jam
packed with useful peering focused presentations and more than qualifies
this as a business need. Last year dozens of participants interacted for
the first time and I’m looking forward to similar introductions later this
month. Putting a face to the name helps significantly in this very
relationship based role which has more to do with international relations
than enable."

I am not willing to fork over my pto or personal cash to "remove the
conflict of interest" . I'd rather buy transit.

And, i am also not willing to spend my corp money with dec-ix or others to
send my competitors on a cruise. Unless ...

why cant this just be business?

That said, i have never been on a peering cruise and all my peering needs
are met with peeringdb and email. So it is just business for me, and no i
am not going to spend money at an IX that does not see things the way i do.

CB

Re: NANOG67 - Tipping point of community and sponsor bashing?

[Zbyněk Pospíchal]

Dne 15.06.16 v 20:10 Mikael Abrahamsson napsal(a):

> Well, the customers also wanted more functions and features. They wanted
> sFLOW statistics to show traffic, customer portals, better SLAs,
> distributed IXes, remote peering, more hand-holding when connecting etc.

Are you sure they still want them if they have to pay for these features
separately?

Currently, such luxury functions are increasing costs also for
networks who don't need/want it.

Best Regards,
Zbynek

Re: RPKI implementation

[Randy Bush]

> When a cache loses connectivity, the entries from that cache
> are purged after a time interval. Default is 60 seconds

why not the poll interval for that cache server?

randy

Re: NANOG67 - Tipping point of community and sponsor bashing?

[Job Snijders]

On Thu, Jun 16, 2016 at 03:52:02PM +0200, Nurani Nimpuno wrote:
> A growing exchange point is not only a "nice-to-have" for those
> operating it, but vital to those networks who peer there. If you stop
> adding value to those networks peering at the IX, you will slowly
> become irrelevant. 

I view this differently: an IXP that does not grow, still provides value
to the existing members/customers. The value is not growth itself, just
like the amount of connected networks is not a metric for value, but a
metric for the potential of value.

The value an IXP brings can be defined as the delta between what it
would cost to 'do it yourself' and run everything over private peering
and using a public internet exchange. And this definition of value is
underlined by the fact that a "public to private" lifecycle exists. 

> We work in a similar way with our pricing. (You mention that there is
> a lot of negotiations on pricing with IXPs.) I would like to be 100%
> clear that for the Netnod IX, we don’t negotiate or give “sweet deals”
> to anyone. We publish our fee schedule and we stick to it.

This is an admirable quality. I believe an IXP is most successful when
it presents a level playing field. Hope you never change this :)

Kind regards,

Job

Re: 1GE L3 aggregation

[Saku Ytti]

On 16 June 2016 at 22:36, Baldur Norddahl  wrote:

Hey,

> If I need to speak BGP with a customer that only has 1G I will simply make
> a MPLS L2VPN to one of my edge routers. We use the ZTE 5952E switch with
> 48x 1G plus 4x 10G for the L2VPN end point. If that is not enough the ZTE
> 8900 platform will provide a ton of ports that can do MPLS.

I wonder if you'd do this, if you could do L3 to the edge. And why is
termination technology dependant on termination rate?

> The tunnel is automatically redundant and will promote link down events, so
> there is not really any downside to doing it this way on low bandwidth
> peers.

When you say redundant, do you mean that label can take any path
between access port and termination IRB/BVI? Or do you actually have
termination redundancy?
If you don't have termination redundancy, you have two SPOF, access
port and termination.
If you do have termination redundancy, you're spending control-plane
resource from two devices, doubling your control-plane scale/cost.


I'm not saying it's bad solution, I know lot of people do it. But I
think people only do it, because L3 at port isn't offered by vendors
at lower rates.

-- 
  ++ytti

Re: NANOG67 - Tipping point of community and sponsor bashing?

[Eric Kuhnke]

 On the point raised by this
index of IXP costs - has anyone put together a table of information on the
opposite side of the question: What is the cost of establishing a PNI
direct crossconnect in a major IX point?

This varies widely by particular location and which real estate investment
company/datacenter developer happens to own a particular property.

There's to-remain-unnamed colo & power providers who like to get people
into a facility with cheap rack & power and then charge hundreds of dollars
per month per fiber pair for a regular 9/125 XC between two racks or cages
in the same facility.

Others where the cost is $0 monthly and you can pull your own fiber through
the overhead trays to establish PNI. Others where you pay monthly rent for
your own 144-strand 4U patch panel in a fiber MMR but the XCs between
panels are $0/MRC.

Also worth knowing for comparison to the spreadsheet in #3 below: To what
extent are intra-facility XC costs available without NDAs to publish and
share? Which colo facilities consider NRCs and MRCs for fiber XCs to be
proprietary information?


3 -
> https://docs.google.com/spreadsheets/d/18ztPX_ysWYqEhJlf2SKQQsTNRbkwoxPSfaC6ScEZAG8/edit#gid=0

Re: 1GE L3 aggregation

[Baldur Norddahl]

On 16 June 2016 at 22:27, Saku Ytti  wrote:

> On 16 June 2016 at 22:36, Baldur Norddahl 
> wrote:
>
> Hey,
>
> > If I need to speak BGP with a customer that only has 1G I will simply
> make
> > a MPLS L2VPN to one of my edge routers. We use the ZTE 5952E switch with
> > 48x 1G plus 4x 10G for the L2VPN end point. If that is not enough the ZTE
> > 8900 platform will provide a ton of ports that can do MPLS.
>
> I wonder if you'd do this, if you could do L3 to the edge. And why is
> termination technology dependant on termination rate?
>

The ZTE 5952E (routing switch) can do L3VPN including BGP. But it is
limited to about 30k routes. It is usable if the customer wants a default
route solution, but not if he wants the full default free zone.

The ZTE M6000S-2S4 (carrier grade router) will do all you want, however it
is more expensive. We use the MPLS routing switch because it is a $2k
device compared to the router which is more like $15k.

As a small ISP we have two edge routers (the slightly larger M6000-S3 which
is about $20k). Our customers are spread out throughout the city and we
have 26 PoPs, so it is much more cost effective to have the cheaper device
put the traffic in a tunnel and haul it back to the big iron.


> > The tunnel is automatically redundant and will promote link down events,
> so
> > there is not really any downside to doing it this way on low bandwidth
> > peers.
>
> When you say redundant, do you mean that label can take any path
> between access port and termination IRB/BVI? Or do you actually have
> termination redundancy?
>

Our PoPs are connected in a ring topology (actually multiple rings). If a
link goes down somewhere, or an intermediate device crashes, the L2VPN will
reconfigure and find another path.


> If you don't have termination redundancy, you have two SPOF, access
> port and termination.
>

For a BGP customer I could offer two tunnels, one to each of our provider
edge routers. But very few of our customers are BGP customers, they just
want normal internet. For them we do VRRP between the two provider edge
routers and have the one tunnel go to both.


> If you do have termination redundancy, you're spending control-plane
> resource from two devices, doubling your control-plane scale/cost.
>

The M6000 devices can handle 64k tunnels and are generally way overpowered
for our current business. It is true that I might be limited to 1x 64k
customers instead of 2x 64k customers, but with that many customers I would
need to upgrade anyway.


> I'm not saying it's bad solution, I know lot of people do it. But I
> think people only do it, because L3 at port isn't offered by vendors
> at lower rates.
>


We actually moved away from a hybrid solution with L3 termination at the
customer edge to simply backhauling everything in L2VPNs. We did this
because the L2VPN tunnels are needed anyway for other reasons and it is
easier to have one way to do things.

Regards,

Baldur

Re: NANOG67 - Tipping point of community and sponsor bashing?

[Phil Rosenthal]

Hello all,

I wasn't able to attend NANOG this time around, but watched Dave Temkin's 
presentation on youtube.

My comments are:
1) Over the past 5 years:
My cost for switch/router ports have gone down a lot.
My cost for transit has gone down a lot.
My cost for exchange ports have gone down, but not quite as fast as my transit 
and switch/router ports, and this does lead to some value questions. Dave is 
right to ask them.

However: exchange port fees are not my biggest enemy today. My cross connect 
fees have not gone down *at all*. On a proportion basis, cross connect fees 
have gone from "not mattering" to being an important part of any deployment 
cost calculation. Why aren't we raising hell about cross connect fees?

2) Exotic features -- Pvlan, L2VPN, L3VPN have absolutely no purpose on an 
exchange. If it could be done 'free' with commodity hardware, then fine -- but 
if it translates to requiring Big Expensive Routers instead of a cheaper but 
fast switch, this should translate to higher pricing for the customers 
requiring these exotic features -- not the customers who just want a big L2 
vlan.

3) Remote peering -- This is mostly a question about distance for value.  There 
is a clear benefit in providing multi-datacenter exchanges within a metro, and 
both FL-IX and SIX are doing this with a very good value proposition. Having 
the ability to join DECIX Frankfurt from NYC and vice versa -- again, this is a 
bizarre service to be offered, and regular users should not be expected to pay 
for this. If there is a market for these services at an unsubsidized price, 
then fine -- but regular members should not be subsidizing this service.

4) sFlow -- I'm not sure why this is even really a topic. Commodity hardware 
does have sFlow capability, and FLIX demonstrates this well. With that said, 
for us, it is of extremely limited value. We might check these graphs to 
validate measurements of our internal netflow/sflow graphing systems, but 
generally, I look at the graphs generated by my exchange vendors less than once 
per year per exchange. I am honestly not even sure if SIX offers this service, 
as I never had a reason to check.

5) Marketting vs Outreach: These things are honestly basically the same thing, 
mostly separated by the question of "is it good marketing or not". I like 
having more members at the exchanges I am a member of. If it translates to an 
additional 3% per year to have an additional 5% of traffic to new members, I am 
fine with this.  If it translates to an extra 50% of cost for 5% of additional 
traffic, I am not fine with it.

Finally -- there is nothing wrong with asking questions. If you are an exchange 
company and you can defend your prices for what you offer, then there is no 
problem.  If you are an exchange and are mostly just hoping nobody asks the 
questions because you won't have any good answers -- well, I think this is 
exactly why Dave asked the question.

Best Regards,
-Phil Rosenthal
> On Jun 16, 2016, at 1:58 PM, Adam Rothschild  wrote:
> 
> I think a fresh conversation is needed around what makes up a
> "minimally viable" feature set for an IXP:
> 
> The days of an IXP "needing" to engineer and support a multi-tenant
> sFlow portal, because the only other option is shelling out the big
> bucks for Arbor, have long passed -- overlooking the plethora of open
> sourced tools, folk like Kentik have broken into that market with
> rationally priced commercial alternatives.  Likewise, one might argue
> that offering layer-2 and layer-3 (!) VPNs is at best non-essential,
> and a distraction that fuels purchasing very expensive hardware, and
> at worse competitive with customers.
> 
> On the other hand, building out a metro topology to cover all relevant
> carrier hotels, with reasonable path diversity, is absolutely table
> stakes.  And outreach is a great function, *when* it nets unique new
> participants.  To cite a recent example, the various R networks and
> smaller broadband and mobile providers showing up here in the US, due
> to excellent efforts by the NYIIX and DE-CIX teams.
> 
> At the end of the day, IXP peering must be significantly cheaper than
> transit alternatives, many of which are priced based on utilization
> (as opposed to port capacity).  We can dance around this point all we
> want, but absent a change in trajectory, I worry some IXPs will
> ultimately price themselves out of the market, and all the gold-plated
> features in the world won't satiate those making purchasing decisions.
> 
> $0.02,
> -a
> 
> On Thu, Jun 16, 2016 at 11:17 AM, Niels Bakker 

Re: NANOG67 - Tipping point of community and sponsor bashing?

[Eric Kuhnke]

> However: exchange port fees are not my biggest enemy today. My cross
connect fees have not gone down *at all*. On a proportion basis, cross
connect fees have gone from "not mattering" to being an important part of
any deployment cost calculation. Why aren't we raising hell about cross
connect fees?

IMHO we should be, in the spirit of:
https://en.wikipedia.org/wiki/Rent_Is_Too_Damn_High_Party

Assuming the existence of overhead fiber trays throughput, when you
consider the actual cost of a two strand XC between two cages in the same
facility:

30 meter SC-SC duplex 9/125 G.657.A1 cable: $11

There should be a community effort to lobby facility managers and colo/IX
real estate management that the value of their facility will be greater if
XCs are free or nearly free, resulting in higher occupancy and a greater
critical mass of carriers, rather than trying to extract revenue from the
tenants by $300/mo MRC per fiber pair between two racks.



On Thu, Jun 16, 2016 at 4:06 PM, Phil Rosenthal  wrote:

> Hello all,
>
> I wasn't able to attend NANOG this time around, but watched Dave Temkin's
> presentation on youtube.
>
> My comments are:
> 1) Over the past 5 years:
> My cost for switch/router ports have gone down a lot.
> My cost for transit has gone down a lot.
> My cost for exchange ports have gone down, but not quite as fast as my
> transit and switch/router ports, and this does lead to some value
> questions. Dave is right to ask them.
>
> However: exchange port fees are not my biggest enemy today. My cross
> connect fees have not gone down *at all*. On a proportion basis, cross
> connect fees have gone from "not mattering" to being an important part of
> any deployment cost calculation. Why aren't we raising hell about cross
> connect fees?
>
> 2) Exotic features -- Pvlan, L2VPN, L3VPN have absolutely no purpose on an
> exchange. If it could be done 'free' with commodity hardware, then fine --
> but if it translates to requiring Big Expensive Routers instead of a
> cheaper but fast switch, this should translate to higher pricing for the
> customers requiring these exotic features -- not the customers who just
> want a big L2 vlan.
>
> 3) Remote peering -- This is mostly a question about distance for value.
> There is a clear benefit in providing multi-datacenter exchanges within a
> metro, and both FL-IX and SIX are doing this with a very good value
> proposition. Having the ability to join DECIX Frankfurt from NYC and vice
> versa -- again, this is a bizarre service to be offered, and regular users
> should not be expected to pay for this. If there is a market for these
> services at an unsubsidized price, then fine -- but regular members should
> not be subsidizing this service.
>
> 4) sFlow -- I'm not sure why this is even really a topic. Commodity
> hardware does have sFlow capability, and FLIX demonstrates this well. With
> that said, for us, it is of extremely limited value. We might check these
> graphs to validate measurements of our internal netflow/sflow graphing
> systems, but generally, I look at the graphs generated by my exchange
> vendors less than once per year per exchange. I am honestly not even sure
> if SIX offers this service, as I never had a reason to check.
>
> 5) Marketting vs Outreach: These things are honestly basically the same
> thing, mostly separated by the question of "is it good marketing or not". I
> like having more members at the exchanges I am a member of. If it
> translates to an additional 3% per year to have an additional 5% of traffic
> to new members, I am fine with this.  If it translates to an extra 50% of
> cost for 5% of additional traffic, I am not fine with it.
>
> Finally -- there is nothing wrong with asking questions. If you are an
> exchange company and you can defend your prices for what you offer, then
> there is no problem.  If you are an exchange and are mostly just hoping
> nobody asks the questions because you won't have any good answers -- well,
> I think this is exactly why Dave asked the question.
>
> Best Regards,
> -Phil Rosenthal
> > On Jun 16, 2016, at 1:58 PM, Adam Rothschild  wrote:
> >
> > I think a fresh conversation is needed around what makes up a
> > "minimally viable" feature set for an IXP:
> >
> > The days of an IXP "needing" to engineer and support a multi-tenant
> > sFlow portal, because the only other option is shelling out the big
> > bucks for Arbor, have long passed -- overlooking the plethora of open
> > sourced tools, folk like Kentik have broken into that market with
> > rationally priced commercial alternatives.  Likewise, one might argue
> > that offering layer-2 and layer-3 (!) VPNs is at best non-essential,
> > and a distraction that fuels purchasing very expensive hardware, and
> > at worse competitive with customers.
> >
> > On the other hand, building out a metro topology to cover all relevant
> > carrier hotels, with reasonable path diversity, is absolutely table
> > stakes.  

Re: NANOG67 - Tipping point of community and sponsor bashing?

[Dave Temkin]

A key point:

On Thu, Jun 16, 2016 at 6:09 PM, Baldur Norddahl 
wrote:

I have studied Netnod extensively because we want to become members
>

You meant a customer, but because of a lack of transparency (and great
marketing) amongst some IXPs, it's very easy to conflate member-mutual
IXPs, commercial, and non-profit IXPs. Being a "member" of NetNod provides
you with a very different set of benefits vs. LINX and unless you read the
letters of incorporation, you may not know that.

Re: NANOG67 - Tipping point of community and sponsor bashing?

[Baldur Norddahl]

Hi,

I have studied Netnod extensively because we want to become members, but we
can not simply because it is too expensive. I just signed a deal with he.net
for a flatrate 10G transit for about the same as the 10G Comix port cost.
The difference being that the he.net port provides much more value and
besides also provides indirect one-step-away peering with everyone on Comix.

So from my perspective it is clear that Netnod has a pricing problem here.
Especially for the lower speeds (10G). There is also a value problem
because the only Comix peer that moves a lot of traffic to us is Akamai,
and they promised that we could peer directly skipping the middleman.

We are based in Copenhagen. The Netnod IX in Stockholm would provide a lot
more value, but to get there we have to add the cost for transport and
after doing that, the comparison to the 10G he.net transit is just silly.

Here is an opinion: If the IX pricing is comparable to transit, the service
needs to be too. Netnod will need to connect the five (I think there are
five) Netnod IX'es into one big domain. I am meeting with NL-IX next week
and as I understand it, that is exactly what they did - we will probably
buy NL-IX and skip Netnod for this single reason.

I feel that smaller providers are being let down by the IX community at
this point. The value of "smaller" is going to get larger if the IX
community does not move with the transit providers. We want to take part
but there is a limit of how much over price you can sign onto and keep your
job.

Regards,

Baldur






On 16 June 2016 at 15:52, Nurani Nimpuno  wrote:

> Hi Dave,
>
> So, I watched your presentation this week at NANOG remotely, sorry I
> couldn’t be there.
>
> Ok, so while you make a lot of very different points in your
> presentations, I *think* the basic argument you are making is that IXPs are
> too expensive. Correct me if I’m wrong. Or more specifically, you are
> saying that Ams-IX, Linx, Netnod and DE-CIX are too expensive. You have not
> looked at US IXPs because they don’t publish their fees, and you have not
> looked at the whole IXP community.
>
> I think you are then also questioning if these IXPs are using their funds
> wisely. You are also stating that you are talking about these IXPs from the
> perspective of a big US provider connecting into Europe (i.e not a small
> local ISP). You question some of the IXP expansions into the US. You
> question the membership model as a viable model for IXPs. You also say that
> those who sustain the IXPs growth should benefit from them. And you
> question why there are so many IXPs, and not only a handful of very big
> ones. I hope I have captured this correctly.
>
> Ok, so firstly, I must say I’m a little disappointed that you or your
> staff have never approach us to discuss any of this. We have Netnod
> meetings twice a year, we have been present at many of the same events in
> the last year and we have always strived to be open, transparent and to
> listen to our entire customer base. I take your point about the Netnod fees
> (even though I would also like to point out that we have actually reduced
> our other port fees for 100mbps, 1G, remote peering). But I’m not sure why
> you haven’t brought it to us directly. Netflix has been at several Netnod
> meetings in the past, so we have had plenty of opportunity to discuss this.
>
> But ok, let’s leave that aside. I will try to address some of your points.
>
> Firstly as many have pointed out, these four IXPs are not representative
> of all IXPs, and the four of us are also very different from each other.
>
> I can’t address the IXP expansion into the US. And I don’t represent a
> membership-based IXP.
>
> The European IXP community is a very diverse one, serving different
> regions, markets and different types of customers. I personally believe
> that this rich diversity is one of the reasons the European interconnection
> scene has been flourishing as well as it has. There is a big difference
> between Europe and the rest of the world, particularly the US. And the
> European IXP community was held up as a model for the rest of the world by
> many. We have been cooperating for many years through the Euro-IX where our
> common goals have been to improve interconnection in the region, share
> information and experience and work to improve services for our customers.
> (I believe you have been trying to do the same through Open-IX.)
>
> The diversity has also been seen as important to serve both the very large
> international providers like yourself, and the small local ISPs. Localising
> traffic and building a local operator community have been seen as an
> important ingredient in the value of the IXPs. Our challenge as IXPs is to
> find the best way to serve all these different needs and wishes from our
> very diverse customer base. Having only a handful of very large IXPs would
> in my view not serve these different needs as well. Personally I am a
> subscriber 

Re: NANOG67 - Tipping point of community and sponsor bashing?

[Nick Hilliard]

Dave Temkin wrote:
> They are representative of the most important IXPs to deliver traffic
> from in Western Europe. 

I don't doubt that they are important IXPs for delivering traffic.
However, no other IXP in europe (both eastern and western) is doing
expansion outside the countries that they operate in, other than three
out of the four that you mentioned; none of the member-owned
organisations in the region are making large profits or in most cases
anything more than marginal profits, and all of them have lower port
costs.  Also, none of their activities suggest that their marketing
budgets are large.  These, I think, were the main points of contention
you were concerned about.

> I would posit that what defines important to me may not be what defines
> important to you and the same can be said when you look at how various
> "internet" companies look at what's important in their vertical.

We're not talking about relative importance; we're talking about whether
the problems you identified with the four IXPs named in your talk are
representative of problems with the larger IXP community.  I cannot find
evidence that they are, at least not in the areas that you identified as
problems.

> Netnod runs a dns root server
> system (i.root-servers.net ) as well as a
> heavy duty time service.  
> 
> There are others who do this for no cost and some who do it for
> government money. Whether or not my port fees should subsidize this is a
> valid question, and was brought up in the Q afterwards.

All root operators do this for no charge, but at substantial cost.

Running a root dns server system is one of the things what Netnod does
because that's one of the things that the organisation is chartered to do.

> Regarding the pricing reduction on page 16 of your preso, the US$ and
> UK£ are not much different than what they were 5 years ago, but the €
> has dropped by 30% against the US$.  
> 
> You speak to this below, however if my business is primarily run in USD
> (which was the relevant use case presented: I'm a US company deciding if
> I should peer in Europe or buy transit) then those currency fluctuations
> have a very different impact than if I'm a European company functioning
> primarily in local currency.

Oh sure, but this is a matter that you need to take up with your
financial people.  I have no doubt that Netflix employs smart financial
people, and that their decisions are the right thing for Netflix.

IXPs are going to operate in their local currency and they cannot be
held responsible for international currency fluctuations.  From this
respect, I don't think it's useful to bring this up in a critical
context because it's not something that they can influence in any way
whatever.

> I did purposefully mention SIX as a polar opposite example - there is
> definitely a happy medium to be found.

This edges into one of the things that is crucial to this discussion,
and it was unfortunate that it wasn't explored more.  The crux is that
there is a substantial cultural difference between how US people view
IXPs and how european people view IXPs.

As far as I can tell there are, for the most part, two types of IXPs in
the US: commercial and co-operative.  How they differ from european IXPs
is that the commercials are almost all run by the data centres and are
tied to those data centres.  Most if not all of the co-operative IXPs
are to some degree or other financed by donations or sponsorship and the
donation types are: cash, equipment and manpower.

In europe, there are three types of IXP: commercial, member based and
non-member, non-profit.  Many of the commercial IXPs are not owned by
the data centres (e.g. NL-IX, ECIX, etc).  The member-owned IXPs are
answerable fully to their membership (e.g. LINX, INEX), and the
non-member, non-profit IXPs (Netnod, VIX, etc) provide a service to the
community as they see fit, but are not required to answer to the
organisations who use them for peering services, even if they are likely
to listen to what those organisations say.

Crucially, almost all of the european non-profit IXPs are 100%
self-funded without donations, sponsorship or subsidisation of manpower.
 They have offices, admin people, support staff and everything else that
a normal business has.  IXP staff are paid market rates because if you
don't do this, they walk.

They do this because this having a dependable income source ensures long
term stability and their members / customers / peering participants need
to know that the organisation that supports their business is built on a
sound structural foundation.  This matters to them and they are prepared
to pay for it.

The flip side of it is that European non-profit IXPs are, by design,
more expensive than US non-profit IXPs and there are almost no free /
zero-recurrent-cost IXPs in the region.

So I would question comparing a european IXP with e.g. SIX or
CommunityIX and would suggest that this is a case of 

Re: NANOG67 - Tipping point of community and sponsor bashing?

[Daniel Golding]

On Thu, Jun 16, 2016 at 12:58 PM Will Hargrave  wrote:

{snip}



> Dan Golding disagreed with me but I can certainly speak for LONAP where
> I feel our mission of “promoting efficient interconnection in the
> UK” is hugely enhanced by our ability to provide services in any of
> our current seven datacentres, across four different operators. London
> would not be the great city of interconnection it is without the east
> London cluster of DCs from different operators.
>
>
Of course, that's not what I said, but only people who were present
actually know that. You said that LONAP's distributed strategy "kept
datacenters honest" to use your exact quote. That implied some sort of
benefit for members in acting as some sort of counterweight to (rapacious?)
data center providers. I made the point that distributed IX's don't really
impact power or space costs in data centers. I can provide actual data on
this, if you would like.

Dan



> {snip}
>
>
> Will
>

Re: NANOG67 - Tipping point of community and sponsor bashing?

[Baldur Norddahl]

On 17 June 2016 at 03:18, Dave Temkin  wrote:

> A key point:
>
> On Thu, Jun 16, 2016 at 6:09 PM, Baldur Norddahl <
> baldur.nordd...@gmail.com> wrote:
>
> I have studied Netnod extensively because we want to become members
>>
>
> You meant a customer, but because of a lack of transparency (and great
> marketing) amongst some IXPs, it's very easy to conflate member-mutual
> IXPs, commercial, and non-profit IXPs. Being a "member" of NetNod provides
> you with a very different set of benefits vs. LINX and unless you read the
> letters of incorporation, you may not know that.
>
>
Ok, customer. Actually I don't care. I will evaluate the business value no
matter who owns it and I will vote with my dollars/euros/kroner.

Yes it would be nice if we had a true self owned entity that would help
people peer the most cost effective way and only that. Apparently they have
something like it in Seattle. But not in my part of the world, hence I view
them all as purely commercial entities. I need a good balance of price and
quality and will buy from whoever comes with the best business proposal.

Regards,

Baldur

Re: NANOG67 - Tipping point of community and sponsor bashing?

[Nurani Nimpuno]

Hi Dave,

So, I watched your presentation this week at NANOG remotely, sorry I couldn’t 
be there.

Ok, so while you make a lot of very different points in your presentations, I 
*think* the basic argument you are making is that IXPs are too expensive. 
Correct me if I’m wrong. Or more specifically, you are saying that Ams-IX, 
Linx, Netnod and DE-CIX are too expensive. You have not looked at US IXPs 
because they don’t publish their fees, and you have not looked at the whole IXP 
community. 

I think you are then also questioning if these IXPs are using their funds 
wisely. You are also stating that you are talking about these IXPs from the 
perspective of a big US provider connecting into Europe (i.e not a small local 
ISP). You question some of the IXP expansions into the US. You question the 
membership model as a viable model for IXPs. You also say that those who 
sustain the IXPs growth should benefit from them. And you question why there 
are so many IXPs, and not only a handful of very big ones. I hope I have 
captured this correctly. 

Ok, so firstly, I must say I’m a little disappointed that you or your staff 
have never approach us to discuss any of this. We have Netnod meetings twice a 
year, we have been present at many of the same events in the last year and we 
have always strived to be open, transparent and to listen to our entire 
customer base. I take your point about the Netnod fees (even though I would 
also like to point out that we have actually reduced our other port fees for 
100mbps, 1G, remote peering). But I’m not sure why you haven’t brought it to us 
directly. Netflix has been at several Netnod meetings in the past, so we have 
had plenty of opportunity to discuss this. 

But ok, let’s leave that aside. I will try to address some of your points.

Firstly as many have pointed out, these four IXPs are not representative of all 
IXPs, and the four of us are also very different from each other. 

I can’t address the IXP expansion into the US. And I don’t represent a 
membership-based IXP. 

The European IXP community is a very diverse one, serving different regions, 
markets and different types of customers. I personally believe that this rich 
diversity is one of the reasons the European interconnection scene has been 
flourishing as well as it has. There is a big difference between Europe and the 
rest of the world, particularly the US. And the European IXP community was held 
up as a model for the rest of the world by many. We have been cooperating for 
many years through the Euro-IX where our common goals have been to improve 
interconnection in the region, share information and experience and work to 
improve services for our customers. (I believe you have been trying to do the 
same through Open-IX.)

The diversity has also been seen as important to serve both the very large 
international providers like yourself, and the small local ISPs. Localising 
traffic and building a local operator community have been seen as an important 
ingredient in the value of the IXPs. Our challenge as IXPs is to find the best 
way to serve all these different needs and wishes from our very diverse 
customer base. Having only a handful of very large IXPs would in my view not 
serve these different needs as well. Personally I am a subscriber to both 
Netflix and HBO. I like diversity. :) But sure, it’s an interesting discussion 
to be had!

As others have pointed out, contrary to common belief, the technical part of an 
IXP is one of the simplest. There is a plethora of examples of IXPs in Africa, 
but also in the US, where IXPs simply are a single switch sitting in a closet 
somewhere, only serving a handful of ASes. One of the biggest challenges for an 
IXP is to gain customers and get enough gravitation and value to the exchange. 
A growing exchange point is not only a "nice-to-have" for those operating it, 
but vital to those networks who peer there. If you stop adding value to those 
networks peering at the IX, you will slowly become irrelevant. 

While some think that a good technical solution would sell itself, I believe 
that is a fallacy (not only in the IXP world). Netnod started out as a very 
small IXPs with only a few local operators connected to it. And I strongly 
believe that if we hadn’t done as much outreach as we do, we would’ve stayed 
tiny until this day. 

As for how we do this outreach and what events we go to, while I can’t speak 
for any other IXes, I seriously doubt that any professional IXP today would not 
carefully assess the business value for each event it attends. At Netnod, we 
evaluate each event we send people to, and assess and measure the value 
afterwards. 

Then I thought I would write some words about Netnod specifically since you 
bring us up. 

(As others have pointed out, the RIPE meeting social is covered partly by the 
RIPE NCC, partly by the sponsor, and partly by the participants themselves, so 
I’ll just leave that there.) 

Firstly, yes we are a 

Re: NANOG67 - Tipping point of community and sponsor bashing?

[Dave Temkin]

Hi Nurani,

Much of what you've asked me below is answered up-thread, so I'm not going
to rehash it for the sanity of the others following this discussion. I have
snipped what hasn't been.


On Thu, Jun 16, 2016 at 8:52 AM, Nurani Nimpuno  wrote:

>
>
>  I take your point about the Netnod fees (even though I would also like to
> point out that we have actually reduced our other port fees for 100mbps,
> 1G, remote peering). But I’m not sure why you haven’t brought it to us
> directly. Netflix has been at several Netnod meetings in the past, so we
> have had plenty of opportunity to discuss this.
>

Nothing in my presentation said "Netflix seeks to get better port fees".
You'll find that I, not once, in my deck or oral presentation, mentioned
Netflix. I spoke at length with LINX after the presentation and pointed out
that I seek to help the entire market, not just my employer, better
understand how IXPs price their services, what things are negotiable, and
what things need to change. Call it thinly-veiled, but I didn't even use my
employer slide master - this was geared as a community discussion.


> And I don’t represent a membership-based IXP.
>

An important distinction. Poring through
http://www.netnod.se/about/documents , there is very little transparency
into the actual operations of NetNod.

>
>  If you stop adding value to those networks peering at the IX, you will
> slowly become irrelevant.
>

And therein lies the rub, we (many of us, not just you and I) disagree
about what "adding value" is defined as. I'm glad we can have this
conversation.


>
> While some think that a good technical solution would sell itself, I
> believe that is a fallacy (not only in the IXP world). Netnod started out
> as a very small IXPs with only a few local operators connected to it. And I
> strongly believe that if we hadn’t done as much outreach as we do, we
> would’ve stayed tiny until this day.
>

Outreach is fantastic!


>
>
> We work in a similar way with our pricing. (You mention that there is a
> lot of negotiations on pricing with IXPs.) I would like to be 100% clear
> that for the Netnod IX, we don’t negotiate or give “sweet deals” to anyone.
> We publish our fee schedule and we stick to it. Whenever someone wants a
> special deal (which happens often, particularly with the larger customers),
> our response is that we treat everyone equally. If you want a cheaper deal,
> then another customer is basically funding your reduction. So we don’t do
> this. We believe this is more fair and transparent.
>

That's fantastic, and I agree with this approach. And that's why it's
important to make this a community discussion, not a "Netflix and Netnod"
discussion.


>
> As for a general discussion about costs, service levels and IXPs, I think
> there is a very interesting discussion that could be had with a more
> focused discussion. How do “we” best serve today's very diverse set of
> operators? How does an IXP strike that balance? How do operators best solve
> their interconnection needs (through IXPs, private peering, transit etc)
> and is that changing? What type of interconnection environment do we
> believe best scales Internet growth in the future? What is the total cost
> of interconnection, where are the big costs, what are the different models
> and where is the whole industry moving? Now THOSE are discussions I
> personally would find very valuable!
>

We agree. I'm really glad that this has sprouted so many threads of
discussion. This seems to have kicked off the discussion within the larger
community beyond just the four examples, and I think that what we've seen
thus far is healthy discourse.

Re: 1GE L3 aggregation

[joel jaeggli]

On 6/16/16 12:51 AM, Saku Ytti wrote:
> Hey,
> 
> I've been bit poking around trying to find reasonable option for 1GE
> L3 full BGP table aggregator. It seems vendors are mostly pushing
> Satellite/Fusion for this application.
> 
> I don't really like the added complexity and tight coupling
> Satellite/Fusion forces me. I'd prefer standards based routing
> redundancy to reduce impact of defects.
> 
> ASR9001 and MX104 are not an options, due to control-plane scale. New
> boxes in vendor pipeline are completely ignoring 1GE.
> 
> I've casually talked with other people, and it seems I'm not really
> alone here. My dream box would be 96xSFP + 2xQSFP28, with pretty much
> full edge features (BGP, LDP, ISIS, +1M FIB, +5M RIB, per-interface
> VLANs, ipfix or sflow, at least per-port QoS with shaper, martini
> pseudowires).
> 
> With tinfoil hat tightly fit on my head, I wonder why vendors are
> ignoring 1GE? Are business cases entirely driven now by Amazon,
> Google, Facebook and the likes? Are SP volumes so insignificant in
> comparison it does not make sense to produce boxes for them?
> Heck even 10GE is starting to become problematic, if your application
> is anything else than DC, because you can't choose arbitrary optics.

There's not a lot of innovation going on in lower end 1G chipsets. The
natural consequent of that is that you can build a high-end gig switch
or router around a chipset supporting 10Gb/s ports or feeds and speeds
your cogs are naturally going to be rather similar to the 10Gb/s offering.




signature.asc
Description: OpenPGP digital signature

Re: Barefoot "Tofino": 6.4 Tbps whitebox switch silicon?

[Peter Phaal]

On Thu, Jun 16, 2016 at 1:19 AM, Saku Ytti  wrote:
> On 16 June 2016 at 06:21, Eric Kuhnke  wrote:
>> Based on their investors, could have interesting results for much lower
>> cost 100GbE whitebox switches.
>
> Why lower cost? The BOM isn't the expensive part, the code is the
> expensive part. Only way I see this happening, is if we get open
> source routing suite for the box, i.e. 0 cost software.

It shouldn't be long before we see open source routing suites (Bird,
Quagga, GoBGP, etc) running on Linux (ONL, OS10, OpenSwitch). There is
a P4 program that implements the Switch Abstraction Interface (SAI),
which provides a common interface, device independent, interface to
merchant silicon.

http://p4.org/p4/an-open-source-p4-switch-with-sai-support/

A quick way to do interesting things with the programmable data plane
is to use P4 to augment the basic switching / routing behavior
provided by SAI: moving resources from layer 2 tables to layer 3
tables, adding telemetry, adding additional control capabilities,
etc.:

http://blog.sflow.com/2016/06/programmable-hardware-barefoot-networks.html

Re: RPKI implementation

[Randy Bush]

> That is also configurable.
>>> When a cache loses connectivity, the entries from that cache
>>> are purged after a time interval. Default is 60 seconds
>> why not the poll interval for that cache server?

i am aware of that.  my point was that cache purge default might better
be set to cache refresh interval than 60 secs.

randy

1GE L3 aggregation

[Saku Ytti]

Hey,

I've been bit poking around trying to find reasonable option for 1GE
L3 full BGP table aggregator. It seems vendors are mostly pushing
Satellite/Fusion for this application.

I don't really like the added complexity and tight coupling
Satellite/Fusion forces me. I'd prefer standards based routing
redundancy to reduce impact of defects.

ASR9001 and MX104 are not an options, due to control-plane scale. New
boxes in vendor pipeline are completely ignoring 1GE.

I've casually talked with other people, and it seems I'm not really
alone here. My dream box would be 96xSFP + 2xQSFP28, with pretty much
full edge features (BGP, LDP, ISIS, +1M FIB, +5M RIB, per-interface
VLANs, ipfix or sflow, at least per-port QoS with shaper, martini
pseudowires).

With tinfoil hat tightly fit on my head, I wonder why vendors are
ignoring 1GE? Are business cases entirely driven now by Amazon,
Google, Facebook and the likes? Are SP volumes so insignificant in
comparison it does not make sense to produce boxes for them?
Heck even 10GE is starting to become problematic, if your application
is anything else than DC, because you can't choose arbitrary optics.

-- 
  ++ytti

Re: NANOG67 - Tipping point of community and sponsor bashing?

[Tom Hill]

On 16/06/16 15:40, Dave Temkin wrote:
> Nothing in my presentation said "Netflix seeks to get better port fees".
> You'll find that I, not once, in my deck or oral presentation, mentioned
> Netflix. I spoke at length with LINX after the presentation and pointed out
> that I seek to help the entire market, not just my employer, better
> understand how IXPs price their services, what things are negotiable, and
> what things need to change. Call it thinly-veiled, but I didn't even use my
> employer slide master - this was geared as a community discussion.

I wasn't sure if you were talking on behalf of Netflix either. Mainly
because the first thing you said at the Nanog presentation was to
correct everyone on your title at Netflix. ;)

Rather than alluding to it, letting everyone come to their own
conclusion, you'd have been better off just saying it outright.

Definitely do not be surprised when anyone's confused as to this fact,
however.

-- 
Tom

Re: NANOG67 - Tipping point of community and sponsor bashing?

[Niels Bakker]

* zby...@dialtelecom.cz (Zbyněk Pospíchal) [Thu 16 Jun 2016, 14:23 CEST]:

Dne 15.06.16 v 20:10 Mikael Abrahamsson napsal(a):

Well, the customers also wanted more functions and features. They 
wanted sFLOW statistics to show traffic, customer portals, better 
SLAs, distributed IXes, remote peering, more hand-holding when 
connecting etc.


Are you sure they still want them if they have to pay for these 
features separately?


Currently, such luxury functions are increasing costs also for 
networks who don't need/want it.


sFlow statistics isn't a luxury function.  Neither is remote peering.  
The others Mikael mentioned are debatable at worst but you'd be 
telling the membership what they really want rather than listening 
to them saying what they want.


This thread is full of people who have never run large L2 networks 
stating their opinions on running large L2 networks, and they 
invariably underestimate their complexity and the logistics required.



-- Niels.

Re: NANOG67 - Tipping point of community and sponsor bashing?

[Niels Bakker]

This thread is full of people who have never run large L2 networks
stating their opinions on running large L2 networks, and they
invariably underestimate their complexity and the logistics required.


* ra...@psg.com (Randy Bush) [Thu 16 Jun 2016, 17:56 CEST]:

maybe the complexity and the logistics required are WHY they don't build
large L2 networks.  SMITH: Doctor, it hurts when I do this. DALE: Don't
do that.


Wait.  I thought vijay "your solution doesn't scale" gill was a hero
of the NANOG community, but now you're telling me that actually
scaling up is a sin?



sFlow statistics isn't a luxury function.  Neither is remote peering.

by 'remote peering' do you mean an exchange essentially selling transit?


I mean the common practice of connecting via a L2 pseudowire with 
a provider that has an arrangement with the IXP to do so, rather than 
putting a router in a datacenter the IXP is in and then connecting to 
that router  with possibly the same provider to the rest of your network.
Mikael Abrahamsson probably meant the same thing when he used the term 
two mails upthread.



-- Niels.

Re: NANOG67 - Tipping point of community and sponsor bashing?

[Randy Bush]

> This thread is full of people who have never run large L2 networks
> stating their opinions on running large L2 networks, and they
> invariably underestimate their complexity and the logistics required.

maybe the complexity and the logistics required are WHY they don't build
large L2 networks.  SMITH: Doctor, it hurts when I do this. DALE: Don't
do that.

> sFlow statistics isn't a luxury function.  Neither is remote peering.

by 'remote peering' do you mean an exchange essentially selling transit?

randy

Re: NANOG67 - Tipping point of community and sponsor bashing?

[Will Hargrave]

On 15 Jun 2016, at 19:23, Sander Steffann wrote:


So here we are now... Where do we want to go?
I think IXPs have indeed become too much like ISPs, providing more 
services but also increasing complexity and cost. I prefer simple, 
scalable and cheap solutions!
I want to go to an IXP being a nice simple ethernet switch. Add some 
nice graphs and a route server, and we're done. Redundancy is a 
separate switch :)


(I spoke on this topic in the session - I regret insufficiently 
coherently, but I’ll try again)



Most of the major IXs in the European market operate in multiple 
datacentres. Why? Because it decreases the monopoly conferred upon one 
particular datacentre in a market which becomes the ‘go to’ 
location.


Dan Golding disagreed with me but I can certainly speak for LONAP where 
I feel our mission of “promoting efficient interconnection in the 
UK” is hugely enhanced by our ability to provide services in any of 
our current seven datacentres, across four different operators. London 
would not be the great city of interconnection it is without the east 
London cluster of DCs from different operators.


We have had a fair few single site IXs in London - e.g. the now defunct 
RBEIX, Sovex, Meriex. I don’t think it is a viable model for an IXP in 
a well-developed market.



Then there is another concern. What’s the plan for SIX if the Westin 
Building colo is sold to someone less benevolent and co-operative? I am 
really pleased their current arrangement seems to work well for SIX, its 
members and datacentre partners. I think our own members would be less 
comfortable with that level of risk.



Will

Call for Presentations - DNS-OARC Fall Workshop, Dallas, Oct. 2016

[Wessels, Duane]

[with apologies to those who see this on multiple lists]

Call for Presentations

As announced at the close of NANOG67, the DNS-OARC 25th Workshop will take
place in Dallas, Texas during October 15th and 16th 2016, the Saturday and
Sunday before NANOG68.  To attract the best DNS minds, DNS-OARC is requesting
proposals for presentations, with a preference for Resolver's Operations
experiences, including running DNSSEC validating resolvers.

This workshop intends to build from previous strong DNS-OARC workshops,
where both operational content and research are welcome. All DNS-related
subjects are accepted. If you are an OARC member, and have a sensitive
topic you would like to present for members-only, we will accommodate
those talks too. A timeslot will be available for lightning talks (5-10
minutes) on Sunday October 16th, for which submissions will be accepted
during October 15th on a first-come first-served basis.

Workshop Milestones
15th June 2016, Call for Presentations posted and open for submissions
14th August 2016, Deadline for submission
1st September 2016, Final Program published
10th October 2016, Final deadline for slideset submission

Details for presentation submission will be published here:

   https://indico.dns-oarc.net/event/25/call-for-abstracts/

The workshop presentations will be organized by common themes, depending
on the topics and the timing of each presentation. There are 30-minute
and 15-minute slots, let us know your preference in your submission. To
ensure the quality of the workshop, submissions should include slides.
Draft slides are acceptable on submission.

You can contact the Programme Committee:

   https://www.dns-oarc.net/oarc/programme

via  if you have questions or concerns.

Sebastian Castro, for the OARC Programme Committee

OARC is also seeking sponsorship for this workshop, please contact
 if your organization is interested in becoming a
sponsor.

(Please note that OARC is run on a non-profit basis, and is not in a
position to reimburse expenses or time for speakers at its meetings.)

Re: NANOG67 - Tipping point of community and sponsor bashing?

[Adam Rothschild]

I think a fresh conversation is needed around what makes up a
"minimally viable" feature set for an IXP:

The days of an IXP "needing" to engineer and support a multi-tenant
sFlow portal, because the only other option is shelling out the big
bucks for Arbor, have long passed -- overlooking the plethora of open
sourced tools, folk like Kentik have broken into that market with
rationally priced commercial alternatives.  Likewise, one might argue
that offering layer-2 and layer-3 (!) VPNs is at best non-essential,
and a distraction that fuels purchasing very expensive hardware, and
at worse competitive with customers.

On the other hand, building out a metro topology to cover all relevant
carrier hotels, with reasonable path diversity, is absolutely table
stakes.  And outreach is a great function, *when* it nets unique new
participants.  To cite a recent example, the various R networks and
smaller broadband and mobile providers showing up here in the US, due
to excellent efforts by the NYIIX and DE-CIX teams.

At the end of the day, IXP peering must be significantly cheaper than
transit alternatives, many of which are priced based on utilization
(as opposed to port capacity).  We can dance around this point all we
want, but absent a change in trajectory, I worry some IXPs will
ultimately price themselves out of the market, and all the gold-plated
features in the world won't satiate those making purchasing decisions.

$0.02,
-a

On Thu, Jun 16, 2016 at 11:17 AM, Niels Bakker 

Re: NANOG67 - Tipping point of community and sponsor bashing?

[Zbyněk Pospíchal]

Dne 16.06.16 v 17:17 Niels Bakker napsal(a):
> * zby...@dialtelecom.cz (Zbyněk Pospíchal) [Thu 16 Jun 2016, 14:23 CEST]:

>> Are you sure they still want them if they have to pay for these
>> features separately?
>>
>> Currently, such luxury functions are increasing costs also for
>> networks who don't need/want it.
> 
> sFlow statistics isn't a luxury function. 

Anything more than plain L2 in an IXP is a kind of luxury. An IXP member
with it's own flow collection (or at least mac accounting) can feel they
don't need sFlow statistics in an exchange. It's also proven it's
possible to run an IXP, including a big one, without sFlow stats.

We can say the same about route servers, SLA, customer portals etc. (ok,
remote peering is a different case).

If IXP members think they have to pay such functionality in their port
fees, ok, it's their own decision, but member's opinion "we don't need
it and we don't want to pay for it" is rational and plausible.

Best Regards,
Zbynek

Re: NANOG67 - Tipping point of community and sponsor bashing?

[Mike Hammett]

I think that's a very limited mindset. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 



Midwest Internet Exchange 
http://www.midwest-ix.com 


- Original Message -

From: "Zbyněk Pospíchal"  
To: nanog@nanog.org 
Sent: Thursday, June 16, 2016 1:19:22 PM 
Subject: Re: NANOG67 - Tipping point of community and sponsor bashing? 

Dne 16.06.16 v 17:17 Niels Bakker napsal(a): 
> * zby...@dialtelecom.cz (Zbyněk Pospíchal) [Thu 16 Jun 2016, 14:23 CEST]: 

>> Are you sure they still want them if they have to pay for these 
>> features separately? 
>> 
>> Currently, such luxury functions are increasing costs also for 
>> networks who don't need/want it. 
> 
> sFlow statistics isn't a luxury function. 

Anything more than plain L2 in an IXP is a kind of luxury. An IXP member 
with it's own flow collection (or at least mac accounting) can feel they 
don't need sFlow statistics in an exchange. It's also proven it's 
possible to run an IXP, including a big one, without sFlow stats. 

We can say the same about route servers, SLA, customer portals etc. (ok, 
remote peering is a different case). 

If IXP members think they have to pay such functionality in their port 
fees, ok, it's their own decision, but member's opinion "we don't need 
it and we don't want to pay for it" is rational and plausible. 

Best Regards, 
Zbynek 

IXP economics Was: Re: NANOG67 - Tipping point of community and sponsor bashing?

[Martin Hannigan]

Well. Its complicated. I think this is far more political than about COGS.
But hey. Why not?

I agree with Dave. Shocking. I know. At least the context. He's right.
Thanks for reminding us. We know these things. We'll have to see how IXP
communities react now. Perhaps espresso service will be defined as good
outreach? If it is, thats certainly up to them.

Btw, have you thanked your local US branded euro IXP? They created market
pressures that saved many of us a Brinks truck full of cash by increasing
competitiveness. A lot of us owe them thanks and support for doing
good.  If you do, grab Job, John, Henk, Pauline, Arnold or Andreas and give
them a big COGS crushing hug. Go easy on Pauline, less crush please.

Best,

Marty

On Thursday, June 16, 2016, Zbyněk Pospíchal  wrote:

> Dne 16.06.16 v 17:17 Niels Bakker napsal(a):
> > * zby...@dialtelecom.cz  (Zbyněk Pospíchal) [Thu 16 Jun
> 2016, 14:23 CEST]:
>
> >> Are you sure they still want them if they have to pay for these
> >> features separately?
> >>
> >> Currently, such luxury functions are increasing costs also for
> >> networks who don't need/want it.
> >
> > sFlow statistics isn't a luxury function.
>
> Anything more than plain L2 in an IXP is a kind of luxury. An IXP member
> with it's own flow collection (or at least mac accounting) can feel they
> don't need sFlow statistics in an exchange. It's also proven it's
> possible to run an IXP, including a big one, without sFlow stats.
>
> We can say the same about route servers, SLA, customer portals etc. (ok,
> remote peering is a different case).
>
> If IXP members think they have to pay such functionality in their port
> fees, ok, it's their own decision, but member's opinion "we don't need
> it and we don't want to pay for it" is rational and plausible.
>
> Best Regards,
> Zbynek
>

Re: 1GE L3 aggregation

[Baldur Norddahl]

Hi

If I need to speak BGP with a customer that only has 1G I will simply make
a MPLS L2VPN to one of my edge routers. We use the ZTE 5952E switch with
48x 1G plus 4x 10G for the L2VPN end point. If that is not enough the ZTE
8900 platform will provide a ton of ports that can do MPLS.

The tunnel is automatically redundant and will promote link down events, so
there is not really any downside to doing it this way on low bandwidth
peers.

Regards

Baldur
Den 16. jun. 2016 09.52 skrev "Saku Ytti" :

Hey,

I've been bit poking around trying to find reasonable option for 1GE
L3 full BGP table aggregator. It seems vendors are mostly pushing
Satellite/Fusion for this application.

I don't really like the added complexity and tight coupling
Satellite/Fusion forces me. I'd prefer standards based routing
redundancy to reduce impact of defects.

ASR9001 and MX104 are not an options, due to control-plane scale. New
boxes in vendor pipeline are completely ignoring 1GE.

I've casually talked with other people, and it seems I'm not really
alone here. My dream box would be 96xSFP + 2xQSFP28, with pretty much
full edge features (BGP, LDP, ISIS, +1M FIB, +5M RIB, per-interface
VLANs, ipfix or sflow, at least per-port QoS with shaper, martini
pseudowires).

With tinfoil hat tightly fit on my head, I wonder why vendors are
ignoring 1GE? Are business cases entirely driven now by Amazon,
Google, Facebook and the likes? Are SP volumes so insignificant in
comparison it does not make sense to produce boxes for them?
Heck even 10GE is starting to become problematic, if your application
is anything else than DC, because you can't choose arbitrary optics.

--
  ++ytti

Re: NANOG67 - Tipping point of community and sponsor bashing?

[Leslie]

On Wed, Jun 15, 2016 at 8:41 PM, Martin  Hannigan  wrote:
>

> SFMIX is great. But poorly distributed. We should support their efforts, but 
> how many IXPs do we need in the Bay area? AMS-IX Bay Area is creating a 
> market along with SFMIX.
>

SFMIX is in 5 physical locations(
https://www.sfmix.org/connect/locations ) and is always open to
talking to other providers about extending into their datacenter. So
I'd say we're in a variety of locations! We've also just celebrated
our 10 year anniversary :)

Leslie

Re: NANOG67 - Tipping point of community and sponsor bashing?

[Martin Hannigan]

> On Jun 16, 2016, at 01:12, Leslie  wrote:
> 
>> On Wed, Jun 15, 2016 at 8:41 PM, Martin  Hannigan  wrote:
> 
>> SFMIX is great. But poorly distributed. We should support their efforts, but 
>> how many IXPs do we need in the Bay area? AMS-IX Bay Area is creating a 
>> market along with SFMIX.
> 
> SFMIX is in 5 physical locations(
> https://www.sfmix.org/connect/locations ) and is always open to
> talking to other providers about extending into their datacenter. So
> I'd say we're in a variety of locations! We've also just celebrated
> our 10 year anniversary :)
> 
> Leslie

Agreed. Extended XC?

Best,

-M<

Re: Barefoot "Tofino": 6.4 Tbps whitebox switch silicon?

[Pavel Odintsov]

Looks very promising!

On Thu, Jun 16, 2016 at 6:21 AM, Eric Kuhnke  wrote:
> a lot of PR fluff, but this may be of interest:
>
> http://www.wired.com/2016/06/barefoot-networks-new-chips-will-transform-tech-industry/
>
> https://barefootnetworks.com/media/white_papers/Barefoot-Worlds-Fastest-Most-Programmable-Networks.pdf
>
>
> Based on their investors, could have interesting results for much lower
> cost 100GbE whitebox switches.



-- 
Sincerely yours, Pavel Odintsov

RPKI implementation

[Jakob Heitz (jheitz)]

During the RPKI presentation there was a question about
resilience of the router if the RPKI cache loses connectivity.
The IOS-XR implementation allows multiple caches to be configured.
When a cache loses connectivity, the entries from that cache
are purged after a time interval. Default is 60 seconds and it is configurable.
A lookup of a prefix that is not loaded will return not-found.
5 seconds after the latest RPKI database update,
a refresh request is sent to each neighbor, provided that the neighbor either:
- dropped any received route due to a policy that contains validation-state, or
- received a route, the validation state of which changed.
If soft reconfiguration inbound is configured, then the refresh is avoided,
because the received paths are stored.

Thanks,
Jakob.