Re: 400G forwarding - how does it work?

[James Bensley]

On Tue, 26 Jul 2022 at 21:39, Lawrence Wobker  wrote:
> So if this pipeline can do 1.25 billion PPS and I want to be able to forward 
> 10BPPS, I can build a chip that has 8 of these pipelines and get my 
> performance target that way.  I could also build a "pipeline" that processes 
> multiple packets per clock, if I have one that does 2 packets/clock then I 
> only need 4 of said pipelines... and so on and so forth.

Thanks for the response Lawrence.

The Broadcom BCM16K KBP has a clock speed of 1.2Ghz, so I expect the
J2 to have something similar (as someone already mentioned, most chips
I've seen are in the 1-1.5Ghz range), so in this case "only" 2
pipelines would be needed to maintain the headline 2Bpps rate of the
J2, or even just 1 if they have managed to squeeze out two packets per
cycle through parallelisation within the pipeline.

Cheers,
James.

Re: 400G forwarding - how does it work?

[James Bensley]

On Wed, 27 Jul 2022 at 15:11, Masataka Ohta
 wrote:
>
> James Bensley wrote:
>
> > The BCM16K documentation suggests that it uses TCAM for exact
> > matching (e.g.,for ACLs) in something called the "Database Array"
> > (with 2M 40b entries?), and SRAM for LPM (e.g., IP lookups) in
> > something called the "User Data Array" (with 16M 32b entries?).
>
> Which documentation?
>
> According to:
>
> https://docs.broadcom.com/docs/16000-DS1-PUB
>
> figure 1 and related explanations:
>
> Database records 40b: 2048k/1024k.
> Table width configurable as 80/160/320/480/640 bits.
> User Data Array for associated data, width configurable as
> 32/64/128/256 bits.
>
> means that header extracted by 88690 is analyzed by 16K finally
> resulting in 40b (a lot shorter than IPv6 addresses, still may be
> enough for IPv6 backbone to identify sites) information by "database"
> lookup, which is, obviously by CAM because 40b is painful for
> SRAM, converted to "32/64/128/256 bits data".

Hi Masataka,

Yes I had read that data sheet. If you have 2M 40b entries in CAM, you
could also have 1M 80 entries (or a mixture); the 40b CAM blocks can
be chained together to store IPv4/IPv6/MPLS/whatever entries.

Cheers,
James.

Re: 400G forwarding - how does it work?

[James Bensley]

Hi Lawrence, thanks for your response.

On Mon, 25 Jul 2022 at 15:34, Lawrence Wobker  wrote:
> This is the parallelism part.  I can take multiple instances of these 
> memory/logic pipelines, and run them in parallel to increase the throughput.
...
> I work on/with a chip that can forwarding about 10B packets per second… so if 
> we go back to the order-of-magnitude number that I’m doing about “tens” of 
> memory lookups for every one of those packets, we’re talking about something 
> like a hundred BILLION total memory lookups… and since memory does NOT give 
> me answers in 1 picoseconds… we get back to pipelining and parallelism.

What level of parallelism is required to forward 10Bpps? Or 2Bpps like
my J2 example :)

Cheers,
James.

Re: 400G forwarding - how does it work?

[James Bensley]

Thanks for the responses Chris, Saku…

On Mon, 25 Jul 2022 at 15:17, Chris Adams  wrote:
>
> Once upon a time, James Bensley  said:
> > The obvious answer is that it's not magic and my understanding is
> > fundamentally flawed, so please enlighten me.
>
> So I can't answer to your specific question, but I just wanted to say
> that your CPU analysis is simplistic and doesn't really match how CPUs
> work now.

It wasn't a CPU analysis because switching ASICs != CPUs.

I am aware of the x86 architecture, but know little of network ASICs,
so I was deliberately trying to not apply my x86 knowledge here, in
case it sent me down the wrong path. You made references towards
typical CPU features;

> For example, it might take 4 times as long to process the first packet,
> but as long as the hardware can handle 4 packets in a queue, you'll get
> a packet result every cycle after that, without dropping anything.  So
> maybe the first result takes 12 cycles, but then you can keep getting a
> result every 3 cycles as long as the pipeline is kept full.

Yes, in the x86/x64 CPU world keeping the instruction cache and data
cache hot indeed results in optimal performance, and as you say modern
CPUs use parallel pipelines amongst other techniques like branch
prediction, SIMD, (N)UMA, and so on, but I would assume (because I
don’t know) that not all of the x86 feature set map nicely to packet
processing in ASICs (VPP uses these techniques on COTS CPUs, to
emulate a fixed pipeline, rather than run to completion model).

You and Saku both suggest that heavy parallelism is the magic source;

> Something can be "line rate" but not push the first packet
> through in the shortest time.

On Mon, 25 Jul 2022 at 15:16, Saku Ytti  wrote:
> I.e. say JNPR Trio PPE has many threads, and only one thread is
> running, rest of the threads are waiting for answers from memory. That
> is, once we start pushing packets through the device, it takes a long
> ass time (like single digit microseconds) before we see any packets
> out. 1000x longer than your calculated single digit nanoseconds.

In principal I accept this idea. But lets try and do the maths, I'd
like to properly understand;

The non-drop rate of the J2 is 2Bpps @ 284 bytes == 4.8Tbps, my
example scenario was a single J2 chip in a 12x400G device. If each
port is receiving 400G @ 284 bytes (164,473,684 pps), that’s one every
6.08 nanoseconds coming in. What kind of parallelism is required to
stop from ingress dropping?

It takes say 5 microseconds to process and forward a packet (seems
reasonable looking at some Arista data sheets which use J2 variants),
which means we need to be operating on 5,000ns / 6.08ns == 822 packets
per port simultaneously, so 9868 packets are being processed across
all 12 ports simultaneously, to stop ingress dropping on all
interfaces.

I think the latest generation Trio has 160 PPEs per PFE, but I’m not
sure how many threads per PPE. Older generations had 20
threads/contexts per PPE, so if it hasn’t increased that would make
for 3200 threads in total. That is a 1.6Tbps FD chip, although not
apples to apples of course, Trio is run to completion too.

The Nokia FP5 has 1,200 cores (I have no idea how many threads per
core) and is rated for 4.8Tbps FD. Again doing something quite
different to a J2 chip, again its RTC.

J2 is a partially-fixed pipeline but slightly programmable if I have
understood correctly, but definitely at the other end of the spectrum
compared to RTC. So are we to surmise that a J2 chip has circa 10k
parallel pipelines, in order to process 9868 packets in parallel?

I have no frame of reference here, but in comparison to Gen 6 Trio of
NP5, that seems very high to me (to the point where I assume I am
wrong).

Cheers,
James.

400G forwarding - how does it work?

[James Bensley]

Hi All,

I've been trying to understand how forwarding at 400G is possible,
specifically in this example, in relation to the Broadcom J2 chips,
but I don't the mystery is anything specific to them...

According to the Broadcom Jericho2 BCM88690 data sheet it provides
4.8Tbps of traffic processing and supports packet forwarding at 2Bpps.
According to my maths that means it requires packet sizes of 300Bs to
reach line rate across all ports. The data sheet says packet sizes
above 284B, so I guess this is excluding some headers like the
inter-frame gap and CRC (nothing after the PHY/MAC needs to know about
them if the CRC is valid)? As I interpret the data sheet, J2 should
supports chassis with 12x 400Gbps ports at line rate with 284B packets
then.

Jericho2 can be linked to a BCM16K for expanded packet forwarding
tables and lookup processing (i.e. to hold the full global routing
table, in such a case, forwarding lookups are offloaded to the
BCM16K). The BCM16K documentation suggests that it uses TCAM for exact
matching (e.g.,for ACLs) in something called the "Database Array"
(with 2M 40b entries?), and SRAM for LPM (e.g., IP lookups) in
something called the "User Data Array" (with 16M 32b entries?).

A BCM16K supports 16 parallel searches, which means that each of the
12x 400G ports on a Jericho2 could perform an forwarding lookup at
same time. This means that the BCM16K "only" needs to perform
forwarding look-ups at a linear rate of 1x 400Gbps, not 4.8Tbps, and
"only" for packets larger than 284 bytes, because that is the Jericho2
line-rate Pps rate. This means that each of the 16 parallel searches
in the BCM16K, they need to support a rate of 164Mpps (164,473,684) to
reach 400Gbps. This is much more in the realm of feasible, but still
pretty extreme...

1 second / 164473684 packets = 1 packet every 6.08 nanoseconds, which
is within the access time of TCAM and SRAM but this needs to include
some computing time too e.g. generating a key for a lookup and passing
the results along the pipeline etc. The BCM16K has a clock speed of
1Ghz (1,000,000,000, cycles per second, or cycle every 1 nano second)
and supports an SRAM memory access in a single clock cycle (according
to the data sheet). If one cycle is required for an SRAM lookup, the
BCM16K only has 5 cycles to perform other computation tasks, and the
J2 chip needs to do the various header re-writes and various counter
updates etc., so how is magic this happening?!?

The obvious answer is that it's not magic and my understanding is
fundamentally flawed, so please enlighten me.

Cheers,
James.

Re: Validating multi-path in production?

[James Bensley]

On Fri, 12 Nov 2021 at 16:54, Adam Thompson  wrote:

> The best I've come up with so far is to have two test systems (typically
> VMs) that use adjacent IP addresses and adjacent MAC addresses, and test
> both inbound and outbound to/from those, blindly trusting/hoping that
> hashing algorithms will *probably* exercise both paths.
>

If the goal is to test that traffic *is* being distributed across multiple
links based on traffic headers, then you can definable roll your own. I
think the problem is orchestrating it (feeding your topology data into the
tool, running the tool, getting the results out, and interpreting the
results etc).

A coupe of public examples:
https://github.com/facebookarchive/UdpPinger
https://www.youtube.com/watch?v=PN-4JKjCAT0

If you do roll your own, you need to taylor the tests to your topology and
your equipment. For example, you can have two VMs as you mentioned, each at
opposite ends of the network. Then, if your network uses a 5-tuple for ECMP
inside the core for example, you could send many flows between the two VMs,
rotating the sauce port for example, to ensure all links in a LAG or all
ECMP paths are used.

It's tricky to know the hashing algo for every type of device you have in
your network, and for each traffic type for each device type, if you have a
multi vendor network. Also, if your network carries a mix of IPv4, IPv6,
PPP, MPLS L3 VPNs, MPLS L2 VPNs, GRE, GTP, IPSEC, etc. The number of
permutations of tests you need to run and the result sets you need to
parse, grows very rapidly.

Cheers,
James.

Announcing UKNOF46

[James Bensley]

Dear Colleagues,

We are pleased to announce UKNOF46, taking place online, on 19th
January followed by an online social event.

The meeting website is live and both the Call for Presentations and
Registrations are now open.


[ REGISTRATION ]

There are a limited number of Free Registration spaces - 50 in total.
Paid tickets are £50 each including VAT and the Eventbrite fee. You
may also pay extra as an add-on to your registration.

Registration is required to participate in the meeting. A recording of
the proceedings will be made available on the UKNOF YouTube channel
for viewing at a later time.

Please register via:

 


[ CALL FOR PRESENTATIONS ]

The Programme Committee are seeking content for UKNOF46, which takes
place online, from the community for this meeting.

UKNOF's remit is "distribution of clue", and for our first meeting of
2021, we are looking for content which is both operationally timely
and our 'normal' operational remit.

The whole session is planned for 3 - 5 hours. We are looking for a mix
of formats

  * 15 or 25 minute presentations (with 5 minutes for Q)
  * 5 minute lightning talks (no Q)
  * 45 minute panel sessions

We are keen for talks and panels in these subject areas:

  * 2020 retrospective and forward looking to 2021
  * Confessions & War Stories / Lessons learnt from the field
  * Data centre design and operations
  * DNS infrastructure
  * Impact of public policy of network operations
  * IPv6 deployment
  * Network Automation and Tooling including SDN, Open Source initiatives
  * Network hardware/software/configuration architecture, design and operations
  * Open protocol standards and best practise initiatives
  * Peering and interconnection / routing.
  * Personal/Network security, inclusiveness and abuse prevention


Please submit your proposals at

 

**Submissions** are welcome at any time, but for UKNOF46 we would like
to have them **no later than 7 January 2021** (We have published an
initial set of accepted talks in the "Confirmed Speakers" section on
the meeting website .


Our expected timeline is:

7 December 2020 - Submissions open via Indico
18 December 2020 - Initial Submission Deadline
22 December 2020 - Initial Contribution list published
7 January 2021 - Final Deadline for submission (23:59 GMT)
12 January 2021 - Full agenda published
13 January 2021- Deadline for slides submission and planned rehearsal date
19 January 2021 - Meeting

Information for speakers about presenting and what to expect at UKNOF
is available at 

Additional information for speakers of UKNOF46:

  * your talk will be live broadcast and recorded for future reference
  * your presentation slides will be available for delegates and
others to download and refer to, before, during and after the meeting
  * you will be expected to attend a rehearsal in the week running up
to the workshop.  This has been tentatively scheduled for Wednesday 13
January. It would be very useful to have your slides (even if draft)
ready for this.

To preserve editorial neutrality, the UKNOF PC does not accept
abstract submissions for presentations by commercial organisations
that are also sponsors of the same UKNOF event. If you would like one
of our sponsored presentation slots, please contact
[spon...@uknof.org.uk]

UKNOF is run on a non-profit basis and is not in a position to
reimburse expenses or time for speakers at its meetings.

Regards
James
UKNOF Programme Committee Member

Re: SRv6

[James Bensley]

On 19 September 2020 03:23:15 BST, Randy Bush  wrote:
>> Information can be in plaintext and private
>
>Three can keep a secret, if two of them are dead.  -- franklin
>
>i know you truely believe the tunnel k00laid.  the security
>community does not.

Hi Randy,

I'm not sure what you're saying here, I never said MPLS VPNs are secure, only 
private. I hope others recognise that they are different concepts.

Cheers,
James.

Re: SRv6

[James Bensley]

On 16 September 2020 22:38:38 CEST, Randy Bush  wrote:
>> Privacy != encryption.
>
>cleartext == privacy * 0
>
>cleartext * complexity == privacy * 0

False. Cleartext and privacy are two different things which are not mutually 
exclusive. Information can be in plaintext and private, it can also be 
encrypted and not private.

Consider multiple devices connected to a single customer instance (A) on an 
MPLS L2 VPN provider's network, consisting of a single VLAN/broadcast domain, 
all the connected devices are able to send information to each other, and they 
can receive the information sent to other devices not intended for itself. Any 
device, for example, can send a gratuitous ARP, update the control plane of the 
switch and pull traffic towards itself and have visibility of all the 
conversation on the VLAN/broadcast domain. Even if the conversations are 
encrypted, meaning no plaintext, which you seem to suggest means privacy, this 
receiving device sees all the conversations which take place, when they are 
taking place, between whom, for how long, how often, and so on. Encryption 
hasn't provided privacy if someone can see all that information.

Now consider a second customer (B) connected to a separate customer instance on 
the same L2 VPN provider network. Customer A can send any traffic they like and 
they can listen all day until the cows come home; they will never be able to 
send traffic to a customer B device in a separate L2 VPN instance, and they 
will never receive any traffic from a customer B device, they can't even see 
that customer B exists, if they are having any conversations, when, for how 
long etc, nothing.

That is privacy, which is completely different to plaintext and ciphertext.

Cheers,
James

Re: SRm6 (was:SRv6)

[James Bensley]

On 17 September 2020 11:05:24 CEST, Saku Ytti  wrote:
>On Thu, 17 Sep 2020 at 11:03, James Bensley 
>wrote:
>
>> MPLSoUDP lacks transport engineering features  like explicit paths,
>FRR LFA and FRR rLFA, assuming only a single IP header is used for the
>transport abstraction [1]. If you want stuff like TI-LFA (I assume this
>is supported in SRm6 and SRv6, but I'm not familiar with these, sorry
>if that is a false assumption) you need additional transport headers or
>a stack of MPLS labels encapped in the UDP header and then you're back
>to square one.
>
>One of us has confusion about what MPLSoUDP is. I don't run it, so
>might be me.
>
>SPORT == Entropy (so non-cooperating transit can balance)
>DPORT == 6635 (NOT label)
>Payload = MPLS label(s)
>
>Whatever MPLS can do MPLSoUDP can, by definition, do. It is just
>another MPLS point-to-point adjacency after the MPLSoUDP
>abstraction/tunnel.

Nope, we have the same understanding. But the email I was responding to was 
talking about using MPLSoUDP for service label encapsulation *only*, not 
transport & services labels:


>>  If you want an IPv6 underlay for a network offering VPN services 
>
> And what's wrong again with MPLS over UDP to accomplish the very same with 
> simplicity ? 
>
> MPLS - just a demux label to a VRF/CE
> UDP with IPv6 header plain and simple 
> 
> + minor benefit: you get all of this with zero change to shipping hardware 
> and software ... Why do we need to go via decks of SRm6 slides and new wave 
> of protocols extensions ???


Cheers,
James.

Re: SRm6 (was:SRv6)

[James Bensley]

On 16 September 2020 23:51:03 CEST, Robert Raszuk  wrote:
>Hi Ron,
>
>>  If you want an IPv6 underlay for a network offering VPN services
>
>And what's wrong again with MPLS over UDP to accomplish the very same
>with
>simplicity ?
>
>MPLS - just a demux label to a VRF/CE
>UDP with IPv6 header plain and simple
>
>+ minor benefit: you get all of this with zero change to shipping
>hardware
>and software ... Why do we need to go via decks of SRm6 slides and new
>wave
>of protocols extensions ???
>
>Best,
>Robert.
>
>

>>
>>
>> Please consider the TE mechanism described in
>> draft-bonica-6man-comp-rtg-hdr and the service labeling mechanism
>described
>> in draft-bonica-6man-vpn-dest-opt. These can be deployed on a mix and
>match
>> basis. For example can deploy:
>>
>>
>>
>>- Draft-bonica-6man-vpn-dest-opt only, allowing traffic to follow
>the
>>least-cost path from PE to PE.
>>- Deploy draft-bonica-6man-vpn-dest-opt only, using a legacy
>method
>>(VXLAN, RFC 4797) to label services.
>>
>>
>>
>> In all cases, the semantic of the IPv6 address is unchanged. There is
>no
>> need to encode anything new in the IPv6 address.

MPLSoUDP lacks transport engineering features  like explicit paths, FRR LFA and 
FRR rLFA, assuming only a single IP header is used for the transport 
abstraction [1]. If you want stuff like TI-LFA (I assume this is supported in 
SRm6 and SRv6, but I'm not familiar with these, sorry if that is a false 
assumption) you need additional transport headers or a stack of MPLS labels 
encapped in the UDP header and then you're back to square one.

Cheers,
James.

[1] I'm interested to hear if anyone has done any large scale MPLSoUDP work. 
Did you hack in this functionality with static egress interface entries/static 
routes pushed from a central controller for specific IPs reserve as "path" IPs?

Re: SRv6

[James Bensley]

On Tue, 15 Sep 2020 at 19:14, Randy Bush  wrote:
>
> > I'm still learning, but, It does seem interesting that the IP layer
> > (v6) can now support vpn's without mpls.
>
> as the packet payload is nekkid cleartext, where is the P in vpn?

Define "privacy". In the kind of VPN I think you're suggesting (e.g.
an IPSEC based VPN) they implement the classic CIA acronym
(Confidentiality, Integrity and Authentication, with the "C"
essentially meaning "encrypted" in practice however, privacy requires
all three of "CIA", encryption alone isn't privacy). "VPN" is not
mutually dependent on "CIA", the two things can and do exist
separately.

WIth MPLS L3 VPNs for example, the traffic isn't encrypted, but by
creating a layer of abstraction (at the control plane, by signalling
via MP-BGP using RDs and RTs, and at the forwarding plane by using
MPLS tunneling) a customer's routing data and forwarding data is kept
private (not encrypted!) from my physical infa forwarding- and
control-planes, and from each other L3 VPN customer on my infra [1].

In fact, the point that customer (control- and forwarding-plane) data
is kept private from MY INFRA, is *the* fundamental aspect of MPLS L3
VPNs; they wouldn't scale at all without it. Privacy != encryption.

Cheers,
James.

[1] This doesn't mean there aren't security flaws in MPLS (there are,
but there are in things like IPSEC too), and "how secure" it is, is a
separate subject.

Re: Devil's Advocate - Segment Routing, Why?

[James Bensley]

On Tue, 30 Jun 2020 at 22:07, Mark Tinka  wrote:
>
>
>
> On 30/Jun/20 20:37, James Bensley wrote:
>
> > Mark, does someone have a gun to your head? Are you in trouble? Blink
> > 63 times for yes, 64 times for no ;)
>
> You're pretty late to this party, mate...

True, but what's changed in two weeks with regards to LDv6 and SR?

What was your use case / requirement for LDv6 - to remove the full
table v6 feed from your core or to remove IPv4 from your IGP or both?

Cheers,
James.

Re: Devil's Advocate - Segment Routing, Why?

[James Bensley]

On Wed, 17 Jun 2020 at 23:19, Mark Tinka  wrote:
> Yes, we all love less state, I won't argue that. But it's the same question 
> that is being asked less and less with each passing year - what scales better 
> in 2020, OSPF or IS-IS. That is becoming less relevant as control planes keep 
> getting faster and cheaper.
>
> I'm not saying that if you are dealing with 100,000 T-LDP sessions you should 
> not consider SR, but if you're not, and SR still requires a bit more 
> development (never mind deployment experience), what's wrong with having 
> LDPv6? If it makes near-as-no-difference to your control plane in 2020 or 
> 2030 as to whether your 10,000-node network is running LDP or SR, why not 
> have the choice?

I'm going to kick the nest in the other direction now :D ... There
would be no need to massively scale an IGP or worry about running
LDPv4 + LDv6 or SR MPLS if we had put more development time into MPLS
over UDP. I think it's a great technology which solves a lot of
problems and I've been itching to deploy it for ages now, but vendor
support for it is nowhere near the level of MPLS over Ethernet.

Cheers,
James.

Re: Devil's Advocate - Segment Routing, Why?

[James Bensley]

On Wed, 17 Jun 2020 at 18:08, Mark Tinka  wrote:
>
> Hi all.
>
> When the whole SR concept was being first dreamed up, I was mildly excited 
> about it. But then real life happened and global deployment (be it basic 
> SR-MPLS or SRv6) is what it is, and I became less excited. This was back in 
> 2015.
>
> All the talk about LDPv6 this and last week has had me reflecting a great 
> deal on where we are, as an industry, in why we are having to think about SR 
> and all its incarnations.
>
> So, let me be the one that stirs up the hornets' nest...
>
> Why do we really need SR? Be it SR-MPLS or SRv6 or SRv6+?

I am clearly very far behind on my emails, but of the emails I've read
so far in this thread though you have mentioned at least twice:

On Wed, 17 Jun 2020 at 18:08, Mark Tinka  wrote:
> What I am less enthused about is being forced

On Wed, 17 Jun 2020 at 23:22, Mark Tinka  wrote:
> it tastes funny when you are forced

Mark, does someone have a gun to your head? Are you in trouble? Blink
63 times for yes, 64 times for no ;)

Cheers,
James.

Re: Devil's Advocate - Segment Routing, Why?

[James Bensley]

On Wed, 17 Jun 2020 at 22:09,  wrote:
>
> > From: NANOG  On Behalf Of Mark Tinka
> > Sent: Wednesday, June 17, 2020 6:07 PM
> >
> >
> > I've heard a lot about "network programmability", e.t.c.,
> First of all the "SR = network programmability" is BS, SR = MPLS, any 
> programmability we've had for MPLS since ever works the same way for SR.

It works because SR != MPLS.

SR is a protocol which describes many aspects, such as how traffic
forwarding decisions made at the ingress node to a PSN can be
guaranteed across the PSN, even though the nodes along the PSN path
use per-hop forwarding behaviour and different nodes along the path
have made different forwarding decisions.

When using SR MPLS segment IDs are used as an index into the label
range (SRGB) and so SIDs don't correlate 1:1 to MPLS labels, equally
with SRv6 the segment IDs are encoded as IPv6 addresses and don't
correlate 1:1 to an IPv6 address. There is a venn diagram with an
overlapping section in the middle which is "generic SR" with a bunch
of core features that are supported agnostic of the encoding
mechanism.

Cheers,
James.

Re: End to End testing

[James Bensley]

On Thu, 12 Dec 2019 at 14:55, Fawcett, Nick via NANOG  wrote:
>
> Anyone have any suggestions on devices that I can put at two points in the 
> network to test packet loss, latency, jitter etc.  I was thinking of maybe 
> engineering my own using a couple of pi’s,  but the downfall is they don’t 
> have SFP ports.  I’m looking for something that’s portable and easy to 
> configure and drop in.  Thanks.
>
>
>
> ~Nick

Hi Nick,

We've had great success with Exfo testers. They can be expensive if
you're a small shop, but they really do "just work". I've used many
open source tools, none of them compare to the Exfo's we have.

Cheers,
James.

Re: FRR as Route-Reflector & Scaling stats

[James Bensley]

On Thu, 7 Nov 2019 at 14:36, Rakesh M  wrote:
>
>
> Hi Nanog,
>
>
> We want to Deploy and use FRR for Route reflection on a Dell Edge. Any one 
> has expereience with it and can give insight into number of routes and scale 
> that you used FRR to do Route Reflection

There is possibly no better place to ask than on the FRR mailing list:
f...@lists.frrouting.org

Cheers,
James.

Re: UK, NL, & Asia LTE Providers for Opengear Console Servers

[James Bensley]

On Thu, 1 Aug 2019 at 02:36, Ryan Gelobter  wrote:
>
> Anyone have recommendations for providers who I can use for LTE on Opengear 
> console servers in the UK, Netherlands, and Singapore? 1 provider for all 3 
> countries would be great but I'll take what I can get. Oddly when talking to 
> Opengear they don't really have any great advice. We use Verizon SIM cards in 
> the US with static IPs.
>
> Thanks,
> Ryan

Hi Ryan,

Vodafone GDSP (Global Data Service Platform) might be what you're
looking for "We offer 4G LTE roaming in over 100 countries." - I've
been looking into this recently but can't find a list of those
countries publically available. I think you'd need to approach
Vodafone directly as an interested customer/party, and I'm working via
a VAR so no direct channel.

Cheers,
James.

Re: few big monolithic PEs vs many small PEs

[James Bensley]

On 27 June 2019 16:26:03 BST, Mark Tinka  wrote:
>
>
>On 27/Jun/19 10:58, James Bensley wrote:
>
>> Hi Adam,
>>
>> Over the years I have been bitten multiple times by having fewer big
>> routers with either far too many services/customers connected to them
>> or too much traffic going through them. These days I always go for
>> more smaller/more routers than fewer/larger routers.
>>
>> One experience I have made is that when there is an outage on a large
>> PE, even when it still has spare capacity, is that the business
>impact
>> can be too much to handle (the support desk is overwhelmed, customers
>> become irate if you can't quickly tell them what all the impacted
>> services are, when service will be restored, the NMS has so many
>> alarms it’s not clear what the problem is or where it's coming from
>> etc.).
>>
>> I’ve seen networks place change freeze on devices, with the exception
>> of changes that migrate customers or services off of the PE, because
>> any outage would create too great an impact to the business, or risk
>> the customers terminating their contract. I’ve also seen changes
>> freeze be placed upon large PEs because the complexity was too great,
>> trying to work out the impact of a change on one of the original PEs
>> from when the network was first built, which is somehow linked to
>> virtually every service on the network in some obscure and
>> unforeseeable way.
>
>I would tend to agree when the edge routers are massive, e.g., boxes
>like the Cisco ASR9922 or the Juniper MX2020 are simply too large, and
>present a real risk re: that level of customer aggregation (even for
>low-revenue services such as Broadband). I don't think I'd ever justify
>buying these towers to aggregate customers, mainly due to the risk.
>
>For us, even the MX960 is too big, which is why we focus on the MX480
>(ASR9906 being the equivalent). It's a happy medium between the small
>and large end of the spectrum.
>
>And as I mentioned before, we just look at a totally different box for
>100Gbps customers.
>
>Mark.

Yeah, if you want to name specific boxes then yes I've made similar experiences 
with the same boxen. Even the MX960 is slightly too big for a PE depending on 
how you load it (port combinations).

Large boxes like the MX2020, ASR9922, NCS6K, etc. these can only reasonably be 
used as P nodes in my opinion.

Cheers,
James.

Re: few big monolithic PEs vs many small PEs

[James Bensley]

On 27 June 2019 16:31:27 BST, Mark Tinka  wrote:
>
>
>On 27/Jun/19 14:48, James Bensley wrote:
>
>> That to me is a simple scenario, and it can be mapped with a
>> dependency tree. But in my experience, and maybe it's just me, things
>> are usually a lot more complicated than this. The root cause is
>> probably a bad design introducing too much complexity, which is
>> another vote for smaller PEs from me. With more service dedicated PEs
>> one can reduce or remove the possibility of piling multiple services
>> and more complexity onto the same PE(s).
>
>Which is one of the reasons we - painfully to the bean counters -
>insist
>that routers are deployed for function.
>
>We won't run peering and transit services on the same router.
>
>We won't run SP and Enterprise on the same router as Broadband.
>
>We won't run supporting services (DNS, RADIUS, WWW, FTP, Portals, NMS,
>e.t.c.) on the same router where we terminate customers.
>
>This level of distribution, although quite costly initially, means you
>reduce the inter-dependency of services at a hardware level, and can
>safely keep things apart so that when bits fail, you aren't committing
>other services to the same fate.
>
>Mark.

Agreed. This is worked well for me over time.

It's costly in the initial capex out-lay but these boxes will have different 
upgrade/capacity increase times and price points, so over time everything 
spreads out.

Massive iron upgrades require biblical business cases and epic battles to get 
the funds approved. Periodic small to medium PE upgrades are nicer on the 
annual budget and the forecasting.

Cheers,
James.

Re: few big monolithic PEs vs many small PEs

[James Bensley]

On Thu, 27 Jun 2019 at 12:46,  wrote:
>
> > From: James Bensley 
> > Sent: Thursday, June 27, 2019 9:56 AM
> >
> > One experience I have made is that when there is an outage on a large PE,
> > even when it still has spare capacity, is that the business impact can be 
> > too
> > much to handle (the support desk is overwhelmed, customers become irate
> > if you can't quickly tell them what all the impacted services are, when 
> > service
> > will be restored, the NMS has so many alarms it’s not clear what the problem
> > is or where it's coming from etc.).
> >
> I see what you mean, my hope is to address these challenges by having a 
> "single source of truth" provisioning system that will have, among other 
> things, also HW-customer/service mapping -so Ops team will be able to say 
> that if particular LC X fails then customers/services X,Y,Z will be affected.
> But yes I agree with smaller PEs any failure fallout is minimized 
> proportionally.

Hi Adam,

My experience is that it is much more complex than that (although it
also depends on what sort of service you're offering), one can't
easily model the inter-dependency between multiple physical assets
like links, interfaces, line cards, racks, DCs etc and logical
services such as a VRFs/L3VPNs, cloud hosted proxies and the P edge.

Consider this, in my opinion, relatively simple example:
Three PEs in a triangle. Customer is dual-homed to PE1 and PE2 and
their link to PE1 is their primary/active link. Transit is dual-homed
to PE2 and PE3 and your hosted filtering service cluster is also
dual-homed to PE2 and PE3 to be near the Internet connectivity.

How will you record the inter-dependencies that an outage on PE3
impacts Customer? Because when that Customer sends traffic to PE1
(lets say all their operations are hosted in a public cloud provider),
and PE1  has learned the shortest-path to 0/0 or ::0/0 from PE2, the
Internet traffic is sent from PE1 to PE2, and from PE2 into your
filtering cluster, and when the traffic comes back into PE2 after
passing through the filters it is then sent to PE3 because the transit
provider attached to PE3 has a better route to Customer's destination
(AWS/Azure/GCP/whatever) than the one directly attached to PE2.

That to me is a simple scenario, and it can be mapped with a
dependency tree. But in my experience, and maybe it's just me, things
are usually a lot more complicated than this. The root cause is
probably a bad design introducing too much complexity, which is
another vote for smaller PEs from me. With more service dedicated PEs
one can reduce or remove the possibility of piling multiple services
and more complexity onto the same PE(s).

Most places I've seen (managed service providers) simply can't map the
complex inter-dependencies they have been physical and logical
infrastructure without having some super bespoke and also complex
asset management / CMDB / CI system.

Cheers,
James.

Re: few big monolithic PEs vs many small PEs

[James Bensley]

On Wed, 19 Jun 2019 at 21:23,  wrote:
>
> Hi folks,
>
> Recently I ran into a peculiar situation where we had to cap couple of PE
> even though merely a half of the rather big chassis was populated with
> cards, reason being that the central RE/RP was not able to cope with the
> combined number of routes/vrfs/bgp sessions/etc..
>
> So this made me think about the best strategy in building out SP-Edge
> nowadays (yes I'm aware of the centralize/decentralize pendulum swinging
> every couple of years).
> The conclusion I came to was that *currently the best approach would be to
> use several medium to small(fixed) PEs to replace a big monolithic chasses
> based system.
> So what I was thinking is,
> Yes it will cost a bit more (router is more expensive than a LC)
> Will end up with more prefixes in IGP, more BGP sessions etc.. -don't care.
> But the benefits are less eggs in one basket, simplified and hence faster
> testing in case of specialized PEs and obviously better RP CPU/MEM to port
> ratio.
> Am I missing anything please?
>
> *currently,
> Yes some old chassis systems or even multi-chassis systems used to support
> additional RPs and offloading some of the processes (e.g. BGP onto those)
> -problem is these are custom hacks and still a single OS which needs
> rebooting LC/ASICs when being upgraded -so the problem of too many eggs in
> one basket still exists (yes cisco NCS6k and recent ASR9k lightspeed LCs are
> an exception)
> And yes there is the "node-slicing" approach from Juniper where one can
> offload CP onto multiple x86 servers and assign LCs to each server (virtual
> node) - which would solve my chassis full problem -but honestly how many of
> you are running such setup? Exactly. And that's why I'd be hesitant to
> deploy this solution in production just yet. I don't know of any other
> vendor solution like this one, but who knows maybe in 5 years this is going
> to be the new standard. Anyways I need a solution/strategy for the next 3-5
> years.
>
>
> Would like to hear what are your thoughts on this conundrum.
>
> adam
>
> netconsultings.com
> ::carrier-class solutions for the telecommunications industry::

Hi Adam,

Over the years I have been bitten multiple times by having fewer big
routers with either far too many services/customers connected to them
or too much traffic going through them. These days I always go for
more smaller/more routers than fewer/larger routers.

One experience I have made is that when there is an outage on a large
PE, even when it still has spare capacity, is that the business impact
can be too much to handle (the support desk is overwhelmed, customers
become irate if you can't quickly tell them what all the impacted
services are, when service will be restored, the NMS has so many
alarms it’s not clear what the problem is or where it's coming from
etc.).

I’ve seen networks place change freeze on devices, with the exception
of changes that migrate customers or services off of the PE, because
any outage would create too great an impact to the business, or risk
the customers terminating their contract. I’ve also seen changes
freeze be placed upon large PEs because the complexity was too great,
trying to work out the impact of a change on one of the original PEs
from when the network was first built, which is somehow linked to
virtually every service on the network in some obscure and
unforeseeable way.

This doesn’t mean there isn’t a place for large routers. For example,
in a typical network, by the time we get to the P nodes layer in the
core we tend to have high levels of redundancy, i.e. any PE is
dual-homed to two or more P nodes and will have 100% redundant
capacity. Down at the access layer customers may be connected to a
single access layer device or the access layer device might have a
single backhaul link. So technically we have lots of customers,
services and traffic passing through larger P node devices, but these
devices have a low rate of changes / low touch, perform a low number
of functions, they are operationally simple, and are highly redundant.
Adversely at the service edge, which I guess is your main concern
here, I’m all about more smaller devices with single service dedicated
devices.

I’ve tried to write some of my experiences here
(https://null.53bits.co.uk/index.php?page=few-larger-routers-vs.-many-smaller-routers).
The tl;dr version though is that there’s rarely a technical
restriction to having fewer large routers and it’s an
operational/business impact problem.

I'd like to hear from anyone who has had great success with fewer larger PEs.

Cheers,
James.

Re: modeling residential subscriber bandwidth demand

[James Bensley]

On Tue, 2 Apr 2019 at 17:57, Tom Ammon  wrote:
>
> How do people model and try to project residential subscriber bandwidth 
> demands into the future? Do you base it primarily on historical data? Are 
> there more sophisticated approaches that you use to figure out how much 
> backbone bandwidth you need to build to keep your eyeballs happy?
>
> Netflow for historical data is great, but I guess what I am really asking is 
> - how do you anticipate the load that your eyeballs are going to bring to 
> your network, especially in the face of transport tweaks such as QUIC and TCP 
> BBR?
>
> Tom

Hi Tom,

Historical data is definitely the way to predict a trend, you can’t
call something a trend if it only started today IMO, something (e.g.
bandwidth profiling) needs to have been recorded for a while before
you can say that you are trying to predict the trend. Without
historical data you're just making predications without any direction,
which I don't think you want J

Assuming you have a good mixture of subs, i.e. adults, children, male,
female, different regions etc. and 100% of your subs aren't a single
demographic like university campuses for example; then I don't think
you need to worry about specifics like the adoption of QUIC or BBR.
You will never see a permeant AND massive increase in your total
aggregate network utilisation from one day to the next.

If for example, a large CDN makes a change that increases per-user
bandwidth requirements, it's unlikely they are going to deploy it
globally in one single big-bang change. This would also be just one of
your major bandwidth sources/destinations, of which you'll likely have
several big-hitters that make up the bulk of your traffic. If you have
planned well so far, and have plenty of spare capacity (as others have
mentioned, in the 50-70% range and your backhaul/peering/transit links
are of a reasonable size ratio to your subs, e.g. subs get 10-20Mbps
services and your links are 1Gbps) there should be no persisting risk
to your network capacity as long as you keep following the same
upgrade trajectory. Major social events like the Super Bowl where you
are (or here in England, sunshine) will cause exceptional traffic
increases, but only for brief periods.

You haven't mentioned exactly what you're doing for modelling capacity
demand (assuming that you wanted feedback on it)?

Assuming all the above is true for you, to give us a reasonable
foundation to build on;
In my experience the standard method is to record your ingress traffic
rate at all your PEs or P nodes, and essentially divide this by the
number of subs you have (egress is important too, it's just usually
negligible in comparison). For example, if your ASN has a total
average ingress traffic rate of 1Gbps at during peak hours and, you
have 10,000 subs, you can model on say 0.1Mbps per sub. That’s
actually a crazily low figure these days but, it’s just a fictional
example to demonstrate the calculation.

The ideal scenario is that you have this info for as long as you can.
Also, the more subs you have the better it all averages out. For
business ISPs, bringing on 1 new customer can make a major difference,
if it’s a 100Gbps end-site site and your backbone is a single 100Gbps
link you could be in trouble. For residential services, subs almost
always have slower links than your backbone/P/PE nodes.

If you have different types of subs it’s also worth breaking down the
stats by sub type. For example; we have ADSL subs and VDSL subs. We
record the egress traffic rate on the BNGs towards each type of sub
separately and then aggregate across all BNGs. For example, today peak
inbound for our ASN was X, of that X, Y went to ADSL subs and Z when
to VDSL subs. Y / $number_of_adsl_subs == peak average for an ADSL
line and, Z / $number_of_vdsl_subs == peak average for a VDSL line.

It’s good to know this difference because a sub migrating from ADSL to
VDSL is not the same as getting a new sub in terms of additional
traffic growth. We have a lot of users upgrading to VDSL which makes a
difference at scale, e.g 10K upgrades is less additional traffic than
10k new subs. Rinse and repeat for you other customer types (FTTP/H,
wireless etc.)


> On Tue, Apr 2, 2019 at 2:20 PM Josh Luthman  
> wrote:
>>
>> We have GB/mo figures for our customers for every month for the last ~10 
>> years.  Is there some simple figure you're looking for?  I can tell you off 
>> hand that I remember we had accounts doing ~15 GB/mo and now we've got 1500 
>> GB/mo at similar rates per month.
>>
>
> I'm mostly just wondering what others do for this kind of planning - trying 
> to look outside of my own experience, so I don't miss something obvious. That 
> growth in total transfer that you mention is interesting.

You need to be careful with volumetric based usage figures. As links
continuously increase in speed over the years, users can transfer the
same amount of data in less bit-time. The problem with polling at any
interval (be it 1 

Re: Free Open Source Network Operating Systems

[James Bensley]

On Sat, 9 Mar 2019 at 16:09, Colton Conor  wrote:
> Right now, the cost of the whitebox plus a paid network operating system 
> seems to equal the same cost as a discounted Juniper, Cisco, or Arista. I am 
> not seeing the savings on paper.

I'm not going to defend the prices of Cisco/Juniper/et al. as they are
often ridiculous however, there are many services you can get from
these larger vendors that you can't from the smaller ones; it's not
only some hardware, software, a bit of support and the occasional bug
fix. Big vendors like Cisco and Juniper (and others, these are just
examples) have a plethora of additional services smaller white box
vendors don't have. My point here is, you seem to be suggesting the
white box vendors offer the exact same service for less money but, in
my experience this isn't apples for apples.

> If we could just buy the whitebox hardware, and have a free operating system 
> on there, then financially whitebox switches would be half the cost of a 
> similar Cisco switch after discount.

If you think about Capex only then possibly yes it could be cheaper.

> Am I missing something?

Yes. Can you imagine what a support nightmare it would be when there
is a compatibility issue between software and hardware and those two
components are from two different vendors and you're having major
packet loss in your network?

How will you manage NMS integration when the hardware vendor
implements counters for that feature you use but the software vendor
doesn’t, or vice versa?

I’m not saying this can’t be done. Cumulus have many happy customers
using white box switches so it definitely can be done. I’m saying,
it’s not just about the Capex.

Cheers,
James.

Re: Cisco ASR's with RSP440 engines...

[James Bensley]

On Tue, 19 Feb 2019 at 15:29, Tom Hill  wrote:
>
> On 18/02/2019 21:50, John Von Essen wrote:

Hi John,

> > If anyone on here has experience with the ASR series running the
> > RSP440-SE or -TR, please contact me off-list. I'm trying to better
> > understand real world performance when it comes to handling a few full
> > BGP tables on these, it would be running as very basic edge router
> > primarily just doing BGP. I know the RSP440 is EOL, but the plan would
> > be to upgrade to RSP880 within a year.
>
> The 440 is a beast. Faster even than the 9001's RP. You'll be fine with
> a LOT of BGP edge work. :)

Tom's experiances mirror my own, the 440 is a work horse, a few BGP
feeds will be no problems.

Cheers,
James.

Re: A Zero Spam Mail System [Feedback Request]

[James Bensley]

On 18 February 2019 06:58:21 GMT, Viruthagiri Thirumavalavan  
wrote:
>Just gone through all your replies.
>
>Literally everyone attacking me here. Could you tell me why? Because I
>have
>been rude to John Levine, right? So you all think you have the right to
>give me "mob justice". But as an innocent man I have to suffer all John
>Levine attacks because he is a most valued member of NANOG. Is that
>what
>you are all saying?
>
>There is only one regret I have in this situation. I shouldn't have
>been
>rude with Töma Gavrichenkov and Suresh Ramasubramanian. That's because
>they
>didn't know what happened between John and me 6 months back. Most
>probably
>they would never have behaved with me in the way John behaved. I wasn't
>paying attention to that part. When I noticed the word "50 words", I
>thought they are mocking me too.
>
>-
>@Töma Gavrichenkov and Suresh Ramasubramanian
>
>I don't think I can go back and correct my mistakes. But trust me. I do
>regret for my words. I'm really sorry for being rude with you two. Take
>this as my sincerest apology. You two deserve that.
>-
>
>@Everyone
>
>It sucks when you sit on the "humiliation" chair when the mistake is
>not
>yours. I'm a farmer's son from a third world country, yet trying to
>contribute to this world in the way I can.
>
>Asking for feedback is not a mistake. But I have been attacked in
>multiple
>lists for that. This is the only thread I was rude and cocky.
>
>What you all think I spend my time only in attacking others? Have you
>ever
>noticed my other threads ? I
>usually give respect to everyone. But I can't give respect to people
>who
>don't care about others feelings. What you all think, I'm a heartless
>man?
>
>One guy was attacking me for my poor english skills. Excuse me for not
>being poetic in my paper. I studied in my local language. English was
>an
>alien language to me. I started to learn "English" only in my early
>twenties. I just turned 30. This is what I picked in the past 10 years.
>
>Just because the ball is in your court doesn't mean, you all can throw
>at
>me in the way you can. I explained what happened between John Levine
>and me
>in my original post. That's because I don't want this man to go and
>create
>another thread to attack me or meddle in my efforts.
>
>I'm a guy who spend day and night in working on things I believe. I'm
>definitely not gonna turn into a Mark Zuckerberg. But I'm gonna make a
>difference to this world one way or another.
>
>None of never completed my paper. Most probably you have no idea what's
>in
>it. But you all think you have the right to attack me, because I was
>rude
>with John? This is an engineering community. Don't convert it into a
>"Prison Brotherhood" where the new guy always has to bend over.


I have no idea who you are, or who John is, or what sort if disagreement you 
guys had. I also don't care. I'm a user of this list, I read the threads that 
look interesting when I can (when I have time) and sometimes post responses. 
You haven't offended me and I don't owe you anything, so here is my impartial 
response;

Your white paper is 300 pages long. That is literally 10x the length of what a 
white paper should be. White papers are not instruction manuals on exactly how 
something works and how to set it up, they're short succinct documents that 
give the highlights of the product, who can benefit from it, how, why etc.

Who is your target audience for your writing? I read somewhere up to about half 
of your medium blog post. You start by explaining what different kinds of email 
are (spam vs phishing, transactional vs promotional) etc. This is an 
introduction for what email is. That's too basic for sending to a technical 
audience. Skip right to the main course, people have short attention spans, I 
hate having to skim through pages of stuff I know to find what I'm looking for.

I didn't finish the blog post. I read that I'd have to create a mailbox 
(Dombox) for each domain I want to receive mail form and I switched off. I read 
further to see the examples but this killed my interest. Its too much effort 
for the average person. I'm emailing you now from throw-away Gmail account, 
which is free with world class spam filtering, I get like 1 spam email in my 
inbox per month. 1! And the reason I have this Gmail account is so that I can 
be reckless with handing out this this email address and it has no negative 
consequences for me. It's signed up to about 50 mailing lists right now, I get 
hundreds of emails. If I had to create an entry for each domain I wanted to 
received mail from I'd pull my eyes out with frustration.

Also, this already exists. I could just run my own mail server and deny a mail 
except from domains or addresses which I have explicitly white listed in my 
server config, so how is your solution bettering that? I can even use a two 
step system: free email with Gmail to use their excellent spam filtering 

Re: [Community bleaching on edge] RTBH no_export

[James Bensley]

On Wed, 6 Feb 2019 at 13:55,  wrote:
>
> Hi folks,
>
>
>
> This “RTBH no_export” thread made me wonder what is the latest view on BGP 
> community bleaching at the edge (in/out).
>
> Anyone filtering extended RT communities inbound on NOSes that accept 
> extended communities by default? Yeah about that…

Hi Adam,

I think Junos is an example of a NOS that advertises extended BGP
communities by default (and accepts them without scrubbing). It seems
"not ideal" to me (by which I mean there could be potential for BGP
NLRIs to be processed in an undesired way). However, I think that
ext-comm information sent in NLRI UPDATES over an AFI/SAFI 1/1 or 2/1
session aren't processed.

I haven't got the time to lab this right now but, I guess one question
would be if (for example) a CPE sends a BGP UPDATE over an 1/1 or 2/1
session into a PE inside a VRF, with ext comm attached, when the
UPDATE is advertised to another PE over a 1/128 or 2/128 session will
that remote PE process the ext-comm value the CPE sent to the initial
PE in the 1/1 or 2/1 session? What if that CPE was in instead a
transit or peering partner and you're running an Internet-in-a-VRF
design, can anyone on the Internet send routes into your edge PE and,
with the correct ext-comm, have them importing into another L3 VPN?

Cheers,
James.

Re: Your opinion on network analysis in the presence of uncertain events

[James Bensley]

On Tue, 15 Jan 2019 at 19:01, Vanbever Laurent  wrote:
>
> Hi NANOG,
>
> Networks evolve in uncertain environments. Links and devices randomly fail; 
> external BGP announcements unpredictably appear/disappear leading to 
> unforeseen traffic shifts; traffic demands vary, etc. Reasoning about network 
> behaviors under such uncertainties is hard and yet essential to ensure 
> Service Level Agreements.
>
> We're reaching out to the NANOG community as we (researchers) are trying to 
> better understand the practical requirements behind "probabilistic" network 
> reasoning. Some of our questions include: Are uncertain behaviors 
> problematic? Do you care about such things at all? Are you already using 
> tools to ensure the compliance of your network design under uncertainty? Are 
> there any good?
>
> We designed a short anonymous survey to collect operators answers. It is 
> composed of 14 optional questions, most of which (13/14) are closed-ended. It 
> should take less than 10 minutes to complete. We expect the findings to help 
> the research community in designing more powerful network analysis tools. 
> Among others, we intend to present the aggregate results in a scientific 
> article later this year.
>
> It would be *terrific* if you could help us out!
>
> Survey URL: https://goo.gl/forms/HdYNp3DkKkeEcexs2
>
> Thanks much!
>
> Laurent Vanbever, ETH Zürich
>
>
> PS: It goes without saying that we would also be extremely grateful if you 
> could forward this email to any operator you know and who may not read NANOG.

Hi Laurent,

I have filled out the survey however, I would just like to request
that in the future you don't use a URL shortner like goo.gl; many
people don't like those because we can't see were you're sending us
until we click that link. Some people also block them because they are
a security issue (our corporate proxy does, I have to drop off the VPN
or use a URL expander to retrieve the original URL).

Also have you seen Batfish? I looks like you guys want to write a tool
that has some overlap with Batfish. Batfish can ingest the configs
from my network and answer questions such as "can host A can reach
host B?" or "will prefix advertisement P from host A will be
filtered/accepted by host B?", "if I ping from this source IP who has
a return route and can respond?" etc.

Kind regards,
James.

Re: Network Speed Testing and Monitoring Platform

[James Bensley]

On Wed, 16 Jan 2019 at 16:54, Colton Conor  wrote:
>
> As an internet service provider with many small business and residential 
> customers, our most common tech support calls are speed related. Customers 
> complaining on slow speeds, slowdowns, etc.
>
> We have a SNMP and ping monitoring platform today, but that mainly tells us 
> up-time and if data is flowing across the interface. We can of course see the 
> link speed, but customer call in saying the are not getting that speed.
>
> We are looking for a way to remotely test customers internet connections 
> besides telling the customer to go to speedtest.net, or worse sending a tech 
> out with a laptop to do the same thing.
>
> What opensource and commercial options are out there?

Hi Colton,

In the past I have used CPEs which support remote loopback. When the
customer complains we enable remote loopback, send the traffic to that
customers connection (rather than requiring a CPE that can generate
the traffic or having an on site device) and measuring what comes
back.

Cheers,
James.

Re: JunOS Fusion Provider Edge

[James Bensley]

Hi Mehmet,

This has been discussed on the Juniper-NSP list several times, here's
a couple of examples:

https://puck.nether.net/pipermail/juniper-nsp/2018-November/036673.html
https://puck.nether.net/pipermail/juniper-nsp/2018-April/035397.html
https://puck.nether.net/pipermail/juniper-nsp/2016-June/032964.html

Cheers,
James.

Re: IGP protocol

[James Bensley]

On 15 November 2018 01:51:28 GMT, Baldur Norddahl  
wrote:
>Also not true that the management network is the last thing to boot. In
>contrary, everything else depends on that being ready first. And that
>would
>also be true if we used is-is.

It is when you out management in a VRF...

>ons. 14. nov. 2018 14.55 skrev James Bensley :
>
>>  Several networks I've seen place
>management
>> in a VRF / L3 VPN, which means that by the time you have remote
>> management access, everything else is already working, it's like the
>> last thing to come up when there's been a problem


Cheers,
James.

Re: IGP protocol

[James Bensley]

On Tue, 13 Nov 2018 at 12:09, Saku Ytti  wrote:
>
> On Tue, 13 Nov 2018 at 12:37, Mark Tinka  wrote:
>
> > Main reasons:
> > - Doesn't run over IP.
>
> Why is this upside? I've seen on two platforms (7600, MX) ISIS punted
> on routers running ISIS without interface having ISIS.  With no
> ability to limit it, so any connected interface can DoS device with
> trivial pps rate, if ISIS is being ran.

I guess the OPs original question wasn't clear enough because, I think
most people are talking about IS-IS vs OSPF2/3 from a theoretical
protocol perspective, and you're talking from a practical vendor
implementation perspective.

>From a purely theoretical perspective I see IS-IS not running over IP
as an advantage too. No mater what routes I inject into my IGP, IS-IS
won't stop working. I may totally fsck my IP reachability but IS-IS
will still work, which means that when I fix the issue, service should
be restored quite quickly. Several networks I've seen place management
in a VRF / L3 VPN, which means that by the time you have remote
management access, everything else is already working, it's like the
last thing to come up when there's been a problem. I like the
"management in the IGP + IS-IS" design.

However, in reality the vendor implementation blows the protocol
design out of the water. You need to consider both when evaluating a
new IGP. Cisco nearly implemented a handy feature with
prefix-suppression, whereby in IOS for OSPF only one would prevent
p-t-p links being advertised into the IGP database. But they didn't
implement this for IS-IS. Then in IOS-XR they removed this feature
from OSPF and implemented it for IS-IS ?!?! So yeah, vendors
implementations are just as important and the theoretical potential of
the protocol.

Oh yeah, forgot to answer the original question. For a greenfield
deployment I'd be happy with either OSPFv3 or IS-IS as long as it's
well designed I don't see much between them, it would come down to
vendor support then.

Cheers,
James.

Re: NAT on a Trident/Qumran(/or other?) equipped whitebox?

[James Bensley]

On Mon, 15 Oct 2018 at 10:07,  wrote:
>
> Interesting, but isn’t stateful tracking once again just swapping, but in 
> this case port 123 in port 32123 out?
>
> So none of the chips you named below support swapping parts of L4 header and 
> that part is actually done with SW assistance please?
>
> So for example the following:
>
> https://eos.arista.com/7150s-nat-practical-guide-source-nat-dynamic/#2Dynamic_Source_NATOverload_Many_to_one
>
> - wouldn’t be at line-rate please?

Hi Adam,

NAT/PAT is an N:1 swapping (map) though so a state/translation table
is required to correctly "swap" back the return traffic. MPLS for
example is 1:1 mapping/action. NAT/PAT state tables tend to fill
quickly so to aid with this we also have timers to time out the
translations and free up space in the translation table, and also
track e.g. TCP RST or TCP FIN to remove entries from the table, so
it's not "just swapping".

Cheers,
James.

Re: Console Servers

[James Bensley]

On Wed, 19 Sep 2018 at 09:50, Saku Ytti  wrote:

> I think WAN indeed is very market situational, and if you need to
> support world, it is beneficial to have solution which supports many
> WAN options, without needing external boxes and external power bricks.
> We try to do just ethernet, but even that is already being provided as
> copper, fibre and in one market with PPPoE, all which are non-issues
> by going with Cisco. I do wish I had second option, I do wish JNPR SRX
> would support async serial ports.

Agreed, this is where the Cisco's shine. We can insert a mixture of
ADSL/VDSL, Ethernet and serial cards into the same box. It's a nice
all in one solution that supports all our various OOB connection types
and the console connectivity, and we could connect a IP management
switch.

I mentioned earlier that OOB inside a DC is different to in a
telephone exchange, and the OP didn't mention which was required. I
forgot to mention that a third kind of OOB is for kit that is outside.
If you need temperature hardened and DC powered OOB kit your options
dramatically shrink and it's worth considering if you're in that boat.

Cheers,
James.

Re: Console Servers

[James Bensley]

On Tue, 18 Sep 2018 at 15:26, Saku Ytti  wrote:
>
> On Tue, 18 Sep 2018 at 16:39, Alan Hannan  wrote:
>
> > Long ago I used Cisco 2511/2611 and was fairly happy.  A little later I 
> > used portmaster and was less so.  Recently I've been using Opengear and 
> > they work fairly well but the price is fairly high.   I use the CM7100 and 
> > IM7100.
>
> Out of curiosity, how do you connect them? I see quotes around
> 200USD/MRC for ethernet in US, implying 12kUSD 5 year cost on just
> connectivity, add rack rental, and power and Opengear price is maybe
> 10% of TCO?
>
> Personally I still prefer Cisco, as not to have new operating system
> to automate. Add conserver to connect persistently to each console
> port, so that you get persistent logs from console to your NMS, and so
> that you can multiplex your console sessions.
> It's hard to recover the CAPEX benefit if you need OOB platform
> specific OPEX costs.
>
> --
>   ++ytti

Hi Saku,

I forgot to mention, it also depends how "out" of band your OOB needs
to be. We use Ciena 6500s for our DWDM infrastructure and they have a
wayside channel (like various DWDM vendors), so it's a separate
channel over the same physical fibre. For anything except a fibre cut
it seems to work.

Cheers,
James.

Re: Console Servers

[James Bensley]

On Tue, 18 Sep 2018 at 14:38, Alan Hannan  wrote:
>
> I'd like your input on suggestions for an alternate serial port manager.
>
> Long ago I used Cisco 2511/2611 and was fairly happy.  A little later I used 
> portmaster and was less so.  Recently I've been using Opengear and they work 
> fairly well but the price is fairly high.   I use the CM7100 and IM7100.
>
> General specs I'm looking for are:
>
>  * 8 to 48 or more rs232 serial ports on rj45
>  * nice-to-have software selectable pinouts (cisco v. straight)
>  * gig-e ethernet port (100mbps ok)
>  * 1U form factor
>  * redundant AC power
>  * access physical serial connections via local port #
>  * access physical serial connections via local IP alias (nice to have)
>
> Can you recommend a serial port server/concentrator that I could use in place 
> of opengear for a better value and/or lower cost?
>
> I'm just ignorant about the current market for serial port concentrators and 
> so far web searches have not revealed ideas, so your input is appreciated!
>
> Thanks!
>
> -alan

Hi Alan,

Ah the trusty Cisco solution - yep, used the 2800 series quite a bit
for exactly this, just last year even I was deploying them. 16-32
serial connections for OOB console, an Ethernet port for an OOB IP
MGMT switch and VDSL for the WAN connection. You can also use low end
Juniper SRX devices for this (SRX200 series) and a cheap console
server, I've used SRX + OpenGear (not so cheap) console server just
fine. You just need a couple of central firewalls to terminate some
IPSEC tunnels (again, cheap SRXs have served fine as the crypto
throughput is typically low).

Some companies don't deploy anything into production which is not
vendor supported, so the 2800s wouldn't fly in that case. We used to
buy 2800s off ebay and 2nd hand tin sellers. However, for lab work
some companies are more relaxed, this is an example 2800 config that I
use for console access in the lab if you want:
https://null.53bits.co.uk/index.php?page=hwic-16a-terminal-server
I'd be reluctant to deploy Cisco 2800s (or similar) today unless there
is a newer variant, is there an ISGv2 variant with serial connectivity
that Cisco will be supporting for a few more years? I know OpenGrear
are expensive but in my current outfit, they do "just work" and the
few we had at my old place, again they did "just work".

Cheers,
James.

Re: Console Servers

[James Bensley]

On Tue, 18 Sep 2018 at 15:26, Saku Ytti  wrote:
>
> On Tue, 18 Sep 2018 at 16:39, Alan Hannan  wrote:
>
> > Long ago I used Cisco 2511/2611 and was fairly happy.  A little later I 
> > used portmaster and was less so.  Recently I've been using Opengear and 
> > they work fairly well but the price is fairly high.   I use the CM7100 and 
> > IM7100.
>
> Out of curiosity, how do you connect them? I see quotes around
> 200USD/MRC for ethernet in US, implying 12kUSD 5 year cost on just
> connectivity, add rack rental, and power and Opengear price is maybe
> 10% of TCO?
>
> Personally I still prefer Cisco, as not to have new operating system
> to automate. Add conserver to connect persistently to each console
> port, so that you get persistent logs from console to your NMS, and so
> that you can multiplex your console sessions.
> It's hard to recover the CAPEX benefit if you need OOB platform
> specific OPEX costs.

For cheap OOB connectivity that scales, I've had some success with
VDSL for OOB console server connections. Note that I didn't say
"great"...

In some DCs I've done mutual OOB swaps with other telcos in the same
suite, this is usually cheap or free (excluding the one time xconnect
cost, in suite xconnects often have no recurring charge) but you need
to track them all, often every swap is bespoke, providers come and go
so you need to replace them, if it's free you sometimes don't get
maintenance alerts ;)

Sometimes the DC provider has an OOB connectivity service that uses
separate transit providers than we use and this often cheap too. Again
this is often bespoke per DC/colo provider though.

The most scalable solution I've been involved in so far is VDSL. Here
in the UK lots of DCs are on-net for the national incumbent VDSL
provider (BT). It means we can have the same style of connection to
most DCs, same physical presentation, same cost, it eases the contract
management for renewing them as we have one supplier etc. The biggest
problem I've experienced with this approach is getting the copper line
to the rack, some DCs charge a small fortune as copper pairs to a rack
is a bespoke service for them, some do it regularly.

I've just moved on from an LLU provider in the UK, a CLEC in US
terminology, we had about 1200 PoPs around the UK most of which were
telephone exchanges. If you want OOB in a DC it's different to a
telephone exchange (well it is here), seeing as the OP hasn't
mentioned if OOB will be in DCs/telephone exchanges/sailing boats/etc.
I think it's worth pointing out tjat VDSL is often not available
within an exchange here and maybe it's the same in the US.

Cheers,
James.

Re: Service provider story about tracking down TCP RSTs

[James Bensley]

On Sat, 1 Sep 2018 at 21:06, Garrett Skjelstad  wrote:
>
> I would love this as a blog post to link folks that are not nanog members.
>
> -Garrett

Hi Garrett,

It is available via the NANOG list archives:
https://mailman.nanog.org/pipermail/nanog/2018-September/096871.html

I've shared this story to non-list member using that URL.

Thanks for the write up Frank!

Cheers,
James.

Re: Multicast traffic % in enterprise network ?

[James Bensley]

On 10 August 2018 at 08:44, Jethro R Binks  wrote:
>
> In terms of other Internet use, the BBC recently published this white
> paper on the R efforts with HTTP Server Push/QUIC, part of which
> describes an "experimental IP multicast profile of HTTP over QUIC".
>
>   https://www.bbc.co.uk/rd/publications/whitepaper336
>
> Jethro.

The BBC publishes this list of multicast partners but I'm not sure if
it's up-to-date and they still offer multicast outside of their own
network? iPlayer is unicast to consumer, so? Do they offer multicast
to iPlayer to these partners or only for set-top-box services?

Cheers,
James.

Re: Multicast traffic % in enterprise network ?

[James Bensley]

On 9 August 2018 at 13:57, Saku Ytti  wrote:
> On Thu, 9 Aug 2018 at 15:27, James Bensley  wrote:
>
>> A recent customer uses multicast to have the same packet arrive at
>> multiple destinations at the same time for resilience (their own
>> internal systems, not IPTV or media etc). Having just refreshed their
>> network for the next 5-10 years it's not going away anytime soon.
>
> I believe the same time delivery is motivation for stock exchanges
> too. One of the larger exchanges used MX and multicast, which of
> course does btree or utree (in this  case utree) replication, which
> makes delivery times are very much variant.
> So it is very much implementation detail what type of delivery time
> differences to expect in different ports and it is not by design
> superior to unicast.


I'm definately not saying it was a good idea / good design :)

I'm just saying that it's another example of multicast in use that is
not the usual IPTV or financial trading (aeronautical in this case).

Cheers,
James.

Re: Multicast traffic % in enterprise network ?

[James Bensley]

On 8 August 2018 at 19:49, Mankamana Mishra (mankamis) via NANOG
 wrote:
> Hi Every one,
> Recently we had good discussion over multicast uses in public internet. From 
> discussion, it was pointed out uses of multicast is more with in enterprise.  
> Wanted to understand how much % multicast traffic present in network
>
>   *   If there is any data which can provide what % of traffic is multicast 
> traffic. And if multicast is removed, how much unicast traffic it would add 
> up?
>   *   Since this forum has people from deployment area, I would love to know 
> if there is real deployment problems or its pain to deploy multicast.
>
>
> These questions is to work / discussion in IETF to see what is pain points 
> for multicast, and how can we simplify it.

A recent customer uses multicast to have the same packet arrive at
multiple destinations at the same time for resilience (their own
internal systems, not IPTV or media etc). Having just refreshed their
network for the next 5-10 years it's not going away anytime soon.

Cheers,
James.

Re: using expect to log into devices

[James Bensley]

Do you need to write this yourself, I've used this expect script too many times 
such that I should be ashamed...It "just works":

https://sourceforge.net/projects/cosi-nms/files/ciscocmd/

Cheers,
James.

Re: Proving Gig Speed

[James Bensley]

> From: "James Bensley" 
> Also I recommend you test to a server on you network near to your
> peering & transit edge. This way users can test up to the point where
> you would have over the "The Internet" and have no further control.
> Testing to a server off-net (like off-net Ookla tells me nothing in my
> opinion).

...

On 17 July 2018 at 15:36, Mike Hammett  wrote:
> It tells you how good your peering is. ;-)

Or transit :)

Cheers,
James.

Re: Proving Gig Speed

[James Bensley]

On 17 July 2018 at 17:18, Mike Hammett  wrote:
> I don't think you understand the gravity of the in-home interference issue. 
> Unfortunately, neither does the IEEE.
>
> It doesn't need to be in lock-step, but if a significant number of homes have 
> issues getting over 100 megabit wirelessly, I'm not sure we need to be 
> concerned about 10 gigabit to the home.

Maybe leaky feeder cables will become the norm, running along the
walls/ceilings of all new build homes?

But assuming a wired connection within the premises for a moment, and
that we get 1Gbps over that wired connection, because we all have
FTTH: For the question of "does it make any difference (1Gbps/10Gbps
instead of 100Mbps)": If I download a 4K movie to watch it should take
an order of magnitude less time at 1Gbps than 100Mbps. Or even when
streaming, my player will fetch a video chunk/segment that is
typically in the 3-10 seconds range, I would fetch each chunk much
quicker, and so my ISPs connection spends more time idle, which means
their backbone carries a higher volume of traffic for a smaller period
of time. There is some benefit to be had in the balance of a user
consuming X Mbps across the backbone for Y seconds or X^2 Mbps but for
Y/10 seconds. It expect it would affect oversubscription and content
ratios.

Cheers,
James.

Re: Proving Gig Speed

[James Bensley]

On 17 July 2018 at 12:50, Mark Tinka  wrote:
> But to answer your questions - for some customers, we insist on JDSU
> testing for large capacities, but only if it's worth the effort.
>
> Mark.

Hi Mark,

Our field engineers have 1G testers, but even at 1G they are costly
(in 2018!), so none have 10Gbps or higher testers and we also only do
this for those that demand it (i.e. no 20Mbps EFM customer ever asks
for a JSDU/EXO test, because iPerf can easily max out such a link,
only those that pay for say 1G over 1G get it). Hardware testers are
the best in my opinion right now but it annoys me that this is the
current state of affairs, in 2018, even for 1Gbps!

Cheers,
James.

Re: Proving Gig Speed

[James Bensley]

On 17 July 2018 at 09:54, Saku Ytti  wrote:
> On Tue, 17 Jul 2018 at 10:53, James Bensley  wrote:
>
>> Virtually any modern day laptop with a 1G NIC will saturate a 1G link
>> using UDP traffic in iPerf with ease. I crummy i3 netbook with 1G NIC
>> can do it on one core/thread.
>
> I guess if you use large packets this might be true. But personally,
> if I'm testing network, I'm interested in latency, jitter, packet, bps
> _AND_ pps goals as well, not just bps goal.

Hi Saku,

Yeah I fully agree with what you are saying however, the OPs question
sounds like he "only" needed to prove bandwidth. With 1500 byte frames
I've run it up to nearly 10Gbps before (it was between VMs in two
different DCs that were having slow transfers and the hyper-visors had
10G NICs, so I dare say, on bare metal with large frames it will do
10Gbps).

> And I've never seen clean
> 1Gbps on iperf with small packets. It just cannot be done, even if
> iPerf was written half decently and it used recvmmsg, it still
> wouldn't be anywhere near.
> Clean 1Gbps with small packets in user space is actually very much
> doable today, just you can't use UDP socket, you must use AF_PACKET on
> Linux or BPF on OSX and you can write portable 1Gbps UDP
> sender/receiver.
> I'm very surprised we don't have iperf like program for netengs which
> does this and reports latency, jitter, packet loss with binary search
> for highest lossless pps/bps rates.

I absolutely agree there is a gap in the open source market for this
exact application. A tool that sends traffic between Tx and Rx (or
bidirectionally) at a specified frame size and frame rate, which can
max out 10Gbps at 64 byte frames if required (I say 10Gbps instead of
1Gbps because 10Gbps as an access circuit speed is being increasingly
common), and throughout the test it should report RTT and one way
latency, jitter and packet loss etc. and then output the results in a
format that is easy to parse. It should also have a JSON API and be
able to run in a "daemon" mode like an iPerf server that is always on
ready for people to test to/from.

> I started to write one with Anton Aksola in Rust (using libpnet[0]),
> and implemented quite flexible protocol (server/client, client can ask
> server exactly what kind of packet to construct/expect, what rate to
> send/receive over JSON based protocol), so you could also use it to
> ask it to DDoS your routers control-plane in lab etc. And actually got
> it working, OSX+Linux ~wirarate (still needs higher end laptop to do
> 1.5Mpps on single core and we didn't implement multicore support). But
> as both of us are trash in Rust (and every other applicable language
> in this domain), we kind of dropped the project once we had sufficient
> POC running on our laptops.
> Someone who actually can code, could easily implement such program in
> a weekend. I'm happy to share the trash we've done if someone intends
> to check this box in open source world. May use it for inspiration, or
> just straight up add polish and enough CLI to make it usable as-is.

I went through a similar process. AF_PACKET is definitely what you
need to use if you want to use user-space in Linux (don't know about
MAC, only use Linux). I wrote a basic multi-threaded load generator
and load sinker (Tx and Rx) in C using various Kernel methods (send(),
sendmsg(), sendmmsg(), and PACKET_MMAP) with AF_PACKET to compare them
all: https://github.com/jwbensley/EtherateMT

The problem is that C is a great language to write high performance
stuff, it's a shit language to create a JSON API in. I have two back
to back lab servers at work with 10G links between them, low end
2.1Ghz Xeons, I get 1Mpps per core, 8 cores-1 for OS means I max out
at 7Mpps :(

I know that XDP is coming to Linux user space so we'll see where that
goes, as it promises the magic performance levels we want. Also
TPACKETv4 is coming for AF_PACKET in Linux which should also get us to
that magic level of performance in user land (it is effectively Kernel
bypass). I'll add this to EtherateMT when I get some time to check
it's performance: https://lwn.net/Articles/737947/

So EtherateMT works OK as a proof of concept, but nothing more. It
requires 100% CPU utilisation to send/receive at such high pps rates,
there is no CPU time for stats collection or fancy rtt/latency/jitter
etc. That can only be done (right now) with something like DPDK,
because it we only need one or two cores for Tx/Rx and then we have
free cores for stats collections/generations etc. I looked into
MoonGen, it creates Lua bindings for DPDK which means you can rapidly
develop DPDK based tools without knowing much about DPDK. It had some
RFC2544 Lua scripts for DPDK and I started to re-write them as they
were old and didn't work with the latest version of MoonGen:

https://github.com/jwbensley/MoonGen-Scripts

The throughput script

Re: Proving Gig Speed

[James Bensley]

On 16 July 2018 at 18:58, Chris Gross  wrote:

Hi Chris,

> I'm curious what people here have found as a good standard for providing 
> solid speedtest results to customers. All our techs have Dell laptops of 
> various models, but we always hit 100% CPU when doing a Ookla speedtest for a 
> server we have on site. So then if you have a customer paying for 600M or 
> 1000M symmetric, they get mad and demand you prove it's full speed. At that 
> point we have to roll out different people with JDSU's to test and prove it's 
> functional where a Ookla result would substitute fine if we didn't have 
> crummy laptops possibly. Even though from what I can see on some google 
> results, we exceed the standards several providers call for.
>
> Most of these complaints come from the typical "power" internet user of 
> course that never actually uses more than 50M sustained paying for a 
> residential connection, so running a circuit test on each turn up is uncalled 
> for.
>
> Anyone have any suggestions of the requirements (CPU/RAM/etc) for a laptop 
> that can actually do symmetric gig, a rugged small inexpensive device we can 
> roll with instead to prove, or any other weird solution involving ritual 
> sacrifice that isn't too offensive to the eyes?

I would say, don't use a browser based speed test - how fast is your
browser? Answer: It can vary wildly!

Also there are SO many variables when testing TCP you MUST test using
UDP if you want to just test the network path. Every OS will behave
differently with TCP, also with UDP but the variance is a lot lower.

Also I recommend you test to a server on you network near to your
peering & transit edge. This way users can test up to the point where
you would have over the "The Internet" and have no further control.
Testing to a server off-net (like off-net Ookla tells me nothing in my
opinion).

Virtually any modern day laptop with a 1G NIC will saturate a 1G link
using UDP traffic in iPerf with ease. I crummy i3 netbook with 1G NIC
can do it on one core/thread.

We have several iPerf servers dotted around the network and our
engineers can test to those at any time and it works well for us.

Cheers,
James.

Re: Linux BNG

[James Bensley]

Hi Baldur,

These guys made a PPPoE client for VPP - you could probably extend
that into a PPP server:

https://lists.fd.io/g/vpp-dev/message/9181
https://github.com/raydonetworks/vpp-pppoeclient

Although, I would agree that deploying PPP now is a bit of a step
backwards and IPoE is the way to be doing this in 2018.

If you want subscribers with a S-TAG/C-TAG landing in unique virtual
interfaces with shared gateway etc, IPv4 + IPv6 (DHCP/v6) and were
deploying this on "real service provider networking kit" [1] then the
way to do this is with pseudowire headend termination. (PWHE/PWHT).
However, you're going to struggle to implement something like PWHT on
the native Linux networking stack. Many of the features you want exist
in Linux like DHCP/v6, IPv4/6, MPLS, LDP, pseudowires etc, but not all
together as a combined service offering. My two pence would be to buy
kit from some like Cisco or Juniper as I don't think the open source
world is quite there yet. Alternatively if it *must* be Linux look at
adding the code to https://wiki.fd.io/view/VPP/Features as it has all
constituent parts (DHCP, IP, MPLS, bridges etc.) but not glued
together. VPP is an order of magnitude faster than the native Kernel
networking stack. I'd be shocked if you did all that you wanted to do
at 10Gbps line rate with one CPU core.

Cheers,
James.

[1] Which means the expensive stuff big name vendors like Cisco and Juniper sell

Re: What are people using for IPAM these days?

[James Bensley]

On 13 June 2018 at 13:54, Paul Ebersman  wrote:
> IPAM? Meh.
>
> Why bother?

So true - when customers want their IP details why should I, the
person they are paying to track this information, spend time
looking-up the info they reqeust?! I normally set them up with a login
to the core and tell them "look it up yourself you lazy git!".

For those that actually believe in "IPAMs" - this is misnomer phrase
these days (has been for ages). If you want to do stuff at scale you
need a "number" tracking system or "abstract resource that is being 0
and 4096 or between 1 and 65535" etc. We assign VLANs, stacks of
VLANs, MPLS labels, stacks of MPLS labels, IPv4 addresses, IPv6
addresses, VPN IDs (pseudowires, VPLS etc.), route targets, route
distinguishers, ASNs, logical interface IDs, physical interfaces, and
so on. They are all finite resources in the network and all "just
numbers". I have made many experiances of people putting a lot of
effort into tracking IP addresses only and none of other stuff. I
don't know why more people aren't asking for recommendations for a
"resource tracker" [1].

Cheers,
James.


[1] Resource tracker != CMDB.

Re: Segment Routing

[James Bensley]

On 22 May 2018 at 09:14, Mark Tinka  wrote:
> I'm more curious about use-cases for folk considering SR, than SR itself.
>
> 4 years on, and I still can't find a reason to replace my LDP network
> with SR.
>
> Your use-case makes sense, as it sounds like Cisco deliberately left LDP
> out of your box, considering SR is in. But that's a whole other
> discussion :-)...

I'm also interested in the uses cases.

As a "typical" service provider (whatever that means) who doesn't have
any SR specific requirements such as service chaining, the only
reason/feature SR has which currently makes me want to deploy it is
TI-FLA (to improve our (r)LFA coverage) - but this is only for failure
scenarios so under normal working conditions as far as I know, there
is no benefit available to us right now.

Cheers,
James.

Re: Route Reflector Client Design Question

[James Bensley]

On 4 May 2018 at 07:01, Erik Sundberg  wrote:
> 1. Can I enable iBGP between the PE's in a full mesh to allow traffic between 
> the PE's without going to the core's. Or does this break the Route Reflector 
> model?

If I have understood your design correctly then don't use
next-hop-self on the RR's. Ideally you'd have out of band RRs so you
don't need NHS but I can see that you're are both RRs and PEs so NHS
is required. If you enabled iBGP sessions between all your PEs you
have defeated the point of RRs :)

I don't know your setup in detail but is there something you can do on
your RRs to allow you to remove NHS? eBGP routes will have NHS by
default so you just need to check the impact on iBGP routes, maybe you
can remove NHS at the expense of redistributing a couple of connected
routes.

Cheers,
James.

Re: How are you configuring BFD timers?

[James Bensley]

On 22 March 2018 at 09:59, Saku Ytti  wrote:
> Ethernet handles unidirectional failure natively through autonego asserting 
> RFI.

I was thinking about this as I wrote that post. I've not had a chance
to test this across our various devices types, I will have to try and
find the time to test which devices support this and how effectively
it works. For now we just use BFD because it's tested as working on
all our device types. We've seen issues where LOS isn't working
properly across third party DWDM platforms so for now, we have no BFD
bugs and it catches these issues for us.

I think it might also be worth testing Ethernet RFI to check the
delays that Jason was talking about; which is quicker to signal link
down and begin the re-convergence process, RFI / LOS / BFD?

Cheers,
James.

Re: How are you configuring BFD timers?

[James Bensley]

On 21 March 2018 at 13:10, Jason Lixfeld  wrote:
> Hey,
>
> For those running BFD on your land-based point-to-point links, I’m interested 
> in hearing about what factors you consider when deciding how to configure 
> your timers and multiplier.
>
> On paper, BFD between two devices over a local or metro dark fibre or wave 
> seems pretty trivial:  Assuming your gear can a) support echo mode b) 
> hardware offloads echo processing c) automatically treats echos as vital and 
> puts them into the appropriate high priority queue, then setting the timers 
> down to their lowest possible values (3ms on some of the gear that I’ve seen) 
> and some low multiplier seems more than reasonable.  But?
>
> From another angle, your link isn’t dark fibre or a wave but, for example, 
> ethernet over some sort of IP based L2 Transport, and is still a low (sub 
> 1ms) one-way latency local or metro link.  How do you set your timers, and 
> what do you base that on?
>
> From yet another angle, what if your link is a long-haul wave, or for that 
> matter a wave of any distance that imposes a one-way latency that is higher 
> than the minimum tx and rx timers that are supported by your gear?  We’ll 
> assume an unprotected wave, because I’m sure if it’s protected, you have no 
> choice but to consider the one-way latency of the longest of the two segments.
>
> I made some assumptions above about support for echo mode and hardware 
> offload, but what if (some of) your gear doesn’t support some or all of that 
> stuff?  How do you factor your configuration decisions?
>
> Thanks!


Going back to the original question;

> From another angle, your link isn’t dark fibre or a wave but, for example, 
> ethernet over some sort of IP based L2 Transport, and is still a low (sub 
> 1ms) one-way latency local or metro link.  How do you set your timers, and 
> what do you base that on?

Personally I don't care if it's a wavelength, dark fibre or L2 VPN
service. I don't treat them differently based on the underlying
connectivity type. The SLAs are probably more important. But if we are
paying for say 10G of capacity on a link which is say a 10G pseudowire
from another carrier, I treat it the same as a dark fibre connected to
10G transceivers at each end. Wave lengths are generally more stable
in my opinion, we did have a 10G L2 Ethernet circuit from a carrier
that was a pseudowire from them essentially, and their PE was under a
DDoS attack so our L2 VPN service was affected (because the pseudowire
was flapping up and down). But once the circuit is up and running for
a while, if you're regularly pushing somewhere near the max circuit
bandwidth and monitoring circuit latency, you'll get a feel for "how
good" the carrier is and then adjust from there. Generally speaking
though, if the carrier is "good" I treat DF/lamda/L2 circuits the same
with regards the BFD/IGP tuning.

> I made some assumptions above about support for echo mode and hardware 
> offload, but what if (some of) your gear doesn’t support some or all of that 
> stuff?  How do you factor your configuration decisions?

Elsewhere in the thread you have mentioned that you are using Cisco
ME3600 devices. If you disable BFD echo mode you will be able to get
low timers on these devices. Echo mode is enabled by default on IOS
when you enable BFD under an interface, which these devices don't
support, so you need to explicitly disable it. See the min/max/avg BFD
timers below between two ME devices when the interfaces are configured
with "bfd interval 50 min_rx 50 multiplier 3":

ME3600#show bfd neighbors interface te0/2 details
...
Session state is UP and using echo function with 50 ms interval.
Session Host: Software
...
Rx Count: 72, Rx Interval (ms) min/max/avg: 1/4976/4323 last: 2348 ms ago
Tx Count: 74, Tx Interval (ms) min/max/avg: 1/4968/4217 last: 1436 ms ago


If you add the command "no bfd echo" to the interface you should see
the following min/max/avg BFD timers:


ME3600#show bfd neighbors interface te0/2 details
...
Session state is UP and not using echo function.
Session Host: Software
...
Rx Count: 3314443, Rx Interval (ms) min/max/avg: 1/72/47 last: 36 ms ago
Tx Count: 3310865, Tx Interval (ms) min/max/avg: 1/72/47 last: 40 ms ago

We have a mixture of devices and they don't all support BFD echo mode.
We have for example Cisco ASR9000s that support both echo / no echo
mode, so it may have one interface towards a Juniper MX running BFD
echo mode and one interface towards a Cisco ME which runs no echo
mode. It's working fine for us.


Cheers,
James.

Re: How are you configuring BFD timers?

[James Bensley]

On 21 March 2018 at 16:37, Luke Guillory  wrote:
> He's asking because if it was dark the interface would go down when the link 
> was lost and the router would pull routes. But PA to FL would lead me to 
> believe it'll be a wave from some type of DWDM gear which brings us to BFD.

Could it not also help with a unidirectional failure of the fibre?
E.g. Loss of signal in only one direction might not bring the link
down, but if one BFD peer stops receiving BFD packets it'll bring the
link down.

On 21 March 2018 at 17:10, Jason Lixfeld  wrote:
> A few years ago I did some testing and found that the time between the 
> transceiver detecting LOS and the routing protocol (ISIS in this case) being 
> informed that the link was down (triggering the recalculation) took longer 
> than it took BFD to signal ISIS to recalculate.

Have you looked at testing and adding this command to your IOS devices:

ip routing protocol purge interface

Also have you tried to set the carrier delay to zero?

carrier-delay down 0

I haven't compared them to BFD explicitly, but I would expect the two
commands together to have the same effect as you're seeing with BFD
(the link down is signaled to the IGP ASAP).

Cheers,
James.

Re: improving signal to noise ratio from centralized network syslogs

[James Bensley]

On 5 February 2018 at 18:57,   wrote:
> On Mon, 05 Feb 2018 10:49:42 -0800, "Scott Weeks" said:
>> I have no knowledge of syslog-ng.  Does it do the
>> real time scrolling like I mention?
>
> Use 'tail -f' or similar.

The only problem is that with BASH based solutions is that they are
slow. They don't scale well.

Some years ago I wrote a script that would periodically (every 5
minutes by default) grep for interesting events / filter uninteresting
events from the syslog file and email you the results. It's here if
anyone is interested: https://null.53bits.co.uk/index.php?page=sysgrep

It's OK for a small network or small number of devices but it doesn't
scale well. Having said that, it's better than nothing and costs $0
(which exactly why I used it in the first place).

Cheers,
James.

Re: Spectre/Meltdown impact on network devices

[James Bensley]

On 7 January 2018 at 19:02, Jean | ddostest.me via NANOG
 wrote:
> Hello,
>
> I'm curious to hear the impact on network devices of this new hardware
> flaws that everybody talk about. Yes, the Meltdown/Spectre flaws.
>
> I know that some Arista devices seem to use AMD chips and some say that
> they might be immune to one of these vulnerability. Still, it's possible
> to spawn a bash shell in these and one with limited privileges could
> maybe find some BGP/Ospf/SNMP passwords. Maybe it's also possible to
> leak a full config.
>
> I understand that one need access but still it could be possible for one
> to social engineer a NOC user, hijack the account with limited access
> and maybe run the "exploit".
>
> I know it's a lot of "if" and "maybe", but still I'm curious what is the
> status of big networking systems? Are they vulnerable?
>
> Thanks
>
> Jean

Some devices run affected Intel chips like the Cisco ASR9000 series
and they run Perl and Python so very exploitable I would expect, IF
you have shell access.

There are much more serious security issues out there to worry about
for networking gear than Meltdown/Spectre, e.g. this great CCC34 preso
where the attacker runs remote code on a Cisco device and removes the
password authentication for Telnet:
https://events.ccc.de/congress/2017/Fahrplan/events/8936.html

The video is on the CCC YouTube channel:
https://www.youtube.com/watch?v=fA6W9_zLCeA

If somebody has shell access you're basically knackered, I'm more
concerned about these kinds of remote exploits as demonstrated. Proper
iACLs/CoPPs and IDS/IPS, good patching cycles etc.

Cheers,
James.

Re: DSL Operators Mailing List?

[James Bensley]

On 7 January 2018 at 17:10, Mike Hammett  wrote:
> Is there a good mailing list for DSL operators? A cursory search really only 
> came up with DSL Reports, which is far from what I'm looking for.

Hi Mike,

I only know of the https://puck.nether.net/mailman/listinfo/cisco-bba
list. It is Cisco specific list and very low volume (see the archives)
however, people might not mind you talking about DSL operations in
general as long as it isn't specific to another vendor.

Cheers,
James.

Re: Bandwidth distribution per ip

[James Bensley]

On 20 December 2017 at 15:52, Saku Ytti  wrote:
> On 20 December 2017 at 16:55, Denys Fedoryshchenko  wrote:
>
>> And for me, it sounds like faulty aggregation + shaping setup, for example,
>> i heard once if i do policing on some models of Cisco switch, on an
>> aggregated interface, if it has 4 interfaces it will install 25% policer on
>> each interface and if hashing is done by dst ip only, i will face such
>> issue, but that is old and cheap model, as i recall.
>
> One such old and cheap model is ASR9k trident, typhoon and tomahawk.
>
> It's actually pretty demanding problem, as technically two linecards
> or even just ports sitting on two different NPU might as well be
> different routers, they don't have good way to communicate to each
> other on BW use. So N policer being installed as N/member_count per
> link is very typical.

Hi,

In the case of ASR9K IOS-XR 6.0.1 added the following command:
"hw-module all qos-mode bundle-qos-aggregate-mode"

This splits the bandwidth over the links and takes into account the
link bandwidth; with bundle bandwidth 50G (with 10G+40G members) the
ratios become 5/1 and 5/4 respectively (it is supporting unbalanced
member link speeds).

Also the NPUs don't need to talk to each other on the ASR9K; the
central fabric arbiter has a view of bandwidth per VoQ and can control
the bandwidth across the LAG member interfaces when they are spread
over multiple lines cards and NPUs.


Cheers,
James,

Re: DevOps workflow for networking

[James Bensley]

On 10 August 2017 at 01:52, Kasper Adel  wrote:
> We are pretty new to those new-age network orchestrators and automation,
>
> I am curious to ask what everyone is the community is doing? sorry for such
> a long and broad question.
>
> What is your workflow? What tools are your teams using? What is working
> what is not? What do you really like and what do you need to improve? How
> mature do you think your process is? etc etc

The wheels here move extremely slowly so it's slowly, slowly catchy
monkey for us. So far we have been using Ansible and GitLab CI and the
current plan is to slowly engulf the existing network device by device
into the process/toolset.

> Wanted to ask and see what approaches the many different teams here are
> taking!
>
> We are going to start working from a GitLab based workflow.
>
> Projects are created, issues entered and developed with a gitflow branching
> strategy.
>
> GitLab CI pipelines run package loadings and run tests inside a lab.

Yes that is the "joy" of GitLab, see below for a more detailed
breakdown but we use docker images to run CI processes, we can branch
and make merge requests which trigger the CI and CD processes. It's
not very complicated and it just works. I didn't compare with stuff
like BitBucket, I must admit I just looked at GitLab and saw that it
worked, tried it, stuck with it, no problems so far.

> Tests are usually python unit tests that are run to do both functional and
> service creation, modification and removal tests.
>
> For unit testing we typically use python libraries to open transactions to
> do the service modifications (along with functional tests) against physical
> lab devices.

Again see below, physical and virtual devices, and also some custom
python scripts for unit tests like checking IPv4/6 addresses are valid
(not 999.1.2.3 or AA:BB:HH::1), AS numbers are valid integeters of the
right size etc.

> For our prod deployment we leverage 'push on green' and gating to push
> package changes to prod devices.
>
> Thanks

Yeah that is pretty much my approach too. Device configs are in YAML
files (actually multiple files). So one git repo stores the
constituent YAML files, when you update a file and push to the repo
the CI process starts which runs syntax checks and semantic checks
against the YAML files (some custom python scripts basically).

As Saku mentioned, we also follow the “replace entire device config”
approach to guarantee the configuration state (or at least “try” when
it comes to crazy old IOS). So this means we have Jinja2 templates
that render YAML files into device specific CLI config files. They
live in a separate repo and again many constituent Jinaj2 files make
one entire device template. So any push to this Jinja2 repo triggers a
separate CI workflow which performs syntax checking and semantic
checking of the Jinja2 templates (again, custom Python scripts).

When one pushes to the YAML repo to update a device config, the syntax
and semantic checks are made against the YAML files; they are then
“glued” together to make the entire device configs in a single file,
the Jinja2 repo is checked out, the entire YAML file is used to feed
the Jinja templates and configs are built and now the vendor specific
config needs to be syntax checked.

This CD part of the process (to a testing area) is a WIP still, for
Junos we can push to a device and use “commit check” for IOS and
others we can’t. So right now I’m working on a mixture of pushing the
config to virtual IOS devices and to physical kit in the lab but this
also causes problems in that interface / line card slot numbers/names
will change so we need to run a few regex statements against the
config to jimmy it into a lab device (so pretty ugly and temporary I
hope).

When the CD to “testing” passes then CD to “production” can be
manually triggered. Another repo stores the running config of all
devices (from the previous push). So we can push the candidate config
to a live device (using Ansible with NAPALM [1]) and get a diff
against the running config, make the “config replace” action, then
download the running config and put that back into the repo. So we
have a local stored copy of device configs so we can see off-line the
diff’s between pushes. It also provides a record that the process of
going form YAML > Jinaj2 > to device produces the config we expected
(although prior to this one will have had to make a branch and then a
merge request, which is peer reviewed, to get the CD part to run and
push to device, so there shouldn’t be any surprises this late in the
process!).

Is it fool proof, no. It is a young system still being design and
developed. Is it better than before, hell yes.

Cheers,
James.

[1] Ansible and NAPALM here might seem like overkill but we use
Ansible for other stuff like x86 box management so this means
configuring a server or a router is abstracted through one single tool
to the operator (i.e. playbooks are use irrelevant of device 

Re: Virtual or Remote Peering

[James Bensley]

On 15 August 2017 at 15:52, Rod Beck  wrote:
> How well does this service work? I understand it usually involves 
> point-to-multipoint Switched Ethernet with VLANs and resold IX ports. Sounds 
> like a service for ISP that would like to peer, but have relatively small 
> volumes for peering purposes or lopsided volumes.
>
>
> Roderick Beck
>
> Director of Global Sales
>
> United Cable Company
>
> DRG Undersea Consulting
>
> Affiliate Member
>
> www.unitedcablecompany.com
>
> 85 Király utca, 1077 Budapest
>
> rod.b...@unitedcablecompany.com
>
> 36-30-859-5144
>
>
> [1467221477350_image005.png]

I think for very samll providers were cost pretty much governs
everything, it is quite handy.

I know of a small provider who takes/took a 1G pipe to such a remote
connectivity provider, the pipe is split with VLANs and over one VLAN
they have a remote peering session to a IXP, over another VLAN they
receive a blended transit service and over another VLAN L2
connectivity to other PoP the remote peering provider is also.

It is several eggs in one basket but for very small providers this can
provide a much needed money saving opportunity.

Cheers,
James.

Re: Creating a Circuit ID Format

[James Bensley]

On 21 August 2017 at 21:26, Colton Conor  wrote:
> We are building a new fiber network, and need help creating a circuit ID
> format to for new fiber circuits. Is there a guide or standard for fiber
> circuit formats? Does the circuit ID change when say a customer upgrades
> for 100Mbps to 1Gbps port?
>
> What do the larger carriers do? Any advice on creating a circuit ID format
> for a brand new fiber network?
>
>
>  Originally we ran a CLEC using a LECs copper, and our circuit ID was
> historically a telephone number for DSL circuits. The ILEC had a complex
> method for assigning circuit IDs.
>
> I am sure anything will work as long as you keep track of it, but any
> advice would be great!

In my opinion the circuit ID should be an abitrary (but unique) value
and nothing more. As Nick suggested start at 1 and go up. If your
company is called ABC Ltd then maybe have your first circuit ID as
ABC0001 and count up from there, it's as simple as that.

For me, all the circuit ID should be is a record number/ID of a
database entry and nothing more (or a search string). Some people like
to have circuit IDs which include circuit types, or circuit speeds, or
interface type, but as you asked, do you then change the circuit ID if
the circuit speed changes, or the interface types changes, or the
medium etc?

No, it's just a pointer to a database record, KISS.

Cheers,
James.

Re: (Network Orchestrators evaluation) : tail-f vs Anuta vs UBIqube vs OpenDaylight

[James Bensley]

On 10 August 2017 at 02:01, Kasper Adel  wrote:
> Hi,

Hi Kim,

> This is not a vendor bashing thread.
>
> We are a group of networking engineers  less experience with software) in
> the middle of the process of procuring a network automation/orchestration
> controller, if that is even a good definition and we are clueless on how to
> evaluate them.

If you don't have much in house software expertise buying an off the
shelf solution with support could be the best bet for you. ODL is
aimed more at somebody who wants to "roll their own" solution as it's
really giving you a unified southbound API to your devices but you
still need to connect with the ODL via its northbound API in order to
orchestrate your tin really fluently. This will require a lot of in
house development work (probably more than you want). Also this is
“just” (albiet a powerful one) an API to your network. It won’t act as
a single source of truth, it’s not a data store, or an IPAM, or an NMS
etc.

Anuta seems quite good, of all the ones you listed in the subject line
I'd recommend that one. We had a demo of Anuta NCX and I was pretty
happy with it, it's vendor neutral with support for the big names
already built it an you can write Python plugins to extending support
to any weird vendor kit you might have lurking in a remote dusty PoP,
they also take feature request if enough customers have vendor X
they’ll add support for it. It is also semi-service orchestrated, you
can define “services” yourself and config templates and push them out
to devices. I quite liked it, I'd recommend you get a demo if you want
an off the self-product that is vendor neutral with support. It ticks
those basic boxes (probably more but I can’t remember as we didn’t
choose it – not because it was flawed, it just wasn’t what we needed
for the project we had in mind). Like ODL it is just for mass
configuration and essentially and zero touch provisioning. You need to
glue it to the rest of your OSS stack probably via the API.

Tail-f - seeing as they were gobbled up by Cisco, do you mean whatever
the Cisco product is called now, NSO I believe (Network Services
Orchestrator). We had a meeting with Cisco a while back to arrange a
demo, at the time it was very Cisco focused however I think it has
become more open (in that it is still a closed source product but more
network device vendor agnostic).

Don’t know UBiqube – I’ll have a read up on it, thanks!

> Other than the obvious, which is to try them out, i wonder what else is
> important to consider/watch out for.

As mentioned above, get something with an API, if you have multiple
systems internally for BSS and OSS, try to move towards only having
systems with APIs so that in the long term when say the BSS system
accepts an order it can push an update to your OSS stack which
configure a port on an edge router ready for that customer to connect
to, and connects to your NMS API and pre-emptively starts to graph the
port; all that jazz. Service orchestration is more than just
automating config deployment which some people seem to forgot, or
focus too much on, the service wrap is also very important (after
accepting an order for a new CPE from a customer, and having fired the
order over your suppliers API, in the response from the supplier with
the new CPEs serial number, that needs to go into your asset database
and be marked against that customer and the end site it’s being
shipped to etc). Flexible products with a standardised API.


> We are presented with 3 different vendors and even OpenDayLight was
> considered as the open source alternative.

NAPALM was already mentioned as an open source alternative. If you
want to get your hands a bit more dirty consider Ansible or SaltStack
(both of which can be used with and without NAPALM but generally you
want to use them with NAPALM) as they are both open source and free to
use but you can pay for support.

We also looked at Blue Planet from Ciena, it’s an impressive product
with some big name customers. It also has a big price tag as it’s
really for large deployments. We didn’t go with it because we wanted
to start (very) small trials and build up.


> My humble thoughts are given below and i would appreciate getting
> 'schooled' on what i need to ask the vendors:
>
> 1) Are they Model driven : But i still don't know how to evaluate that.

By model driven do you mean like YANG models to infer the
configuration state, or model driven from a service perspective? Anuta
NCX, Blue Planet, Tail-F/NSO, ODL all have support for YANG models if
you meant YANG. Anuta and NSO will let you create “services” which can
be config templates that are deployed as a whole and verified, if you
mean “service” model driven.

> 2) Do they parse Cisco/Juniper CLI or they are limited to SNMP and YANG.

If you have IOS devices you NEED a product that supports IOS CLI. IOS
is a pain in the backside to automate and the CLI is the only really
way of doing it sadly.

> 3) If they 

Re: Long AS Path

[James Bensley]

On 24 June 2017 at 13:10, Mel Beckman  wrote:
> James,
>
> By "experienced by someone else" I mean someone who is not one of your 
> customers.
>
> The better strategy, I think, is to not filter long paths unless you have a 
> reason to see their creating a problem. Otherwise you're just operating on 
> superstition, no?
>
> -mel via cell

Hi Mel,

I mean this as a rhetorical question as we could talk until the end of
time about this; what is the difference between operating on
superstition and trying to be pro-active? Both for me fall under the
category of "risk management".

Cheers,
James.

Re: Long AS Path

[James Bensley]

On 23 Jun 2017 17:03, "Mel Beckman"  wrote:

James,

The question is whether you would actually hear of any problems. Chances
are that the problem would be experienced by somebody else, who has no idea
that your filtering was causing it.

 -mel beckman


Hi Mel,

For us this the answer is almost definitely a yes. We are an MSP (managed
service provider) as opportunities to a traditional ISP, so our customers
know they can open a ticket with us for pretty much anything and we'll try
and look into it.

We have had some weird issues with far away sites, first line can't find
any issue, it works it's way up to somebody who knows how to check if we
would be filtering a route on our transit and peering sessions.

Earlier when I said that care is required when filtering long AS_PATH
routes or certain AS numbers, we looked at the BGP table to see exactly
which routes we'd drop before hand and communicated out these changes. I
think for an MSP this shouldn't be hard to implement and manage, I can
appreciate for a "flat" ISP ("he's some transit, help yourself") it could
be more challenging.

In relation to the OPs question, long AS_PATH routes can be filtered I just
wouldn't bother except for very long paths to drop as little as possible
and be sure of whY you drop/filter.

Cheers,
James.

Re: Long AS Path

[James Bensley]

On 21 Jun 2017 17:51, "Mel Beckman"  wrote:

Steinar,

What reason is there to filter them?


The main reason I know of is this:

On 22 Jun 2017 17:17, "Steve Lalonde"  wrote:

Mel,

There was a Cisco bug many years ago that caused lots of issues. Since then
we have limited max-as to 50 and it has not caused any reported issues yet.

Link that does not require a CCO login to view.
http://blog.ipspace.net/2009/02/oversized-as-paths-cisco-ios-bug.html


Like many providers we do still have legacy kit tucked away with shameful
firmware versions running and also multiple vendors in play so I can't be
sure we don't have devices still susceptible to such a bug in view of the
DFZ.

On 21 Jun 2017 18:45, "Bob Evans"  wrote:

My cut off is 6 ASNs - more than 6 and it never makes it to the FIB.


So for the above reason we do have an AS_PATH limit but it is quite high
like 63 or 120 ASNs (on mobile can't remember and right now). I don't want
to get near a ^2 boundary like 255 or 128 but also don't want to drop
prefixes if possible like a last resort fail-safe, so it's a very high
number and will remove it at some point.

On 22 Jun 2017 14:49, "jim deleskie"  wrote:

I see 5+ prepends as maybe not reason to have your "BGP driving license
revoked" but if I can continue with the concept that you have your BGP
learners permit.


That (6) seems pretty low to me. Apart from a potential bug the other
reason we thought of to block routes with a massive AS_PATH is that it
could be a sign that the operator of a network doesn't know what their
doing or doesn't have good processes in place (YMMV). If you have a path
twice in BGP, once with a "giant" path length and once with a "normal"
length the provider offering the "longer" path may have any manner of
problems internally if they can't even manage their BGP routing policies
appropriately.

I have seen genuine reasons for up to about 6 pre-prends and beyond so
you're probably dropping a decent amount of routes. At the time I set our
filter I think we were dropping one single route. No one has complained so
its still in place.

Giant AS_PATHs can be debated in a similar fashion to AS numbers
used/restricted by RFCs. We have AS number filters in place to block
prefixes with a private ASN in the path, any transit provider should block
such routes, again if they aren't it's a sign to me they're not doing a
really great job. But it's very hard to know if you can drop those routes
without affecting your customer base or that a suitable alternative exists.
Great care must be taken when doing this.

Otherwise the following issue arises:

On 21 Jun 2017 19:13, "Saku Ytti"  wrote:

Hey,

Uou're saying, you drop long AS_PATH, to improve customer observed
latency? Implication being, because you dropped the long AS_PATH
prefixes, you're now selecting shorter AS_PATH prefixes to the FIB?

Absent of this policy, in which scenario would you have inserted the
filtered longer AS_PATH prefix to the FIB? I assume only scenario
where this happens is where there is larger aggregate route, which is
lower latency than the more specific route with longer as_path.


So we drop "giant" AS_PATH length routes and routes with certain ASNs in
the path, but we are fairly certain we don't need them / our customers
don't need them etc. Not because we believe we are getting better (lower
latency routes) routes from somewhere else as Saku said above, we can't
feasibly "test" and compare the performance of every route we receive or
need/want, but to protect our infrastructure.

On 22 Jun 2017 16:54, "Jakob Heitz (jheitz)"  wrote:

23456 is AS_TRANS. Either your router does not support 4 byte AS or there
is a bug at AS 12956 or AS 12956 is intentionally prepending 23456.


This ties in with my point about specific ASN filtering. Its not a problem
seeing 23456 in your table but again perhaps an indicator of a problem or
someone using legacy kit. No problem with 23456 routes like AS_PATH length
of up to say 50 but beyond that, I can't thing of a genuine reasons and
could be a sign that along the path there may be stability issues for
example. Again, too difficult to measure.

Depending on your customer base and requirements it can be safe to drop
giant paths but care is needed, and if you do it, I think a number like 6
is way too low.

Cheers,
James.

Re: PCIe adapters supporting long distance 10GB fiber?

[James Bensley]

On 20 June 2017 at 17:10, Denys Fedoryshchenko  wrote:
> On 2017-06-20 18:59, Hunter Fuller wrote:
>>
>> On Tue, Jun 20, 2017 at 10:29 AM Chris Adams  wrote:
>>
>>> For Linux at least, the standard driver includes a load-time option to
>>> disable vendor check.  Just add "options ixgbe allow_unsupported_sfp=1"
>>> to your module config and it works just fine.
>>
>> For anyone who may be going down this road, if you have a two-port Intel
>> NIC, I discovered you have to pass "allow_unsupported_sfp=1,1" or it will
>> only apply to the first port. Hope that helps someone.
>
> Also it wont work with X710, you need to do NVRAM hack for it, SFP are
> checked in firmware.

We have third party SFPs in X710 "based" NIC. To be exact we some HPE
servers which have a "HPE Ethernet 10Gb 2-port 562SFP+ Adapter" which
uses an OEM X710 controller branded as HP. So it maybe that the HP SoC
on the NIC is allowing us to use any SFP. We haven't added any kernel
module load parameters to make that work or NVRAM hacks, we just
flashed the NIC to the lasted firmware version upon arrival as a
general good practice move and compiled the latest i40e & i40evf
drivers.

We have third party 10G SFP+'s (single more / 10Km / LC) working and
1G copper SFPs too, the ports are multi-rate. Again this may be
something special about the HP NIC and the X710 controller is ignorant
of the PHY <> MAC conversation or something.

Cheers,
James.

Re: Templating/automating configuration

[James Bensley]

On 7 June 2017 at 19:52, Brian Knight  wrote:
> The import process to the database runs directly on our rancid server, 
> reading the downloaded configs out of the appropriate directory within 
> rancid. Most of our gear is Cisco, so the ciscoconfparse module for Python 
> helps a lot with organizing and querying the config.  From there, the config 
> is parsed for key items like interface name, description, configured 
> bandwidth, etc., and that info is then added or updated as necessary in the 
> database.
>
>
>
> Because it's dependent on rancid, there is some lag time between when a 
> change is made and when the database gets updated, so we still strongly 
> encourage running the pre-config checks for new circuits.  But with PyEZ, it 
> looks like you easily could, after grabbing that lock, validate the existing 
> config before pushing down new config.  Lots of possibilities there.  I'm 
> envious that you have a vendor-written Python module as a choice!
...
>
> Or, at least, rebuild the existing configs based on the new source of truth, 
> so that subsequent config parsing conforms to a single standard.

Cisco IOS and IOS-XE have config locks on the CLI, as well as
automatic configuration rollbacks and the ability to generate a config
diff on deice. For some reason lots of people seem to forget/ignore
this.

If you are using NAPALM then I believe you can also implement this
through NAPALM.

Cheers,
James.

Re: Templating/automating configuration

[James Bensley]

On 7 June 2017 at 00:43, Vincent Bernat  wrote:
>  ❦  6 juin 2017 14:30 +0100, Oliver Elliott  :
>
>> I echo Ansible. I'm using it with NAPALM and jinja2 templates to push and
>> verify config on switches.
>
> Why not using the builtin ability of ansible for most vendors? (genuine
> question)
>
>  http://docs.ansible.com/ansible/list_of_network_modules.html

One reason, which is our reason for using NAPALM with Ansible, is that
the built in Ansible modules often just edit certain lines of config
in the target device. For example, the Cisco IOS module within Ansible
scans the device config for say the line starting with "Interface
Etherernet 1/1" and then I tell it to ensure the lines " ip vrf
customer A" and " ip address x.x.x.x n.n.n.n" are under the search
line. It's OK but its text matching and not fool proof. It also
doesn't help me to guarantee the state of our tin (I might push an
update to one interface on a device and simultaneously someone else
might pushes an update to a different interface, our respective views
of the device config might not include each other’s updates).

We use the NAPALM module although it needs to be a bit more than just
NAPALM, its not a panacea. We generate a full device config (even for
a one line interface update) and push that into atomic storage (git),
when then pass that file from git to NAPALM. NAPALM will copy the file
to the device and do a full config replace for us, and we can get a
diff from before and after that process and report that back and
ensure that exactly what we wanted to change has been changed only.
All changes come through git which act’s like a queue meaning that if
two people make simultaneous updates to different interfaces there’ll
be a git commit/push error. [1]

Cheers,
James.

[1] That’s the plan at least, the reality though is that vendor bugs
are plentiful.

Re: RFC2544 Testing Equipment

[James Bensley]

On 30 May 2017 at 16:41, James Harrison  wrote:
> On 30/05/17 16:22, Nick Olsen wrote:
>>  Looking to test up to 1Gb/s at various packet sizes, Measure Packet loss,
>> Jitter..etc. Primarily Copper, But if it had some form of optical port, I
>> wouldn't complain. Outputting a report that we can provide to the customer
>> would be useful, But isn't mandatory. Doesn't need anything fancy, Like
>> MPLS awareness, VLAN ID's..etc.

...
> if
> you have a VPLS setup then probably you'd go from a 1U box next to your
> VPLS box through the VPLS pipe through to the endpoint.

If you are using VPLS then you need to send 1Gbs of broadcast traffic
and see how that cripples your network and send 1Gbps of BPDUs and ARP
requests/responses etc. to see how that ruins everything, as your
customer will loop it at some point. Also to check how your PEs work
and if storm-control or similar is working.

We had an issue with a VPLS instance where different model edge PEs
had their core facing interfaces built in different ways; some had a
physical interface configured facing the core, some a sub-interface,
some an SVI/BVI/BDI etc, it turned out that device X won't tunnel PPP
packets over VPLS/pseudowires when the core facing interface is an
SVI, model Y will but not when using EVCs etc.

People usually test TCP/UDP over IPv4 which doesn't tell you much
about what your equipment/service can or can't do and how it will
fail.

Cheers,
James.

Re: RFC2544 Testing Equipment

[James Bensley]

On 31 May 2017 at 11:56, Saku Ytti  wrote:
> Cool. Seems you're using AF_PACKET, which makes it actually unique.
> iperf/netperf etc use UDP or TCP socket, so UDP performance is just
> abysmal, you can't saturate 1GE link with any reliability. So
> measuring for example packet loss is not possible at all.
>
> I've been meaning to write AF_PACKET based UDP sender/receiver and
> have gotten pretty far with friend of mine on rust version, we can
> congest 1GE (on minimum size frames) on Linux reliably and actually
> tell if you're lossy. It has server/client design, where client
> requests via JSON based messages through control-channel server to
> receive or send, and what exactly.
> Alas, we're only 80% there, and seem to struggle to find time to
> polish it for initial release.
>
> We definitely need tool like iperf, which performs at least to 1GE,
> and AF_PACKET can do that, UDP socket cannot. Alas 10GE is still pipe
> dream for anything as portable as iperf, as you'd need to use DPDK,
> netmap or equivalent which will remove the NIC from userland, there
> are quite few options for that use-case, but no good option for
> use-case when you want at least 1GE but you cannot remove NIC from
> userland.

Hi Saku,

Yeah AF_PACKET sockets are used and you really need to be on a 4.x
Kernel for better performance (update your NIC firmware etc). The
problem with Etherate is that is uses Ethernet for the test data and
control data and since Ethernet is loss-less is does some strange
(read: lame) things like send some control or data frames three times
to try and ensure the other side receives it when there is frame loss.

Yeah 1G with large frames is do-able. 10G with large frames is also
do-able with a fast CPU. Etherate is single threaded though so you’ll
not get anywhere near 10G with 64 byte frames in Etherate. I have
started writing a multi-threaded version which will use TCP sockets to
exchange control data but still use AF_PACKET sockets for data plane
traffic.

10G with 64 byte packets should be achievable (still writing it so not
100% confirmed yet) when using the PACKET_MMAP Tx/Rx rings in
AF_PACKET which is what the new aptly named EtherateMT
(multi-threaded) uses. One can then use multiple threads (each on a
difference CPU core) and each with its own Tx or Rx ring buffer to
push packets to the NIC and we can use RSS on the NIC and assign each
NIC Tx queue to a separate core also for processing NET_TX and NET_RX
IRQs.

So it might take 12 or 16 cores but it should be do-able in EtherateMT
still with the iperf like portability, whereas DPDK can do this on a
single core (pkt-gen and moon-gen etc). However EtherateMT would
ideally use only Kernel native features (no 3rd party libraries
required or custom Kernel complication to enable an optional modules).

Yeah Rust seems cool, it's on my "to-learn" list along with Go and
seven thousand over things so writing in C for now.

Cheers,
James.

Re: RFC2544 Testing Equipment

[James Bensley]

On 30 May 2017 at 16:22, Nick Olsen  wrote:
>  Greetings all,
>
>  Looking for a good test set. Primary use will be testing L2 circuits
> (It'll technically be VPLS, But the test set will just see L2). Being able
> to test routed L3 would also be useful. Most of the sets I've seen are two
> sided, A "reflector" at the remote side, And the test set in hand run by
> the technician.
>
>  Looking to test up to 1Gb/s at various packet sizes, Measure Packet loss,
> Jitter..etc. Primarily Copper, But if it had some form of optical port, I
> wouldn't complain. Outputting a report that we can provide to the customer
> would be useful, But isn't mandatory. Doesn't need anything fancy, Like
> MPLS awareness, VLAN ID's..etc.
>
>
>Nick Olsen
>  Sr. Network Engineer
>  Florida High Speed Internet
>  (321) 205-1100 x106


If you are just testing the forwarding at layer 2 and have no budget
you can use free software and a laptop (for your copper requirement).
I've been writing this (https://github.com/jwbensley/Etherate) and we
use that as well as hardware testers.

Cheers,
James.

Re: Linux container (Docker) or linux VM RFC2544/EtherSAM appliance

[James Bensley]

I think you can do this with either of Pktgen[1] or Moongen[2] which
both use DPDK.

Cheers,
James.

[1] http://dpdk.org/download
[2] https://github.com/emmericp/MoonGen

Re: BGP Route Reflector - Route Server, Router, etc

[James Bensley]

On 20 March 2017 at 11:03, Greg Hankins  wrote:
> On Mon, Mar 20, 2017 at 12:35:24PM +0200, Mark Tinka wrote:
>>On 14/Jan/17 00:39, Brandon Ewing wrote:
>>> Work is being done to allow RRs to compute metrics from the client's
>>> position in the IGP: See
>>> https://tools.ietf.org/html/draft-ietf-idr-bgp-optimal-route-reflection-13
>>> for more information
>>
>>BGP-ORR is currently supported in Junos and IOS XR (ASR9000, I
>>believe... I haven't confirmed for other IOS XR platforms).
>>
>>I feel, is one of those features that doesn't need a business case -
>>much like ketchup at a fast-food joint.
>
> Mark is spot on, this is an important point.  We just added ORR to SR OS
> 15.0.R1 on the 7x50/VSR.

Yes, which means we all know what we have do to. Everyone needs to
join in to increase the pressure.

Cheers,
James.

Re: RNP Santa Catarina Point of Presence (PoP-SC)

[James Bensley]

On 9 March 2017 at 08:24, James Bensley <jwbens...@gmail.com> wrote:
> Hi All,
>
> Anyone from RNP on-list that can reach out to me off-list, minor
> technical problem in your network.
>
> Thanks,
> James.


Thanks all, contact has been made off-list!

Cheers,
James.

RNP Santa Catarina Point of Presence (PoP-SC)

[James Bensley]

Hi All,

Anyone from RNP on-list that can reach out to me off-list, minor
technical problem in your network.

Thanks,
James.

Re: BGP Route Reflector - Route Server, Router, etc

[James Bensley]

On 13 January 2017 at 04:02, Hugo Slabbert <h...@slabnet.com> wrote:
>
> On Thu 2017-Jan-12 22:59:21 +, James Bensley <jwbens...@gmail.com>
> wrote:
>
>> On 12 January 2017 at 20:32, Justin Krejci <jkre...@usinternet.com> wrote:
>>>
>>> . I have not found many resources discussing using a non-router box as a
>>> route reflector (ie a device not necessarily in the forwarding path of the
>>> through traffic). I am thinking things like OpenBGPd and BIRD could make a
>>> good route reflector though they are most often discussed in the context of
>>> IXPs (ie eBGP sessions).
>>
>>
>> The CSR1000v (IOS-XE),IOS-XRv and vMX are production ready. People are
>> deploying these in production and its increasing in popularity.
>
>
> Any thoughts on vRR vs. vMX for this use case?  I see Mark called out vRR as
> having morphed into vMX, but AFAIK vRR is just vMX minus the forwarding
> plane, is targeted as an out-of-path reflector, and coexists with vMX as a
> different deployment option rather than having been replaced by it.  I would
> assume that vRR should come in a few bucks lower than the vMX as a result,
> but I've only previously gotten quotes on vRR not vMX.
>
>
>> Mark Tinka gave a good preso at a recent Nanog:
>>
>> https://www.nanog.org/sites/default/files/2_Tinka_21st_Century_iBGP_Route_Reflection.pdf
>>
>> https://www.youtube.com/watch?v=wLEjOj2fyp8=PLO8DR5ZGla8hcpeEDSBNPE5OrZf70iXZg=21
>>
>> Cheers,
>> James.
>
>
> --
> Hugo Slabbert   | email, xmpp/jabber: h...@slabnet.com
> pgp key: B178313E   | also on Signal


Sorry I don't know about the pricing, but the newer vMX product is now
split into two VMs, the virtual control plane and virtual forwarding
plane. I think the vRR product is still like the "older" style vMX
which was one combined control and forwarding plane image. At a guess,
perhaps its heavy throughput limited?

We have used the "older" style vMX images in the lab (14.something)
which is the combined all in one VM, it works fine for us for actual
network traffic testing as well as various BGP tests like router
reflectors so I see know reason why it wouldn't work as a vRR. I think
the actual "vRR" product from Juniper is just a more light weight VM,
perhaps someone can clarify the tech behind it?

We don't have any virtual RRs in production yet but we are running
CSR1000v in lab tests right now which is working fine for us so we'll
probably push that out to prod at some point in both scenarios (as an
in path virtual router and out of path virtual route-reflector) but
that is 12+ months away as we still have lots more testing to do.

Cheers,
James.

Re: BGP Route Reflector - Route Server, Router, etc

[James Bensley]

On 12 January 2017 at 20:32, Justin Krejci  wrote:
> . I have not found many resources discussing using a non-router box as a 
> route reflector (ie a device not necessarily in the forwarding path of the 
> through traffic). I am thinking things like OpenBGPd and BIRD could make a 
> good route reflector though they are most often discussed in the context of 
> IXPs (ie eBGP sessions).

The CSR1000v (IOS-XE),IOS-XRv and vMX are production ready. People are
deploying these in production and its increasing in popularity.

Mark Tinka gave a good preso at a recent Nanog:
https://www.nanog.org/sites/default/files/2_Tinka_21st_Century_iBGP_Route_Reflection.pdf
https://www.youtube.com/watch?v=wLEjOj2fyp8=PLO8DR5ZGla8hcpeEDSBNPE5OrZf70iXZg=21

Cheers,
James.

Re: BRAS/BNG Suggestion

[James Bensley]

On 2 December 2016 at 10:37, t...@pelican.org  wrote:

> On Friday, 2 December, 2016 05:55, "Mark Tinka" 
> said:
>
> > Redback used to be popular - I believe they got picked up by Ericsson.
>
> I'd steer clear at a small scale like 20k subscribers.  In my experience,
> Ericsson as an organisation just aren't set up to deal with a company that
> want to buy a couple of boxes, install and run them themselves, and call
> support when something breaks - it's all about multi-month PoCs and
> fully-managed rollouts at incumbent-telco scale.  Think 20M subs, not 20k.
>
>
I've heard Redback aren't so good anymore. We are very happy with Cisco
(ASR1000s) and Juniper (MX480s) for this function.

Cheers,
James.

Re: OSPF vs ISIS - Which do you prefer & why?

[James Bensley]

On 10 November 2016 at 05:59, Mark Tinka  wrote:
>
>
> On 9/Nov/16 19:12, Michael Bullut wrote:
>
>> Greetings Team,
>>
>> While I haven't worked with IS-IS before but the only disadvantage I've
>> encountered with OSPF is that it is resource intensive on the router it is
>> running on which is why only one instance runs on any PE & P device on an
>> ISP network. OSPF is pretty good in handling the core network routing while
>> BGP & EGP handle the last-mile routing between PE & CE devices. BGP & EGP
>> can run on top of OSPF. I came across this *article*
>> 
>> when
>> scrolling the web a while back and I still want to find out if am the only
>> one who thinks its a matter of choice between the two. Although there isn't
>> distinct 1:1 argument, it's good we discuss it here and figure out why one
>> prefer one over the other *(consider a huge flat network)**.* What say you
>> ladies and gentlemen?
>
> I've given a talk about this a couple of times since 2008. But our
> reasons are to choosing IS-IS are:

I don't think there is much of a debate to be had any more, the gap
between them is so small now (OSPFv3 and ISIS that is, no one would
deploy OSPFv2 now in greenfield right?):

>   * No requirement to home everything back to Area 0 (Virtual Links are
> evil).

This is a good point I think.

>   * Integrated IPv4/IPv6 protocol support in a single IGP implementation.

This is in OSPv3.

>   * Single level (L2) deployment at scale.

Single area 0 deployment at scale? Bit of a moot point unless you
compare a specific device model and specific code version in two
identical deployments, its not much to do with the protocol but the
vendor implementation and the brute force.

>   * Scalable TLV structure vs. Options structure for OSPFv2. OSPFv3
> employs a TLV structure, however.

OSPv3 has this.

>   * Inherent scaling features, e.g., iSPF, PRC, e.t.c. Some of these may
> not be available on all vendor implementations.

OSPF has these too.

> Ultimately, router CPU's are way faster now, and I could see a case for
> running a single-area OSPFv2. So I'd likely not be religious about
> forcing you down the IS-IS path.

Yeah this ^ I don't think there is a stronge case for either protocol.

Somenoe mentioned the AOL NANOG talk about migrating from OSPF to
ISIS. There was a NANOG talk recently-ish about someone migrating from
OSPF to BGP. There wasn't even a need for an IGP, BGP scalled better
for them (in the DC).

BGP these days supports PIC and BFD etc, how much longer to IGPs have? :)

James.

Re: Large BGP Communities beacon in the wild

[James Bensley]

On 27 October 2016 at 16:47, Owen DeLong  wrote:
> I don’t mind the move to 32, but I hope the vendors are getting appropriately 
> smacked for squatting and that those attributes are not allowed to be 
> misappropriated by the vendors.
>
> We have a standards process for a reason and vendors simply squatting on 
> numbers is a violation of that process which cannot be allowed to stand 
> unless we wish to establish that as precedent and simply allow vendors to 
> claim numbers as they wish.
>
> This already happened with the BSD community in their implementation of a 
> pseudo-VRRP like capability and now two different vendors have abused BGP 
> path attributes.
>
> This is not a good path for us to continue.
>
> Owen

Here here!

Name and shame, it is not acceptable!


James.

Re: Host.us DDOS attack -and- related conversations

[James Bensley]

On 3 August 2016 at 15:16, Alain Hebert  wrote:
> PS:
>
> I will like to take this time to underline the lack of
> participation from a vast majority of ISPs into BCP38 and the like.  We
> need to keep educating them at every occasion we have.
>
> For those that actually implemented some sort of tech against
> it, you are a beacon of hope in what is a ridiculous situation that has
> been happening for more than 15 years.


At the risk of starting a "NANOG war" [1], BCP isn't a magic wand.

If I find a zero day in the nasty customised kernels that OVH run on
their clients boxes, I only need 300 compromised hosts to send 300Gbps
of traffic without spoofing the IP or using amplification attacks [2].

I can rent a server with a 10Gbps connection for 1 hour for a few
quid/dollars. I could generate hundreds of Gbps of traffic for about
£1000 from legitimate IPs, paid for with stolen card details. How will
BCP save you then? Can everyone stop praising it like it was a some
magic bullet?

James.


[1] A pathetic and futile one, so different from the rest.

[2] Subsitute OVH for any half decent provider that isn't really oversubscribed.

Re: IX in Iran by TIC

[James Bensley]

On 12 July 2016 at 14:36, Bevan Slattery  wrote:
> EXAMPLE 1.
> There maybe for example an enterprise  that is looking for a service
> provider in a facility (XYZ in NY for example) but that provider actually
> "peers" their transit routers at the ABC facility down the street.  Because
> the provider doesn't peer in XYZ there is no public record of them being
> there in peering DB.  Providers are in heaps of DC's/locations that they
> just don't peer.  So they effectively have no central location where people
> can see that they are "available to service".  This is more of a directory
> of where providers are and what services they can provide.

Hmm, so maybe I'm just a maverick, we are not using any public peering
fabrics at minute due to what can only be described as a senior
management cluster foook [1], so on peeringdb we list some pops that we are
in that we are willing (and do) have private peering sessions in. It
doesn't say on peeringdb that there are IX's in some of these PoPs but
hopefully when we need to establish a private interconnect with
someone they will see we are in the same PoP as them even though there
is no IX in that PoP and put 2 and 2 together, and contact us to
discuss a cross connect.

For the avoidance of doubt, I'm not trying to poo poo the site, just
trying to work out where the different is feature set lies exactly.


Cheers,
James.


[1] Is this a list for adults or children, my original email bounced
back because I used the work f*ck?

Re: IX in Iran by TIC

[James Bensley]

On 12 July 2016 at 14:36, Bevan Slattery  wrote:
> EXAMPLE 1.
> There maybe for example an enterprise  that is looking for a service
> provider in a facility (XYZ in NY for example) but that provider actually
> "peers" their transit routers at the ABC facility down the street.  Because
> the provider doesn't peer in XYZ there is no public record of them being
> there in peering DB.  Providers are in heaps of DC's/locations that they
> just don't peer.  So they effectively have no central location where people
> can see that they are "available to service".  This is more of a directory
> of where providers are and what services they can provide.

Hmm, so maybe I'm just a maverick, we are not using any public peering
fabrics at minute due to what can only be described as a senior
management cluster fuck, so on peeringdb we list some pops that we are
in that we are willing (and do) have private peering sessions in. It
doesn't say on peeringdb that there are IX's in some of these PoPs but
hopefully when we need to establish a private interconnect with
someone they will see we are in the same PoP as them even though there
is no IX in that PoP and put 2 and 2 together, and contact us to
discuss a cross connect.

For the avoidance of doubt, I'm not trying to poo poo the site, just
trying to work out where the different is feature set lies exactly.


Cheers,
James.

Re: IX in Iran by TIC

[James Bensley]

On 12 July 2016 at 13:46, Bevan Slattery  wrote:
> Great work.  Might be worthwhile to also look at throwing your fabric/IX on
> Cloud Scene www.cloudscene.com .  Provides visibility for people looking
> for DC's, providers and fabrics that just aren't limited to IX locations or
> peers.
>
> Cheers

That's a nifty site but isn't it largely overlapping with peeringdb
which is already more established?

Just my two pence.

James.

Re: Real world power consumption of a 7604-S or 7606-S

[James Bensley]

On 28 June 2016 at 01:26, Tom Hill  wrote:
> On 28/06/16 00:26, Eric Kuhnke wrote:
>> Example:
>> 7604S chassis with dual 2700W DC power - chassis and fans use how much
>> power?
>> 2 x RSP720-3CXL at 310W each
>> WS-X6704 with DFC4 - ???W each
>
> Way too much, is the simple answer.
>
> I did have a 7604 (non-S) with the same PSUs, 1x SUP720-3BXL, 1x
> WS-X6724-SFP and 1x WS-X6708-3CXL was drawing near 2kW.
>
> It's not healthy, please consider how much you'll spend in electricity
> vs. something else. For example, the ASR9001 uses a 5th of the power.


+1 for the asr9001 route.

For the record though see below, I've tried to supply you figures from
different model 7600s but we mostly hvae 7606-S's. These are all
single SUP/RSPsystems.

Cheers,
James.


7606-S#show power
system power redundancy mode = redundant
system power total = 2669.10 Watts (63.55 Amps @ 42V)
system power used =  1620.78 Watts (38.59 Amps @ 42V)
system power available = 1048.32 Watts (24.96 Amps @ 42V)
Power-Capacity PS-Fan Output Oper
PS   Type   Watts   A @42V Status Status State
 -- --- -- -- -- -
1PWR-2700-AC2669.10 63.55  OK OK on
2PWR-2700-AC2669.10 63.55  OK OK on
Pwr-Allocated  Oper
Fan  Type   Watts   A @42V State
 -- --- -- -
1FAN-MOD-6SHS180.18  4.29  OK
Pwr-Requested  Pwr-Allocated  Admin Oper
Slot Card-Type  Watts   A @42V Watts   A @42V State State
 -- --- -- --- -- - -
1WS-X6748-SFP322.14  7.67   322.14  7.67  onon
2WS-X6704-10GE   362.46  8.63   362.46  8.63  onon
5RSP720-3CXL-10GE378.00  9.00   378.00  9.00  onon
6(Redundant Sup)   - -  378.00  9.00  - -


7606-S#show power
system power redundancy mode = redundant
system power total = 2669.10 Watts (63.55 Amps @ 42V)
system power used =  2137.38 Watts (50.89 Amps @ 42V)
system power available =  531.72 Watts (12.66 Amps @ 42V)
Power-Capacity PS-Fan Output Oper
PS   Type   Watts   A @42V Status Status State
 -- --- -- -- -- -
1PWR-2700-AC2669.10 63.55  OK OK on
2PWR-2700-AC2669.10 63.55  OK OK on
Pwr-Allocated  Oper
Fan  Type   Watts   A @42V State
 -- --- -- -
1FAN-MOD-6SHS180.18  4.29  OK
Pwr-Requested  Pwr-Allocated  Admin Oper
Slot Card-Type  Watts   A @42V Watts   A @42V State State
 -- --- -- --- -- - -
1WS-X6748-GE-TX  325.50  7.75   325.50  7.75  onon
2WS-X6748-SFP254.94  6.07   254.94  6.07  onon
3WS-X6704-10GE   295.26  7.03   295.26  7.03  onon
4WS-X6748-GE-TX  325.50  7.75   325.50  7.75  onon
5(Redundant Sup)   - -  378.00  9.00  - -
6RSP720-3CXL-10GE378.00  9.00   378.00  9.00  onon




7604(non -S)#show power
system power redundancy mode = redundant
system power total = 2669.10 Watts (63.55 Amps @ 42V)
system power used =   760.20 Watts (18.10 Amps @ 42V)
system power available = 1908.90 Watts (45.45 Amps @ 42V)
Power-Capacity PS-Fan Output Oper
PS   Type   Watts   A @42V Status Status State
 -- --- -- -- -- -
1PWR-2700-AC/4  2669.10 63.55  OK OK on
2PWR-2700-AC/4  2669.10 63.55  OK OK on
Pwr-Allocated  Oper
Fan  Type   Watts   A @42V State
 -- --- -- -
1FAN-MOD-4HS  57.54  1.37  OK

Pwr-Requested  Pwr-Allocated  Admin Oper
Slot Card-Type  Watts   A @42V Watts   A @42V State State
 -- --- -- --- -- - -
1WS-SUP720-3B282.24  6.72   282.24  6.72  onon
2WS-X6704-10GE   295.26  7.03   295.26  7.03  onon
3WS-X6548-GE-TX  125.16  2.98   125.16  2.98  onon

Re: Network traffic simulator

[James Bensley]

On 24 May 2016 at 13:17, Mitchell Lewis  wrote:
> Hi,I am looking to validate the performance specs of a core router. I am 
> looking for a network traffic simulator which can simulate 40 gbps of 
> traffic. I am looking for a simulator with sfp+ ports.
> I am interested in any input as to brands to look at, build one myself etc.
> Thanks,Mitchell

COTS hardware is cheap enough, TRex should do what you want:

http://trex-tgn.cisco.com/

Cheers,
James.

Re: Question on peering strategies

[James Bensley]

>> On 22 May 2016, at 07:33, Max Tulyev  wrote:
>>
>> Hi All,
>>
>> I wonder why a "VLAN exchange" does not exists. Or I do not know any?
>>
>> In my understanding it should be a switch, and people connected can
>> easily order a private VLAN between each other (or to private group)
>> through some kind of web interface.
>>
>> That should be a more easy and much less expensive way for private
>> interconnects than direct wires.

On 23 May 2016 at 09:59, Marty Strong via NANOG  wrote:
> This does exist, often called an elastic fabric, e.g. Megaport
>
> Regards,
> Marty Strong
> --
> CloudFlare - AS13335
> Network Engineer
> ma...@cloudflare.com
> +44 7584 906 055
> smartflare (Skype)
>
> http://www.peeringdb.com/view.php?asn=13335


As Marty said, it does exist. AN example from LONAP in the UK:
https://www.lonap.net/fees.php

Private VLANs between members = FREE

Another option is using a provider like IXReach (now "Console"), take
a peering to them, and then down multiple VLANs they can through you
peerings to different IXs from around the world and to other networks:
http://www.ixreach.com/

Cheers,
James.

Re: Network Weathermap

[James Bensley]

On 28 April 2016 at 20:33, Peter Phaal  wrote:
> Many drawing tools support SVG as a file export format. Exporting or
> converting the map to SVG format allows the map attributes (link
> colors, widths, etc) to be modulated using JavaScript embedded in the
> web page.
>
> As an example, the following SC15 weathermap was created by converting
> a PDF diagram of the network into an SVG file:
>
> http://blog.sflow.com/2015/11/sc15-live-real-time-weathermap.html
>
> The code is on GitHub and it wouldn't be hard to re-purpose:
>
> https://github.com/pphaal/sc15-weather
>
> The ESnet weathermap is very cool and they have open sourced the code:
>
> https://my.es.net/
> http://www.hpcwire.com/2015/10/05/esnet-releases-software-for-building-interactive-network-portals/

https://github.com/esnet/react-timeseries-charts/
https://github.com/esnet/react-network-diagrams/

Fwoor!

Sorry Weathermap, you've been usurped! I think I'll write a wrapper in
PHP to pull the data from RRDs and feed into that.

Cheers,
James.

Re: Network Weathermap

[James Bensley]

On 28 April 2016 at 19:41, Ishmael Rufus  wrote:
> You could probably build the converter in PHP and make it a plugin of
> weathermap.
>
> You kids and your Python :)

I would prefer it to be PHP actually, people keep moaning at me for
using PHP, which I am much more fluent in. However if it were in PHP
it comes with the advantage of being in the same language the the rest
 of Cacti (more or less).

Cheers,
James.

Re: Network Weathermap

[James Bensley]

Hi all,

I know its been a while since I posted this thread, I've been swamped.
Finally I'm getting time to look back at this. I think I had 0 on-list
replies and about 10 off-list private replies, so clearly others are having
the same problem but not speaking openly about it.

There were two main themes in the off list replies;

1. Several people are drawing in a tool like Visio and then importing the
picture as a background to the weathermap plugin and adding the links and
nodes over the top.

2. A couple of people were drawing in something else other than Visio that
would spit out files containing objects and coordinates and then had
written scripts to convert those coordinates to Weathermap plugin file
format.

Method 1 is OK, I really want it to be less hassle than that so 2 seems
like the best idea. Only one person would share their conversion script
with me briefly on PasteBin then it expired and it wasn't for Visio format
files, so I didn't save it.

Having a quick play in Visio just now the files are saved as XML formatted
X/Y axis values. Bit of a Python novice but I'm thinking I could basically
ingest a Visio file and parse the the XML and then iterate over it
converting each "object" into weathermap syntax.

That isn't too difficult however for the maps to be any good I need to
think about the "via" feature for links in Weathermap to map them  more
clearly if they cross over each other. There might still also be a lot of
hackery when it comes to mapping the imported nodes and links to actual
ones in Cacti. It might be that you have to match all the imported nodes
and links to RRDs the first time you import the diagram then on all future
imports just new links and nodes.

Before I commit the time to this, has anyone done this already or is anyone
a absolute Lord of Python who wants to do it quicker than I can do it? :)

Cheers,
James.

Re: NTT communications horrible routing, unresponsive NOC

[James Bensley]

On 23 March 2016 at 23:39, Paras Jha  wrote:
> At a loss as to what to do now, since their NOC isn't receptive. Anyone
> have someone I can contact off-list to get this issue resolved? It's
> especially frustrating because the problem absolutely cannot be resolved on
> our end, even with a no-export since NTT is keeping the routes in their
> router.

It would be helpful to show how you came to these conclusions about
prefix caching and routing issues etc (share the data you gathered).

If you are an NTT customer (it's sounds a bit like you're a customer
of an NTT customer, it's not clear) then this should be account
management 101, you should have an escalations process/matrix, kick
off that process now.

There are NTT dudes on this list so probably they will reach out to
you off list, if not post back requesting an NTT contact.

James.

Re: AWS Direct Connect - Peering VPCs to Tier 1's and MPLS

[James Bensley]

On 1 March 2016 at 20:41, Michael O'Connor  wrote:
> Jay,
>
> VPC is supported over IPsec if your public path is sufficient into the AWS
> cloud.

^ This.

I work for a DirectConnect provider, albeit in the UK though. We have
fibre links to a AWS edge routers and we have multiple customers
seperated by VLANs over a fibre link, each terminating into different
VRFs on our edge and the AWS edge. For each customer we have an eBGP
session with a virtual gateway that lives inside the customer's VPC
domain.

Also for each customer they have backup tunnels using IPSec over the
Internet. Again we run eBGP over the IPSec tunnels to the virtual
gateway inside each customers VPC domain.

"just works".

James.

SunGard On List?

[James Bensley]

Hi All,

Any SunGard on list?

Having a path issue from multiple ISPs in the UK.

Cheers,
James.

Re: IX ARP Timeout

[James Bensley]

How often does your peering router change IP address?

For the majority of people I would expect the answer to be almost
nevery/very rarely.

James.

Re: interconnection costs

[James Bensley]

On 23 Dec 2015 20:06, "Reza Motamedi"  wrote:
>
> All the costs of HW, SW, personnel, administration, and perhaps
transmission between colos (including remote peering, being waved to
another location, tethering) would be the same, right?

Usually yes but with transit you are paying for global
connectivity/reachability, for an extreme comparison, I only need to
exchange 50mbps of data with Facebook before they will peering with me
(last time I checked), I'm not going to put a 100Mbps link into a port that
can do 10G. So a direct transit feed is more commercially viable for me as
a consumer than private or public peering unless a particular business
demand can cover the cost risk of peering.

James.

Re: interconnection costs

[James Bensley]

On 22 December 2015 at 19:11, Reza Motamedi <motam...@cs.uoregon.edu> wrote:
> Thanks guys for the replies.
>
> I wanted to clarify two things in my questions. First by peering I did not
> necessarily mean "settlement free" interconnection. I meant any inter-AS
> connection. My understanding is that in addition to the cost of transit that
> should be paid to the transit provider, there also exists the cost of the
> xconnect that is charged by the colocation provider. Secondly, my question
> was more about the expenses, as opposed to the technical costs/benefits. I
> have browsed through the "Peering Playbook", but I think its more about
> providing a case "settlement free" peering.

Dude, how are you going to weigh up the costs and benefits of peering
if you don't include the "costs". I refer you back to the same
documented I referred you to yesterday...

> On Tue, Dec 22, 2015 at 9:33 AM, James Bensley <jwbens...@gmail.com> wrote:
>> https://docs.google.com/document/d/1i2bPZDt75hAwcR4iKMqaNSGIeM-nJSWLZ6SLTTnuXNs/edit?pref=2=1#


This time look at section 4 of this huge and hard to navigate
document, "4. Peering Costs":

https://docs.google.com/document/d/1i2bPZDt75hAwcR4iKMqaNSGIeM-nJSWLZ6SLTTnuXNs/edit#heading=h.nqqauszv8vj


Loosely extrapolating:

Network transport: You need to be physically connected unless you
blagged space in the same rack and can patch in for free.

Hardware: You need tin to route packets.

Software: You need software to monitor the packet routing.

Colocation: You need space/power/cooler/security in which the tin can operate.

Staffing costs: Someone has to configure that tin.

Admin/engineering overhead: Someone has to manage the peering process

Peering port: You probably have to pay to peer.

Reseller ports: You might need remote connectivity to the LAN and
"network transport" above would refer to a cross connect to the remote
peering provider instead of directly into the IXP LAN.



James.

Re: interconnection costs

[James Bensley]

On 22 December 2015 at 16:44, Reza Motamedi  wrote:
> I think there is no single answer as different businesses may have
> different pricing models. I hope the discussion can help me understand the
> whole ecosystem a little bit better.


Hi Reza,

I have a list of example items that need to be costed in below, it is
by no means a definitive list though:

https://docs.google.com/document/d/1i2bPZDt75hAwcR4iKMqaNSGIeM-nJSWLZ6SLTTnuXNs/edit?pref=2=1#


Cheers,
James.

Re: EoMPLS vlan rewrite between brands; possibly new bug in Cisco IOS 15

[James Bensley]

On 15 November 2015 at 01:31, Jonas Bjork  wrote:
> Dear Mr. Jeff,
>
> Thank you for your reply. Below is the complete output in question (l2 is 
> short for l2transport).
> You are mentioning platform capabilities and that the default might have 
> changed. How do I alter this?
>
> pe#sh mpls l2 vc 42 d
> Local interface: Po190.42 up, line protocol up, Eth VLAN 42 up
>   Destination address: X.X.1.89, VC ID: 42, VC status: down
> Last error: Imposition VLAN rewrite capability mismatch with peer
> Output interface: none, imposed label stack {}
> Preferred path: not configured
> Default path: no route
> No adjacency
>   Create time: 00:00:59, last status change time: 00:31:40
> Last label FSM state change time: 00:00:18
> Last peer autosense occurred at: 00:00:18
>   Signaling protocol: LDP, peer X.X.1.89:0 up
> Targeted Hello: X.X.0.2(LDP Id) -> X.X.1.89, LDP is UP
> Graceful restart: not configured and not enabled
> Non stop routing: not configured and not enabled
> Status TLV support (local/remote)   : enabled/not supported
>   LDP route watch   : enabled
>   Label/status state machine: remote invalid, LruRnd
>   Last local dataplane   status rcvd: No fault
>   Last BFD dataplane status rcvd: Not sent
>   Last BFD peer monitor  status rcvd: No fault
>   Last local AC  circuit status rcvd: No fault
>   Last local AC  circuit status sent: DOWN PW(rx/tx faults)
>   Last local PW i/f circ status rcvd: No fault
>   Last local LDP TLV status sent: No fault
>   Last remote LDP TLVstatus rcvd: Not sent
>   Last remote LDP ADJstatus rcvd: No fault
> MPLS VC labels: local 242, remote 1199
> Group ID: local 0, remote 0
> MTU: local 9216, remote 9216
> Remote interface description:
> Remote VLAN id: 42
>   Sequencing: receive disabled, send disabled
>   Control Word: Off (configured: autosense)
>   SSO Descriptor: X.X.1.89/42, local label: 242
>   Dataplane:
> SSM segment/switch IDs: 0/0 (used), PWID: 142
>   VC statistics:
> transit packet totals: receive 0, send 0
> transit byte totals:   receive 0, send 0
> transit packet drops:  receive 0, seq error 0, send 0
> pe#
>
> Anyone else: feel free to join in. Maybe we have any L2VC/PW ninjas watching.
>
> Best regards,
> Jonas Bjork

Hi Jonas,

In that output you have "Remote VLAN id: 42" -What is the local VLAN
ID on your Cisco PE? Do you need to VLAN rewrite here?

Since you using different VLANs at each end, can you build the
pseudowire at a point in the network stack where the VLAN tag has been
popped off already and transport the frames untagged, so they will be
pushed on again at the other end? (Is this is a VC type 4 pseudowire,
check with "show mpls l2transport binding 42", if so, a dummy VLAN
should be pushed on and popped off transparently if all hardware in
use supports it).

I don't know HP but with the Cisco 7600 for example, if it's VLAN 50
then you could add "interface vlan 50; xconnecy X.X.1.89 42 encaps
mpls", if your hardware supports that. Or use mux-uni; "int gix/x.y;
encaps dot1q y; xconnecy X.X.1.89 42 encaps mpls". Then add vice versa
on the HP kit.

What IOS have you tried to upgrade to, 15.2(4)S4a? If this is a VC
type 4 pseudowire and either the HP or Cisco isn't supporting
inserting a dummy VLAN tag, why is this a VC type 4 pseudowire? The
VLAN re-write I guess. Certainly in IOS 15.3 (so probably also in 15.2
but I'm not 100% certain of that) Cisco IOS should be defaulting to VC
type 5 unless the other end negotiates down to VC type 4. VC type 5 in
IOS supports transporting of untagged, tagged and double tagged
frames, I don't think there is any functionality in VC type 5 that
wasn't in older type 4's so I'd try and work out why your devices are
negotiating type 4 and maybe try to fix that, or as above, transport
untagged.

Cheers,
James.

Re: Long-haul 100Mbps EPL circuit throughput issue

[James Bensley]

On 5 Nov 2015 21:50, "Eric Dugas"  wrote:
>
> Hello NANOG,
>
> We've been dealing with an interesting throughput issue with one of our
> carrier. Specs and topology:
>
> 100Mbps EPL, fiber from a national carrier. We do MPLS to the CPE
providing
> a VRF circuit to our customer back to our data center through our MPLS
> network. Circuit has 75 ms of latency since it's around 5000km.
>
> Linux test machine in customer's VRF <-> SRX100 <-> Carrier CPE (Cisco
> 2960G) <-> Carrier's MPLS network <-> NNI - MX80 <-> Our MPLS network <->
> Terminating edge - MX80 <-> Distribution switch - EX3300 <-> Linux test
> machine in customer's VRF
>
> We can full the link in UDP traffic with iperf but with TCP, we can reach
> 80-90% and then the traffic drops to 50% and slowly increase up to 90%.
>
> Any one have dealt with this kind of problem in the past? We've tested by
> forcing ports to 100-FD at both ends, policing the circuit on our side,
> called the carrier and escalated to L2/L3 support. They tried to also
> police the circuit but as far as I know, they didn't modify anything else.
> I've told our support to make them look for underrun errors on their Cisco
> switch and they can see some. They're pretty much in the same boat as us
> and they're not sure where to look at.
>
> Thanks
> Eric

Hi Eric,

Sounds like a TCP problem off the top of my head, however just throwing it
out there, we use a mix of wholesale access circuit providers and carriers
for locations we haven't PoP'ed and we are an LLU provider (CLEC in US
terms). For such issues I have been developing an app to test below TCP/UDP
and for pseudowires testing etc:

https://github.com/jwbensley/Etherate

It may or may not shed some light when you have an underlying problem
(although yours sounds TCP related).

Cheers,
James.