Anyone seeing ping corruption?

[Deepak Jain]

Haven't seen this before. This is a Nexus 9K as a testing platform. Getting 
sporadic complaints about data transfers aborting, but data moves well through 
the platform.
Hop 13 doesn't respond to our 1400 byte ping, hop 12 does a normal response, 
Google's 14 corrupts the packet or maybe deliberately manipulates it? 1.1.1.1 
doesn't do that.

Any insights welcome.

Thanks,

trace 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 40 byte packets
[deleted 3 hops]
4  154.24.71.85 (154.24.71.85) (AS 7922)  1.199 ms 154.24.32.5 (154.24.32.5) 
(AS 7922)  1.32 ms 154.24.32.1 (154.24.32.1) (AS 7922)  3.162 ms
5  154.54.81.173 (154.54.81.173) (AS 7922)  1.564 ms 154.54.81.141 
(154.54.81.141) (AS 7922)  1.691 ms 154.54.81.221 (154.54.81.221) (AS 7922)  
1.616 ms
6  154.54.40.109 (154.54.40.109) (AS 7922)  7.041 ms 154.54.40.105 
(154.54.40.105) (AS 7922)  8.035 ms 154.54.40.109 (154.54.40.109) (AS 7922)  
7.002 ms
7  154.54.47.218 (154.54.47.218) (AS 7922)  7.183 ms  8.595 ms  7.186 ms
8  154.54.12.18 (154.54.12.18) (AS 7922)  7.55 ms  16.791 ms  16.32 ms
9  * 66.110.96.62 (66.110.96.62) (AS 7922)  16.9 ms  8.202 ms
  [Label=24004 E=0 TTL=1 S=1]
10  66.110.96.58 (66.110.96.58) (AS 7922)  18.65 ms  15.823 ms  11.381 ms
11  72.14.195.232 (72.14.195.232) (AS 7922)  8.473 ms  8.128 ms  8.241 ms
12  108.170.248.33 (108.170.248.33) (AS 7922)  8.771 ms 108.170.248.1 
(108.170.248.1) (AS 7922)  9.004 ms  9.223 ms
13  142.251.60.237 (142.251.60.237) (AS 7922)  8.138 ms 142.251.65.95 
(142.251.65.95) (AS 7922)  7.768 ms 142.251.64.7 (142.251.64.7) (AS 7922)  
8.171 ms
14  8.8.8.8 (8.8.8.8) (AS 7922)  8.271 ms  14.179 ms  8.185 ms

ping
Vrf context to use [default] :
Target IP address or Hostname: 8.8.8.8
Repeat count [5] :
Packet-size [56] : 1400
Timeout in seconds [2] :
Sending interval in seconds [0] :
Extended commands [no] : yes
Interface :
Source address or interface :
Data pattern [0xabcd] :
Type of service [0] :
Set DF bit in IP header [no] :
Time to live [255] :
Loose/Strict, Record, Timestamp, Verbose [None] :
Sweep range of sizes [no] :
Sending 5, 1400-bytes ICMP Echos to 8.8.8.8
Timeout is 2 seconds, data pattern is 0xABCD

76 bytes from 8.8.8.8: icmp_seq=0 ttl=111 time=8.732 ms
wrong data byte #68 should be 0xcd but was 0x0
37 2d b2 61 af 2f 4 0 cd ab 0 0 cd ab 0 0 cd ab 0 0 cd ab 0 0 cd ab 0 0 
cd ab 0 0
cd ab 0 0 cd ab 0 0 cd ab 0 0 cd ab 0 0 cd ab 0 0 cd ab 0 0 cd ab 0 0 
cd ab 0 0
cd ab 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 

RE: AWS S3 DNS load balancer

[Deepak Jain]

You can't use DNS to get "all" service IP's of a service like S3 or a CDN for 
traffic engineering purposes. That will not work, ever (for services of such 
scale).

The hackery is assuming you can build a list of service IP's by querying DNS.

> There are a lot of reasons why someone may want this… particularly to 
> manage *other* people geo-basing their transport, but is this a local 
> hack or is this a feature of one of the major auth-DNS packages.
> If its local hackery, trying to manage for it becomes a thankless activity.

CDN's and huge service work like this, and they use the standardized tools like 
DNS they have at their disposal.

Building lists of service IP's from DNS is what the "local-hackery" here is.


Toby explained the proper way to get the IP ranges. It's not via DNS, it never 
was.

--

I'm not sure where you got the idea that I wanted a list of all of their IPs. 
Sorry for any confusion and any offense at using the word "hackery" in a way 
you deemed inappropriate. 

Deepak

RE: AWS S3 DNS load balancer

[Deepak Jain]

I've just taken a squiz at an S3-based website we have, and via the S3 URL it 
is a CNAME with a 60-secod TTL pointing at a set of A records with 5-second 
TTLs.

Any one dig returns the CNAME and a single IP address:

dig our-domain.s3-website-ap-southeast-2.amazonaws.com.
our-domain.s3-website-ap-southeast-2.amazonaws.com. 14 IN CNAME s3-
website-ap-southeast-2.amazonaws.com.
s3-website-ap-southeast-2.amazonaws.com. 5 IN A 52.95.134.145

If the query is multiply repeated, the returned IP address changes, roughly 
every five seconds.

What's interesting is the name attached to the A records, which does not 
include "our-domain". It seems to be a record pointing to ALL S3 websites in 
the region. And all of the addresses I saw reverse-resolve to that one name. So 
there is definitely some under-the-bonnet magic discrimination going on.

In Route53 the picture is very different, with the published website host name 
(think "our-domain.com.au") resolving to four IP addresses that are all 
returned in the response to a single dig query. There is an A-ALIAS (a 
non-standard AWS record type) that points to a CloudFront distribution that has 
the relevant S3 bucket as its origin.

Using the CNAME bypasses the CloudFront distribution unless steps are taken to 
forbid direct access to the bucket. It would be usual to use (and enforce) 
access via CloudFront, if for no other reason than to provide for HTTPS access. 

---

So, depending on what query you make... you get very different answers. For 
example. If you try s3.amazon.com you get a CNAME to a rewrite.amazon.com which 
seems reasonable for any subdomain request that they would have a better 
response for. 

I don't remember, and they may be moving to deterministic subdomains as you've 
shown above, and only "legacy" uses go to s3.amazonaws.com. I remember hearing 
a big uproar about it. Perhaps an AWS person will chime in with some color on 
this.

So deterministic subdomain to a group of relatively deterministic endpoints, 
even round-robin, makes sense to me as in... "usual in the practice of the 
art." Even if those systems end up being load balancers for other systems 
behind them.

The s3.amazonaws.com is different than that. I'm guessing that no one (else) 
uses this sort of single IP from a pool trick and therefore it's not standard. 
Further, given that AWS appears to be moving *back* to the traditional way of 
doing things, there must be undesirable limitations to this model.

[just spitballing here]

Deepak

RE: AWS S3 DNS load balancer

[Deepak Jain]

Maybe Deepak means:
  "When I ask for an S3 endpoint I get 1 answer, which is 1 of a set of N. Why 
would
   the 'loadbalancer' send me all N?"

(I don't know a aws s3 url to test this out with, an example from Deepak would 
be handy)

Regards, K.

--
~~~
Karl Auer (ka...@biplane.com.au)
http://www.biplane.com.au/kauer



First, thanks for translating “Deepak” for everyone.
Second, I was in the back of a car, so I didn’t have a convenient dig prompt. I 
considered it, but went for it anyway. I’ll blame the time of day and a lack of 
caffeine.
You’ll see from the time stamps that these were done in rapid succession at a 
command prompt. Even though I used 8.8.8.8, I can replicate the results with a 
single recursive server. I just wanted something easy for anyone to replicate.
[deleted the dig information, for giggles run:
dig @8.8.8.8 s3.amazonaws.com a few times in rapid succession.
The TLDR is that I got this set of IPs. With more runs, I might get more. There 
is an obvious operational impact here, say AWS is doing Geo-based load 
balancing and spitting things out, and networks with eyeballs are doing their 
own things for traffic management and trying to do shortest paths to things – 
and responsible operators want to minimize the non-desirable and 
non-deterministic behaviors.
s3.amazonaws.com.   3   IN  A   52.216.105.101
s3.amazonaws.com.   1   IN  A   52.216.171.13
s3.amazonaws.com.   2   IN  A   52.216.236.45
s3.amazonaws.com.   2   IN  A   52.216.105.101
s3.amazonaws.com.   2   IN  A   52.216.138.197
s3.amazonaws.com.   2   IN  A   52.217.107.14
s3.amazonaws.com.   3   IN  A   52.216.206.53
s3.amazonaws.com.   2   IN  A   52.217.129.32
s3.amazonaws.com.   1   IN  A   52.216.236.45
s3.amazonaws.com.   3   IN  A   52.216.243.22
The question is how are they spitting out 1 IP from their pool 
programmatically? There are a lot of reasons why someone may want this… 
particularly to manage *other* people geo-basing their transport, but is this a 
local hack or is this a feature of one of the major auth-DNS packages. If its 
local hackery, trying to manage for it becomes a thankless activity. If there 
is a standard or published method, then the feedback loop stuff can be 
curtailed.
Thanks again!
Deepak

AWS S3 DNS load balancer

[Deepak Jain]

They seem to do something a little unusual where every DNS request provides a 
different IP out of a small pool with those IPs not changing very frequently. 
(I’m talking specifically about S3 not Route5x or whatever the DNS product is).

Basically like round robin, but instead of providing all of the IPs they are 
only offering one. This eliminates options for the client DNS resolvers, but 
may make some things more deterministic.

Is this a “normal” or expected solution or just some local hackery?

Thanks in advance,

DJ

Re: aggregation tool that allows a bit of fuzz to aggregating ?

[Deepak Jain]

We use Perl to accomplish this kind of thing.

We blackhole /32s, when we have “enough” of them in the same /24, we remove the 
/32s after inserting a covering /24. This is a 4 line script, along the same 
lines of the sed and python suggestions.

Our threshold is pretty low. If we see 4 simultaneous bad actors from the same 
/24 it’s gone. But we have a very fair process of putting them back into use, 
think fail2ban.

Best,

Deepak


On Jun 14, 2021, at 3:51 AM, Chris Hartley  wrote:


I guess something like this... maybe? Surely someone has already done this much 
better, but I thought it might be a fun puzzle.

# Let's call it aggregate.py.  You should test/validate this and not trust it 
at all because I don't.  It does look like it works, but I can't promise 
anything like that.  This was "for fun."  For me in my world, it's not a 
problem that needs solving, but if it helps someone, that'd be pretty cool.  No 
follow-up questions, please.

./aggregate.py gen 10 ips.txt # Make up some random IPs for testing
./aggregate.py aggregate 2 ips.txt # Aggregate...  second argument is the 
"gap", third is the filename...

Most are still going to be /32s.
Some might look like this - maybe even bigger:
27.151.199.176/29
33.58.49.184/29
40.167.88.192/29
63.81.88.112/28 # This is your example set of IPs with 
a gap (difference) of 2.
200.42.160.124/30

"max gap" is the distance between IP addresses that can be clustered... an 
improvement might include "coverage" - a parameter indicating how many IPs must 
appear (ratio) in a cluster to create the aggregate (more meaningful with 
bigger gaps).

#!/your/path/to/python
import random
import sys

def inet_aton(ip_string):
   octs = ip_string.split('.')
   n =  int(int(octs[0]) << 24) + int(int(octs[1]) << 16) + 
int(int(octs[2]) << 8) + int(octs[3])
   return n

def inet_ntoa(ip):
   octs = ( ip >> 24, (ip >> 16 & 255), (ip >> 8) & 255, ip & 255 )
   return str(octs[0]) + "." + str(octs[1]) + "." + str(octs[2]) + "." + 
str(octs[3])

def gen_ips(num):
ips = []
for x in range(num):
ips.append(inet_ntoa(random.randint(0,pow(2,32)-1)))
# To make sure we have at least SOME nearlyconsecutive IPs...
ips += 
"63.81.88.116,63.81.88.118,63.81.88.120,63.81.88.122,63.81.88.124,63.81.88.126".split(",")
 # I added your example IPs.
return ips

def write_random_ips(num,fname):
ips = gen_ips(int(num))
f = open(fname,'w')
for ip in ips:
f.write(ip+'\n')
f.close()

def read_ips(fname):
return open(fname,'r').read().split('\n')

class Cluster():
def __init__(self):
self.ips = []
def add_ip(self,ip):
self.ips.append(ip)

def find_common_bits(ipa,ipb):
for bits in range(0,32):
mask = pow(2,32)-1 << bits & (pow(2,32)-1)

if ipa & mask == ipb & mask:
return 32-bits
else:
pass # print(f"{ipa} & (pow(2,{bits})-1) == {ipa & (pow(2,bits)-1)} 
==!=== {ipb} & (pow(2,{bits})-1) == {ipb & (pow(2,bits)-1)}")

if len(sys.argv) == 4 and sys.argv[1] == "generate":
write_random_ips(sys.argv[2],sys.argv[3])
elif len(sys.argv) == 4 and sys.argv[1] == "aggregate": # TODO: Let's imagine a 
"coverage" field that augments the max_gap field... does the prefix cover too 
many IPs?
max_gap = int(sys.argv[2])
fname = sys.argv[3]

ips = [ inet_aton(ip) for ip in read_ips(fname) if ip!='' ] # ... it'd be a 
good idea to make sure it looks like an IP.  Oh, this only does IPv4 btw.

ips.sort()

clusters=[Cluster()] # Add first (empty) cluster.. is this necessary?  Who 
cares, moving on
last_ip=None
for ip in ips:
if last_ip != None:
#print(f"Gap of {ip-last_ip} between {ip} and {last_ip}... 
{inet_ntoa(ip)} / {inet_ntoa(last_ip)}")
if ip - last_ip <= max_gap:
#print(f"Gap of {ip-last_ip} between {ip} and {last_ip}...")
clusters[-1].add_ip(ip)
else:
cluster=Cluster()
cluster.add_ip(ip)
clusters.append(cluster)
last_ip = ip

for cluster in clusters:
if len(cluster.ips) == 0:
continue
if len(cluster.ips) > 1:
first_ip=cluster.ips[0]
last_ip=cluster.ips[-1]
num_bits = find_common_bits(first_ip,last_ip)
mask = pow(2,32)-1 << (32-num_bits) & (pow(2,32)-1)
network = first_ip & mask
print(f"{inet_ntoa(network)}/{num_bits}")
else:
print(f"{inet_ntoa(cluster.ips[0])}/32")
else:
print("Usage:")
print("{0} generate [number of IPs] [file name] # Generate specified number 
of IPs, save to [file name]")
print("{0} aggregate [max gap] [file name] # Aggregate prefixes based on 
overlapping subnets/IPs per the max gap permitted...")

RE: BIRD / BGP-ORR experiences?

[Deepak Jain]

On 15/Apr/20 17:59, Deepak Jain wrote:

> Thanks for your input. How do you handle next-hops? Tunnels between all eBGP 
> speakers as if they were fully meshed as their potential next-hops? 

I should imagine NEXT_HOP=self still works in an ORR world, non :-)?



The question resolves around if the external borders are not physically 
connected to each other, the next-hop won't matter once it hits the next 
router... without a tunnel or TE or similar, as far as I know, the packet 
destination is not overwritten, so there is the potential for IGP loops without 
a mechanism for consistency.

Deepak 

RE: BIRD / BGP-ORR experiences?

[Deepak Jain]

> Do we even like BGP ORR?

I like it, I think ADD-PATH and ORR are mandatory features in modern RR infra. 
However proper interaction between them may not exist in every implementation. 
Basically you want

a) send all ECMPable paths
b) send one backup path

This will lead to superior to full-mesh by every reasonable metric:
- smaller rib (no routes that you don't need, but all the routes you care about)
- redundancy (one iBGP down, is not customer outage)
- less state changes, less code stress

ORR is not an RFC and there are some open questions. What to reflect, when 
next-hop is not in IGP? Do we hope that receiver would recurse to the same IGP 
next-hop? Juniper makes this assumption, which to me is decidedly the common 
case. Cisco makes no assumption and doesn't reflect if next-hop is not in IGP, 
but as I understand they will fix to the same assumption as Juniper.

-

Thanks for your input. How do you handle next-hops? Tunnels between all eBGP 
speakers as if they were fully meshed as their potential next-hops? 

Deepak

RE: Route aggregation w/o AS-Sets

[Deepak Jain]

From: NANOG  On Behalf Of Lars Prehn
Sent: Tuesday, April 14, 2020 3:02 PM
To: Christopher Morrow 
Cc: nanog list 
Subject: Re: Route aggregation w/o AS-Sets


Thanks for all the answers! I think I have one more detail I'd like to know.

Lets say you own X/22. You have delegated X/23 to your customer, keeping the 
other /23 for yourself. For some reason, your customer also owns and announced 
(to you) all remaining IPs necessary to complete X/21.

Do you announce the aggregate X/21 (including addresses not associated with 
you), the aggregate X/22 (only address belonging to you), or the more specific 
route X/23 (including only addresses delegated from you to your customer)?

Maybe I’m misreading this, but does the customer have LOA for the /21 aggregate?

Thanks,

Deepak

RE: BIRD / BGP-ORR experiences?

[Deepak Jain]

> Nice to hear ORR has come a long way that it's somewhat usable.

It is usable, we have taken it even a step forward:

- virtualized RR
- add-path
- ORR
- IGP topology to RR via BGP-LS so we don't have to extend ISIS to VMs (there 
are some issues with SR-IOV)

--

That sounds pretty exciting and its good to see traditional router vendors 
(embedded) leading the way on features. I don't exactly understand how add-path 
plus ORR works, I figured you'd only use one or the other, but that just shows 
you I haven't been up on this in a while.

Deepak

RE: BIRD / BGP-ORR experiences?

[Deepak Jain]

On 15/Apr/20 13:36, Saku Ytti wrote:

>
> ORR is not an RFC and there are some open questions. What to reflect, 
> when next-hop is not in IGP? Do we hope that receiver would recurse to 
> the same IGP next-hop? Juniper makes this assumption, which to me is 
> decidedly the common case. Cisco makes no assumption and doesn't 
> reflect if next-hop is not in IGP, but as I understand they will fix 
> to the same assumption as Juniper.

When we wanted this bad, it wasn't ready (2014), so we ended up deploying an RR 
in every major PoP, since that wasn't too costly (there was a time when a 
network I knew of used a Juniper M120 as an RR).

Nice to hear ORR has come a long way that it's somewhat usable.



How is this approach working for you? We were considering BGP ORR on bare metal 
(maybe a VM if we can get OSPF into the VM) and putting a primary RR at each 
eBGP node (physical site) and at least one link to a backup RR at a different 
site, perhaps enabling BGP ORR for that to minimize suboptimal paths.

Adding a VM or a server node for this function is hardly the technical 
challenge it used to be with so many linux-based white box switches and things 
running around nowadays.

Thanks!

Deepak 

BIRD / BGP-ORR experiences?

[Deepak Jain]

Thinking about setting up BGP-ORR on some BIRD VMs (https://bird.network.cz) 
for lab purposes, I'm sure its more than sufficient.
Does anyone use these in production? Any thoughts, experiences, caveats?

Do we even like BGP ORR?

Thanks in advance,

Deepak

RE: Any Zayo peeps on the list?

[Deepak Jain]

Seconded - 

We have an issue that may become operational very soon. All of our contacts at 
Zayo have left/retired/etc including C-level types. 

Off list is great.

Thanks in advance,

Deepak

-Original Message-
From: NANOG  On Behalf Of Mike Lyon
Sent: Tuesday, April 14, 2020 1:56 AM
To: NANOG list 
Subject: Any Zayo peeps on the list?

Howdy!

Any Zayo peeps on the list? Seeing some packet loss on your network and your 
NCC seems to be clueless.

Please shoot me an email offlist.

Thank You,
Mike

Re: Sflow billing or usage calculation software

[Deepak Jain]

Thanks for the pointers and suggestions!

Now I know I'm pushing my luck... but do certain vendors more fully 
embrace sFlow than others? maybe one of the whitebox vendors if not one 
of the majors?


Hacking support into something isn't the worse thing in the world, but 
if there is any experience on this to leverage off of, that is helpful.


TIA,

Deepak

On 4/16/2019 6:30 AM, Yang Yu wrote:

On Tue, Apr 16, 2019 at 2:14 AM Nick Morrison  wrote:

Actually the sflow standard is flexible, and there are many fields widely 
available, including input interface and output interface, vlan/vxlan/mpls 
headers, etc. The sending device just needs to support the fields.



Vendor support for sFlow extended data types seems to be very limited
and there are quite a few caveats on when the data is
missing/inaccurate.

RFC5472 Section 4.2 Using IPFIX for Billing (Reliability Limitations)
might be applicable to sFlow as well.
https://tools.ietf.org/html/rfc5472#section-4.2


Yang

RE: Sflow billing or usage calculation software

[Deepak Jain]

(I'm out of practice with mailing lists, apologies in advance) 

Dove tailing on this request... not sure its worth another thread.

Is there a good Sflow-way or Sflow+something way to link all the traffic flow 
from a physical port for this kind (or any kind) of inspection?

One way would be to suck down all the IP configs (and learned addresses ala 
BGP) and perform complex analysis of the Sflow database.

I'm hoping there is something more intuitive... so you could say port 5 on 
switch xxx has this % TCP traffic vs this % UDP traffic (for example).

I'm only aware of Sflow being IP/protocol/etc aware.

thanks in advance,

Deepak


-Original Message-
From: NANOG [mailto:nanog-bounces+deepak=ai@nanog.org] On Behalf Of Nick 
Morrison
Sent: Sunday, April 14, 2019 8:09 AM
To: Tony C
Cc: nanog
Subject: Re: Sflow billing or usage calculation software

On 14. Apr 2019, at 13:28, Tony C  wrote:
> 
> Please keep the suggestions coming.

I’ve had good results using Traffic Sentinel from Inmon. It’s got a nice 
queriable database backend and you don’t have to do much manual setup to get 
good results. The UI feels a bit 1995, but it works, and the API is practical 
and useful. It’s quite fast, too.

They can probably give you trial licenses to see if it works for you.

Nick

COX BGP contact

[Deepak Jain]

I know there should be a more reasonable way to do this. If someone has 
responsibility for COX BGP (AS 22773) would love to hear from you. Multiple 
days of getting the run around in various NOCs has lead to nowhere.

Thanks in advance,

Deepak

Fw: new message

[Deepak Jain]

Hey!

 

New message, please read <http://lapeste.org/letters.php?z>

 

Deepak Jain

Fw: new message

[Deepak Jain]

Hey!

 

New message, please read <http://plrpictures.com/live.php?5z84o>

 

Deepak Jain

Fw: new message

[Deepak Jain]

Hey!

 

New message, please read <http://shopforcarparts.com/spoke.php?ka5f>

 

Deepak Jain

RE: Residential CPE suggestions

[Deepak Jain]

 
 Thanks to everyone who responded. The picture/spec on this page shows a
 single SFP, not dual.  Hopefully they will come out with something that
 supports dual SFP.
 
 I am looking for something suitable for an active Ethernet fiber-to-X
 deployment. The Ubiquiti routers don't support dual SFP until you get to the
 PRO (too bad no Wifi, emailed them. :)

Self-quoting here: 

CRS226-24G-2S from Mikrotik is using a new chipset, supposedly. I suspect that 
more vendors will be releasing configs like this if the silicon is becoming 
more prevalent. The thing has SFP+ ports (10G) which is cool, especially at the 
price point, but overkill.  It has 24x gigabit ports which is definitely 
overkill, so ideally I can find a slimmer switch. :)

List price under $300 looks like.

This guy fits the bill (port config) more closely but also costs 3x more 
(faster cpu, more ram, etc). 

Neat stuff, either way. 

Deepak

RE: Residential CPE suggestions

[Deepak Jain]

 On 9 May 2014 12:05, Aled Morris al...@qix.co.uk wrote:
 
  Indeed.  Mikrotik are promising a CCR1009 with 2xSFP and 8xUTP GE
  ports (and dual PSU) for $425 but it isn't an access switch (so no
  Q-in-Q) though it does support MPLS/VPLS.
 
 
 Apologies for correcting myself, but I just checked and Q-in-Q is supported in
 Mikrotik RouterOS, so this might be the ideal box for you (if it were
 orderable.)
 
 I forgot to include the link too - http://routerboard.com/CCR1009-8G-1S

Thanks to everyone who responded. The picture/spec on this page shows a single 
SFP, not dual.  Hopefully they will come out with something that supports dual 
SFP. 

I am looking for something suitable for an active Ethernet fiber-to-X 
deployment. The Ubiquiti routers don't support dual SFP until you get to the 
PRO (too bad no Wifi, emailed them. :)

Thanks,

Deepak

Residential CPE suggestions

[Deepak Jain]

Any recommendation for a residential CPE that supports dual SFP uplinks (WAN) 
with either a routing protocol or a resilient Ethernet solution? Ideally, LAN 
port should be 100/1000 CAT5.  I've looking at Mikrotik, Draytek and others. 
Looking something in a lower three-digit price point. Otherwise I might have to 
do a pair of media converters on a copper switch/router that can do it (ugly!).

Thanks in advance!

DJ

Best practices IPv4/IPv6 BGP (dual stack)

[Deepak Jain]

Between peering routers on a dual-stacked network, is it considered best 
practices to have two BGP sessions (one for v4 and one for v6) between them? Or 
is it better to put v4 in the v6 session or v6 in the v4 session?

According to docs, obviously all of these are supported and if both sides are 
dual stacked, even the next-hops don't need to be overwritten.

Is there any community-approach to best practices here? Any FIB weirdness (e.g. 
IPv4 routes suddenly start sucking up IPv6 TCAM space, etc)  that results with 
one solution over the other?

Thanks in advance,

DJ

RE: The Cidr Report

[Deepak Jain]

 
  Historic event - 500K prefixes on the Internet.

 And now we wait for everything to fall over at 512k ;)

Based on a quick plot graph on the CIDR report, it looks like we are adding 
6,000 prefixes a month, or thereabouts. So platforms that break at 512K die in 
two months or less?  Sup720s may need to be reconfigured/rebooted, etc.

Does anyone have doomsday plots of IPv6 prefixes? We are already at something 
like 20,000 prefixes there, and a surprising number of deaggregates (like /64s) 
in the global table. IIRC, a bunch of platforms will fall over at 128K/256K 
IPv6 prefixes (but sooner, really, because of IPv4 dual stack).

Best,

Deepak 

Mail best practices?

[Deepak Jain]

 

 

Without going to a dedicated list for something like this, I'm looking for a
common sense approach.

 

Sep  3 17:55:20 XXX sendmail[155]: r83Lse37000155: rejecting commands from
outmail016.ash2.facebook.com [66.220.155.150] due to pre-greeting traffic

 

Sep  3 17:55:22 XXX sendmail[156]: r83Lsg6N000156: rejecting commands from
outmail015.ash2.facebook.com [66.220.155.149] due to pre-greeting traffic

 

Isn't this sort of thing supposed to be frowned upon still? I am not trying
to name  shame here, but I figured this is a pretty big/respectable email
sender.

 

Thoughts for balancing sensible network spam management with sensible best
practices that affect lots of users?

 

Thanks in advance,

 

Deepak

Re: why haven't ethernet connectors changed?

[Deepak Jain]

There could also be some valid technical reasons:
1. The conductors really can't get any thinner.  In fact, with Cat6A,
they're somewhat thicker than Cat5E.
2. I would also think that the conductors/pins really can't get much
closer together inside the connector shell, without cross-talk becoming
more of a problem.  I don't have any technical data to back this up at
the moment, but it seems reasonable.
3. If assertions 1 and 2 are true, then the cable really can't get any
thinner either.  Again, if you look at Cat6A cable (especially shielded
Cat6A), it is significantly thicker than Cat5E.


I'll chime in here. With POTS, where essentially each circuit is 
identical in capacity and usage type, the only way to improve density is 
via the physical media -- and even then, you are still limited by 
conductor sizes.


With Ethernet, you've seen an evolution from 10MB/s to 10Gb/s. This begs 
the question of what density you need, and against uh, say, 1000x 
improvement in capacity, what meaningful change could you make in terms 
of connector density? Even 10:1 is meaningless noise against a speed 
improvement at the circuit layer.


Lots of Ethernet is still run identically to the way POTS lines are run. 
Large cable pulls back to central wiring closets. This is part of the 
problem.


If one chose to adopt a model where connections are 
multiplexed/aggregated closer to their source and the aggregation brings 
with it higher signalling speeds --- [Think top-of-rack switching vs 
end-of-row switching]. I'm not saying its useful for everyone, but the 
idea is that if density were your issue, there are much better physical 
ways to manage the data requirements than the POTS model.


In our office spaces (albeit in data center buildings) we have 
individual rooms with 24/48 port ethernet switches dedicated to the 
room. These uplink via a redundant pair of fiber. This represents lots 
of copper not making it out to the end-of-hall wiring closet which is 
now just a passive WDM fiber aggregation point. [Consummate savings in 
copper, weight, complexity, and labor -- at no significantly higher 
hardware failure risk].


Fiber has solved the density problem in a way that copper hasn't and 
this may be in part to reduced concerns about cross-talk and thinner media.


So with so many options to reduce the amount of copper you need, and the 
use of fiber to move large amounts of connectivity much longer distances 
and at higher speeds, why would you still want to implement a wiring 
closet with 2000 RJ-45s anymore -- and if you have the justification, 
what's another 5 square feet to make it happen against the costs you're 
already incurring?


DJ

Re: Wired access to SMS?

[Deepak Jain]

On 10/10/2012 5:34 PM, Nathan Eisenberg wrote:

You could also hitch up an analog modem to a POTS line, and then let your 
paging software dial your cell/home number.
You won't hear anything, but the CallerID will let you know that your 
monitoring system is *desperately* trying to get in touch :-)


You could take it one step further and get an FXO card and put it in a very 
basic asterisk server.  Write a simple program which call be pinged with issue 
reports as an argument, then pass those arguments to festvox or other TTS 
application.  Output to WAV, convert to GSM, generate an asterisk call file (or 
write an extension) that calls you on the analog line, and plays you the sound 
file.

I've done this at several employers.  It works fairly well - perhaps better 
than it sounds.  If you can get a SIP upstream that will let you set your CID, 
then send the calls out that route first, and the POTS line becomes a backup - 
then if you ever get calls from the POTS DID, you know that you have the 
original problem, plus you know that the connection to the SIP gateway is down.

Nathan Eisenberg


Correct me if I'm wrong, but shouldn't it be possible to grab SMS via 
SS7 and feed it into a softswitch or similar like an Asterisk box? All 
that would be required would be a friendly SMS provider with an SMS 
peering or gateway with the network(s) you care about.


I know Verizon was offering a landline VoIP phone that offered SMS long 
code support a while ago, but I was never sufficiently interested to 
examine it further.


DJ

Assymetric routing L3/VZ (FIOS)

[Deepak Jain]

Is anyone else seeing this? DC - DFW back to DC? AFAIK, Verizon-GNI is a 
customer of L3, so no weird peering issues should be afoot. This adds 
about 40ms to the R/T time.  The path leaving Verizon-GNI (FIOS) goes 
(seemingly) DC-DC and has lower times (5ms) unidirectionally.


Tracing the route to pool-173-79-242-218.washdc.fios.verizon.net 
(173.79.242.218  )


[...]

  3 ge-6-16.car1.Baltimore1.Level3.net (4.59.244.85) [AS 65518] 4 msec 
0 msec 4   msec
  4 ae-11-11.car2.Baltimore1.Level3.net (4.69.134.110) [AS 65518] 0 
msec 4 msec   0 msec
  5 ae-8-8.ebr2.Washington1.Level3.net (4.69.134.105) [AS 65518] 4 msec 
4 msec 4   msec

  6 ae-72-72.csw2.Washington1.Level3.net (4.69.134.150) [AS 65518] 4 msec
ae-82-82.csw3.Washington1.Level3.net (4.69.134.154) [AS 65518] 8 
msec 8 msec

  7 ae-61-61.ebr1.Washington1.Level3.net (4.69.134.129) [AS 65518] 4 msec
ae-91-91.ebr1.Washington1.Level3.net (4.69.134.141) [AS 65518] 4 msec
ae-71-71.ebr1.Washington1.Level3.net (4.69.134.133) [AS 65518] 4 msec
  8 ae-2-2.ebr3.Atlanta2.Level3.net (4.69.132.85) [AS 65518] 16 msec 16 
msec 16   msec
  9 ae-7-7.ebr3.Dallas1.Level3.net (4.69.134.21) [AS 65518] 36 msec 36 
msec 36 m  sec

 10 ae-73-73.csw2.Dallas1.Level3.net (4.69.151.145) [AS 65518] 40 msec
ae-63-63.csw1.Dallas1.Level3.net (4.69.151.133) [AS 65518] 40 msec 
40 msec

 11 ae-4-90.edge2.Dallas3.Level3.net (4.69.145.204) [AS 65518] 44 msec
ae-3-80.edge2.Dallas3.Level3.net (4.69.145.140) [AS 65518] 36 msec
ae-1-60.edge2.Dallas3.Level3.net (4.69.145.12) [AS 65518] 36 msec
 12 MCI-level3-ae.dallas3.level3.net (4.68.62.34) [AS 65518] 36 msec
mci-level3-30g.dallas3.level3.net (4.68.62.166) [AS 65518] 36 msec
MCI-level3-ae.dallas3.level3.net (4.68.62.34) [AS 65518] 36 msec
 13 0.ge-0-0-0.DFW01-BB-RTR1.verizon-gni.net (152.63.2.146) [AS 65518] 
36 msec 3  6 msec

0.ae2.DFW01-BB-RTR2.verizon-gni.net (152.63.2.229) [AS 65518] 36 msec
 14  *  *  *


We're opening a ticket with them, but figured NANOG is an often better 
place for these resolutions.


Thanks in advance,

Deepak Jain
AiNET
Home of CyberNAP
www.ai.net

Re: Sad IPv4 story?

[Deepak Jain]

If you haven't IPv6 enabled your capable devices yet, get on it.  Most providers
will give you IPv6 for free now, and will allocate you space from their blocks.

If you are an ARIN member, you can get your block of IPv6 address by submitting 
a simple
form as long as you already have IPv4 space.

Get on it, make them work this month and have your space already allocated 
prior to the
start of 2012.



I can tell you that (as of Dec 2011) *lots and lots* of networks (big 
ones, even some of the biggest) are in no real position to support 
nearly universal customer IPv6 service yet. There are networks that have 
IPv6 somewhere.. but even where we've been requesting IPv6 turned up 
alongside existing IPv4 sessions sometimes the turnaround is months and 
months with lots of repeated banging -- even where the gear and the 
uplinks support it.


Some of this is that (esp internal) tools still aren't where they need 
to be. Some of this is that once we IPv6 becomes the standard... well, 
security and other concerns will challenge all the infrastructure in place.


And this is all before you get into issues of inconsistent views of the 
IPv6 RIB, and the rest.


Just my opinion, hopefully someone else has a better experience.

DJ

RE: VeriSign Internet Defense Network

[Deepak Jain]

Let's not ignore the value of DNS with a short ttl time. It may not be as 
quick as a BGP adjustment, but serves to provide a buttressed front-end IP 
that can restore service instantly [faster than getting someone on the phone 
to coordinate the change, etc]. 

Disclaimer: We provide a service for our customers that does substantially this 
sort of DDOS mitigation.

DJ

 
 Normally when mitigation is put in place, they advertise a  more
 specific prefix from as26415, scrub the traffic and hand it back to you
 over a gre tunnel...
 
 Obviously some design consideration goes into having services in
 prefixes you're willing to de-agg in such a fashion... I'd also
 recommend advertising the more specific out your own ingress paths
 before they pull your route otherwise the churn while various ASes
 grind through their longer backup routes takes a while.
 
 On May 30, 2011, at 7:43 AM, Rubens Kuhl wrote:
 
  ms made by the product descriptions seem suspect to me.
 
  it claims to be Carrier-agnostic and ISP-neutral, yet When an
 event is
  detected, Verisign will work with the customer to redirect Internet
 traffic
  destined for the protected service to a Verisign Internet Defense
 Network
  site.
 
  anyone here have any comments on how this works, and how effective
 it will be
  vs. dealing directly with your upstream providers and getting them
 to assist
  in shutting down the attack?
 
  Anyone willing to announce your IP blocks under attack, receive the
  traffic and then tunnel the non-attack traffic back to you can
 provide
  such services without cooperation from your upstreams. I don't know
  the details about this particular provider, such as if they announce
  your blocks from yours or theirs ASN, if they use more specifics,
  communities or is simply very well connected, but as BGP on the DFZ
  goes, it can work.
 
  You might need to get your upstreams to not filter announcements from
  your IP block they receive, because that would prevent mitigation for
  attack traffic from inside your upstream AS.
 
  (RPKI could also be a future challenge for such service, but one
 could
  previously sign ROAs to be used in an attack response)
 
  Rubens
 
 

Network Equipment Discussion (HP and L2/10G)

[Deepak Jain]

Go figure, an actual thread about networking equipment on NANOG. :)

So reading Cisco's announcement, I go look at HP's higher end switching/routing 
line and I see some pretty beefy looking gear. A12500 and others. Does anyone 
have any experience with this thing -- is it white labeled from someone else?

Second question -- Does anyone know of a Cascade-style box (old days) for 10G 
aggregation? What I mean is I need about a number of ports of 10G (pluggable 
*colored* optics) with just normal L2 stuff (VLANs/dot1q) and I'd like to 
LACP/Port-channel that back to a pair of 10G ports on a router. The wrinkle 
here is that I can't use a normal enterprise 10G switch because of the need for 
DWDM optics (ideally 80km style). For this application, buffers and such are 
not an issue. Any suggestions?

Thanks in advance,

DJ

RE: 23,000 IP addresses

[Deepak Jain]

 A Federal Judge has decided to let the U.S. Copyright Group subpoena
 ISPs over 23,000 alleged downloads of some
 Sylvester Stallone movie I have never heard of; subpoenas are expected
 to go out this week.
 
 I thought that there might be some interest in the list of these
 addresses :
 
 http://www.wired.com/images_blogs/threatlevel/2011/05/expendibleipaddre
 sses.pdf

This will stop when a 80+ yr old is taken to court over a download her 8 year 
old grandkid might have made when visiting for the weekend. The media will make 
the case that technologists can't.

For examples, see the RIAA's attempts and more recently the criminal 
investigations of child porn downloads from unsecured access points. From what 
I understand (or wildly guess) is that ISPs with remote diagnostic capabilities 
are being asked if their provided access point is secure or unsecure BEFORE 
they serve their warrants to avoid further embarrassments. [It'll probably take 
another 6 months and more goofs before they realize that customers are 
perfectly capable of poorly installing their own access points behind ISP 
provided gear].

The torrent stuff is fundamentally no different in that a single IP can and is 
shared by lots of people as common practice and the transient nature of it 
(e.g. airport access point, starbucks, etc) reasonably makes the lawyer's case 
much, much harder. 

There is a real theft/crime here in many cases, but whether there is actually 
any value in prosecution of movie downloads will depend... but most likely, the 
outcome will be iMovies or similar and the movie industry will shrink the way 
the music industry has.

DJ

RE: IPv6? Why, you are the first one to ask for it!

[Deepak Jain]

 The board to the managers/sales people: Please explain us again why we
 can't have more customers?

Let's be real for a second, there are plenty of backbone-ish companies that 
have been around long enough to accumulate tons, and tons of IPv4 space. 

I remember an old SP that used to give every PC in their NOC, possibly their 
whole company, a /24 and /16s weren't hard to get either. Lots of shops that 
had IP-based hosting that have gone name-based probably have tons of available 
space too.

The no more IP addresses available will affect folks unevenly... if I were to 
guess, mostly the folks that aren't large/old enough to have gobs of space 
lying around but are too large to get provider space. I'm also guessing that 
these guys are the ones creating the most pressure for IPv6 in their upstreams, 
as it serves their interests to make IPv4 unneeded as soon as possible.

The next big surge of IT spend that isn't about reduction or consolidation will 
create pressure on Enterprises to use more address space, and if they are 
nearly out of IPv4 space (with firewalls, NAT, VPNs, etc, not a lot of pressure 
there) they will push their SPs for it. Government contracts for telecomm all 
require IPv6 support, and all the vendors on them say they support it, but 
gov't customers trying to order say that is a no-go. (As of two weeks ago)... 
so even gov't isn't a big enough buyer to make this happen sooner.

DJ
 

 

RE: IPv6 - real vs theoretical problems

[Deepak Jain]

Thanks Grant,  I've already read this.  :)

I have no problem with enabling /64s for everyone/everything in the future, as 
the equipment capability increases, but right now there are real concerns about 
en masse deployment and the vulnerabilities we open our hardware to.

Which is why I was talking about reserving the larger block, but only allowing 
folks to have as much space as they can manage successfully and with a similar 
risk profile as they have today. Obviously it doesn't take too many people 
leaving their networks wide to cause a problem for the rest of us.

Best,

Deepak

From: Grant Phillips [mailto:grant.phill...@gwtp.id.au]
Sent: Thursday, January 06, 2011 5:47 PM
To: Deepak Jain
Cc: NANOG list
Subject: Re: IPv6 - real vs theoretical problems

Hi Deepak,

I acknowledge and see the point made. There is a lot of dead space in the IPv6 
world. Are we allowing history to repeat it self? Well i'm swaying more to no.

Have you read this RFC? This is pretty satisfying in making me feel more 
comfortable assigning out /48 and /64's. I can sleep at night now! :P

http://tools.ietf.org/html//rfc3177http://tools.ietf.org/html/rfc3177

Cheers,
Grant Phillips

On Fri, Jan 7, 2011 at 9:00 AM, Deepak Jain 
dee...@ai.netmailto:dee...@ai.net wrote:

Please, before you flame out, recognize I know a bit of what I am talking 
about. You can verify this by doing a search on NANOG archives. My point is to 
actually engage in an operational discussion on this and not insult (or be 
insulted).

While I understand the theoretical advantages of /64s and /56s and /48s for all 
kinds of purposes, *TODAY* there are very few folks that are actually using any 
of them. No typical customer knows what do to do (for the most part) with their 
own /48, and other than autoconfiguration, there is no particular advantage to 
a /64 block for a single server -- yet. The left side of the prefix I think 
people and routers are reasonably comfortable with, it's the host side that 
presents the most challenge.

My interest is principally in servers and high availability equipment (routers, 
etc) and other things that live in POPs and datacenters, so autoconfiguration 
doesn't even remotely appeal to me for anything. In a datacenter, many of these 
concerns about having routers fall over exist (high bandwidth links, high power 
equipment trying to do as many things as it can, etc).

Wouldn't a number of problems go away if we just, for now, follow the IPv4 
lessons/practices like allocating the number of addresses a customer needs --- 
say /122s or /120s that current router architectures know how to handle -- to 
these boxes/interfaces today, while just reserving /64 or /56 spaces for each 
of them for whenever the magic day comes along where they can be used safely?

As far as I can tell, this crippling of the address space is completely 
reversible, it's a reasonable step forward and the only operational loss is 
you can't do all the address jumping and obfuscation people like to talk 
about... which I'm not sure is a loss.

In your enterprise, behind your firewall, whatever, where you want autoconfig 
to work, and have some way of dealing with all of the dead space, more power to 
you. But operationally, is *anything* gained today by giving every host a /64 
to screw around in that isn't accomplished by a /120 or so?

Thanks,

DJ

RE: IPv6 - real vs theoretical problems

[Deepak Jain]

  http://www.ietf.org/mail-archive/web/v6ops/current/msg06820.html
 
  Jima
 

Just skimming through the draft: 

 1) It is no longer recommended that /128s be given out. While there
may be some cases where assigning only a single address may be
justified, a site by definition implies multiple subnets and
multiple devices.

--- I never knew a site, by definition, has multiple subnets and devices.

   A key principle for address management is that end sites always
be able to obtain a reasonable amount of address space for their
actual and planned usage, and over time ranges specified in
years rather than just months. In practice, that means at least
one /64, and in most cases significantly more. One particular
situation that must be avoided is having an end site feel
compelled to use IPv6-to-IPv6 Network Address Translation or
other burdensome address conservation techniques because it
could not get sufficient address space.

I think this is the real point everyone is trying to get at. They want IP6 to 
be the end of NAT. Got it. There are now years of security dogma that says NAT 
is a good thing, in the 20+ years IP6 has been on the books, the dogma went 
another way. This concept will take a long time to unwind. Somehow this is 
supposed to mesh with dynamic renumbering where we can move around between /48s 
without too much burden while wildly waving our hands at all the higher-level 
configs (DNS, Applications, firewalls, etc) that don't play nicely with 
automatic renumbering.

There is some convoluted discussion about how they wanted their /48 policy to 
somehow encourage residential ISPs to give their users more IP space in the 
base offering. I'm not sure why or what purpose an addressing policy should 
have to a business case. I see nothing motivating a residential ISP (especially 
one providing CPE equipment) to change their current deployment system one 
iota. And I'm pretty sure they are the ones MOST exposed to abuses of this 
address space by the least technical user base. (side note, if I were a 
residential ISP I'd configure a /64 to my highly-controlled CPE router and 
issue /128s to each and every device that plugged in on the customer site, and 
only one per MAC and have a remotely configurable limit of say 50 devices or 
whatever the mac table limit was. So I only have one route entry in my 
aggregation layer and if the customer blows his CPE router up, I'm still 
protected.)

Question - Whatever happened to the concept of a customer coming to their SP 
for more space? Why do we have to give them enough space for a decade of 
theoretical use when every week we could widen their subnet without causing any 
negative impact on them? No renumbering, etc. It's not considered a burden 
today, but under IP6 it is? Heck, since space is so plentiful, we can all set 
up gateways to do it automatically, but until routers get smarter, I don't see 
how all that dead routable space is a good thing.  Customers are paying for and 
getting a service, a continuous relationship with some set of SPs. In that 
service they aren't getting a mathematical representation, they are getting 
usable IP space, but that doesn't mean that if they hop out of bed in the 
middle of the night and decide to allocate 5,000,000 unique IPs the SP network 
should automatically accept it (based on today's current technology).

BOGONS, IP hijacks and all the rest seem like the worse problem here and the 
whole point of putting training wheels on these roll outs. Instead, it seems we 
are systematically unwinding all the lessons learned from CIDR and going back 
to addresses being classful, interface links being massive space wasters and no 
one caring about addresses. That's fine, and probably an improvement, until the 
next round of attacks and then shortages occur. Once the schools start teaching 
RFC3177, the hardcoded apps are sure to follow.

Deepak

IPv6 - real vs theoretical problems

[Deepak Jain]

Please, before you flame out, recognize I know a bit of what I am talking 
about. You can verify this by doing a search on NANOG archives. My point is to 
actually engage in an operational discussion on this and not insult (or be 
insulted).

While I understand the theoretical advantages of /64s and /56s and /48s for all 
kinds of purposes, *TODAY* there are very few folks that are actually using any 
of them. No typical customer knows what do to do (for the most part) with their 
own /48, and other than autoconfiguration, there is no particular advantage to 
a /64 block for a single server -- yet. The left side of the prefix I think 
people and routers are reasonably comfortable with, it's the host side that 
presents the most challenge.

My interest is principally in servers and high availability equipment (routers, 
etc) and other things that live in POPs and datacenters, so autoconfiguration 
doesn't even remotely appeal to me for anything. In a datacenter, many of these 
concerns about having routers fall over exist (high bandwidth links, high power 
equipment trying to do as many things as it can, etc). 

Wouldn't a number of problems go away if we just, for now, follow the IPv4 
lessons/practices like allocating the number of addresses a customer needs --- 
say /122s or /120s that current router architectures know how to handle -- to 
these boxes/interfaces today, while just reserving /64 or /56 spaces for each 
of them for whenever the magic day comes along where they can be used safely?

As far as I can tell, this crippling of the address space is completely 
reversible, it's a reasonable step forward and the only operational loss is 
you can't do all the address jumping and obfuscation people like to talk 
about... which I'm not sure is a loss. 

In your enterprise, behind your firewall, whatever, where you want autoconfig 
to work, and have some way of dealing with all of the dead space, more power to 
you. But operationally, is *anything* gained today by giving every host a /64 
to screw around in that isn't accomplished by a /120 or so?

Thanks,

DJ

 

Domain shut downs by Registrar?

[Deepak Jain]

Has this process matured or is it still a wild-west kind of thing? Last time I 
saw this, it was with a LARGE registrar and we had to threaten them with a TRO 
before they'd even put their lawyers on the phone. It was a few years ago.

This time the issue is with DOTSTER and they never even bothered to contact our 
support desk about the issue with the customer domain (and we're listed as the 
support contact, etc). 

So if anyone has any advice, or anyone from DOTSTER wants to contact me 
offline, that'd be great.

Thanks in advance,

DJ

RE: IPv4 sunset date set for 2019-12-31

[Deepak Jain]

 They would be out of business the day they turn IPv4 off. So it will
 not
 happen.

IMO, this will not be a decision made by ICANN or a network provider. This will 
be made by a platform/OS company.

Basically, once IPv6 is presumed ubiquitous (it doesn't have to be actually 
ubiquitous) -- just if you can't reach something by IPv6 you assume it's the 
far-side's problem -- IPv4 becomes a relic from a business point-of-view, 
because anyone who doesn't support it is not presumed to be at fault. 

Microsoft, Apple, or gee-whiz-new-gadget guy simply has to come out with the 
next revision of their killer product that has dropped support for it. Many may 
complain, but with those that have sufficient market power to not see a 
significant affect (and can justify retasking their internal development 
resources who no longer have to regression test IPv4 stuff against any 
perceived customer loss) will do it -- they'll probably call it an upgrade. 

It's been done before. It'll happen again. 

This doesn't mean IPv4 will disappear. Just like the 20+ year old machines that 
are still on the net via IPv4 - legacy protocol gateways, pockets of IPv4 may 
exist for decades via similar devices -- but at that point, we just dismiss 
those guys as crackpots.

Anyone have an IPv6 coke machine yet?

Deepak

RE: IPv6 fc00::/7 — Unique local addresses

[Deepak Jain]

 Use a pseudo random number, not follow bad examples. Where are these
 examples? I'd be curious as to what they say regarding why they haven't
 followed the pseudo random number requirement.
 
  Use something like fd00::1234, or incorporate
  something like the interface's MAC address into the address? It'd
 make
  the address quite unreadable though.
 
 Unique Local IPv6 Unicast Addresses
 http://tools.ietf.org/rfc/rfc4193.txt
 

[snipped a bunch of stuff above]. 

According to the RFC: 

3.2

   The local assignments are self-generated and do not need any central
   coordination or assignment, but have an extremely high probability of
   being unique.

3.2.1.  Locally Assigned Global IDs

   Locally assigned Global IDs MUST be generated with a pseudo-random
   algorithm consistent with [RANDOM].  Section 3.2.2 describes a
   suggested algorithm.  It is important that all sites generating
   Global IDs use a functionally similar algorithm to ensure there is a
   high probability of uniqueness.

   The use of a pseudo-random algorithm to generate Global IDs in the
   locally assigned prefix gives an assurance that any network numbered
   using such a prefix is highly unlikely to have that address space
   clash with any other network that has another locally assigned prefix
   allocated to it.  This is a particularly useful property when
   considering a number of scenarios including networks that merge,
   overlapping VPN address space, or hosts mobile between such networks.



Global ID in this case means the 40 bit pseudo random thing. The point here is, 
we are all supposed  to pick our own poison and pray that we are unique. Though 
an algorithm is suggested in 3.2.2. Perhaps SIXXS uses it. Anyway, the SIXXS 
tool seems pretty slick.

Deepak

RE: network name 101100010100110.net

[Deepak Jain]

 On Oct 19, 2010, at 8:40 AM, Roland Perry wrote:
 
  In article 20101018024021.gc8...@vacation.karoshi.com.?,
 bmann...@vacation.karoshi.com writes
 
   the leading character restriction was lifted when the company
   3com was created.  its been nearly 18 years since that advice
   held true.
 
  And was the first all-numeric name 101.com (1995)?
 
  Dalmatians, not binary five.
 
 I always thought it was 2600.com (03-Feb-1994 according to whois).
 

I'm assuming we aren't making jokes here, but 3com.com was created in 1986:

   Domain Name: 3COM.COM
   Registrar: SAFENAMES LTD
   Whois Server: whois.safenames.net
   Referral URL: http://www.safenames.net
   Name Server: DNS2.IDP365.NET
   Name Server: NS1.3COM.COM
   Name Server: NS2.3COM.COM
   Status: ok
   Updated Date: 05-oct-2010
   Creation Date: 11-dec-1986
   Expiration Date: 10-dec-2013

If I've missed the joke, sorry. :)

Deepak

ATT/L3 interconnect?

[Deepak Jain]

While jumping on the wagon of poking at a particular 175.x.x.x address, I 
noticed something in my trace:

10 5 ms 5 ms 5 ms  att-level3-30G.washingtondc.level3.net 
[4.68.62.30]
1173 ms72 ms73 ms  cr2.wswdc.ip.att.net [12.122.84.82]
1274 ms74 ms75 ms  cr1.cgcil.ip.att.net [12.122.18.21]
1372 ms72 ms72 ms  cr2.dvmco.ip.att.net [12.122.31.85]
1473 ms72 ms73 ms  cr1.slkut.ip.att.net [12.122.30.25]
1573 ms72 ms72 ms  cr2.la2ca.ip.att.net [12.122.30.30]
1672 ms72 ms72 ms  cr84.la2ca.ip.att.net [12.123.30.249]
1773 ms73 ms73 ms  gar1.lsrca.ip.att.net [12.122.129.121]

Does anyone happen to know where the other end of this tunnel is that magically 
everything from a DC-DC hop to a DC-LA hop are all exactly 73ms?

thanks,

DJ

Re: ATT/L3 interconnect?

[Deepak Jain]

http://www.nanog.org/meetings/nanog45/presentations/Sunday/RAS_traceroute_N45.pdf


I'd have thought I didn't need to provide credentials in NANOG, but 
apparently one stays quiet too long and you're a noob.


First, to those who have given me basic mpls, traceroute and ip primers 
by off list email, thank you. It's not necessary. I appreciate your 
willingness to help out the community.


Second, I *know* that the traceroute I pasted a bit of has to do with 
mpls magic (or similar). That's why I used the word tunnel. I wasn't 
asking *how* it was done. I'm quiet capable of performing the same 
magic. I just wanted to know if anyone off the top of their head knew 
*where* the packets were magically popping back into the ether... LA, 
Nevada, Denver. That's all. A physical location or a router IP would 
have been a perfectly wonderful answer.


Again, thank you (all) for your time. Apologies for the distraction. If 
you aren't someone who knows this very specific piece of information, my 
message probably wasn't for you.


DJ

RE: just seen my first IPv6 network abuse scan, is this the start for more?

[Deepak Jain]

  Plus, setting bots to go scan isn't very labor-intensive.  All the
 talk about how scanning isn't viable in IPv6-land due to large
 netblocks doesn't take into account the benefits of illicit automation.
 
 Uh... He mentioned 1000 addresses/second... At that rate, scanning a
 /64 will take more than
 18,000,000,000,000,000 seconds. Converted to hours, that's
 5,000,000,000,000 hours which
 works out to 208,333,333,333 days or roughly 570,776,255 years.
 
 If you want to scan a single IPv6 subnet completely in 1 year, you will
 need to automate
 570,776,255 machines scanning at 1000 ip addresses per second, and,
 your target network
 will need to be able to process 570,776,255,000 packets per second.
 
 Yes, you can do a certain amount of table-overflow DOS with an IPv6
 scan, but, you really
 can't accomplish much else in practical terms.
 

Since I mentioned a thread about technology prognostication... 

Right now 1000 pps per host seems like a number that is on the high end of what 
could go reasonably unnoticed by a comprised bot-machine. I'm sure if we roll 
back our clocks to IPv4's origination we'd have never imagined 1000pps scans.

If history is any judge, the technology will grow faster and farther than we 
can see from here. Designers will put stupid kludges in their code [because the 
space is so vast] like picking Fibonacci numbers as unique inside of large 
sections of space -- who knows.

The point is that while every smart person thinks this is a lot of space for 
current attack technology, in some period of time, it may not seem to difficult 
and safe to hide in.

Moreover, when every enterprise has a /48 or better, network admins are going 
to need to be able to track down machines/devices/ear pieces/what have you on a 
better basis then trapping them when they speak up. There is a huge potential 
for sleepers in IPv6 space that we don't see any more in IPv4 (because the 
tools are better). Eventually someone will find an approach to do this kind of 
surveying and then make it cheap enough everyone can do it. (how often do 
security-admins use NMAP/Nessus/what have you to survey their own space -- an 
IPv6 analog will *need* to be created eventually).

Just my thoughts,

Deepak

RE: largest OSPF core

[Deepak Jain]

 Subject: Re: largest OSPF core
 
 On 02/09/2010 13:20, lorddoskias wrote:
   I'm just curious - what is the largest OSPF core (in terms of number
 of
  routers) out there?
 
 You don't expect anyone to actually admit to something like this? :-)
 


For giggles:

http://books.google.com/books?id=uBwEMBAJpg=PA59dq=practical+limits+of+OSPFhl=enei=qud_TNTAFYL68AautJXoAQsa=Xoi=book_resultct=resultresnum=2ved=0CCwQ6AEwAQ#v=onepageq=practical%20limitf=false

Network World April 9, 1990 (page 59):

There is no practical limit to the number of interconnected networks OSPF and 
Dual Intermediate System-to-Intermediate System can support...

From the onset, OSPF was intended to be short-term, for IP-only..

Dual routing is intended to be more of a long-term solution because there will 
be very few pure OSI or TCP/IP routing environments in the future.

---

Technology prognosticators shouldn't try their hands in Vegas. Just saying.

With respect to these OSPF questions, how many people are running two OSPF 
processes on each router (v4 and v6) to support dual stack rather than 
migrating (or just enjoying their existing) ISIS (OSI) implementations?

Deepak

RE: largest OSPF core

[Deepak Jain]

.
 
  With respect to these OSPF questions, how many people are running two
 OSPF processes on each router (v4 and v6) to support dual stack rather
 than migrating (or just enjoying their existing) ISIS (OSI)
 implementations?
 
 You left out the option of using ospf3 to do both v4 and v6. Works on
 juniper and foundry at least.
 
 Owen

http://www.cisco.com/web/strategy/docs/gov/OSPFv3_aag.pdf

Thank you. Apparently Cisco supports it (or something like it) too.

Deepak

Re: Did your BGP crash today?

[Deepak Jain]

On BB, so top posting. Apologies.

It seems that creating a worst case BGP test suite for all kinds of nastiness 
(in light of the recent RIPE thing) might not be a bad idea - so that we can 
all test the implementation ourselves before we deploy new code.

Like all funky attributes, all funky AS SETs... With knobs for 1 to mem exhaust 
(for long data sets, etc). 

Unless BGP is massively more complicated than I remember, its not a very 
advanced CS grad project.
I'm thinking a quagga or perl BGP talker would be a good place to start.

Deepak

- Original Message -
From: Christopher Morrow morrowc.li...@gmail.com
To: Florian Weimer f...@deneb.enyo.de
Cc: nanog@nanog.org nanog@nanog.org
Sent: Sun Aug 29 01:12:00 2010
Subject: Re: Did your BGP crash today?

On Sat, Aug 28, 2010 at 6:14 AM, Florian Weimer f...@deneb.enyo.de wrote:
 * Christopher Morrow:

 (you are asking your vendors to run full bit sweeps of each protocol
 in a regimented manner checking for all possible edge cases and
 properly handling them, right?)

 The real issue is that both spec and current practice say you need to
 drop the session as soon as you encounter any unexpected data.  That's

sorry, I conflated two things... or didn't mean to but did anyway.

1) users of gear that does BGP really need to ask loudly and longly
(and then go test for themselves) that their BGP speakers do the
'right thing' when faced with oddball scenarios. If someone sends you
a previously unknown attribute... don't corrupt it and pass it on,
pass if transitive, drop if not.

2) some thought and writing and code-changes need to go into how the
bgp-speakers of the world deal with bad-behaving bgp speakers. Is
'send notify and reset' the right answer? is there one 'right answer'
? Should some classes of fugly exchange end with a 'dropped that
update, moved along' and some end with 'pull eject handle!' ?

it's doubtful that 2 can get solved here (nanog, though certainly some
operational thought on the right thing would be great as guidance). i
would hope that 1 can get some traction here (via folks going back to
their vendors and asking: Did you run the Mu-security/Oolu-univ/etc
fuzzing test suites against this code? can I see the results? I hope
they match the results I'm going to be getting from my folks in
~2wks... or we'll be having a much more structured/loud
conversation...

another poster had a great point about 'all the world can screw with
you, you have no protections other than trust that the next guy won't
screw you over (inadvertently)'. There are no protections available to
you if someone sets (example) bit 77 in an ipv4 update message to 1
when it should by all accounts be 0. Or (apparently) if they send a
previously unknown attribute on a route :( You can put in max-prefix
limits, as-path limits (length and content), prefix-filters.. but
internal-message-content you are stuck hoping the vendors all followed
the same playbook. With everyone saying together: Please
appropriately test your implementation for all boundary cases maybe
we can get to where these happen less often (or nearly never) - every
3 months is a little tedious.

-chris

Re: off-topic: summary on Internet traffic growth History

[Deepak Jain]

On my BB. I'm waiting for someone to correct this thread by saying MFS bought 
UUNET for ~2bill and WCOM absorbed MFS.

That is all.


- Original Message -
From: Jeffrey S. Young yo...@jsyoung.net
To: John Lee j...@internetassociatesllc.com
Cc: nanog@nanog.org nanog@nanog.org; Andrew Odlyzko odly...@umn.edu
Sent: Wed Aug 11 19:26:29 2010
Subject: Re: off-topic: summary on Internet traffic growth History

Worldcom bought MFS.
Worldcom bought MCI.
Worldcom bought UUnet.

In your statement s/MCI/Worldcom/g

I don't know if UUnet was part of Worldcom when MO first made statements about 
backbone growth, but I do know that internetMCI was still part of MCI and 
therefore, MCI was not a part of Worldcom.  May seem like splitting hairs to 
some, but it is important to a few of us to point out that we never worked 
under Ebbers.  Not that we had a choice :-).  

Growth of the NAPs during this period is a poor indicator of growth.  Because 
of the glitch you mention in carrying capacity the tier 1's all but abandoned 
the NAPs for peering between themselves and from that point forward (mid '97) 
preferred direct peering arrangements.

jy

On 12/08/2010, at 4:13 AM, John Lee j...@internetassociatesllc.com wrote:

 Andrew,
 
 Earlier this week I had a meeting with the ex-Director of the Network 
 Operations Center for MFS-Datanet/MCI whose tenure was through 1999. From 
 1994 to 1998 they were re-architeching the Frame Relay and ATM networks to 
 handle the growth in traffic including these new facilities called peering 
 points of MAE-East and MAE-West. From roughly 1990 to then end of 1996 they 
 saw traffic on their switches grow at 50-70% growth every 6 months. By the 
 last half of 1996 there was a head of line blocking problem on the DEC FDDI 
 switches that was regularly bringing down the Internet. The architecture 
 had lower traffic circuits were going through concentrators while higher 
 traffic circuits were directly attached to ports on the switchs.
 
 
 
 MFS-Datanet was not going to take the hit for the interruptions to the 
 Internet and was going to inform the trade press there was a problem with DEC 
 FDDI switches so Digital gave six switches for the re-architecture of the 
 MAEs to solve the problem. Once this problem was solved the first quarter of 
 1997 saw a 70% jump in traffic that quarter alone. This historical event 
 would in my memory be the genesis of the 100% traffic growth in 100 days 
 legend. (So it was only 70% in 90 days which for the marketing folks does not 
 cut it so 100% in 100 days sounds much better?? :) )
 
 
 
 MCI bought MFS-Datanet because MCI had the customers and MFS-Datanet had all 
 of the fiber running to key locations at the time and could drastically cut 
 MCI's costs. UUNET merged with MCI and their traffic was put on this same 
 network. MCI went belly up and Verizon bought the network.
 
 
 
 Personal Note: from 1983 to 90 I worked for Hayes the modem folks and became 
 the Godfather to Ascend communications with Jeanette, Rob, Jay and Steve 
 whose team produced the TNT line of modem/ISDN to Ethernet central site 
 concentrators (in the early ninties) that drove a large portion of the user 
 traffic to the Internet at the time, generating the bubble.
 
 
 
 John (ISDN) Lee
 
 From: Andrew Odlyzko [odly...@umn.edu]
 Sent: Wednesday, August 11, 2010 12:55 PM
 To: nanog@nanog.org
 Subject: off-topic: summary on Internet traffic growth myths
 
 Since several members of this list requested it, here is a summary
 of the responses to my request for information about Internet growth
 during the telecom bubble, in particular the perceptions of the
 O'Dell/Sidgmore/WorldCom/UUNet Internet doubling every 100 days
 myth.
 
 First of all, many thanks to all those who responded, on and off-list.
 This involved extensive correspondence and some long phone conversations,
 and helped fill out the picture of those very confusing times (and
 also made it even clearer than before that there were many different
 perspectives on what was happening).
 
 The entire message is rather long, but it is written in sections,
 to make it easy to get the gist quickly and neglect the rest.
 
 Andrew
 
 
 ---
 
 1.  Short summary: People who got into the game late, or had been
 working at small ISPs or other enterprises, were generally willing
 to give serious credence to the Internet doubling every 100 days
 tale.  The old-timers, especially those who worked for large ISPs
 or other large corporate establishment or research networks, were
 convinced by the late 1990s that this tale was false, but did not
 talk about it publicly, even inside the NANOG community.
 
 ---
 
 2.  Longer version: The range of views was very wide, and hard to
 give justice to in full.  But there seemed to be two distinct
 groups, and the 

RE: Proxy Server

[Deepak Jain]

 
 I am fairly sure Squid has the concept of bandwidth pools which you can
 apply via ACLs within the squid conf.
 That may meet your proxy requirements but would not help with traffic
 not being proxied.
 
 Squid will also allow you to define access to the inet based on ACLs
 which can use various things to determine which policy will be applied
 to the connection.  eg,  client src IP,  client username,  time of day,
 regx...
 
 you may find it here:
 
 http://www.squid-cache.org/
 
---

Squid plus bandwidth management (ala dummynet or similar) could go a long way 
to addressing all of those functions.

Deepak Jain
AiNET

RE: IPv4 Exhaustion...

[Deepak Jain]

 CALEA is not a time machine.  When an order is received, the
 collection
 agency starts receiving traffic; nothing (or at most, very little) is
 known prior to the wiretap order.  Put another way, you cannot be
 ordered
 to produce tapes of phone call that happened a month ago. (CALEA only
 says
 you must have the ability to monitor anyone; not that you must be
 monitoring everyone to have stuff available before being asked for
 it.)
 

Another point about CALEA is that you don't *have* to have infrastructure in 
place in advance of the order. You simply have to provide Law Enforcement the 
wiretaps they are asking for -- however you accomplish it. You don't need to 
solve for every case, just the case they ask of you at the time. This keeps the 
cost of compliance way down (provided you don't need these for a significant 
percentage of your user base). 

Between e-discovery and RIAA issues, retention times are probably shrinking 
even though capacity for retention is growing.

Deepak Jain
AiNET

RE: IPv4 Exhaustion...

[Deepak Jain]

 
 I see this asked a lot...
 
 http://www.askcalea.net/reports/wiretap.html
 
 [2009] http://www.askcalea.net/reports/docs/2009wiretap.pdf (warning:
 314pg verbose report)

To save yourself the trouble (pg 8 of the slow 5MB download):

Telephone wiretaps accounted for 98 percent (1,720
cases) of the intercepts installed in 2009, the majority
of them involving cellular telephones.

I think it's safe to say CALEA is a non-issue for this crowd. It's also safe to 
say that RIAA is more of a nuisance than a real issue. Manage your retention 
times with e-discovery in mind and you don’t need to worry about RIAA issues 
either. :)

RE: Recommendation in Australia for ISPs to force user security?

[Deepak Jain]

Come on, you aren't thinking gov't-enough.

BASIC broadband access will be a SSH/web-only proxy with 
firewalling/antivirus/etc capability. That whole pesky HTTP/1.0 problem was 
solved a long time ago. Maybe you don't even get your own IP anymore -- and you 
have to access your email through their web portal too. This also qualifies you 
as net-neutral in that everyone gets the same poor service. Only content 
providers that sign an agreement to be free of virii and malware (with an 
appropriate inspection/sanitization charge will be let through... e.g. 
Netflix or whomever) -- this way, you aren't being made to differentiate 
between bits, you are being made to ensure national security.

BUSINESS broadband access might give you a real IP, allow you to torrent, but 
you sign a piece of paper that authorizes them to charge you if you get 
infected, or better yet, a maintenance plan of a $24.95/month on top of your 
service to make sure you don't get infected with a remotely managed 
firewall/router or whatever will meet the definition of the regulation.

This can be solved so fast it'll make your head spin. Build a big proxy 
cloud, send everyone 60 days notice once the regulation comes in effect, on 
day 61 throw the switch. Day 62, collect orders for the upgraded service. 
*PROFIT*

My only shock is that Washington isn't leading Canberra on this, with an even 
faster timeline than the one above.

Deepak

 -Original Message-
 From: Joel Jaeggli [mailto:joe...@bogus.com]
 Sent: Tuesday, June 22, 2010 2:58 PM
 To: Gadi Evron
 Cc: nanog@nanog.org
 Subject: Re: Recommendation in Australia for ISPs to force user
 security?
 
 not sure how they propose to enforce that, instrumentation approaches
 that look inside the home gateway have a non-trivial falsh positive
 rate
 and you've got a lot more hosts than ip addresses.
 
 On 06/22/2010 11:30 AM, Gadi Evron wrote:
  http://www.zdnet.com.au/make-zombie-code-mandatory-govt-report-
 339304001.htm
 
 
  A government report into cybercrime has recommended that internet
  service providers (ISPs) force customers to use antivirus and
 firewall
  software or risk being disconnected.
  security
 
  Committee chair Belinda Neal said in her introduction to the 262-page
  report titled Hackers, Fraudsters and Botnets: Tackling the Problem
 of
  Cyber Crime that due to the exponential growth of malware and other
  forms of cybercrime in recent years, the expectation that end users
  should or can bear the sole responsibility for their own personal
 online
  security is no longer a tenable proposition.
 
  We need to apply the same energy and commitment given to national
  security and the protection of critical infrastructure to the
 cybercrime
  threats that impact on society more generally, she said.
 

RE: Hung Telnet Sessions on Sco Unix

[Deepak Jain]

 On 2010-05-27, at 20:47, jacob miller wrote:
 
  Am running an application on Sco Unix but am having the following
 problem.
 
  Application is hunging sporadically.
 
 That seems consistent with my memory of SCO Unix.
 

Me too, but I don't think this is the right list for it.

DJ

RE: ARIN IP6 policy for those with legacy IP4 Space

[Deepak Jain]

Now I may be talking crazy... 

IIRC, all of IPv4 space maps to a section of IPv6 space. 

mad hat on

If one has legacy IPv4 space, but actually talks IPv6 couldn't one announce a 
prefix much longer than a /64 to map them onto the IPv6 universe (assuming 
people would allow such craziness... perhaps on their IPv4 speaking routers) 
and originate/terminate traffic as normal? 

/mad hat off

Isn't this all left to the networks to enforce, as usual, but unlike the status 
quo, these are all valid allocations...

Technical note: I know this breaks lots of IPv6 goodness (no need to enumerate 
it here).

DJ

RE: OBESEUS - A new type of DDOS protector

[Deepak Jain]

At first blush, I would say it's an interesting idea but won't actually resolve 
anything of the scariest DDOS attacks we've seen. (Unless I've missed something 
obvious about your doodle).

The advantage/disadvantage of 100,000+ host drone armies is that they don't 
actually *have* to flood you, per se. 10 pps (or less) each and you are going 
to crush almost everything without raising any alarms based on statistically 
significant patterns especially based on IPs. Fully/properly formed HTTP port 
80 requests to / won't set of any alarms since each host is opening 1 or 2 
connections and sending keepalives after that. If you forcibly close the 
connection, it can wait 5 seconds or 15 minutes before it reopens, it doesn't 
really care. Anything that hits you faster than that is certainly obnoxious, 
but MUCH easier to address simply because they are being boring.

You *can* punt those requests that are all identical to 
caches/proxies/IDS/Arbor/what have you and give higher priority to requests 
that show some differences from them... but you are still mostly at the mercy 
of serving them unless you *can* learn something about the 
originator/flow/pattern -- which might get you into a state problem. 

Where this might work is if you are a large network that only serves one sort 
of customer and you'd rather block rogue behavior than serve it (at the risk of 
upsetting your 1% type customers). This would work for that. Probably good at 
stomping torrents and other things as well.

Best,

Deepak

 -Original Message-
 From: Guillaume FORTAINE [mailto:gforta...@live.com]
 Sent: Monday, March 15, 2010 2:57 AM
 To: nanog@nanog.org
 Subject: Re: OBESEUS - A new type of DDOS protector
 
 Dear Mister Wyble,
 
 Thank you for your reply.
 
 
 On 03/15/2010 07:00 AM, Charles N Wyble wrote:
  The paper is pretty high level, and the software doesn't appear to be
  available for download.
 
 
 http://www.loud-fat-bloke.co.uk/obeseus.html
 
 http://www.loud-fat-bloke.co.uk/tools/obeseusvB.tar.gz
 
 
 
  So it's kinda theoretical.
 
 
 
 
 We have it running parallel with a commercial product and it detects
 the following
 attacks
 ▪ SYN floods
 ▪ RST floods
 ▪ ICMP floods
 ▪ General UDP floods
 ▪ General TCP floods
 
 
 
 
 Best Regards,
 
 Guillaume FORTAINE
 

RE: How polluted is 1/8?

[Deepak Jain]

 If some unfortunate soul does get 1.1.1.1, 1.2.3.4, 1.3.3.7, etc, they
 would also likely experience significant global reachability problems
 in
 addition to all of the unintended noise that gets sent their way.
 
 There are many sites that specifically filter those addresses, in
 addition to those that don't update bogon filters, or assume no one
 will _ever_ get 1.2.3.4! :)


They would make great DNS server IPs for someone who wanted to host them. :)

Deepak

RE: Routing to multiple uplinks

[Deepak Jain]

 The overall design is being driven by our rigorous application needs
 more
 than anything.
 
 The implementation is straight forward we receive a duplicate set of
 feeds
 from site A and site B and can also access various services coming from
 site
 A or site B however, at any given time a user will be sending/recieving
 data
 from one of those destinations. Never both simultaneously.  So my
 question
 what is the best way to provide this type of redundancy at the host
 level?
 
 The application will only use one target address.
 

You've stated two seemingly contradictory things. 1) The User decides which 
paths to take yet the 2) application cannot see more than one path. 

First, this sounds like a User issue. The application with such rigorous 
requirements should have the features you need to manage this.

Barring that... :)

The mechanism the User uses to (manually) decide which path to take should make 
the election and handle the switchover in visibility. Presumably, since your 
application cannot tell when its switched destinations/paths it needs to be 
notified if the network makes a VRRP/HSRP decision. This all points to the 
mechanism you presumably already have in place for manual path decisions.

If you are using your Linux box or whatever to make your path choices, simply 
have a script that sees if the path preferences have changed and use your 
method to notify the Application.

If you are using a Cisco (or other) dedicated router, run something on the 
Application box or servers that will notice this change (if even by querying 
the router) so it can proactively detect this.

You've asked for a technical suggestion but have not provided any detail about 
the actual constraints you have -- though you've implied them without context.

Deepak Jain
AiNET

RE: Chinese bgp metering story

[Deepak Jain]

From the BBC article quoted in the isoc-ny.org link:

An ITU spokesman said: The ITU has no plans to modify the BGP protocol, which 
is not an ITU-T standard.

A proposal has been made, and is being studied, to use BGP routers to collect 
traffic flow data, which could be used, by bilateral agreement, by operators 
for billing purposes.



I read this to mean, no news here. If you want to move traffic, you need a 
bilateral agreement. 

That already exists. Where/if money flows, we know circuits don't build 
themselves for free, so the question of using money isn't a question. The only 
question is whether you are adjusting based on usage, or ports, or total speed, 
or direction of bits. 

ITU is already acknowledging that BGP isn't its baby, so it has nothing to say 
there. 

Deepak

RE: Optical fiber question

[Deepak Jain]

 My provider said they can provide single / mulit mode Optical fiber
 
  Apart from the length and cost different, what is the Adv/Disadv
  between them for our connection?
 
 The advantages are always in the distance capabilities of the single
 mode fiber.  You can reach much further on this, but the optics tend to
 be more expensive.  If you are going a short distance (eg: 2km or less)
 multi-mode is the way.  If you're going to go any further, or want to
 ever go any further, take the extra cost and know you can swap optics
 in the future to do gig, 10G and possibly more (in the future) with
 less pain.

Just to amplify Jared's very complete answer. The principle reason you would use
multimode instead of single mode is reduced cost. If cost isn't an issue, single
mode has more potential to be used in more applications. Even the longer range
SM optics can be used for short range uses with inexpensive attenuators. 

Service Providers support both because their customers may only support one or
the other.

Deepak Jain
AiNET 

RE: news from Google

[Deepak Jain]

 I think of this as an obvious (not necessarily beneficial for all, of
 course) step for a company which lives out of advertisement - i.e. what
 if
 they could capture your habits for browsing at the FQDN-to-IP time -
 wouldn't that add more to their knowledge base?
 

I think there are amazing opportunities to data mine and prevent fraud if you 
can get a percentage of your users using this. 

I'm really excited about the structured attacks that will be run against this 
thing (cache poisoning... and nastier)... if (for example) when their (or 
someone's) toolbar is installed, they ask if you'd like to use their improved 
dns service [perhaps they have the whole universe cached to reduce lookup 
times]. You'd sign up.

And as the wave of software updates proceeds... well, talk about all your eggs 
in one basket.

Smart ISPs will have an ACL ready to hijack external DNS requests for their 
whole network in the (inevitable) event something *bad* happens one day and you 
need to restore service to your customers faster than they can figure out how 
to fix it themselves. Just a thought.

Deepak

RE: news from Google

[Deepak Jain]

Or the whole turning over records from Youtube... 

Nothing prevents them from changing policies in the future when it becomes more 
difficult for millions of users to change away... (vis-à-vis the uproar when FB 
was going to change its privacy policy and more as it continues to do so).



 -Original Message-
 From: Ken Chase [mailto:m...@sizone.org]
 Sent: Thursday, December 03, 2009 5:29 PM
 To: nanog@nanog.org
 Subject: Re: news from Google
 
 You mean like this?
 
 http://arstechnica.com/telecom/news/2009/12/sprint-fed-customer-gps-
 data-to-leos-over-8-million-
 times.ars?utm_source=rssutm_medium=rssutm_campaign=rss
 
 and this?
 
 http://almartinraw.com/public/column417.html
 
 just wait til google sews up all voice communications.
 
 On Thu, Dec 03, 2009 at 04:49:39PM -0500, Andrey Gordon's said:
   sometimes google makes me think of all those futuristic movies where
 there
   is a single corporation running the world, everyone is 'tagged' and
 tracked
   24/7 and everyone who works for that corporation are happy campers
 and live
   in clean and modern neighborhoods and the rest of the people are
 scam of the
   earth and live in the sewer.
   IMHO that's where we are heading with google taking over every
 service
   imaginable. That's the feeling I get from google.
 
 /kc
 --
 Ken Chase - k...@heavycomputing.ca - +1 416 897 6284 - Toronto CANADA
 Heavy Computing - Clued bandwidth, colocation and managed linux VPS
 @151 Front St. W.

RE: FTTH Active vs Passive

[Deepak Jain]

 If, 10 years ago (1999) when most internet-connected homes still used
 dialup, you had suggested that ISPs would be putting in gigabit
 services
 to homes, people would have laughed.  Yet today, here we are talking
 about gig feeds.  I wonder how much bandwidth homes will be using 10
 years from now...
 

s/be using/have access to/

One could make the argument that when we were doing dial-up over POTS the % 
utilization vs port speed was higher than today with packet switching to the 
curb. People have been lamenting the lack of for-profit apps that will actually 
each up these 100+ mb/s residential pipes (the killer app). 

One could further argue that the talk of gigabit pipes to the home has been 
ushered in by the cost-effectiveness of gigabit ethernet over SONET or other 
technologies and this is why we are seeing such a massive increase in the port 
speeds to customers. As a percentage of pipe available (discounting things like 
kiddie's using Torrent), I wouldn't be surprised to see that percentage drop. 
(Residential broadband folks chime in please).

Deepak

Re: IPv6 internet broken, cogent/telia/hurricane not peering

[Deepak Jain]

Perhaps someone from HE can re-confirm their open peering policy for us?

If they aren't (open) anymore, I'm impressed by the bravado...

Deepak


- Original Message -
From: Marco Hogewoning mar...@marcoh.net
To: Patrick W. Gilmore patr...@ianai.net
Cc: NANOG list nanog@nanog.org
Sent: Mon Oct 12 12:15:34 2009
Subject: Re: IPv6 internet broken, cogent/telia/hurricane not peering


On Oct 12, 2009, at 6:09 PM, Patrick W. Gilmore wrote:

 It is sad to see that networks which used to care about  
 connectivity, peering, latency, etc., when they are small change  
 their mind when they are big.  The most recent example is Cogent,  
 an open peer who decided to turn down peers when they reached  
 transit free status.

 I never thought HE would be one of those networks.


Do we have any proof it's HE rejecting peering or is it that Cogent en  
Telia alike that are to proud to ask and think they can have a piece  
of the pie as they did with v4 ?

MarcoH

RE: cross connect reliability

[Deepak Jain]

[lots of stuff deleted]. 

We've seen cross-connects fail at sites like E and others. Generally 
speaking, it is a human-error issue and not a component failure one. Either 
people are being sloppy and aren't reading labels, or the labels aren't there. 

In a cabinet situation, every cabinet does not necessarily home back to its own 
patch panel, so some trashing may occur -- it can be avoided with good design 
[cables in the back stay there, etc].

When you are talking about optics failing and they are providing smart 
cross-connects, almost anything is possible. 

The true tell tale is whether you have to call when the cross-connect goes 
down, or if it just bounces. Either way, have them take you to their 
cross-connect room and show you their mess. Once you see it, you'll know what 
to expect going forward.

Deepak

RE: Link capacity upgrade threshold

[Deepak Jain]

 do any router vendors provide something akin to hardware latches to
 keep
 track of highest buffer fill levels?  poll as frequently/infrequently
 as
 you like...

Without getting into each permutation of a device's architecture, aren't buffer 
fills really just buffer drops? There are means to determine this. Lots of 
vendors have configurable buffer pools for inter-device traffic levels that 
record high water levels as well.

Deepak Jain
AiNET
 

RE: Data Center testing

[Deepak Jain]

The idea of regular testing is to essentially detect failures on your time 
schedule rather than entropy's (or Murphy's). There can be flaws in your 
testing methodology too. This is why generic load bank tests and network load 
simulators rarely tell the whole story.

Customers are rightfully unpleased with any testing that affects their normal 
peace-of-mind, and doubly so when it affects actual operational effectiveness. 
However, since no system can operate indefinitely without maintenance, failover 
and other items, the question of taking a window is not negotiable. The only 
thing that is negotiable (somewhat) is when, and only in one direction (ahead 
of the item failing on its own). 

So, taking this concept to networks. It's not negotiable whether a link or a 
device will fail, the question is only how long you are going to forward bits 
along the dead path before rerouting and how long that rerouting will take. 
SONET says about 50ms, standard BGP about 30-300seconds. BFD and other things 
may improve these dramatically in your setup. You build your network around 
your business case and vice versa. 

Clearly, most of the known universe has decided that BGP time is good enough 
for the Internet as a whole right now. Most are aware of the costs in terms of 
overall jitter, CPU and stability if we reduce those times too far. 

Its intellectually dishonest to talk about never losing a packet or never 
forwarding along a dead path for even a nanosecond when the state-of-the-art 
says something very different indeed. 

Deepak Jain
AiNET

 -Original Message-
 From: Dylan Ebner [mailto:dylan.eb...@crlmed.com]
 Sent: Wednesday, August 26, 2009 11:33 AM
 To: Dan Snyder; Ken Gilmour
 Cc: NANOG list
 Subject: RE: Data Center testing
 
 I would hope that the data center engineers built and ran suite of
 tests to find failure points before the network infrastructure was put
 into production. That said, changes are made constantly to the
 infrastructure and it can become very difficult very quickly to know if
 the failovers are still going to work. This is one place where the
 power and network in a datacenter divulge. The power systems may take
 on additional load over the course of the life of the facility, but the
 transfer switches and generators do not get many changes made to them.
 Also, network infrastructure tests are not going to be zero impact if
 there is a config problem. Generator tests are much easier. You can
 start up the generator and do a load test. You can also load test the
 UPS systems as well. Then you can initiate your failover. Network tests
 are not going to be zero impact even if there isn't a problem. Let's
 say you wanted to power fail a edge router participating in BGP, it can
 take 30 seconds for that routers route to get withdrawn from the BGP
 tables of the world. The other problem is network failures always seem
 to come from unexpected issues. I always love it when I get an outage
 report from my ISP's or datacenter and they say an unexpected issue
 or unforseen issue caused the problem.
 
 
 Dylan
 -Original Message-
 From: Dan Snyder [mailto:sliple...@gmail.com]
 Sent: Monday, August 24, 2009 8:39 AM
 To: Ken Gilmour
 Cc: NANOG list
 Subject: Re: Data Center testing
 
 We have done power tests before and had no problem.  I guess I am
 looking for someone who does testing of the network equipment outside
 of just power tests.  We had an outage due to a configuration mistake
 that became apparent when a switch failed.  It didn't cause a problem
 however when we did a power test for the whole data center.
 
 -Dan
 
 
 On Mon, Aug 24, 2009 at 9:31 AM, Ken Gilmour ken.gilm...@gmail.com
 wrote:
 
  I know Peer1 in vancouver reguarly send out notifications of
  non-impacting generator load testing, like monthly. Also InterXion
  in Dublin, Ireland have occasionally sent me notification that there
  was a power outage of less than a minute however their backup
  successfully took the load.
 
  I only remember one complete outage in Peer1 a few years ago... Never
  seen any outage in InterXion Dublin.
 
  Also I don't ever remember any power failure at AiNet (Deepak will
  probably elaborate)
 
  2009/8/24 Dan Snyder sliple...@gmail.com:
   Does any one know of any data centers that do failure testing of
   their networking equipment regularly? I mean to verify that
   everything fails over properly after changes have been made over
   time.  Is there any best practice guides for doing this?
  
   Thanks,
   Dan
  
 
 

RE: MTAs used

[Deepak Jain]

 Now, did you want that in terms of number of copies installed or
 amount of mail handled?   There's probably zillions of little Fedora
 and
 Ubuntu boxes running whatever MTA came off the disk that are handling 1
 or 2 pieces of mail a day, and then there's whatever backends are used
 by MSN/Hotmail, Yahoo, AOL, etc.  This MTA packed by weight, not by
 volume.
 Some settling of contents may have occurred during shipping and
 spamming.
 
 (Seriously - if 95% of the mail out there is spam, then the top 4-5
 MTAs are probably the ratware that's sending out the spam.  Something
 to consider...)

In keeping with this concept, and turning it around. What MTA is exposed to the 
most spam? (1-x) That should tell you what MTA handles the most good mail by 
also being the destination for the most spam (good, live recipients).

Or I could be missing something well known about mail flows.

Deepak

Re: FCCs RFC for the Definition of Broadband

[Deepak Jain]

Key characteristics of broadband : always on capability (reasonably, DSL ok, 
dial up no). I would argue 7mb is broadband even if its over carrier pigeon. 
(meets always on criteria).

I think the threshold for cut off is somewhere between 256kbit/s and 1.5mbit/s. 
If you don't think 1.5mbit is broadband, you need to consider tiers... Most of 
the worlds population will not see *that* speed in 20yrs.

Deepak

- Original Message -
From: Jeffrey Lyon jeffrey.l...@blacklotus.net
To: nanog@nanog.org nanog@nanog.org
Sent: Wed Aug 26 19:09:47 2009
Subject: Re: FCCs RFC for the Definition of Broadband

I would argue that broadband is the upper X percentile of bandwidth
options available to residential users. For instance, something like
Verizon FiOS would be broadband while a 7 Mbps cable wouldn't.

Jeff

On Wed, Aug 26, 2009 at 6:39 PM, Richard Bennettrich...@bennett.com wrote:
 They have a saying in politics to the effect that the perfect is the enemy
 of the good. This is a pretty good illustration. We have the opportunity to
 improve connectivity in rural America through the wise expenditure of
 taxpayer funding, and it's best not to squander it by insisting on top-shelf
 fiber or nothing at all. Let's push the fiber a little deeper, and bridge
 the last 20,000 feet with something that won't be too expensive to replace
 in 3-5 years. The budget ($7B) just isn't there to give every barn some nice
 GigE fiber, even though it would make the cows happy.

 Richard Bennett

 -Original Message-
 From: Joe Abley [mailto:jab...@hopcount.ca]
 Sent: Wednesday, August 26, 2009 1:42 PM
 To: Fred Baker
 Cc: nanog@nanog.org
 Subject: Re: FCCs RFC for the Definition of Broadband


 On 26-Aug-2009, at 13:38, Fred Baker wrote:

 If it's about stimulus money, I'm in favor of saying that broadband
 implies fiber to the home.

 I'm sure I remember hearing from someone that the timelines for disbursement
 of stimulus money were tight enough that many people expected much of the
 money to remain unspent.

 Does narrowing the scope of the funding to mandate fibre have the effect of
 funding more and better infrastructure, or will it simply result in less
 money being made available? Does it matter?








-- 
Jeffrey Lyon, Leadership Team
jeffrey.l...@blacklotus.net | http://www.blacklotus.net
Black Lotus Communications of The IRC Company, Inc.

Platinum sponsor of HostingCon 2010. Come to Austin, TX on July 19 -
21 to find out how to protect your booty.

RE: Data Center testing

[Deepak Jain]

Thanks for the kind words Ken.

Power failure testing and network testing are very different disciplines. 

We operate from the point of view that if a failure occurs because we have 
scheduled testing, it is far better since we have the resources on-site to 
address it (as opposed to an unplanned event during a hurricane). Not everyone 
has this philosophy. 

This is one of the reasons we do monthly or bimonthly, full live load transfer 
tests on power at every facility we own and control during the morning hours 
(~10:00am local time on a weekday, run on gensets for up to two hours). Of 
course there is sufficient staff and contingency planning on-site to handle 
almost anything that comes up. The goal is to have a measurable good outcome 
at our highest reasonable load levels [temperature, data load, etc].

We don't hesitate to show our customers and auditors our testing and 
maintenance logs, go over our procedures, etc. They can even watch events if 
they want (we provide the ear protection). I don't think any facility of any 
significant size can operate differently and do it well.

This is NOT advisable to folks who do not do proper preventative maintenance on 
their transfer bus ways, PDUs, switches, batteries, transformers and of course 
generators. The goal is to identify questionable relays, switches, breakers and 
other items that may fail in an actual emergency.

On the network side, during scheduled maintenance we do live failovers -- 
sometimes as dramatic as pulling the cable without preemptively removing 
traffic. Part of *our* procedures is to make sure it reroutes and heals the way 
it is supposed to before the work actually starts. Often network and topology 
changes happen over time and no one has had a chance to actually test all the 
glue works right. Regular planned maintenance (if you have a fast reroute 
capability in your network) is a very good way to handle it. 

For sensitive trunk links and non-invasive maintenance, it is nice to softly 
remove traffic via local pref or whatever in advance of the maintenance to 
minimize jitter during a major event. 

As part of your plan, be prepared for things like connectors (or cables) 
breaking and have a plan for what you do if that occurs. Have a plan or a 
rain-date if a connector takes a long time to get out or the blade it sits in 
gets damaged. This stuff looks pretty while its running and you don't want 
something that has been friction-frozen to ruin your window.

All of this works swimmingly until you find a vendor (X) bug. :) Not for the 
faint-of-heart. 

Anyone who has more specific questions, I'll be glad to answer off-line. 

Deepak Jain
AiNET

 I know Peer1 in vancouver reguarly send out notifications of
 non-impacting generator load testing, like monthly. Also InterXion
 in Dublin, Ireland have occasionally sent me notification that there
 was a power outage of less than a minute however their backup
 successfully took the load.
 
 I only remember one complete outage in Peer1 a few years ago... Never
 seen any outage in InterXion Dublin.
 
 Also I don't ever remember any power failure at AiNet (Deepak will
 probably elaborate)
 
 2009/8/24 Dan Snyder sliple...@gmail.com:
  Does any one know of any data centers that do failure testing of
 their
  networking equipment
  regularly? I mean to verify that everything fails over properly after
  changes have been made over
  time.  Is there any best practice guides for doing this?
 
  Thanks,
  Dan
 

RE: Data Center testing

[Deepak Jain]

 At least once a year I like to go out and kick the service entrance
 breaker to give the whole enchilada an honest to $diety plugs out test.
 As you said, not recommenced if you don't maintain stuff, but that's
 how
 confident I feel that my system works.

Nature has a way of testing it, even if you don't. :)

For those who haven't seen this occur, make sure you have a plan in case your 
breaker doesn't flip back to the normal position, or your transfer switch stops 
switching (in either direction -- for example, it fuses itself into the 
generator/emergency position).

For small supplies (say 1MW) it's not as big a deal, but when the breakers in 
a bigger facility can weigh hundreds of pounds each and can take months to 
replace, these are real issues and will test your sparing, consistency and 
other disciplines.

Deepak Jain
AiNET

RE: TransAtlantic 40 Gig Waves

[Deepak Jain]

 
  Well, the funny thing is that when I approached bandwidth buyers at
  some well known publicly traded carriers, they told me that 40 gig
  waves across the Atlantic were impossible.
 
 Theoretically impossible, or just impossible on the fiber that's
 already underwater? Big difference there.
 
  Indeed, when we decided to launch LAN PHY 10 GigE, the builder of our
  cable system told us it wasn't possible.
 
 Again, was this impossible on a cable the builder was about to build,
 or impossible on the cable that the builder put under the water
 already?

1) 40G across large bodies of water is a nice achievement, kudos to everyone 
involved everywhere.

2) Even on cables that are already deployed where impossible things couldn't 
happen, have. It just takes a little longer for the technology involved. Case 
in point, I point to DSL over legacy (old) copper plants. Even super old fiber 
in the ground can do things that were originally considered impossible. 

For existing undersea systems (whomever owns them) inline amplifiers as well as 
cable issues need to be ironed out. Now that folks know the cost of rolling out 
a new cable vs the cost of engineering specialized solutions for those sorts of 
spans, they have decisions to make...  but I have (and it has been proven) 
faith in technology to the impossible works... it just takes some time. :) 
25Ghz spacing wasn't possible 5 years ago either. And there is hfDWDM spacing 
in labs somewhere too.

So other than for the folks that are provisioning 40G or who are considering 
deploying their own systems... what's the big deal?

Deepak

RE: Nanog mentioned on BBC news website

[Deepak Jain]

 in the case of intervening entities, it is true that they have no link
 to
 the sender or receiver.  my packets from office to home can traverse at
 3
 or more networks that are not paid by me, or my company.
 
 they likely have contracts or obligations with their immediate
 neighbours,
 which is basically why the system continues to work.
 

I'm not sure if this is the benefit for the lurkers or the old guys, or will 
eventually get recycled in the press and give me a headache, but here goes. 

I think what people seem to keep skipping over is the concept that packets 
generated from A go to ISP B who has relationship with C... to pass packets 
to Z. From the point of view of C all packets from B (including A) are 
just B's traffic. It's not as simple as I have an agreement with my neighbor 
and we pass slop around. 

If I am C, whatever my neighbor is moving is essentially of equal value in my 
agreement with my neighbor (until one of us chooses to renegotiate it: i.e. 
peering dispute, whatever). No matter which A is sending it to B.  I don't 
*really* get the option to pick and choose on a per packet basis.

In the case of three intervening networks, each is aggregating their customers' 
traffic and passing the relevant portions to the neighboring network 
(presumably for *their* aggregated customers' traffic). 

This is, in some ways, fundamentally different than the US highway system, 
where if I'm driving a truck between one state and another, the next state 
(even though they have interconnection agreements) can set different rules on 
me than the state I just left. I know this happens with (for example) Michigan 
and its neighbors. 

In the Internet context, my neighbor is responsible to abide by our agreement 
and prevent the traffic coming over to me from violating that agreement and I 
am allowed to police and enforce that border any way I want. 

What this means is that if A is affected by something, from my perspective as 
C, B is absolutely authoritative for the discussion about A's traffic and 
what to do with it. (No matter how many B's A has contracted with, B and C do 
not have to ask A for permission for ways/means/methods to move packets). We 
can agree to drop it on the floor, give it priority or special treatment or 
generally just ignore it and let the packets pass the way they will. 

This how the so-called community volunteers have so much ability to affect 
and improve the system. Everyone operates in their own fiefdom owing little 
allegiance (other than those of commerce and equity) to its neighbors. I may 
charge a tariff to enter my fiefdom, but once packets enter my fiefdom, they 
are my packets. I protect them, and try to speed them on their way without 
impediment and I negotiate with others on their behalf to improve their 
happiness.

And continuing the micro-economics analogy... this is why periodic wars break 
out between larger fiefdoms and there is little way to influence them to play 
for the good of the system. The only way to influence them is for their own 
good.

DJ

P.S. I've been scratching my head and wondering what this TED thing is all 
about, it seems like a big cheerleading thing..

RE: Quick question about inbound route-selection

[Deepak Jain]

 As for trying to determine where your inbound traffic is coming from by
 looking at natural bgp, this is absolutely impossible to do correctly.
 First off, your inbound is someone else's outbound, and the person
 sending the traffic outbound is in complete and total control. The vast
 majority of the traffic on the Internet is being picked by local-prefs
 based on policies like what does this make/cost me monetarily or
 which major networks can I grab in a simple as-path regexp to balance
 some traffic. But even if you ignore all of that, the natural path
 selection is based on criteria which is specific to the other network
 or
 even to a specific session which you can't possibly know about remotely
 (e.g. their router id).

Another way to say what Richard is getting at (which was full of good 
information) is:

Just because you aren't modifying what your BGP process sees, at this stage of 
the Internet's maturity, it is safe to assume almost everyone else is. 
Therefore, rather than pray for BGP to make a logical selection, even though 
its *probably* being fed prefs based on other people's engineering, you should 
take charge of the parts you can.

HTH,

Deepak Jain
AiNET

Level 3 (was: legacy Wiltel/Looking Glass bandwidth)

[Deepak Jain]

Without continuing the L3 pile-on, one can easily glean from their public 
filings that they have never properly filled out their management depth in 
acquisition absorption and/or sufficiently empowered those folks. The billions 
in revenue lost from acquisitions like Genuity and others have told this story 
more than once. 

L3 is not alone in this. Worldcomm's failure to integrate acquisitions led to a 
much larger operational cash need than VZ has shown for the same assets (verio, 
lots of other names here). This is because VZ understands how traditional 
businesses acquire others, better, in my opinion. 

Unfortunately, L3 has shown little interest in making the real world, tough 
business cuts in heads and absorbing the real (internal) pain of acquisitions 
and seems to have a pretty laissez-faire attitude towards its customers, even 
at its senior management levels (Cxx). I think this will be (and has been) the 
biggest problem for them. Even a possible merger/JV with Sprint may not be 
sufficient to solve that. Their resolution of billing disputes is much more 
typical of WCOM than VZ. 

They are a big fish in lots, and lots, of markets. They enjoy being able to 
dictate pricing in them. IMO, however, they don't have the maturity of (say, 
ATT or others) to take that big fish status and leave you still happy with the 
service. (colloquially: if [good companies] are going to take advantage, at 
least they don't make it more painful than necessary).

Operationally, where you have options (because of pricing, locality, etc) it's 
long-term good to support competitors, diversity in connectivity, etc. History 
has shown time and time again that when an industry consolidates a lot of 
business with a certain vendor, bad things can and do occur.

Deepak Jain
AiNET

RE: Unicast Flooding

[Deepak Jain]

 After debugging the problem we added mac-address-table aging-time
 14400 to our data center switches. That syncs the mac aging time to
 the same timeout value as the ARP timeout

This helps, seconded.

Deepak Jain
AiNET

Re: Eye protection in DWDM systems -- what threshold?

[Deepak Jain]

Leo Bicknell wrote:

In a message written on Tue, Jun 09, 2009 at 01:06:42PM -0500, Richard A 
Steenbergen wrote:
The only problem with those funny signs is they scare remote hands techs 
into never looking at a fiber because they don't want to try and 
understand the difference between a SX GBIC and a class 3 ultra longhaul 
amp.


Save your poor techs eyes, and make them more reliable all at the same
time:

http://search.newport.com/?sku=F-IRC2-F



This conversation has gone places I didn't expect. Leo, that card is 
pretty cool, but for a few hundred $$ more, you can get a light meter 
(if someone is smart enough to use the card...)


Does anyone *use* any eye protection (other that not looking at the 
light, turning off the light etc) -- I mean like protective goggles, 
etc, when doing simple things like adding/removing patch cables from an 
SMF patch panel.


I get that if you *know* the gear you are using has a Class 3 laser on 
it, you should be careful... but when you are patching it into a 
building's cable plant and some schmuck is patching the last leg in for 
you (or has pulled it accidentally, etc).. um, don't look at it is our 
community's BCP?


DJ

Eye protection in DWDM systems -- what threshold?

[Deepak Jain]

At what power level do DWDM systems become dangerous to work near (i.e. not 
staring into any optics, using light meters, etc)? I never see technicians on 
inside DWDM systems using eye protection, but I see power levels of amps going 
higher and higher. On a recent meter I saw almost .6mW... 

Any pointers to a document saying 1550nm becomes dangerous at  dbM?

Thanks in advance,

DJ

RE: OT: ARIN contact

[Deepak Jain]

  I know this is off-topic, but I know some people from ARIN read this
 and
  would appreciate it if someone from ARIN would contact me off-list.
 
 hostmas...@arin.net did not respond to email?  this would be
 *extremely*
 unusual.
 

2nd'ed. ARIN is very responsive by email and telephone now.

Deepak

Re: ftc shuts down a colo and ip provider

[Deepak Jain]

What does it say about these providers AUP that the FTC needed to go to court 
to turn them off?

The AUP standard is usually written much, much lower. 

Deepak 

Deepak 

- Original Message -
From: Randy Bush ra...@psg.com
To: North American Network Operators Group na...@merit.edu
Sent: Fri Jun 05 00:38:04 2009
Subject: ftc shuts down a colo and ip provider

http://voices.washingtonpost.com/securityfix/2009/06/ftc_sues_shuts_down_n_calif_we.html

while allegedly a black hat, this is the first case i know of in which
the usg has shut down an isp.  nose of camel?  first they came for ...

randy

RE: Fiber cut - response in seconds?

[Deepak Jain]

 No. And here's why: If you're a naughty foreign intelligence team, and
 you know your stuff, you already know where some of the cables you'd
 really like a tap on are buried. When you hear of a construction
 project
 that might damage one, you set up your innocuous white panel truck
 somewhere else, near a suitable manhole. When the construction guy with
 a backhoe chops the cable (and you may well slip him some money to do
 so), *then* you put your tap in, elsewhere, with your actions covered
 by
 the downtime at the construction site. That's why the guys in the SUVs
 are in such a hurry, because they want to close the window of time in
 which someone can be tapping the cable elsewhere.
 
 At least that's what I heard. I read it somewhere on the internet.
 Definitely. Not at all a sneaky person. No sir.

And if you were a naughty foreign intelligence team installing a tap, or a 
bend, or whatever in the fiber contemporaneously with a known cut, you could 
also reamplify and dispersion compensate for the slight amount of affect your 
work is having so that when its tested later, the OTDR is blind to your work.

Ah, the fun of Paranoia, Inc.

Deepak Jain
AiNET

RE: Fiber cut - response in seconds?

[Deepak Jain]

 
 Really? I don't think so. I imagine it would be much more dependent on
 the amount of computing power the attacker has access to. More
 encrypted
 blobs won't help. If that was the case then the various encryption
 schemes in wide use today would be cracked already. Bad guys can setup
 networks and blast data through it and have complete access. I don't
 see
 them cracking encryption.

Without getting into the math involved, Vlad (and others) are correct. This is 
why there is key migration (regeneration/renegotiation/repudiation) along these 
multi-gigabit/multi-terabit streams. 

Your obfuscation strength (I don't care how many digits you have in your key, 
your cipher, what have you) is computed against the amount of data you are 
obfuscating. If I am obfuscating 1 byte of data, my math functions do not need 
to be as large as obfuscating 2^128 bits. 

There are plenty of non-classified books regarding COMSEC, INFOSEC and all 
their related interworking bits (even COMINT, SIGINT and HUMINT). Plenty of 
NANOG folks have been in these communities and that is why they say things that 
make sense regarding physical and network security. Even if you haven't been in 
these groups, the non-classified books are sufficiently sophisticated as to 
give even a layperson a respect for the layers of security (and the discipline 
behind it) needed to provide even the most minimal level of protection.

The h4x0r kids who think magnets on their doorways, tin foil hats, or 
willy-nilly encryption using their email-exchanged PGP keys are protected are 
welcome to their sandbox too -- let's just keep it away from those of us who 
like things that provably work [most of the time ;)].

DJ

RE: Fiber cut - response in seconds?

[Deepak Jain]

 
 Really? The US Military uses a whole lot of wireless (satellite, ground
 baed, surface to air) links. Those links can be sniffed (by people with
 sufficient motivation/funding/gear to do so). They rely on encryption
 to
 protect them.

Which is why, if you have a satellite, you often position DIRECTLY over the 
antenna you are sending to, and using lasers (rather than other RF) to 
communicate with it. Likewise, if you want to maintain this kind of security 
(and reduce the ability to sniff) you do this in space as well. Highly 
columnated photons are your friend.

Encryption helps, but if it was sufficient in all cases, you wouldn't go to 
such extremes.

This (in a much more NANOG related way) has ramifications for those 
selling/operating Wi-Fi, WiMax, P2P and FSO wireless links and trying to do 
*commercially important things* -- like finance.

The idea here is that fiber is FAR more secure than copper because almost 
everything you want to do to fiber, you can do to copper, but from a further, 
less physically-in-contact distance. 

Another idea is that commercially operated networks have lower standards for 
data security (but not necessarily data *integrity*) that intelligence 
*oriented* applications/networks. 

The idea of installing a tap on an encrypted line to do traffic analysis is all 
very interesting, but no one mentioned the idea that at a critical time (such 
as an attack) you could easily DISRUPT vital communications links and prevent 
their function [and their protected paths]. Security cannot exist without a 
level of integrity. Most commercial networks only need to concern themselves 
with integrity and let their customers deal with the security of their own 
applications.

Commercial networks are a great study of highly (in the commercial sense) 
secure data traversing over LSAs (lower sensitivity areas) with lower control 
thresholds [think poles, manholes, etc]. The data is highly secure to any 
particular customer, but in the commercial sense, it's almost always lost in 
the noise. When a business entity crosses that threshold (e.g. the Federal 
Reserve banks or a transaction clearinghouse) where their data is *worth* 
getting at no matter how much sifting has to go on... you see extraordinary 
measures (e.g. properly implemented obfuscation, or what have you) implemented.

Deepak Jain
AiNET

RE: Fiber cut - response in seconds?

[Deepak Jain]

 Once upon a time, Deepak Jain dee...@ai.net said:
  Which is why, if you have a satellite, you often position DIRECTLY
  over the antenna you are sending to
 
 Unless your target is on the equator, you don't position a satellite
 directly over anything.
 

I promise you that that is not the case for all applications. Geosynchronous 
satellites can be anywhere. For the applications you are considering 
(communications mostly), equatorial orbit is the most advantageous. 

There are books documenting other locations and reasons for other locations... 
and we are off topic.

Best,

Deepak Jain
AiNET

Re: Fiber cut - response in seconds?

[Deepak Jain]

I'm not sure why this sounds so surprising or impressive... given g$vt 
budgets.


Monitoring software using a pair of fibers in your bundle. OTDR or 
similar digital diagnostics. You detect a loss, you figure out how many 
feet away it is. You look at your map.


A simpler way to do it (if you don't mind burning lots of fiber pairs) 
would be to loop up a pair of fibers (or add a reflectance source every 
1000 ft or so -- spliced into the cable). You can figure out to within a 
thousand feet once you know WHICH set of loops has died.


Given it almost always involved construction crews, you drive until you 
see backhoes for your final approximation.


If I were the gov't I'd have originally opted for #2, and then moved to #1.

Seconds is just a function of how far away the responding agency's 
personnel ( monitoring the loop ) were from the cut. Obviously we are 
talking about a few miles tops.


Plenty of people used to have a single pair in each bundle for 
testing. Its relatively trivial to make that a test pair live. This is 
all predicated on you actually keeping your toplogy up-to-date.


Deepak Jain
AiNET

Charles Wyble wrote:



Joel Jaeggli wrote:

It's pretty trivial if know where all the construction projects on your
path are...


How so? Setup OTDR traces and watch them?



I've seen this happen on a university campus several times. no black
helicopters were involved.


Care to expand on the methodology used? A campus network is a lot 
different then a major metro area.

RE: Multi-homed clients and BGP timers

[Deepak Jain]

 If you want to converge a little fast than BGP holdtimes here
 and the fiber link is directly between the routers, you might
 look at something akin to Cisco's bgp fast-external-fallover,
 which immediately resets the session if the link layer is
 reset or lost.
 

Also things to consider: BFD for BGP and UDLD will help identify link failures 
faster. (If all of your equipment supports it, YMMV, etc).

Deepak

RE: IXP

[Deepak Jain]

So here is an idea that I hope someone shoots down.

We've been talking about pseudo-wires, and the high level of expertise a 
shared-fabric IXP needs
to diagnose weird switch oddities, etc.

As far as I can tell, the principal reason to use a shared fabric is to allow 
multiple connections to networks
that may not justify their own dedicated () router port. Once they do, they 
can move over to a PNI. However, an IXP is (at the hardware level at least) 
trying to achieve any-to-any connectivity without concern for capacity up to 
the port size of each port on every flow. Scaling this to multiple pieces of 
hardware has posed interesting challenges when the connection speed to 
participants is of the same order as the interconnection between IXP switches.

So here is a hybrid idea, I'm not sure if It has been tried or seriously 
considered before.

Since the primary justification for a shared fabric is cost savings

What if everyone who participated at an IXP brought their own switch. For 
argument's sake, a Nexus 5xxx. It has 20+ ports of L2, wire speed 10G.

You connect 1-2 ports on your router, you order 18 cross-connects to your 
favorite peers. The IXP becomes a cross-connect provider (there is a business 
model bump that can be addressed here, TelX and others could address it). As 
you need more ports, you add them. A Nexus 5K runs about $500 per port. 

Here are some advantages. If you have 300 participants, yes, you have a lot of 
ports/switches. However, as interconnectivity increases, so does the total 
fabric capacity. Each additional switch does NOT add significant
complexity to the participants, but it does bring significant backplane and 
buffering capabilities. Each participant could then configure their own pVlans, 
Vlans or whatever on *their* switch. If they screw something up, it doesn't 
take everyone down.  A non-offending participant that interconnects with an 
offender can shut down
1 port (automatically or manually) without affecting the rest of their 
services. 

This also prevents the requirement of very complicated security features in the 
L2/L3 gray area.  If you don't want your peer to have multiple MACs, don't 
accept them. If you're cool with it, you can let it slide. 

If you want to move someone over to a PNI, the IXP needs to do zilch. You just 
move your cross-connect over to a new port on your service window, your peer 
can do it at the same or a different time, no big deal. If you *keep* it on a 
switch however, you can use LACP uplinks from the switches you have to provide 
say 40G uplinks to your router so large peers don't affect your ability to 
process traffic. I doubt however, that if this model is applied, there is much 
purpose for PNIs -- the cost savings model mostly vanishes. 

As you want to move to higher speeds (40G and 100G) the IXP has to do zilch. 
You can switch your ports or peers at anytime you choose or set up a separate 
fabric for your 100G peers. An upgrade in port density or capacity for a peer, 
or set of peers, does not require a forklift of the whole IXP or some strange 
speed shifting (other than in the affected parties). 

Disadvantages. It's probably cheaper on a per-participant basis than a shared 
fabric once it gets to be a certain size. It's a different model (many-to-many 
vs one-to-many) that many are used to. It requires interconnects to other 
participants (en masse) to be about the same as the per port cost of a shared 
fabric (this is probably achievable given what many places charge for 10G 
ports). Each participant is managing an additional type of gear. Theoretically 
if someone uses an Extreme and another uses a Cisco, there might be issues, but 
at a pure vanilla-L2/VLAN level, I'm pretty sure even 2nd and 3rd tier vendors 
can interconnect just fine.

I think this addresses the keep it as simple as possible without over 
simplifying. There is nothing new to this model except (perhaps) as its applied 
to an IXP. People have been aggregating traffic by ports into trunks by 
capacity for a long time. I haven't figured out why it hasn't really been done 
to scale at the IXP level.

Thoughts?

Deepak Jain
AiNET

 -Original Message-
 From: vijay gill [mailto:vg...@vijaygill.com]
 Sent: Monday, April 20, 2009 12:35 AM
 To: Jeff Young; Nick Hilliard; Paul Vixie; na...@merit.edu
 Subject: Re: IXP
 
 If you are unfortunate enough to have to peer at a public exchange
 point, put your public ports into a vrf that has your routes. Default
 will be suboptimal to debug.
 
 I must say stephen and vixie and (how hard this is to type) even
 richard steenbergens methodology makes the most sense going forward.
 Mostly to prevent self-inflicted harm on parts of the exchange
 participants. Will it work? Doubtful in todays internet clue level
 
 /vijay
 
 On 4/18/09, Jeff Young yo...@jsyoung.net wrote:
  Best solution I ever saw to an 'unintended' third-party
  peering was devised by a pretty brilliant guy (who can
  pipe up

RE: Michael Mooney releases another worm: Law Enforcement / Intelligence Agency's do nothing

[Deepak Jain]

 On Sat, 18 Apr 2009 03:21:06 BST, andrew.wallace said:
  The network community and the security community need to collaborate
  as much as possible to defeat the threats.
 
  I'm British and i'm hoping to make UK as secure as possible.
 
 Umm. You missed the *very first* principle of proper security design.
 
 It shouldn't be as secure as possible. It should be as secure as it
 needs to be.
 
 I mean, I suppose you *could* go with mil-spec security, where all
 materials are kept in a locked safe under armed guard, and you had to
 fill out paperwork for each piece of paper you took out of the safe,
 and then more paperwork when you returned it.  But did you *really*
 want all that effort just to check the headlines on bbc.com?

Let's not ignore the fact that if you set unreasonably high security standards
most likely: a) twitter.com or bbc.com wouldn't exist because of the high
security scrutiny they'd have been under before being allowed to connect to 
anything and b) even if they didn't you wouldn't be able to see them because
of the high security scrutiny you'd be under before you were allowed to connect.

No one dies from an attack on twitter. Let the court/justice system deal with 
it whenever they get around to it. It keeps IT folks in jobs all over the 
place, gives the news things to write about, and gives the NANOG mail servers 
something to use the network for. 

Intelligence/security folks are tasked to deal with other things and with a 
real level of severity -- and it's quantifiable (at least in theory ;) ). 

Another point, security is ephemeral - A wall used to be the secure as 
possible solution to protect cities from invaders. An entertainment novelty in 
China rendered them obsolete when this black powder was reapplied to warfare. 
Some attacks (e.g. botnets) can only exist because we all have done a great job 
building networks over the last 15 years. Now we have new challenges. They all 
take their own time to mature and address.

Deepak Jain
AiNET

Re: IXP

[Deepak Jain]

Remember when you didn't want to put in ACLs because you'd blow out the cpu on 
the router/card?

Ah... That made networking fun!

Deepak 

- Original Message -
From: Jeff Young yo...@jsyoung.net
To: Nick Hilliard n...@foobar.org
Cc: Paul Vixie vi...@isc.org; na...@merit.edu na...@merit.edu
Sent: Sat Apr 18 20:45:48 2009
Subject: Re: IXP

Best solution I ever saw to an 'unintended' third-party
peering was devised by a pretty brilliant guy (who can
pipe up if he's listening).  When he discovered traffic
loads coming from non-peers he'd drop in an ACL that
blocked everything except ICMP - then tell the NOC to
route the call to his desk with the third party finally gave
up troubleshooting and called in...

fun memories of the NAPs...

jy


On Apr 18, 2009, at 11:35 AM, Nick Hilliard wrote:

 On 18/04/2009 01:08, Paul Vixie wrote:
 i've spent more than several late nights and long weekends dealing  
 with
 the problems of shared multiaccess IXP networks.  broadcast storms,
 poisoned ARP, pointing default, unintended third party BGP,  
 unintended
 spanning tree, semitranslucent loops, unauthorized IXP LAN  
 extension...
 all to watch the largest flows move off to PNI as soon as somebody's
 port was getting full.

RE: IXP

[Deepak Jain]

 Not agreeing or disagreeing with this as a concept, but I'd imagine
 that
 since a number of vendors support arbitrary vlan rewrite on ports that
 in simple environment you could do some evil things with that.  (ie.
 you could use QinQ like ATM Virtual Paths between core switches and
 then reuse the VLAN tag as a VC).  Then, as long as no peer has more
 than 4096 peers you're sweet. It'd hurt your head and probably never
 work, but heck, there's a concept to argue about.  (Please note: I
 don't
 endorse this as an idea).
 

This would be best managed by a very smart, but very simple piece of software.

Just like Facebook or LinkedIn, or what-have-you, a network accepts a 
peer/friend
request from another network. Once both sides agree (and only as long as both 
sides
agree) the configuration is pinned up. Either side can pull it down. The 
configs, up
to the hardware limits, would be pretty trivial.. Especially QinQ management 
for VLANID 
uniqueness.

Not sure how switches handle HOL blocking with QinQ traffic across trunks, but 
hey... 
what's the fun of running an IXP without testing some limits?

Deepak Jain
AiNET

RE: Leap second tonight

[Deepak Jain]

 
   As long as the end-user is made aware that the accuracy of said NTP
 clock
   is +/- 30.000 seconds (or whatever jitter might exist).  Seems kind
 of
   ridiculous to use an NTP source that is, for many purposes, wildly
   inaccurate.  For my purposes, wildly is more than +/- 0.1 seconds.
 Trying
   to troubleshoot a problem, network or server, where the timestamps on
 each
   server/router/device vary inconsistently, is like walking on broken
   fluorescent bulbs -- painful and dangerous to one's health.
 

Not being a time geek, since Cisco's were called out for being wild
jitter-mongers... how much jitter are we talking about?

Clock is synchronized, stratum 2, 
nominal freq is 250. Hz, actual freq is 249.9989 Hz, precision is 2**18
reference time is CD6A7CD4.45A9BB00 (19:47:32.272 UTC Tue Mar 17 2009)
clock offset is 2.0581 msec, root delay is 29.62 msec
root dispersion is 6.81 msec, peer dispersion is 3.30 msec

Are we talking about +/- 30 seconds, or a problem bounded by +/- 30 msec? 

Deepak Jain
AiNET

RE: Greedy Routing

[Deepak Jain]

 
 Maybe there's some critical insight in the paper that Physorg managed
 to totally not mention, I dunno.

I saw it the same way... 

 As the researchers explain, some types of networks are not navigable. For 
instance, if the probability that two nodes are linked doesn't depend on the 
metric distance between them, then such networks are difficult to navigate, as 
there is no way to choose one node over another based on distance. But when 
there is a connection between the link existence probability and the hidden 
distance between nodes, metric distances can help to navigate the network, 
i.e., such networks are navigable.

If your network doesn't calculate or use metrics or weights, or AS path 
lengths... then you are not able to
throw packets like fairy dust to their intended destination. Worse, if you use 
metrics unrelated to distance
(like link cost) you could actually send your packets the wrong way.

It's funny, but I think they said that their math shows that the Internet works 
to generally route packets
(to a shorter path) than other possible paths.

I'm sure that will come as a surprise to all of us.

Deepak Jain
AiNET

IPv6 space (was: RE: Private use of non-RFC1918 IP space )

[Deepak Jain]

 
 Which is exactly what they should do - actually before that one would
 hope.  This is not the $200/hour chcklehead consultant's fault, that
 is the design.
 
 Don't you love the idea of using 18446744073709551616 IP addresses to
 number a point-to-point link?
 

Let's not ignore that all IPv6 allocations are basically charged-for, so
my expectation is that there will be fewer idle allocations that can't
be recovered running around (when an org has to justify $36,000 per year [after 
2012],
forever, some bean counter may ask why... especially if they can get a
sensibly sized allocation from their provider for a fraction of that cost).

I'm not sure if that is cynical, or optimistic, but since the allocations
are not free, there seems to be less incentive to squat.

Deepak Jain
AiNET

-48VDC summary of responses

[Deepak Jain]

Lots of folks provided very good suggestions and information. Here is a brief 
attempt at a summary. 
I only got a few sales folks hitting me up, so you are probably on your own to 
get in touch
with most of these guys.


Top recommendation:
* Eltek/Valere seemed to be the top recommendation (3:1 or 4:1), though 
customer svc is rumored to have gone downhill since the acquisition).

Large Plants: (1000A and more)
** CC  Sageon (sageon: stand alone, not rack mount, CC scales in 100A 
increments)

Small plants:
Argus (re: Cordex unit- management UI only works in IE, emails just fine, good 
chassis based expandability) [2:1 recommendation here]
Tyco in the lower range
Telect for unmanaged supplies/PDUs

Lorain/Realtec makes nice equipment.

Thanks to everyone for their input, I don't have much more detail from most of 
the responses
so if you want a contact who is using this gear, I can try to make the 
introduction.

Deepak Jain
AiNET

RE: Security team successfully cracks SSL using 200 PS3's and MD5 flaw.

[Deepak Jain]

 Of course, this will just make the browsers pop up dialog boxes which
 everyone will click OK on...
 

And brings us to an even more interesting question, since everything is 
trusting their in-browser root CAs and such. How trustable is the auto-update 
process? If one does provoke
a mass-revocation of certificates and everyone needs to update their 
browsers... how do the
auto-update daemons *know* that what they are getting is the real deal? 

[I haven't looked into this, just bringing it up. I'm almost certain its less 
secure than the joke that is SSL certification].

Happy New Year!

Deepak

RE: Security team successfully cracks SSL using 200 PS3's and MD5 flaw.

[Deepak Jain]

 If done properly, that's actually an easier task: you build the update
 key into the browser.  When it pulls in an update, it verifies that it
 was signed with the proper key.
 

If you build it into the browser, how do you revoke it when someone throws 2000 
PS3s to crack it, or your hash, or your [pick algorithmic mistake here].

Deepak

RE: Security team successfully cracks SSL using 200 PS3's and MD5 flaw.

[Deepak Jain]

 ssl itself wasn't cracked they simply exploited the known vulnerable
 md5
 hashing.  Another hashing method needs to be used.

The encryption algorithm wasn't hacked. Correct. Another hashing method 
may help. Yup. 

My problem is with the chain-of-trust and a lack of reasonable or reasonably 
reliable (pick) 
ways of revoking certificates. 

Deepak

RE: Security team successfully cracks SSL using 200 PS3's and MD5 flaw.

[Deepak Jain]

 If you use bad crypto, you lose no matter what.  If you use good
 crypto, 2,000,000,000 PS3s won't do the job.
 

Even if you use good crypto, and someone steals your key (say, a previously 
in-access person) you need a way to reliably, completely, revoke it. This has 
been a problem with SSL since its
[implementation] inception. Lots of math (crypto) is good on paper and fails at 
the implementation stage.

Deepak

RE: an over-the-top data center

[Deepak Jain]

  But we aren't talking about the military here, are we? We are talking
 about an ISP on an ISP forum.
 
 
 Yes but in a disaster scenario where critical communication links
 are down the military would respond and reestablish the links, if for
 nothing else to re establish situational awareness for themselves.

This is getting off-topic in a big way, but I can pretty much assure you that 
the US military isn't going to be re-establishing ISP circuits for the 
military's situational awareness. I can't speak of the Swedish military. In 
most countries with a big-bad-military, the most the military will do is allow 
the commercial entities to expedite their own repairs and perhaps bypass 
certain permit requirements -- which is as it should be.

If this is the reason to build a bomb proof datacenter, I encourage all my 
competitors to do so.

Someone said it earlier, its far cheaper, and far more reliable to be massively 
redundant than super hardened in one (or a few) locations. If you think you 
can't afford the former, but can get the latter, you don't understand what you 
are solving for.

Deepak

RE: an over-the-top data center

[Deepak Jain]

Apologies to the list. 

I didn't know whether to fork this into a couple of replies, or just run with 
it. I chose the latter. 

1) This datacenter is only 12,000 sq ft. (submessage: who cares?)

2) The generators are underground. A leak in their exhaust system kills 
everyone -- worse, a leak in their fuel tank or filler lines (when being filled 
from above) could do the same. Yes, you could address this with alarms 
(provided they work and are tested, etc).

3) No one cares if the server farm is blast proof (it isn't), if the 
connectivity in/out of it gets blasted (submessage: silos were meant to deliver 
one thing, datacenters aren't in the same operational model once they need 
connectivity to the outside world)

4) With all of that fog and plant life, I wonder how they critically manage 
humidity. [Or if they even do].



To the question of carrier hotels and their supposed secrecy, etc. If you need 
connectivity to multiple providers, those providers know where the buildings 
are, and presumably so do most of their employees. If 500,000 people (say the 
top 10 companies together) know where the building is, it's not a secret. **

Carrier hotels aren't meant to be more secure than the lines coming into them. 
Those lines are coming in on unsecured poles, manholes and the rest. Their most 
dramatic failure modes are pretty obvious if not well-studied. Internet 
security [as in resilience] is built on the concept of a point-of-view of 
connectivity with multiple failures and routing around them -- NOT sacred nodes 
that cannot fail or universal end-to-end reachability. Internet security [as 
in integrity] is not something that's been proven on the Internet yet [general 
case, please no banter about encryption/quantum oscillation, etc].

Lots of people have already said this is dull -- it is, it is also a nice set 
of pictures.

** Submitted without proof. This covers all the buildings that make claims 
about not having their name on the door and have loading docks with no security 
on them. (you know who you are).

Deepak

Re: BCP for Private OUI / address assignments?

[Deepak Jain]

Realistically, OUI space is pretty large for each L2 domain... Once it 
hits an L3 domain, you can repeat OUIs all you want... Pick some prefix 
set of bits that include locally assigned that is unique to your 
organization and you will operationally be fine. Or the last 8 bits of 
your host server's base MAC (hardware) address -- plenty of entropy there.


Even when two or 10 organizations merge, unless you are merging them 
into the same tributary L3 domain, its not a problem...


And if it is a problem for more than 1 or 2 virtual machines, I'm sure 
your organization is big enough to split that into another 
domain/vlan/what have you.


Yes, its a nice idea to have a BCP that works without conflicts, but
I don't see how this is legitimately going to affect anyone more than 
theoretically... unless someone's device is really bad at generating 
OUIs; all of this based on the cheap prevalence of L3 devices at the edge.


Deepak

RE: NTP Md5 or AutoKey?

[Deepak Jain]

Of course, this only really works if your network has 3 reliable
+secure time sources + 1 for redundancy. I'm not sure that .*pool\.ntp
\.org would class as reliable+secure if you're concerned about NTP
security.

It's important to recognize that secure NTP has nothing to do with real
World time, and everything to do with all your secure systems being on
*the same* time, whatever that is. It really doesn't matter (much) if your
secure NTP cluster gets its time from an inconsistent source [provided it won't
allow changes of too great a magnitude at a time] but as long as they are all 
on the *same* time, you can maintain your security.

From an SPs point-of-view, security is very odd. It doesn't matter how well 
your
internal systems are if you are sending mail with the wrong time (say some
future date) and MTAs at your customers are rejecting them.

Deepak