Re: CIDR string replacement

[Jon Meek]

This is what I have done using R:

https://github.com/meekj/netblockr

I still use similar tools in Perl with Net::Netmask

Jon

On Fri, Oct 2, 2020 at 11:50 AM Royce Williams 
wrote:

> The recent thread on CIDR aggregation cleanup scripts reminds me that I'm
> looking for a similarly efficient implementation of a related tool. (I'm
> gearing up to write my own in Perl, but don't want to reinvent the wheel.)
>
> I'd like a fast, Unix-pipeline-ready tool that *replaces* all IPs within
> that range with a supplied string, using a simple config file as input, and
> ideally with autodetection of IP-address "word" boundaries, as in:
>
> $ cat cidr-replace.cfg
> 105.170.75.0/24|[Unitel] 
> 209.112.128.0/18|[ACS] 
> 209.165.128.0/18|[GCI] 
> 192.0.2.0/24|[TEST-NET-1] 
> 198.51.100.0/24|[TEST-NET-2] 
> 203.0.113.0/24|[TEST-NET-3] 
>
> $ echo "source,data1,data2,209.112.130.2,data3" | cidr-replace
> cidr-replace.cfg
> source,data1,data2,[ACS],data3
>
>
> And I know this is kludgy, but it would also be useful for quick-and-dirty
> work if it had a flag to "append" the string using a known delimiter, as in:
>
> $ echo "source,data1,data2,209.112.130.2,data3" | cidr-replace --append
> ',' cidr-replace.cfg
> source,data1,data2,209.112.130.2,[ACS],data3
>
> (But I'm happy to hack that last functionality into an existing script.)
>
> --
> Royce
>

Re: CIDR cleanup

[Jon Meek]

The Perl Net::Netmask module is also worth checking out. It may not be
better at aggregation but it does have other functions that could be
helpful. I use the shortest match address lookup functions of Net::Netmask
very heavily and have reproduced them in a R / C++ package.

Jon

On Thu, Oct 1, 2020 at 9:47 AM Tim Jackson  wrote:

> #!/usr/bin/perl
> use strict;
> use warnings;
> use Data::Dumper;
> use NetAddr::IP qw(Compact);
>
> my @ips = ( '105.170.72.0/24', '105.170.73.0/24', '105.170.74.0/24' );
>
> my @agged = aggregate(\@ips);
>
> sub aggregate {
> my @naddr = map { NetAddr::IP->new($_) } @{$_[0]};
> my @output = Compact(@naddr);
> return @output;
> }
>
>
> On Thu, Oct 1, 2020 at 8:36 AM John Von Essen  wrote:
>
>> Sorry if this is slightly off-topic, but I am writing some code for a
>> custom GeoDNS routemap. My starting data set is a raw list of /24 subnets,
>> no prefix aggregation has been done. In other words, its the entire BGP
>> routing table in /24 prefixes - tagged by Geo region. Each region is its
>> own txt file with a dump of /24’s. As a result, these lists are HUGE. I
>> want to aggregate the prefixes as much as possible to create a smaller
>> routemap.
>>
>> So right now it looks like:
>>
>> ...
>> 105.170.72.0/24 brs
>> 105.170.73.0/24 brs
>> 105.170.74.0/24 brs
>> 105.170.75.0/24 brs
>> 105.170.76.0/24 brs
>> 105.170.77.0/24 brs
>> 105.170.78.0/24 brs
>> 105.170.79.0/24 brs
>> 105.170.80.0/24 brs
>> 105.170.81.0/24 brs
>> 105.170.82.0/24 brs
>> 105.170.83.0/24 brs
>> 105.170.84.0/24 brs
>> …
>>
>> and so on. Obviously, 105.170.72.0/24 thru 105.170.79.0/24 can be
>> aggregated to 105.170.72.0/21 and so on. I normally use Perl, does
>> anyone now if there is a perl module that will automatically do this prefix
>> aggregation? I tried to write my code to do this, and its not trivial, just
>> lookinh for a shortcurt. I did a breif glance at some CIDR related Perl
>> cpan modules, and nothing has jumped out.
>>
>> Thanks
>> John
>>
>

Re: Proving Gig Speed

[Jon Meek]

On Mon, Jul 16, 2018 at 2:00 PM Chris Gross 
wrote:

> I'm curious what people here have found as a good standard for providing
> solid speedtest results to customers. All our techs have Dell laptops of
> various models, but we always hit 100% CPU when doing a Ookla speedtest for
> a server we have on site. So then if you have a customer paying for 600M or
> 1000M symmetric, they get mad and demand you prove it's full speed. At that
> point we have to roll out different people with JDSU's to test and prove
> it's functional where a Ookla result would substitute fine if we didn't
> have crummy laptops possibly. Even though from what I can see on some
> google results, we exceed the standards several providers call for.
>
> Most of these complaints come from the typical "power" internet user of
> course that never actually uses more than 50M sustained paying for a
> residential connection, so running a circuit test on each turn up is
> uncalled for.
>
> Anyone have any suggestions of the requirements (CPU/RAM/etc) for a laptop
> that can actually do symmetric gig, a rugged small inexpensive device we
> can roll with instead to prove, or any other weird solution involving
> ritual sacrifice that isn't too offensive to the eyes?
>

My practice is to use iperf with packet capture on both sides. The packet
capture can then be analyzed for accurate per-second, or less, throughput,
re-transmit rates, etc. This was implemented in a corporate network in
several ways including dedicated servers (that also did other monitoring),
and bootable CDs or USB sticks that a user in a small office could run on a
standard desktop. Many interesting issues were discovered with this
technique, and a fair number of perceived issues were debunked.

Here is a wrapper to run iperf + tcpdump on each side of a connection (it
could use some automation):

 https://github.com/meekj/perl-packet-tools/blob/master/run_iperf

I originally did the analysis in Perl, but that can be fairly slow when
processing 30 seconds of packets on a saturated GigE link. If anyone is
interested there is now a C++ version along with analysis code in R at:

 https://github.com/meekj/iperfsum

That version currently has only one second resolution. I have a R interface
to libpcap files that could be used for analysis at any time resolution:

 https://github.com/meekj/libpcapR

I have a plan to implement the complete test environment in a Docker
container at some point. I also have a collection of small, mostly
low-cost, computers that I plan to benchmark for network throughput and
data analysis time. Some of the tiny computers can saturate a GigE link but
are very slow processing the data.

Jon

Re: ticketmaster.com 403 Forbidden

[Jon Meek]

Another way to get on their block list is to have a lot of users behind a
single NAT or proxy IP address. In my experience they blocked single IPs.
The first time it was easy to explain that there were 30,000 users behind
the single address and get the block cleared. After that it became more
difficult to get someone to listen. In one case I gave up because we were
about to make a data center change and the blocked address would no longer
be used.

However, I don't believe that the problem ever came back, maybe because we
had fewer users behind individual IP addresses, or because they finally
note that the netblocks were owned by $LARGE_CORPORATION.


On Mon, Feb 6, 2017 at 11:49 AM, Suresh Ramasubramanian  wrote:

> My guess is you have or had sometime in the long distant past a scalper
> operating on your network, using automated ticket purchase bots.
>
> If you still have that scalper around, you might want to turf him.  If
> he’s ancient history, saying so might induce them to remove the block.
>
> --srs
>
> On 06/02/17, 8:45 AM, "nanog-boun...@nanog.org on behalf of
> mike.l...@gmail.com"  mike.l...@gmail.com> wrote:
>
> Yup, i have a /22 that has the same problem. Support is useless...
>
> > On Feb 6, 2017, at 08:35, Ethan E. Dee 
> wrote:
> >
> > It gives me a Forbidden error.
> > It has for over a year.
> > There support says they are not allowed to me why by their policy.
> > it is across an entire /19.
> > I gave up after the fifth time and encourage the customers to call
> them individually.
> >
> >> On 02/06/2017 11:09 AM, Niels Bakker wrote:
> >> * charles.man...@charter.com (Manser, Charles J) [Mon 06 Feb 2017,
> 16:21 CET]:
> >>> It seems that browsing to ticketmaster.com or any of the
> associated IP addresses results in a 403 Forbidden for our customers today.
> Is anyone else having this issue?
> >>
> >> http://help.ticketmaster.com/why-am-i-getting-a-blocked-
> forbidden-or-403-error-message/
> >>
> >>
> >>-- Niels.
> >
>
>
>
>


-- 
Jon T. Meek, Ph.D.
 https://linkedin.com/in/meekjt
 https://meekj.github.io

Re: NIST NTP servers

[Jon Meek]

A note on using a Raspberry Pi as a NTP server. In my limited home lab
testing the RPi server had enough instability that Internet time sources
were always preferred by my workstation after ntpd had been running for a
while. Presumably this was due to the RPi's clock frequency drifting. At
some point I will look at it again.

If you do want to build your own Stratum 1 server you might want to glance
at:

https://github.com/meekj/ntp/blob/master/jon_meek_ntp_poster2009a.pdf

and the references there.

I had hoped to use the very low cost RPi Stratum 1 servers at $DAY_JOB, but
the test device was clearly not up to the job. At some point I hope to
revisit this and do some more testing like I did for that poster. I'll add
in a CDMA server and a dedicated WWVB receiver.

Jon

Re: [Q] What is your favorite Network Tools Live CD / USB, which you could have running in remote offices?

[Jon Meek]

On Thu, Aug 22, 2013 at 1:06 PM, Stefan netfort...@gmail.com wrote:

 I've been toying with Live distros (CD, then USB) for many years, in
 support of security toolsets, to which I kept adding my own stuff, or
 customizing existing components.

 I am now trying to build a network toolset LiveCD/USB, but this time with
 a completely different purpose: I would like to put it in the hands of all
 remote offices we have on our network, and use it to have local systems
 boot out of it, and help us then run troubleshooting tools, from the
 central office, by SSH/X-ing into the remote live system (e.g. iperf,
 hping3, httping, tcping, mtr, tcpdump, voip tools, some thin
 clients/apps, synthetic transactions scripted to run at diff time
 intervals, and report back to us the health seen form the remotes, etc.).
 Has anybody used a base network tools Live CD/USB that they would
 recommend, having used as basis for such a network probe functionality?

 NOTE: I assume *nix based (Linux or BSD flavors), not Windows ...

 TIA,
 ***Stefan


I use Voyage Linux: http://linux.voyage.hk/

In several modes:

 - Bootable USB flash drive

 - On PC Engines ALIX boards from Compact Flash

 - And in a few instances on servers with spinning disks, and desktop with
minimal window system

The bootable USB stick has been used extensively for iperf + tcpdump +
analysis from PCs are remote locations. We either have people copy an image
to the USB stick, or mail them a stick. Then they can turn (almost) any PC
into a network analysis tool. We have the system report it's IP address at
boot time, and then we ssh in.

Jon

Re: [Q] What is your favorite Network Tools Live CD / USB, which you could have running in remote offices?

[Jon Meek]

On Thu, Aug 22, 2013 at 9:17 PM, Christopher X. Candreva
ch...@westnet.comwrote:

 On Thu, 22 Aug 2013, Stefan wrote:

  a completely different purpose: I would like to put it in the hands of
 all
  remote offices we have on our network, and use it to have local systems
  boot out of it, and help us then run troubleshooting tools, from the
  central office, by SSH/X-ing into the remote live system (e.g. iperf,
  hping3, httping, tcping, mtr, tcpdump, voip tools, some thin
  clients/apps, synthetic transactions scripted to run at diff time
  intervals, and report back to us the health seen form the remotes,
 etc.).

 I'm toying with a similar idea, though of putting a Raspberry Pi in remote
 offices to do tests from. I'm just looking for something I can ssh too,
 however, it also doesn't seem like much of a stretch to put some kind of
 web-based screen that someone in the office could run an automated scan,
 and
 read us off information that might help.


 ==
 Chris Candreva  -- ch...@westnet.com -- (914) 948-3162
 WestNet Internet Services of Westchester
 http://www.westnet.com/


There is a lot to be said for the RaspberryPi, but network throughput, and
especially processing power are limited. My tests show that the RaspberryPi
could push only about 46 Mbps of iperf while most PCs configured the same
way get almost to wire speed (100 Mbps or 1Gbps), and processing 30 seconds
of 45 Mbps traffic on the RaspberryPi takes many minutes. But, if you want
to test slower circuits, it can't be beat for cost, size, flexibility.

I am expecting delivery of a Parallella board in October and will be
testing it for iperf capability at GigE speed.

Jon

Re: Old Annex question

[Jon Meek]

On Sun, Feb 13, 2011 at 11:36 AM, Nick Hilliard n...@foobar.org wrote:

 On 13/02/2011 15:30, Joe Hamelin wrote:

 day.  I remember days spent hunting down ring-no-answers in a 400 POTS
 line hunt group.


 It was much easier to detect those by looking for strange port connectivity
 patterns in the logs.

 re: annexes, it was a happy day when we upgraded from annex 3 to
 portmaster.  No idea what the escape key was.

 Nick


I have a couple of Micro Annex's in the recycle pile in my basement and,
after a bit of rummaging, found that I have the paper documentation as well.

In the User's Guide it says: While in a session with a host, pressing an
attention key returns you to the CLI prompt. Somewhere else it indicates
that BREAK is the attention key however that may be configurable.

If anything further is needed contact me, probably off-list, and I can look
in the docs including the full CLI manual.

Jon

Re: Jumbo frame Question

[Jon Meek]

I have the opposite problem. I use iperf to test WAN and VPN
throughput and packet loss, but find that the sending Linux system
starts out with the expected MTU / MSS but then ramps up the packet
size to way beyond 1500. The result is that network equipment must
fragment the packets. On higher bandwidth circuits there are a lot of
re-transmits that mask any real packet loss that might exist in the
path.

I have tried multiple methods to clamp the MTU, but nothing has worked
so far. This leads me to wonder how often real bulk transfer
applications start using jumbo packets that just end up getting
fragmented downstream.

The jumbo packets from iperf occur on various versions of the Linux
kernel and different distributions. It might only happen on GigE.

Suggestions on clamping the MTU are welcome.

Thanks,

Jon

On Thu, Nov 25, 2010 at 7:13 PM, Harris Hui harris@hk1.ibm.com wrote:

 Hi

 Does anyone have experience on design / implementing the Jumbo frame
 enabled network?

Re: A New TransAtlantic Cable System

[Jon Meek]

One of the ways that I have tormented WAN vendors over the years is
with a plot of RTT vs. great circle distance between the end points of
a circuit. Most RTTs usually sit at some constant offset above that
Physics limit straight line. Circuits taking a less than ideal have
their RTT far above the Physics limit line and we have used that
information to get routes fixed.

Using my great circle program that accounts for the non-spherical
Earth for locations we have West of London and North of NYC, assuming
a 1.5 index of refraction I get:

One way distance: 5520.6 km   Round Trip Delay: 55.2 ms

So Heath's estimate is right on, although depending on where he got
the distance maybe it does account for the shape of the Earth.

Jon

On Sat, Oct 2, 2010 at 6:17 AM, Heath Jones hj1...@gmail.com wrote:
 On 2 October 2010 10:52, Rod Beck rod.b...@hiberniaatlantic.com wrote:
 Is that a straight line calculation or did you take into account that a
 straight line is not the shortest path on a curved surface?

 Well that is pretty obvious to most, but no - I didn't go to the
 effort of factoring in curvature of the earth - especially given that
 1.5 is very rough figure anyway for RI of glass. If anything, my
 comment was compliment to your network being close to minimum possible
 latency!

Re: capirca : Google Network Filtering Management

[Jon Meek]

On Fri, Apr 9, 2010 at 5:57 PM, Christopher Morrow
morrowc.li...@gmail.com wrote:
 On Fri, Apr 9, 2010 at 2:09 PM, William Duck na...@qualitymail.com wrote:
   http://code.google.com/p/capirca/
   Developed internally at Google, this system is designed to utilize
   common definitions of networks and services and high-level policy
   files to facilitate the development and manipulation
   of network access control filters (ACLs) for various platforms.

 would be interesting (to the community to get the authors to present
 some material about this at a meeting? (a nanog meeting)

 -Chris

The authors gave an excellent tag-team presentation at USENIX LISA
'09. Video might be available. It would be good at a NANOG meeting.

Jon

Re: NTP clock source

[Jon Meek]

I use both EndRun Technologies and the Garmin 18x LVC + old PC solution.

I am currently seeing 8+ satellites out a North facing window almost
all of the time with the Garmin. The window method may not work if the
window is coated with a metallic layer (common in newer buildings).
Also, be careful extending the serial line. The rise time of the PPS
signal is already degraded by the length of wire that is supplied.

Jon

On Thu, Mar 25, 2010 at 8:51 AM, Kyle Bader kyle.ba...@gmail.com wrote:
 Can anyone recommend a solid clock souce (stratum 0) that's not overly
 expensive? The only stuff I've found so far is ESE, can anyone
 recommend them or conversely has anyone had any problems with their
 hardware?

 --

 Kyle

Re: ethernet to serial converters with ACLs

[Jon Meek]

Avocent / Cyclades boxes have ACL capability (they run Linux) and can
be used with EV-DO/GSM modems. They may not be the lowest cost
solution, but there is a central management system and a wide range of
serial interface units from single port to at least 32 ports.

Jon
Full disclosure: I was a member of their Customer Advisory Board

On Wed, Mar 10, 2010 at 10:06 PM, R. Benjamin Kessler r...@mnsginc.com wrote:

On Mar 10, 2010, at 3:17 PM, Michael Holstein wrote:

 Can anyone recommend a cheap Ethernet to Serial (RS232/422/485)
 converter with functionality like the Lantronix boxes .. except one
 that
 supports access lists (nothing complicated .. maybe a list of 5
 approved
 hosts). I need a bunch of single port devices, not an access-server
 for
 a rack.

We use SENA PS110 boxes (
http://www.sena.com/products/device_servers/hd_ps_x10.php ).  They work
very well, have various ACL features (dunno if it supports 5 named IP
 or not), and other configurables.

Caleb

 On a similar topic, any good solutions for out-of-band serial
 console/Ethernet solutions that use EV-DO/GSM wireless Internet?

Re: Ubiquti NanobridgeM

[Jon Meek]

There is a wealth of information in Ubiquti's forums:

 http://ubnt.com/forum/

Jon

On Thu, Mar 4, 2010 at 1:44 PM, Todd Mueller t...@velocitytelephone.com wrote:
 Anyone have any real-world experience with Ubiquti's MIMO PTP equipment?
 We're looking to shoot data at distances of a few hundred feet up to 2-3
 miles. Reliability? Latency? Other issues? Any feedback is appreciated.

 http://www.ubnt.com/nanobridge

 Thanks!

 Todd

Re: Speed Testing and Throughput testing

[Jon Meek]

I use iperf with packet capture on both sides, then analyze the packet
capture for per-second throughput and re-transmits. I usually do 10
TCP streams for 30 seconds.

Note that on GigE with significant RTTs (5-15 ms) some TCP tuning is
needed to deal with the bandwidth delay product. It is also possible
that Ethernet drivers will have an effect. Local testing of the pair
of test machines should be done if you can't get to about 980 Mbps on
a Gig link (keeping in mind the comment about TCP tuning as latency
increases).

Jon

On Mon, Nov 2, 2009 at 4:56 PM, Mark Urbach mark.urb...@pnpt.com wrote:
 Anyone have a good solution to get accurate speed results when testing at 
 10/100/1000 Ethernet speeds?

 Do you have a server/software that customer can test too?



 Thanks,
 Mark Urbach
 PinPoint Communications, Inc.
 100 N. 12th St  Suite 500
 Lincoln, NE 68508
 402-438-6211  ext 1923  Office
 402-660-7982  Cell
 mark.urb...@pnpt.com
 [cid:image003.jpg@01CA5BD5.1A5CEE20]

Re: Traceroute management

[Jon Meek]

mon ( http://mon.wiki.kernel.org/index.php/Main_Page )
comes with traceroute.monitor

It keeps a state file of current routes and logs only changes. You can
specify equivalent hops, hops to ignore, StopAt addresses, and
UnexpectedHops.

Since it is part of mon, it is easy to alert on a route change.

The IgnoreHop feature was probably added after the mon release. I can
provide a newer version if IgnoreHop would be useful.

Jon

Re: Sprint Leap Second

[Jon Meek]

A visual comparison of my Sprint phone and xclock with second hand on a
synchronized workstation suggests that they have not yet implemented the
leap second.

Our single CDMA NTP clock did handle the leap second at the correct moment.
However, that CDMA clock is West of Philadelphia and I am in New Jersey.
As I mentioned in a previous message,  in 2005 our CDMA clock got the
leap second a few hours early.

Jon

On Sun, Jan 4, 2009 at 4:06 PM, Marshall Eubanks t...@multicasttech.com wrote:
 Has anyone seen evidence that Sprint's cellular network has not
 adopted the leap second yet ?

 (I have reports, but cannot check myself.)

 Marshall

Re: Leap second tonight

[Jon Meek]

My Solaris 10 boxes are all happy (and did not reboot). I monitor NTP
on a number
of devices, including one router. The router was off by one second for
a while, but
is OK after an hour. Everything else was fine immediately.

In 2005, our CDMA clock got the leap second between 15:08 and 15:38
EST creating
some issues due to disagreement with the (too few) GPS clocks.

Jon

On Wed, Dec 31, 2008 at 7:53 PM, Wil Schultz wschu...@bsdboy.com wrote:
 At which point my Solaris 10 v490's reboot in unison, lovely.

 Anyone else see anything interesting?

 -wil

Re: Stress Testing LAN/WAN

[Jon Meek]

We use iperf running off of a bootable Linux CD
with a 2.4 Kernel and can push 960 to 980 Mbps
with no drops or errors on any pair of PCs with
Gig interfaces we have tried so far.

We usually 10 TCP streams, or tune the TCP stack
and use a single stream.

We also capture the traffic on both sides, run a
standard analysis, and plot the send and receive
per-second utilization and the re-transmit rate
from the send side.

Jon

Re: Hardware capture platforms

[Jon Meek]

I have had the same problem and solved it with a rare (even then)
100BT Only hub. I still have at least one stashed away.

For years though, I have been using bonding on Linux to combine multiple
tap streams. We also use hardware aggregators for the higher volume
applications.

Jon

On Thu, Jul 31, 2008 at 12:31 PM, Jay R. Ashworth [EMAIL PROTECTED] wrote:

 And, note carefully: some dual-speed hubs are actually a 10BT hub and
 a 100BT hub *with a switch between them*.  I forget which brand I
 caught this on, but it bit me a couple of years back.

 Which speed cable you plug in determines which hub you're talking to.

 Yes, it's weird.

 Cheers,
 -- jra
 --
 Jay R. Ashworth   Baylink  [EMAIL 
 PROTECTED]
 Designer The Things I Think   RFC 2100
 Ashworth  Associates http://baylink.pitas.com '87 e24
 St Petersburg FL USA  http://photo.imageinc.us +1 727 647 1274

 Those who cast the vote decide nothing.
 Those who count the vote decide everything.
   -- (Josef Stalin)