Re: CIDR cleanup
On 02.10.2020 at 12:03 Job Snijders wrote: Marco Marzetti (PCCW) wrote an even faster compression tool! https://github.com/lamehost/aggregate-prefixes Both these python implementations are meant as replacements for ISC's vintage 'aggregate' Unix utility, with the notable difference that they also support IPv6. Example: job@bench ~$ pip3 install aggregate-prefix job@bench ~$ wc -l dfz_ipv4 810607 job@bench ~$ cat dfz_ipv4 | time aggregate-prefixes - | wc -l 141645 1m40.17s real 1m37.39s user 0m01.60s system Compressing the whole IPv4 DFZ prefix list takes only 100 seconds. First time I uploaded/publish something to/on Github ... so please be kind: https://github.com/FvDxxx/pfxaggr In case you need it even faster (and can accept the little known issues and that it's old, ugly and never reviewed): > wc -l dfz-pfx-20201002-A-v4.txt 813542 dfz-pfx-20201002-A-v4.txt > time cat dfz-pfx-20201002-A-v4.txt | ./pfxagg -a1 > dfz-4-agg-pfx.log real0m1.034s user0m0.909s sys 0m0.232s > time cat dfz-pfx-20201002-A-v4.txt | aggregate-prefixes > dfz-4-agg-pyth.log real1m11.691s user1m10.879s sys 0m0.786s > diff dfz-4-agg-pyth.log dfz-4-agg-pfx.log > wc -l dfz-4-agg-pyth.log dfz-4-agg-pfx.log 141754 dfz-4-agg-pyth.log 141754 dfz-4-agg-pfx.log 283508 total Cheers, Markus
Re: CIDR cleanup
>> ok, i gotta ask. has someone tested to see if they all produce the >> same result givem the same input? i do not mean to imply they do >> not. i just have to wonder. > > Yes, of course. Marco and I collaborated on the tool's regression > testing. > > job@bench $ aggregate6 < dfz_ipv4 | md5 > 066bfea49c4c20fed7d86d355044764a > job@bench $ aggregate-prefixes < dfz_ipv4 | md5 > 066bfea49c4c20fed7d86d355044764a > > job@bench $ aggregate6 < dfz_ipv6 | md5 > 1193796d41cc47f32230da281e3ad419 > job@bench $ aggregate-prefixes < dfz_ipv6 | md5 > 1193796d41cc47f32230da281e3ad419 great. thanks. glad to see folk thinking this way. randy
Re: CIDR cleanup
On Fri, Oct 02, 2020 at 03:39:00AM -0700, Randy Bush wrote: > > Marco Marzetti (PCCW) wrote an even faster compression tool! > > https://github.com/lamehost/aggregate-prefixes > > > > Both these python implementations are meant as replacements for ISC's > > vintage 'aggregate' Unix utility, with the notable difference that they > > also support IPv6. > > ok, i gotta ask. has someone tested to see if they all produce the same > result givem the same input? i do not mean to imply they do not. i > just have to wonder. Yes, of course. Marco and I collaborated on the tool's regression testing. job@bench $ aggregate6 < dfz_ipv4 | md5 066bfea49c4c20fed7d86d355044764a job@bench $ aggregate-prefixes < dfz_ipv4 | md5 066bfea49c4c20fed7d86d355044764a job@bench $ aggregate6 < dfz_ipv6 | md5 1193796d41cc47f32230da281e3ad419 job@bench $ aggregate-prefixes < dfz_ipv6 | md5 1193796d41cc47f32230da281e3ad419 Kind regards, Job
Re: CIDR cleanup
> Marco Marzetti (PCCW) wrote an even faster compression tool! > https://github.com/lamehost/aggregate-prefixes > > Both these python implementations are meant as replacements for ISC's > vintage 'aggregate' Unix utility, with the notable difference that they > also support IPv6. ok, i gotta ask. has someone tested to see if they all produce the same result givem the same input? i do not mean to imply they do not. i just have to wonder. randy
Re: CIDR cleanup
On Thu, Oct 01, 2020 at 02:15:01PM -0300, Marcos Manoni wrote: > Check https://github.com/job/aggregate6 (thank you, Job) Marco Marzetti (PCCW) wrote an even faster compression tool! https://github.com/lamehost/aggregate-prefixes Both these python implementations are meant as replacements for ISC's vintage 'aggregate' Unix utility, with the notable difference that they also support IPv6. Example: job@bench ~$ pip3 install aggregate-prefix job@bench ~$ wc -l dfz_ipv4 810607 job@bench ~$ cat dfz_ipv4 | time aggregate-prefixes - | wc -l 141645 1m40.17s real 1m37.39s user 0m01.60s system Compressing the whole IPv4 DFZ prefix list takes only 100 seconds. Kind regards, Job
Re: CIDR cleanup
On Thu, 1 Oct 2020 13:32:53 +
John Von Essen wrote:
> I tried to write my code to do this, and its not trivial, just
> lookinh for a shortcurt. I did a breif glance at some CIDR related
> Perl cpan modules, and nothing has jumped out.
I wrote the code below some time ago. I've not used it in awhile, but
presumably it does what I think it does. I called it compactaddrs.pl.
Feed it a list of CIDR blocks via stdin.
John
#!/usr/bin/perl -T
use strict;
use warnings;
# compactaddrs - aggregate addr blocks
use NetAddr::IP qw( Compact );
my @v4blocks;
my @v6blocks;
while( defined(my $line=<>) ) {
chomp $line;
if ( $line =~ /:/ ) {
push @v6blocks, NetAddr::IP->new($line);
}
else {
push @v4blocks, NetAddr::IP->new($line);
}
}
if ( scalar @v4blocks > 0 ) {
print "# IPv4 aggregate prefixes\n";
my @aggregates = Compact(@v4blocks);
print 'WHERE';
for my $prefix (@aggregates) {
print ' saddr <<= ';
print "'$prefix' OR";
#print "$prefix\n";
}
}
if ( scalar @v6blocks > 0 ) {
print " #IPv6 aggregate prefixes\n";
my @aggregates = Compact(@v6blocks);
for my $prefix (@aggregates) {
print "$prefix\n";
}
}
print "\n";
Re: CIDR cleanup
Hi, Check https://github.com/job/aggregate6 (thank you, Job) El jue., 1 oct. 2020 a las 10:36, John Von Essen () escribió: > > Sorry if this is slightly off-topic, but I am writing some code for a custom > GeoDNS routemap. My starting data set is a raw list of /24 subnets, no prefix > aggregation has been done. In other words, its the entire BGP routing table > in /24 prefixes - tagged by Geo region. Each region is its own txt file with > a dump of /24’s. As a result, these lists are HUGE. I want to aggregate the > prefixes as much as possible to create a smaller routemap. > > So right now it looks like: > > ... > 105.170.72.0/24 brs > 105.170.73.0/24 brs > 105.170.74.0/24 brs > 105.170.75.0/24 brs > 105.170.76.0/24 brs > 105.170.77.0/24 brs > 105.170.78.0/24 brs > 105.170.79.0/24 brs > 105.170.80.0/24 brs > 105.170.81.0/24 brs > 105.170.82.0/24 brs > 105.170.83.0/24 brs > 105.170.84.0/24 brs > … > > and so on. Obviously, 105.170.72.0/24 thru 105.170.79.0/24 can be aggregated > to 105.170.72.0/21 and so on. I normally use Perl, does anyone now if there > is a perl module that will automatically do this prefix aggregation? I tried > to write my code to do this, and its not trivial, just lookinh for a > shortcurt. I did a breif glance at some CIDR related Perl cpan modules, and > nothing has jumped out. > > Thanks > John > > >
Re: CIDR cleanup
Not Perl, though this may be useful depending on your environment: https://github.com/rus-cert/compress-cidr The examples are for IPv6, though I use it to consolidate lists of IPv4 in a variety of jobs/scripts without issue. YMMV. From: NANOG on behalf of John Von Essen Sent: Thursday, October 1, 2020 6:32 AM To: NANOG Subject: CIDR cleanup Sorry if this is slightly off-topic, but I am writing some code for a custom GeoDNS routemap. My starting data set is a raw list of /24 subnets, no prefix aggregation has been done. In other words, its the entire BGP routing table in /24 prefixes - tagged by Geo region. Each region is its own txt file with a dump of /24’s. As a result, these lists are HUGE. I want to aggregate the prefixes as much as possible to create a smaller routemap. So right now it looks like: ... 105.170.72.0/24 brs 105.170.73.0/24 brs 105.170.74.0/24 brs 105.170.75.0/24 brs 105.170.76.0/24 brs 105.170.77.0/24 brs 105.170.78.0/24 brs 105.170.79.0/24 brs 105.170.80.0/24 brs 105.170.81.0/24 brs 105.170.82.0/24 brs 105.170.83.0/24 brs 105.170.84.0/24 brs … and so on. Obviously, 105.170.72.0/24 thru 105.170.79.0/24 can be aggregated to 105.170.72.0/21 and so on. I normally use Perl, does anyone now if there is a perl module that will automatically do this prefix aggregation? I tried to write my code to do this, and its not trivial, just lookinh for a shortcurt. I did a breif glance at some CIDR related Perl cpan modules, and nothing has jumped out. Thanks John
Re: CIDR cleanup
The Perl Net::Netmask module is also worth checking out. It may not be
better at aggregation but it does have other functions that could be
helpful. I use the shortest match address lookup functions of Net::Netmask
very heavily and have reproduced them in a R / C++ package.
Jon
On Thu, Oct 1, 2020 at 9:47 AM Tim Jackson wrote:
> #!/usr/bin/perl
> use strict;
> use warnings;
> use Data::Dumper;
> use NetAddr::IP qw(Compact);
>
> my @ips = ( '105.170.72.0/24', '105.170.73.0/24', '105.170.74.0/24' );
>
> my @agged = aggregate(\@ips);
>
> sub aggregate {
> my @naddr = map { NetAddr::IP->new($_) } @{$_[0]};
> my @output = Compact(@naddr);
> return @output;
> }
>
>
> On Thu, Oct 1, 2020 at 8:36 AM John Von Essen wrote:
>
>> Sorry if this is slightly off-topic, but I am writing some code for a
>> custom GeoDNS routemap. My starting data set is a raw list of /24 subnets,
>> no prefix aggregation has been done. In other words, its the entire BGP
>> routing table in /24 prefixes - tagged by Geo region. Each region is its
>> own txt file with a dump of /24’s. As a result, these lists are HUGE. I
>> want to aggregate the prefixes as much as possible to create a smaller
>> routemap.
>>
>> So right now it looks like:
>>
>> ...
>> 105.170.72.0/24 brs
>> 105.170.73.0/24 brs
>> 105.170.74.0/24 brs
>> 105.170.75.0/24 brs
>> 105.170.76.0/24 brs
>> 105.170.77.0/24 brs
>> 105.170.78.0/24 brs
>> 105.170.79.0/24 brs
>> 105.170.80.0/24 brs
>> 105.170.81.0/24 brs
>> 105.170.82.0/24 brs
>> 105.170.83.0/24 brs
>> 105.170.84.0/24 brs
>> …
>>
>> and so on. Obviously, 105.170.72.0/24 thru 105.170.79.0/24 can be
>> aggregated to 105.170.72.0/21 and so on. I normally use Perl, does
>> anyone now if there is a perl module that will automatically do this prefix
>> aggregation? I tried to write my code to do this, and its not trivial, just
>> lookinh for a shortcurt. I did a breif glance at some CIDR related Perl
>> cpan modules, and nothing has jumped out.
>>
>> Thanks
>> John
>>
>
Re: CIDR cleanup
#!/usr/bin/perl
use strict;
use warnings;
use Data::Dumper;
use NetAddr::IP qw(Compact);
my @ips = ( '105.170.72.0/24', '105.170.73.0/24', '105.170.74.0/24' );
my @agged = aggregate(\@ips);
sub aggregate {
my @naddr = map { NetAddr::IP->new($_) } @{$_[0]};
my @output = Compact(@naddr);
return @output;
}
On Thu, Oct 1, 2020 at 8:36 AM John Von Essen wrote:
> Sorry if this is slightly off-topic, but I am writing some code for a
> custom GeoDNS routemap. My starting data set is a raw list of /24 subnets,
> no prefix aggregation has been done. In other words, its the entire BGP
> routing table in /24 prefixes - tagged by Geo region. Each region is its
> own txt file with a dump of /24’s. As a result, these lists are HUGE. I
> want to aggregate the prefixes as much as possible to create a smaller
> routemap.
>
> So right now it looks like:
>
> ...
> 105.170.72.0/24 brs
> 105.170.73.0/24 brs
> 105.170.74.0/24 brs
> 105.170.75.0/24 brs
> 105.170.76.0/24 brs
> 105.170.77.0/24 brs
> 105.170.78.0/24 brs
> 105.170.79.0/24 brs
> 105.170.80.0/24 brs
> 105.170.81.0/24 brs
> 105.170.82.0/24 brs
> 105.170.83.0/24 brs
> 105.170.84.0/24 brs
> …
>
> and so on. Obviously, 105.170.72.0/24 thru 105.170.79.0/24 can be
> aggregated to 105.170.72.0/21 and so on. I normally use Perl, does anyone
> now if there is a perl module that will automatically do this prefix
> aggregation? I tried to write my code to do this, and its not trivial, just
> lookinh for a shortcurt. I did a breif glance at some CIDR related Perl
> cpan modules, and nothing has jumped out.
>
> Thanks
> John
>
>
>
>
