On 02.10.2020 at 12:03 Job Snijders wrote:
Marco Marzetti (PCCW) wrote an even faster compression tool!
https://github.com/lamehost/aggregate-prefixes
Both these python implementations are meant as replacements for ISC's
vintage 'aggregate' Unix utility, with the notable difference that they
also support IPv6.
Example:
job@bench ~$ pip3 install aggregate-prefix
job@bench ~$ wc -l dfz_ipv4
810607
job@bench ~$ cat dfz_ipv4 | time aggregate-prefixes - | wc -l
141645
1m40.17s real 1m37.39s user 0m01.60s system
Compressing the whole IPv4 DFZ prefix list takes only 100 seconds.
First time I uploaded/publish something to/on Github ... so please be kind:
https://github.com/FvDxxx/pfxaggr
In case you need it even faster (and can accept the little known issues
and that it's old, ugly and never reviewed):
> wc -l dfz-pfx-20201002-A-v4.txt
813542 dfz-pfx-20201002-A-v4.txt
> time cat dfz-pfx-20201002-A-v4.txt | ./pfxagg -a1 > dfz-4-agg-pfx.log
real 0m1.034s
user 0m0.909s
sys 0m0.232s
> time cat dfz-pfx-20201002-A-v4.txt | aggregate-prefixes >
dfz-4-agg-pyth.log
real 1m11.691s
user 1m10.879s
sys 0m0.786s
> diff dfz-4-agg-pyth.log dfz-4-agg-pfx.log
> wc -l dfz-4-agg-pyth.log dfz-4-agg-pfx.log
141754 dfz-4-agg-pyth.log
141754 dfz-4-agg-pfx.log
283508 total
Cheers, Markus
