Re: ipfw divert filter for IPv4 geo-blocking

[Julian Elischer]

On 1/08/2016 7:16 PM, Dr. Rolf Jansen wrote:

Am 01.08.2016 um 03:17 schrieb Julian Elischer :
On 30/07/2016 10:17 PM, Dr. Rolf Jansen wrote:

I finished the work on CIDR conformity of the IP ranges tables generated by the 
tool geoip. The main constraint is that the start and end address of an IP 
block given by the delegation files MUST BE PRESERVED during the transformation 
to a set of CIDR records. This target is achieved by:

  1. Finding the largest common netmask boundary of the start address utilizing
 int(log2(addr_count)); then iteration like Euclid's algorithm in computing
 a GCD.

  2. Output the CIDR with the given start address and the masklen belonging
 to the found netmask.

  3. If the CIDR does not match the whole original IP range then set the start
 address of the next CIDR block to the next boundary of the common netmask,
 and loop over starting at 1. until the original range has been satisfied.


check out the appletalk code I pointed out  to you.. I wrote that in 93 or so 
but I remember sweating blood
over it to get it right.

I read the description of the code and the following sentence made me suspicious that 
aa_dorangeroute() would guarantee the above mentioned main constraint  "start and 
end address of an IP block given by the delegation files MUST BE PRESERVED" can be 
matched. Start/end address are said to be anything (even undefined) but fixed in the 
description

So you get your data as ranges (i.e. a start and an end address)


...
Split the range into two subranges such that the middle
of the two ranges is the point where the highest bit of difference
between the two addresses makes its transition.
...

I do not want this.

I think I may explain it better by example..

Any range of addresses can conceptually be broken into an optimal 
range of binary chunks.
for example, for the range 1. 62   we have an optimal set of 
binary subranges:

 1-1  2-3  4-7 8-15 16-31 (*) 32-47 48-55 56-59 60-61 and 62-62
there is always a point (*)  where we separate the left hand side from 
the right hand side.  You can also only merge the two items either 
side of the (*) point, and even then, only if they have the same 
width. Ranges not immediately adjacent to the (*) point can never be 
merged into any other range.
In some ranges, there are no items to the right, and for some ranges 
there are no items to the left. There may also be skipped ranges. The 
above example is "worst case".  All 10 subranges are present, and only 
2 can be merged, leading to needing 9 binary subranges to correctly 
express the whole range.
So the term "middle" is misleading.  It however allows you to take an 
arbitrary range of addresses and generate the optimal set of cidr 
addresses.
In this case if you had 10.0.1.0 through 10.0.62.255 you would need 9 
cidr addresses to express the range correctly (the middle two can be 
joined).

That is what the code referred to does.

if you have exceptions as you do in routing tables,  you can express 
it as 3 cidr addresses..  0-64  MINUS 0 and MINUS 63.
i.e 10.0.0.0/18 minus 10.0.0.0/24 minus 10.0.63.0/24. However this is 
probably not of interest to your use case.
It however IS possible to do this in ipfw since it allows overlapping 
ranges of different widths. (you will note that the code pointed to 
does NOT do that. it would be an order of magnitude (or more) harder 
to do.)





I carefully tested the algorithm and a table that I pipe by the new geoip tool 
into ipfw is 100 % identical to the output of the ipfw command 'table N list'.

though that doesn't mean it is semantically identical to the original table due to 
'most specific rule wins" behaviour.

for example:
if you type in ;

1.2.3.0/24 -> A
and
1.2.3.0/26 -> B
then both rules will be listed the same as what you put in
but if you wanted to get all rules that point to A, without having rules that 
point to B, then you would have to export
1.2.3.64/26  -> A
1.2.3.128/25 -> A
  (i.e. TWO rules)

This is definitely not the usage case. The origin of the data to be passed to 
ipfw tables are RIR delegation statistics files, which is guaranteed to be 
consolidated, namely resolved overlaps and joined adjacencies, long before any 
tables for ipfw are generated. Each range entry got a well defined, i.e. fixed, 
i.e non-variable starting address, and anything that changes the starting 
address of the ranges renders the table useless. Every entry got a well defined 
range length, and that one also must not be changed, or the table would be 
useless as well.

I think you are misunderstanding what I meant.


In addition, we are talking about automatic generation of thousands of entries, 
and I never ever won't rely on something like 'most specific rule wins' 
behaviour, I want the behaviour as explicit as possible, and for this reason I 
am happy with 'INPUT is 100 % identical to the OUTPUT'.


I agree. I was saying that getting the same rules out of 

Re: ipfw divert filter for IPv4 geo-blocking

[Dr. Rolf Jansen]

> Am 01.08.2016 um 03:17 schrieb Julian Elischer :
> On 30/07/2016 10:17 PM, Dr. Rolf Jansen wrote:
>> I finished the work on CIDR conformity of the IP ranges tables generated by 
>> the tool geoip. The main constraint is that the start and end address of an 
>> IP block given by the delegation files MUST BE PRESERVED during the 
>> transformation to a set of CIDR records. This target is achieved by:
>> 
>>  1. Finding the largest common netmask boundary of the start address 
>> utilizing
>> int(log2(addr_count)); then iteration like Euclid's algorithm in 
>> computing
>> a GCD.
>> 
>>  2. Output the CIDR with the given start address and the masklen belonging
>> to the found netmask.
>> 
>>  3. If the CIDR does not match the whole original IP range then set the start
>> address of the next CIDR block to the next boundary of the common 
>> netmask,
>> and loop over starting at 1. until the original range has been satisfied.
>> 
> check out the appletalk code I pointed out  to you.. I wrote that in 93 or so 
> but I remember sweating blood
> over it to get it right.

I read the description of the code and the following sentence made me 
suspicious that aa_dorangeroute() would guarantee the above mentioned main 
constraint  "start and end address of an IP block given by the delegation files 
MUST BE PRESERVED" can be matched. Start/end address are said to be anything 
(even undefined) but fixed in the description.

   ... 
   Split the range into two subranges such that the middle
   of the two ranges is the point where the highest bit of difference
   between the two addresses makes its transition.
   ...

I do not want this.

>> I carefully tested the algorithm and a table that I pipe by the new geoip 
>> tool into ipfw is 100 % identical to the output of the ipfw command 'table N 
>> list'.
> though that doesn't mean it is semantically identical to the original table 
> due to 'most specific rule wins" behaviour.
> 
> for example:
> if you type in ;
> 
> 1.2.3.0/24 -> A
> and
> 1.2.3.0/26 -> B
> then both rules will be listed the same as what you put in
> but if you wanted to get all rules that point to A, without having rules that 
> point to B, then you would have to export
> 1.2.3.64/26  -> A
> 1.2.3.128/25 -> A
>  (i.e. TWO rules)

This is definitely not the usage case. The origin of the data to be passed to 
ipfw tables are RIR delegation statistics files, which is guaranteed to be 
consolidated, namely resolved overlaps and joined adjacencies, long before any 
tables for ipfw are generated. Each range entry got a well defined, i.e. fixed, 
i.e non-variable starting address, and anything that changes the starting 
address of the ranges renders the table useless. Every entry got a well defined 
range length, and that one also must not be changed, or the table would be 
useless as well.

In addition, we are talking about automatic generation of thousands of entries, 
and I never ever won't rely on something like 'most specific rule wins' 
behaviour, I want the behaviour as explicit as possible, and for this reason I 
am happy with 'INPUT is 100 % identical to the OUTPUT'.

> you could also export
> 1.2.3.0/24 -> A
> 1.2.3.0/26 -> 0  (think of it as an "EXCEPT for these" rule)
> 
> which is ALSO two rules but you would need to be sure that the receiver knows 
> what to do with them.

This is simply a ridiculous example in the given respect, this sounds like you 
are suggesting fuzzying the input data in order to bring ipfw to its limits. 
This makes life less boring, doesn't it? No thanks.

>> It is worth to note, that already the original RIR delegation files contain 
>> 457 non CIDR conforming IPv4 ranges in a total of 165815 original records. I 
>> guess that this number will increase in the future because the RIR's ran 
>> empty on new IPv4 ranges and are urged to subdivide returned old ranges for 
>> new delegations. The above algorithm is ready for this.
>> 
>> Generally, CIDR conforming tables are more than twice as large as optimized 
>> (joined adjacencies) IP range tables. All said changes have been pushed to 
>> GitHup already.
>> 
> Unfortunately there is no way to specify (using cidr notation) a.b.1.x AND 
> a.b.2.x without including a.b.[03].x.
> 
> HOWEVER
> if you specified the FULL table you could use the "except" feature of routing 
> table behaviour where
> a.b.0.x/22  -> A
> a.b.0.x/24  -> B
> a.b.3.x/24  -> B
> gives you the same thing because of the 'most specific rule wins" nature of 
> routing table evaluation.
> I believe this is the case in the tables you imported.
> the trick is to be able to take an "optimised" table such as that above and 
> produce, given a required subset, just the required part, while changing the 
> rules as needed on the fly to "de-optimise" them enough to maintain 
> correctness.

Again, this is not the usage case.

>> I am still a little bit amazed how ipfw come to accept incorrect CIDR ranges 
>> and 

Re: ipfw divert filter for IPv4 geo-blocking

[Julian Elischer]

On 30/07/2016 10:17 PM, Dr. Rolf Jansen wrote:

Am 29.07.2016 um 10:23 schrieb Dr. Rolf Jansen :

Am 29.07.2016 um 06:50 schrieb Julian Elischer :
On 29/07/2016 5:22 PM, Julian Elischer wrote:

On 29/07/2016 4:53 PM, Dr. Rolf Jansen wrote:

Am 28.07.2016 um 23:48 schrieb Lee Brown :

That makes sense to me.  Your /20 range encompasses 201.222.16.0 -
201.222.31.255.
If you want 201.222.20.0-201.222.31.255, you'll need 3 ranges:

201.222.20.0/22 (201.222.20.0-201.222.23.255)
201.222.24.0/22 (201.222.24.0-201.222.27.255)
201.222.28.0/22 (201.222.28.0-201.222.31.255)

Ian, Julian and Lee,

Thank you vary much for your responses. In order not bloat the thread, I answer 
only to one message.

I found the problem. As a matter of fact, the respective IP ranges in the 
LACNIC delegation statistics file are 3 adjacent blocks with 1024 addresses, 
i.e. those that you listed in your message above:

$grep 201.222.2 /usr/local/etc/ipdb/IPRanges/lacnic.dat
lacnic|BR|ipv4|201.222.20.0|1024|20140710|allocated|164725
lacnic|BR|ipv4|201.222.24.0|1024|20140630|allocated|138376
lacnic|BR|ipv4|201.222.28.0|1024|20140701|allocated|129095

However, my database compilation combines adjacent blocks with the same country 
code, and the ranges above turn into one block of 3072 addresses, which 
obviously doesn't have a valid netmask - log(3072) = 11,5849625.
...
..., it is not sufficient to forget about optimization but I need to check also 
whether, the delegation files contain already some non-CIDR ranges, which need 
to be broken down.

there is code to take ranges and produce cidr sets.

We used to have exactly that code in the appletalk code before we took it out. 
Appletalk uses ranges.
https://svnweb.freebsd.org/base/release/3.2.0/sys/netatalk/at_control.c?view=annotate#l703

though htat uassumes input in the form af an appletak sockaddr..
there is also this python module
https://pythonhosted.org/netaddr/tutorial_01.html#support-for-non-standard-address-ranges


maybe you can find other versions on the net.
however if you fully populate the table, you will get the correct result 
because more specific entries will
override less specific entries. To do that you would have to have a way to 
describe to your program what
value each table entry should output.
If you did what you do now, then you would specify the value for the required countries, 
and give a default falue for "all others".
aggregation of adjacent ranges with same value would be an optimisation.

Don't worry, breaking down an arbitrary IP-range into a CIDR conforming set of 
ranges, doesn't seem too difficult. ...
...
Once I come to a conclusion, I will post it to this mailing list.

I finished the work on CIDR conformity of the IP ranges tables generated by the 
tool geoip. The main constraint is that the start and end address of an IP 
block given by the delegation files MUST BE PRESERVED during the transformation 
to a set of CIDR records. This target is achieved by:

  1. Finding the largest common netmask boundary of the start address utilizing
 int(log2(addr_count)); then iteration like Euclid's algorithm in computing
 a GCD.

  2. Output the CIDR with the given start address and the masklen belonging
 to the found netmask.

  3. If the CIDR does not match the whole original IP range then set the start
 address of the next CIDR block to the next boundary of the common netmask,
 and loop over starting at 1. until the original range has been satisfied.


check out the appletalk code I pointed out  to you.. I wrote that in 
93 or so but I remember sweating blood

over it to get it right.


I carefully tested the algorithm and a table that I pipe by the new geoip tool 
into ipfw is 100 % identical to the output of the ipfw command 'table N list'.
though that doesn't mean it is semantically identical to the original 
table due to 'most specific rule wins" behaviour.


for example:
if you type in ;

1.2.3.0/24 -> A
and
1.2.3.0/26 -> B
then both rules will be listed the same as what you put in
but if you wanted to get all rules that point to A, without having 
rules that point to B, then you would have to export

1.2.3.64/26  -> A
1.2.3.128/25 -> A
 (i.e. TWO rules)

you could also export
1.2.3.0/24 -> A
1.2.3.0/26 -> 0  (think of it as an "EXCEPT for these" rule)

which is ALSO two rules but you would need to be sure that the 
receiver knows what to do with them.




It is worth to note, that already the original RIR delegation files contain 457 
non CIDR conforming IPv4 ranges in a total of 165815 original records. I guess 
that this number will increase in the future because the RIR's ran empty on new 
IPv4 ranges and are urged to subdivide returned old ranges for new delegations. 
The above algorithm is ready for this.

Generally, CIDR conforming tables are more than twice as large as optimized 
(joined adjacencies) IP range tables. All said changes have been pushed to 
GitHup 

Re: ipfw divert filter for IPv4 geo-blocking

[Julian Elischer]

On 30/07/2016 10:17 PM, Dr. Rolf Jansen wrote:


I am still a little bit amazed how ipfw come to accept incorrect CIDR ranges 
and arbitrarily moves the start/end addresses in order to achieve CIDR 
conformity, and that without any further notice, and that given that ipfw can 
be considered as being quite relevant to system security. Or, may I assume that 
ipfw knows always better than the user what should be allowed or denied. 
Otherwise, perhaps I am the only one ever who input incorrect CIDR ranges for 
processing by ipfw.
it's not so amazing when you think about it. The code comes from the 
routing table..


In this context  a.b.c.d/N means "the range of addresses containing 
a.b.c.d, masked to a length of N".  there is no specification that 
a.b.c.d is the first address of the range.  I have relied upon this 
behaviour many times.




Best regards

Rolf

___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"



___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: ipfw divert filter for IPv4 geo-blocking

[Dr. Rolf Jansen]

> Am 31.07.2016 um 15:38 schrieb Ian Smith :
> On Sat, 30 Jul 2016 11:17:13 -0300, Dr. Rolf Jansen wrote:
>> I finished the work on CIDR conformity of the IP ranges tables 
>> generated by the tool geoip. The main constraint is that the start 
>> and end address of an IP block given by the delegation files MUST BE 
>> PRESERVED during the transformation to a set of CIDR records. This 
>> target is achieved by:
>> 
>> 1. Finding the largest common netmask boundary of the start address utilizing
>>int(log2(addr_count)); then iteration like Euclid's algorithm in computing
>>a GCD.
>> 
>> 2. Output the CIDR with the given start address and the masklen belonging
>>to the found netmask.
>> 
>> 3. If the CIDR does not match the whole original IP range then set the start
>>address of the next CIDR block to the next boundary of the common netmask,
>>and loop over starting at 1. until the original range has been satisfied.
>> 
>> I carefully tested the algorithm and a table that I pipe by the new 
>> geoip tool into ipfw is 100 % identical to the output of the ipfw 
>> command 'table N list'.
> 
> Great.  I suppose that caters for some of the odd delegations one sees, 
> such as perhaps a /16 then a /15 (ie 3/4 of a /14) followed by maybe a 
> /12, maybw with another /15 tacked on the end .. but I'm unsure if that 
> applies to country allocations as much as it does within countries.
> 
>> It is worth to note, that already the original RIR delegation files 
>> contain 457 non CIDR conforming IPv4 ranges in a total of 165815 
>> original records. I guess that this number will increase in the 
>> future because the RIR's ran empty on new IPv4 ranges and are urged 
>> to subdivide returned old ranges for new delegations. The above 
>> algorithm is ready for this.
> 
> Yes, and just as well.  I'm surprised it's as few as 457 .. I looked 
> into it a bit back when 115.70/17 was first allocated to my ISP, after 
> previously having been, as I recall, in China .. so of course we fell 
> foul of a number of (probably perennially) out-of-date geoip blockers, 
> for months in some cases .. malevolent beasts if not kept well fed :)
> 
>> Generally, CIDR conforming tables are more than twice as large as 
>> optimized (joined adjacencies) IP range tables. All said changes have 
>> been pushed to GitHup already.
> 
> So how many table entries does 'the world' come to, around 400,000?

No, it is not that bad. The total number of original entries in the delegation 
statistics files of all 5 RIR's is about 166000. The ipdb tool which compiles 
these ranges into a consolidated sorted binary table, that is suitable for 
loading it directly into a binary search tree, reduces the number of entries to 
a bit more than one half, namely ca. 83500.

Consolidation primarily means, resolving of overlaps, because these could not 
be handled in a meaningful way by a binary search tree. Only as an additional 
benefit in the same go, that routine combines adjacencies with the same country 
code, although, skipping the combination is technically not a show stopper for 
the BST, this is only to increase the performance.

The geoip tool which generates the tables of CIDR ranges per country code out 
of the consolidated tables would output a count of 167500 entries for all 
countries. That is a little bit more than the original count, however this 
table is still optimized, because original ranges that when combined form a new 
valid CIDR are not broken down again, but the combined CIDR is passed.

>> I am still a little bit amazed how ipfw come to accept incorrect CIDR 
>> ranges and arbitrarily moves the start/end addresses in order to 
>> achieve CIDR conformity, and that without any further notice, and 
>> that given that ipfw can be considered as being quite relevant to 
>> system security. Or, may I assume that ipfw knows always better than 
>> the user what should be allowed or denied. Otherwise, perhaps I am 
>> the only one ever who input incorrect CIDR ranges for processing by 
>> ipfw.
> 
> You've lost me here, Rolf.  Do you mean that ipfw adds incorrect table 
> entries for a given IPv4 address and mask length?  Or that it c/should 
> itself accept IP ranges and generate the needed CIDR entries to match?

Perhaps an example may explain it better. Remember that the first incarnation 
of geoip passed the incorrect range 201.222.20.0/20 to ipfw. This is an 
incorrect CIDR because the start address does not match a mask boundary defined 
by the given masklen. The point now is that this error is caused by EITHER the 
masklen is incorrect OR the start address is incorrect. ipfw can determine only 
that the CIDR is incorrect, and does rectify it for further processing:

  # ipfw table 1 add 201.222.20.0/20
  # ipfw table 1 list
  -->  201.222.16.0/20 0

So actually ipfw happily takes an incorrect CIDR and transforms this into a 
correct one under the arbitrary assumption that the masklen is the correct 

Re: ipfw divert filter for IPv4 geo-blocking

[Ian Smith]

On Sat, 30 Jul 2016 11:17:13 -0300, Dr. Rolf Jansen wrote:

 > I finished the work on CIDR conformity of the IP ranges tables 
 > generated by the tool geoip. The main constraint is that the start 
 > and end address of an IP block given by the delegation files MUST BE 
 > PRESERVED during the transformation to a set of CIDR records. This 
 > target is achieved by:
 > 
 >  1. Finding the largest common netmask boundary of the start address 
 > utilizing
 > int(log2(addr_count)); then iteration like Euclid's algorithm in 
 > computing
 > a GCD.
 > 
 >  2. Output the CIDR with the given start address and the masklen belonging
 > to the found netmask.
 > 
 >  3. If the CIDR does not match the whole original IP range then set the start
 > address of the next CIDR block to the next boundary of the common 
 > netmask,
 > and loop over starting at 1. until the original range has been satisfied.
 > 
 > I carefully tested the algorithm and a table that I pipe by the new 
 > geoip tool into ipfw is 100 % identical to the output of the ipfw 
 > command 'table N list'.

Great.  I suppose that caters for some of the odd delegations one sees, 
such as perhaps a /16 then a /15 (ie 3/4 of a /14) followed by maybe a 
/12, maybw with another /15 tacked on the end .. but I'm unsure if that 
applies to country allocations as much as it does within countries.

 > It is worth to note, that already the original RIR delegation files 
 > contain 457 non CIDR conforming IPv4 ranges in a total of 165815 
 > original records. I guess that this number will increase in the 
 > future because the RIR's ran empty on new IPv4 ranges and are urged 
 > to subdivide returned old ranges for new delegations. The above 
 > algorithm is ready for this.

Yes, and just as well.  I'm surprised it's as few as 457 .. I looked 
into it a bit back when 115.70/17 was first allocated to my ISP, after 
previously having been, as I recall, in China .. so of course we fell 
foul of a number of (probably perennially) out-of-date geoip blockers, 
for months in some cases .. malevolent beasts if not kept well fed :)

 > Generally, CIDR conforming tables are more than twice as large as 
 > optimized (joined adjacencies) IP range tables. All said changes have 
 > been pushed to GitHup already.

So how many table entries does 'the world' come to, around 400,000?

 > I am still a little bit amazed how ipfw come to accept incorrect CIDR 
 > ranges and arbitrarily moves the start/end addresses in order to 
 > achieve CIDR conformity, and that without any further notice, and 
 > that given that ipfw can be considered as being quite relevant to 
 > system security. Or, may I assume that ipfw knows always better than 
 > the user what should be allowed or denied. Otherwise, perhaps I am 
 > the only one ever who input incorrect CIDR ranges for processing by 
 > ipfw.

You've lost me here, Rolf.  Do you mean that ipfw adds incorrect table 
entries for a given IPv4 address and mask length?  Or that it c/should 
itself accept IP ranges and generate the needed CIDR entries to match?

If the former, how to reproduce for a bug report?  If the latter, might
you contemplate adding that functionality to ipfw - or is ipfw better 
off being driven to generate tables from the output of such as geoip?

cheers, Ian
___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: ipfw divert filter for IPv4 geo-blocking

[Dr. Rolf Jansen]

> Am 29.07.2016 um 10:23 schrieb Dr. Rolf Jansen :
>> Am 29.07.2016 um 06:50 schrieb Julian Elischer :
>> On 29/07/2016 5:22 PM, Julian Elischer wrote:
>>> On 29/07/2016 4:53 PM, Dr. Rolf Jansen wrote:
> Am 28.07.2016 um 23:48 schrieb Lee Brown :
> 
> That makes sense to me.  Your /20 range encompasses 201.222.16.0 -
> 201.222.31.255.
> If you want 201.222.20.0-201.222.31.255, you'll need 3 ranges:
> 
> 201.222.20.0/22 (201.222.20.0-201.222.23.255)
> 201.222.24.0/22 (201.222.24.0-201.222.27.255)
> 201.222.28.0/22 (201.222.28.0-201.222.31.255)
 
 Ian, Julian and Lee,
 
 Thank you vary much for your responses. In order not bloat the thread, I 
 answer only to one message.
 
 I found the problem. As a matter of fact, the respective IP ranges in the 
 LACNIC delegation statistics file are 3 adjacent blocks with 1024 
 addresses, i.e. those that you listed in your message above:
 
 $grep 201.222.2 /usr/local/etc/ipdb/IPRanges/lacnic.dat
 lacnic|BR|ipv4|201.222.20.0|1024|20140710|allocated|164725
 lacnic|BR|ipv4|201.222.24.0|1024|20140630|allocated|138376
 lacnic|BR|ipv4|201.222.28.0|1024|20140701|allocated|129095
 
 However, my database compilation combines adjacent blocks with the same 
 country code, and the ranges above turn into one block of 3072 addresses, 
 which obviously doesn't have a valid netmask - log(3072) = 11,5849625.
 ...
 ..., it is not sufficient to forget about optimization but I need to check 
 also whether, the delegation files contain already some non-CIDR ranges, 
 which need to be broken down.
>>> 
>>> there is code to take ranges and produce cidr sets.
>>> 
>>> We used to have exactly that code in the appletalk code before we took it 
>>> out. Appletalk uses ranges.
>>> https://svnweb.freebsd.org/base/release/3.2.0/sys/netatalk/at_control.c?view=annotate#l703
>>>  
>> 
>> though htat uassumes input in the form af an appletak sockaddr..
>> there is also this python module
>> https://pythonhosted.org/netaddr/tutorial_01.html#support-for-non-standard-address-ranges
>> 
>>> maybe you can find other versions on the net.
>>> however if you fully populate the table, you will get the correct result 
>>> because more specific entries will
>>> override less specific entries. To do that you would have to have a way to 
>>> describe to your program what
>>> value each table entry should output.
>>> If you did what you do now, then you would specify the value for the 
>>> required countries, and give a default falue for "all others".
>>> aggregation of adjacent ranges with same value would be an optimisation.
> 
> Don't worry, breaking down an arbitrary IP-range into a CIDR conforming set 
> of ranges, doesn't seem too difficult. ...
> ...
> Once I come to a conclusion, I will post it to this mailing list.

I finished the work on CIDR conformity of the IP ranges tables generated by the 
tool geoip. The main constraint is that the start and end address of an IP 
block given by the delegation files MUST BE PRESERVED during the transformation 
to a set of CIDR records. This target is achieved by:

 1. Finding the largest common netmask boundary of the start address utilizing
int(log2(addr_count)); then iteration like Euclid's algorithm in computing
a GCD.

 2. Output the CIDR with the given start address and the masklen belonging
to the found netmask.

 3. If the CIDR does not match the whole original IP range then set the start
address of the next CIDR block to the next boundary of the common netmask,
and loop over starting at 1. until the original range has been satisfied.

I carefully tested the algorithm and a table that I pipe by the new geoip tool 
into ipfw is 100 % identical to the output of the ipfw command 'table N list'.

It is worth to note, that already the original RIR delegation files contain 457 
non CIDR conforming IPv4 ranges in a total of 165815 original records. I guess 
that this number will increase in the future because the RIR's ran empty on new 
IPv4 ranges and are urged to subdivide returned old ranges for new delegations. 
The above algorithm is ready for this.

Generally, CIDR conforming tables are more than twice as large as optimized 
(joined adjacencies) IP range tables. All said changes have been pushed to 
GitHup already.

I am still a little bit amazed how ipfw come to accept incorrect CIDR ranges 
and arbitrarily moves the start/end addresses in order to achieve CIDR 
conformity, and that without any further notice, and that given that ipfw can 
be considered as being quite relevant to system security. Or, may I assume that 
ipfw knows always better than the user what should be allowed or denied. 
Otherwise, perhaps I am the only one ever who input incorrect CIDR ranges for 
processing by ipfw.

Best regards

Rolf

___

Re: ipfw divert filter for IPv4 geo-blocking

[Dr. Rolf Jansen]

> Am 29.07.2016 um 06:50 schrieb Julian Elischer :
> On 29/07/2016 5:22 PM, Julian Elischer wrote:
>> On 29/07/2016 4:53 PM, Dr. Rolf Jansen wrote:
 Am 28.07.2016 um 23:48 schrieb Lee Brown :
 
 That makes sense to me.  Your /20 range encompasses 201.222.16.0 -
 201.222.31.255.
 If you want 201.222.20.0-201.222.31.255, you'll need 3 ranges:
 
 201.222.20.0/22 (201.222.20.0-201.222.23.255)
 201.222.24.0/22 (201.222.24.0-201.222.27.255)
 201.222.28.0/22 (201.222.28.0-201.222.31.255)
>>> 
>>> Ian, Julian and Lee,
>>> 
>>> Thank you vary much for your responses. In order not bloat the thread, I 
>>> answer only to one message.
>>> 
>>> I found the problem. As a matter of fact, the respective IP ranges in the 
>>> LACNIC delegation statistics file are 3 adjacent blocks with 1024 
>>> addresses, i.e. those that you listed in your message above:
>>> 
>>> $grep 201.222.2 /usr/local/etc/ipdb/IPRanges/lacnic.dat
>>> lacnic|BR|ipv4|201.222.20.0|1024|20140710|allocated|164725
>>> lacnic|BR|ipv4|201.222.24.0|1024|20140630|allocated|138376
>>> lacnic|BR|ipv4|201.222.28.0|1024|20140701|allocated|129095
>>> 
>>> However, my database compilation combines adjacent blocks with the same 
>>> country code, and the ranges above turn into one block of 3072 addresses, 
>>> which obviously doesn't have a valid netmask - log(3072) = 11,5849625.
>>> ...
>>> ..., it is not sufficient to forget about optimization but I need to check 
>>> also whether, the delegation files contain already some non-CIDR ranges, 
>>> which need to be broken down.
>> 
>> there is code to take ranges and produce cidr sets.
>> 
>> We used to have exactly that code in the appletalk code before we took it 
>> out. Appletalk uses ranges.
>> https://svnweb.freebsd.org/base/release/3.2.0/sys/netatalk/at_control.c?view=annotate#l703
>>  
> 
> though htat uassumes input in the form af an appletak sockaddr..
> there is also this python module
> https://pythonhosted.org/netaddr/tutorial_01.html#support-for-non-standard-address-ranges
> 
>> maybe you can find other versions on the net.
>> however if you fully populate the table, you will get the correct result 
>> because more specific entries will
>> override less specific entries. To do that you would have to have a way to 
>> describe to your program what
>> value each table entry should output.
>> If you did what you do now, then you would specify the value for the 
>> required countries, and give a default falue for "all others".
>> aggregation of adjacent ranges with same value would be an optimisation.

Don't worry, breaking down an arbitrary IP-range into a CIDR conforming set of 
ranges, doesn't seem too difficult. I quickly hacked into the   for() loop for 
table generation in geoip.c a nested do while() for if necessary breaking down 
the given range:

for (i = 0; i < n; i++)
{
   if (!*ccList || findCC(CCTable, sortedIPSets[i][2]))
   {
  ipv4_lo = sortedIPSets[i][0];
  do
  {
 m = intlb(sortedIPSets[i][1] - ipv4_lo  + 1);
 while (ipv4_lo  % (k = (int)lround(exp2(m m--;
 printRange(ipv4_lo, 32 - m, table_number, table_value);
 
  }
  while ((ipv4_lo += k) < sortedIPSets[i][1]);
   }
}

This seems to work as expected, however, I need to check this more carefully 
(in the course of the weekend) until pushing it to GitHub. At the first glance, 
the tables become quite large. For example, the table for Brazil of 824 joined 
entries is bloated to 6621 CIDR records.

Once I came to a conclusion, I will post it to this mailing list.

Best regards

Rolf

___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: ipfw divert filter for IPv4 geo-blocking

[Ian Smith]

On Thu, 28 Jul 2016 23:21:01 -0300, Dr. Rolf Jansen wrote: > Am 
27.07.2016 um 12:31 schrieb Julian Elischer :
 [..]
 >> wow, wonderful!

 >> with that tool, and ipfw tables we have a fully functional geo
 >> blocking/munging solution in about 4 lines of shell script.

 > Unfortunately, I finally discovered that ipfw tables as they are, are 
 > unsuitable for the given purpose, because for some reason ipfw
 > mangles about 20 % of the passed IP address/masklen pairs.

 > For example:

 > # ipfw table 1 add 201.222.20.0/20
 > # ipfw table 1 list
 > -->  201.222.16.0/20 0

 > $ geoip 201.222.20.1
 > --> 201.222.20.1 in 201.222.20.0-201.222.31.255 in BR

 > $ geoip 201.222.16.1
 > --> 201.222.16.1 in 201.222.16.0-201.222.19.255 in AR

Just to add to what Julian and Lee observed, testing IPs at 

(sourced from LACNIC thence whois.registro.br)

inetnum: 201.222.20/22
aut-num: AS61902
abuse-c: CSJ45
owner: Bahialink - Technology
ownerid: 004.724.687/0001-69
country: BR

So the geoip result for 201.222.20.1 is just wrong - it should return

  201.222.20.0 - 201.222.23.255 (ie, /22)

and not

  201.222.16.0 - 201.222.31.255 (ie, /20)

While the range for 201.222.16.1 is in fact a /22:


[..]
inetnum: 201.222.16/22
status: allocated
aut-num: N/A
owner: G2KHosting S.A.
ownerid: AR-GKSA-LACNIC
responsible: Mauro Ferraro
address: Maipu, 33,
address: 2900 - San Nicolas de los Arroyos - BA
country: AR

 > Effectively, I asked ipfw to add an IP-range of Brazil to table 1, 
 > but it actually added another one which belongs to Argentina. This 
 > doesn't make too much sense, does it?

Not if geoip is returning the wrong address range for 201.222.20.1, no.

 > For the time being I switched my servers back to geo-blocking with 
 > the divert filter daemon.

I don't know what's wrong or where, just that it is ..

How are you getting from geoip's IP range to /maskbits?

cheers, Ian
___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: ipfw divert filter for IPv4 geo-blocking

[Julian Elischer]

On 29/07/2016 10:48 AM, Lee Brown wrote:

That makes sense to me.  Your /20 range encompasses 201.222.16.0 -
201.222.31.255.
If you want 201.222.20.0-201.222.31.255, you'll need 3 ranges:
whether it makes sense depends on whether you add the other ranges as 
well with  the default result.
Your daemon has the entire set loaded so it can exclude other internal 
ranges, however it looks like you have only provided

ipfw with partial information.


you'd need to add:

201.222.20.0/20  1
201.222.16.0/22  0

or more correctly what Lee showed above

either the imput data is incorrect, or your cidr aggregation  code has a bug.
because 201.222.20.0/20 (well more correctly "misleading") cidr description).
It implies 201.222.16.0/20  because if you mask 201.222.20.0 with 255.255.240.0 
 (/20) that's what you get.

observe:
soekris# route add 201.222.20.0/20 127.0.0.1
add net 201.222.20.0: gateway 127.0.0.1
soekris# netstat -rn
Routing tables

Internet:
DestinationGatewayFlagsRefs  Use  Netif Expire
default192.168.0.1UGS 0  9082198vr2
[...]
201.222.16.0/20127.0.0.1  US  00lo0
[...]

to make this work correctly you'd either need lee's answer, or you 
need to give it more information:

201.222.16.0/20  {result for brazil}
201.222.16.0/22  {default result, to get subtracted from the above.
your program give s the correct result because it has all the 
information, including the AR information

but you didn't tell ipfw about that.
I suggest you need either input data sanitization, or to fix our 
output code to give the correct subranges.

becasue 201.222.16.0/20 is definately bad input to ipfw.



201.222.20.0/22 (201.222.20.0-201.222.23.255)
201.222.24.0/22 (201.222.24.0-201.222.27.255)
201.222.28.0/22 (201.222.28.0-201.222.31.255)

this  helps :)

On Thu, Jul 28, 2016 at 7:21 PM, Dr. Rolf Jansen  wrote:


Am 27.07.2016 um 12:31 schrieb Julian Elischer :
On 27/07/2016 9:36 PM, Dr. Rolf Jansen wrote:

Am 26.07.2016 um 23:03 schrieb Julian Elischer :
On 27/07/2016 3:06 AM, Dr. Rolf Jansen wrote:

There is another tool called geoip , that I uploaded to GitHub, and

that I use for looking up country codes by IP addresses on the command line.

 https://github.com/cyclaero/ipdb/blob/master/geoip.c

This one could easily be extended to produce sorted IP ranges per CC

that could be fed into tables of ipfw. I am thinking of adding a command
line option for specifying CC's for which the IP ranges should be exported,
something like:

geoip -e DE:BR:US:IT:FR:ES

And this could print sorted IP-Ranges belonging to the listed

countries. For this purpose, what would be the ideal format for directly
feeding the produced output into ipfw tables?

The format for using tables directly is the same as that used for

routing tables.

…
table 5 add 1.1.1.0/32 1000
…
your application becomes an application for configuring the firewall.
(which you do by feeding commands down a pipe to ipfw, which is

started as 'ipfw -q /dev/stdin')

I finished adding a second usage form for the geoip tool, namely

generation of ipfw table construction directives filtered by country codes.

wow, wonderful!

with that tool, and ipfw tables we have a fully functional geo

blocking/munging solution in about 4 lines of shell script.

Unfortunately, I finally discovered that ipfw tables as they are, are
unsuitable for the given purpose, because for some reason ipfw mangles
about 20 % of the passed IP address/masklen pairs.

For example:

# ipfw table 1 add 201.222.20.0/20
# ipfw table 1 list
-->  201.222.16.0/20 0

$ geoip 201.222.20.1
--> 201.222.20.1 in 201.222.20.0-201.222.31.255 in BR

$ geoip 201.222.16.1
--> 201.222.16.1 in 201.222.16.0-201.222.19.255 in AR

Effectively, I asked ipfw to add an IP-range of Brazil to table 1, but it
actually added another one which belongs to Argentina. This doesn't make
too much sense, does it?

For the time being I switched my servers back to geo-blocking with the
divert filter daemon.

Best regards

Rolf

___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"




___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: ipfw divert filter for IPv4 geo-blocking

[Lee Brown]

That makes sense to me.  Your /20 range encompasses 201.222.16.0 -
201.222.31.255.
If you want 201.222.20.0-201.222.31.255, you'll need 3 ranges:

201.222.20.0/22 (201.222.20.0-201.222.23.255)
201.222.24.0/22 (201.222.24.0-201.222.27.255)
201.222.28.0/22 (201.222.28.0-201.222.31.255)

this  helps :)

On Thu, Jul 28, 2016 at 7:21 PM, Dr. Rolf Jansen  wrote:

>
> > Am 27.07.2016 um 12:31 schrieb Julian Elischer :
> > On 27/07/2016 9:36 PM, Dr. Rolf Jansen wrote:
> >>> Am 26.07.2016 um 23:03 schrieb Julian Elischer :
> >>> On 27/07/2016 3:06 AM, Dr. Rolf Jansen wrote:
>  There is another tool called geoip , that I uploaded to GitHub, and
> that I use for looking up country codes by IP addresses on the command line.
> 
>  https://github.com/cyclaero/ipdb/blob/master/geoip.c
> 
>  This one could easily be extended to produce sorted IP ranges per CC
> that could be fed into tables of ipfw. I am thinking of adding a command
> line option for specifying CC's for which the IP ranges should be exported,
> something like:
> 
> geoip -e DE:BR:US:IT:FR:ES
> 
>  And this could print sorted IP-Ranges belonging to the listed
> countries. For this purpose, what would be the ideal format for directly
> feeding the produced output into ipfw tables?
> >>> The format for using tables directly is the same as that used for
> routing tables.
> >>> …
> >>> table 5 add 1.1.1.0/32 1000
> >>> …
> >>> your application becomes an application for configuring the firewall.
> >>> (which you do by feeding commands down a pipe to ipfw, which is
> started as 'ipfw -q /dev/stdin')
> >> I finished adding a second usage form for the geoip tool, namely
> generation of ipfw table construction directives filtered by country codes.
> > wow, wonderful!
> >
> > with that tool, and ipfw tables we have a fully functional geo
> blocking/munging solution in about 4 lines of shell script.
>
> Unfortunately, I finally discovered that ipfw tables as they are, are
> unsuitable for the given purpose, because for some reason ipfw mangles
> about 20 % of the passed IP address/masklen pairs.
>
> For example:
>
> # ipfw table 1 add 201.222.20.0/20
> # ipfw table 1 list
> -->  201.222.16.0/20 0
>
> $ geoip 201.222.20.1
> --> 201.222.20.1 in 201.222.20.0-201.222.31.255 in BR
>
> $ geoip 201.222.16.1
> --> 201.222.16.1 in 201.222.16.0-201.222.19.255 in AR
>
> Effectively, I asked ipfw to add an IP-range of Brazil to table 1, but it
> actually added another one which belongs to Argentina. This doesn't make
> too much sense, does it?
>
> For the time being I switched my servers back to geo-blocking with the
> divert filter daemon.
>
> Best regards
>
> Rolf
>
> ___
> freebsd-ipfw@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
> To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"
___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: ipfw divert filter for IPv4 geo-blocking

[Dr. Rolf Jansen]

> Am 27.07.2016 um 12:31 schrieb Julian Elischer :
> On 27/07/2016 9:36 PM, Dr. Rolf Jansen wrote:
>>> Am 26.07.2016 um 23:03 schrieb Julian Elischer :
>>> On 27/07/2016 3:06 AM, Dr. Rolf Jansen wrote:
 There is another tool called geoip , that I uploaded to GitHub, and that I 
 use for looking up country codes by IP addresses on the command line.
 
 https://github.com/cyclaero/ipdb/blob/master/geoip.c
 
 This one could easily be extended to produce sorted IP ranges per CC that 
 could be fed into tables of ipfw. I am thinking of adding a command line 
 option for specifying CC's for which the IP ranges should be exported, 
 something like:
 
geoip -e DE:BR:US:IT:FR:ES
 
 And this could print sorted IP-Ranges belonging to the listed countries. 
 For this purpose, what would be the ideal format for directly feeding the 
 produced output into ipfw tables?
>>> The format for using tables directly is the same as that used for routing 
>>> tables.
>>> …
>>> table 5 add 1.1.1.0/32 1000
>>> …
>>> your application becomes an application for configuring the firewall.
>>> (which you do by feeding commands down a pipe to ipfw, which is started as 
>>> 'ipfw -q /dev/stdin')
>> I finished adding a second usage form for the geoip tool, namely generation 
>> of ipfw table construction directives filtered by country codes.
> wow, wonderful!
> 
> with that tool, and ipfw tables we have a fully functional geo 
> blocking/munging solution in about 4 lines of shell script.

Unfortunately, I finally discovered that ipfw tables as they are, are 
unsuitable for the given purpose, because for some reason ipfw mangles about 20 
% of the passed IP address/masklen pairs.

For example:

# ipfw table 1 add 201.222.20.0/20
# ipfw table 1 list
-->  201.222.16.0/20 0

$ geoip 201.222.20.1
--> 201.222.20.1 in 201.222.20.0-201.222.31.255 in BR

$ geoip 201.222.16.1
--> 201.222.16.1 in 201.222.16.0-201.222.19.255 in AR

Effectively, I asked ipfw to add an IP-range of Brazil to table 1, but it 
actually added another one which belongs to Argentina. This doesn't make too 
much sense, does it?

For the time being I switched my servers back to geo-blocking with the divert 
filter daemon.

Best regards

Rolf

___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: ipfw divert filter for IPv4 geo-blocking

[olli hauer]

On 2016-07-27 23:15, Dr. Rolf Jansen wrote:
>> Am 27.07.2016 um 17:08 schrieb olli hauer :
>> On 2016-07-27 15:36, Dr. Rolf Jansen wrote:
>>>
>>> I finished adding a second usage form for the geoip tool, namely generation 
>>> of ipfw table construction directives filtered by country codes.
>>>
>>> __
>>> $ geoip -h
>>> geoip v1.0.1 (16), Copyright © 2016 Dr. Rolf Jansen
>>>
>>> Usage:
>>>
>>> 1) look-up the country code belonging to an IPv4 address given by the last 
>>> command line argument:
>>>
>>>   geoip [-r bstfile] [-h] 
>>>  a dotted IPv4 address to be looked-up.
>>>
>>> 2) generate a sorted list of IPv4 address/masklen pairs per country code, 
>>> formatted as ipfw table construction directives:
>>>
>>>   geoip -t [CC:DD:EE:..] [-n table number] [-v table value] [-r bstfile] 
>>> [-h]
>>>
>>>  -t [CC:DD:EE:..]  output all IPv4 address/masklen pairs belonging to 
>>> the listed countries, given by 2 letter
>>>capital country codes, separated by colon. An empty 
>>> CC list means any country code.
>>>  -n table number   the ipfw table number between 0 and 65534 [default: 
>>> 0].
>>>  -v table valuethe 32-bit unsigned value of the ipfw table entry 
>>> [default: 0].
>>>
>>> valid arguments in both usage forms:
>>>
>>>  -r bstfilethe path to the binary file with the consolidated IP 
>>> ranges that has been.
>>>generated by the 'ipdb' tool [default: 
>>> /usr/local/etc/ipdb/IPRanges/ipcc.bst].
>>>  -hshow these usage instructions.
>>> __
>>>
>>> With that, the ipfw configuration script may contain something alike:
>>>
>>>…
>>># allow only web access from DE, BR, US:
>>>/usr/local/bin/geoip -t DE:BR:US -n 7 | /sbin/ipfw -q /dev/stdin
>>>/sbin/ipfw -q add 70 deny tcp from not table\(7\) to any 80,443 in recv 
>>> WAN_if setup
>>>…
>>>
>>> OR, the other way around:
>>>…
>>># deny web access from certain disgraceful regions:
>>>/usr/local/bin/geoip -t KO:TR:SA:RU:GB -n 66 | /sbin/ipfw -q /dev/stdin
>>>/sbin/ipfw -q add 70 allow tcp from not table\(66\) to any 80,443 in 
>>> recv WAN_if setup
>>>…
>>> 
>>
>> Nice work :)
>>
>> Now it is also possible to use geoip to create files usable for pf.
>> (just pipe the output through sed -e 's/table 0 add //')
>>
>> Perhaps the following diff for Makefile is useful.
>> - use PREFIX instead hard coded path
>> - use "install -s" instead "strip -x -o"
>> - use "install -m" instead "cp ; chmod"
> 
> I changed the Makefile according to your suggestions, and I added another 
> command line option to the geoip tool:
> 
> …
>   -p   plain IP table generation, i.e. without ipfw construction 
> directives, -n and -v flags are ignored.
> …
> 
> The changes are already uploaded to GitHub.


Thank you :)

-- 
Regards,
olli
___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: ipfw divert filter for IPv4 geo-blocking

[Dr. Rolf Jansen]

> Am 27.07.2016 um 17:08 schrieb olli hauer :
> On 2016-07-27 15:36, Dr. Rolf Jansen wrote:
>> 
>> I finished adding a second usage form for the geoip tool, namely generation 
>> of ipfw table construction directives filtered by country codes.
>> 
>> __
>> $ geoip -h
>> geoip v1.0.1 (16), Copyright © 2016 Dr. Rolf Jansen
>> 
>> Usage:
>> 
>> 1) look-up the country code belonging to an IPv4 address given by the last 
>> command line argument:
>> 
>>   geoip [-r bstfile] [-h] 
>>  a dotted IPv4 address to be looked-up.
>> 
>> 2) generate a sorted list of IPv4 address/masklen pairs per country code, 
>> formatted as ipfw table construction directives:
>> 
>>   geoip -t [CC:DD:EE:..] [-n table number] [-v table value] [-r bstfile] [-h]
>> 
>>  -t [CC:DD:EE:..]  output all IPv4 address/masklen pairs belonging to 
>> the listed countries, given by 2 letter
>>capital country codes, separated by colon. An empty 
>> CC list means any country code.
>>  -n table number   the ipfw table number between 0 and 65534 [default: 
>> 0].
>>  -v table valuethe 32-bit unsigned value of the ipfw table entry 
>> [default: 0].
>> 
>> valid arguments in both usage forms:
>> 
>>  -r bstfilethe path to the binary file with the consolidated IP 
>> ranges that has been.
>>generated by the 'ipdb' tool [default: 
>> /usr/local/etc/ipdb/IPRanges/ipcc.bst].
>>  -hshow these usage instructions.
>> __
>> 
>> With that, the ipfw configuration script may contain something alike:
>> 
>>…
>># allow only web access from DE, BR, US:
>>/usr/local/bin/geoip -t DE:BR:US -n 7 | /sbin/ipfw -q /dev/stdin
>>/sbin/ipfw -q add 70 deny tcp from not table\(7\) to any 80,443 in recv 
>> WAN_if setup
>>…
>> 
>> OR, the other way around:
>>…
>># deny web access from certain disgraceful regions:
>>/usr/local/bin/geoip -t KO:TR:SA:RU:GB -n 66 | /sbin/ipfw -q /dev/stdin
>>/sbin/ipfw -q add 70 allow tcp from not table\(66\) to any 80,443 in recv 
>> WAN_if setup
>>…
>> 
> 
> Nice work :)
> 
> Now it is also possible to use geoip to create files usable for pf.
> (just pipe the output through sed -e 's/table 0 add //')
> 
> Perhaps the following diff for Makefile is useful.
> - use PREFIX instead hard coded path
> - use "install -s" instead "strip -x -o"
> - use "install -m" instead "cp ; chmod"

I changed the Makefile according to your suggestions, and I added another 
command line option to the geoip tool:

…
  -p   plain IP table generation, i.e. without ipfw construction 
directives, -n and -v flags are ignored.
…

The changes are already uploaded to GitHub.

Best regards

Rolf

___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: ipfw divert filter for IPv4 geo-blocking

[olli hauer]

On 2016-07-27 15:36, Dr. Rolf Jansen wrote:
>> Am 26.07.2016 um 23:03 schrieb Julian Elischer :
>> On 27/07/2016 3:06 AM, Dr. Rolf Jansen wrote:
>>> There is another tool called geoip , that I uploaded to GitHub, and that I 
>>> use for looking up country codes by IP addresses on the command line.
>>>
>>> https://github.com/cyclaero/ipdb/blob/master/geoip.c
>>>
>>> This one could easily be extended to produce sorted IP ranges per CC that 
>>> could be fed into tables of ipfw. I am thinking of adding a command line 
>>> option for specifying CC's for which the IP ranges should be exported, 
>>> something like:
>>>
>>>geoip -e DE:BR:US:IT:FR:ES
>>>
>>> And this could print sorted IP-Ranges belonging to the listed countries. 
>>> For this purpose, what would be the ideal format for directly feeding the 
>>> produced output into ipfw tables?
>> The format for using tables directly is the same as that used for routing 
>> tables.
>> …
>> table 5 add 1.1.1.0/32 1000
>> …
>> your application becomes an application for configuring the firewall.
>> (which you do by feeding commands down a pipe to ipfw, which is started as 
>> 'ipfw -q /dev/stdin')
> 
> I finished adding a second usage form for the geoip tool, namely generation 
> of ipfw table construction directives filtered by country codes.
> 
> __
> $ geoip -h
> geoip v1.0.1 (16), Copyright © 2016 Dr. Rolf Jansen
> 
> Usage:
> 
> 1) look-up the country code belonging to an IPv4 address given by the last 
> command line argument:
> 
>geoip [-r bstfile] [-h] 
>   a dotted IPv4 address to be looked-up.
> 
> 2) generate a sorted list of IPv4 address/masklen pairs per country code, 
> formatted as ipfw table construction directives:
> 
>geoip -t [CC:DD:EE:..] [-n table number] [-v table value] [-r bstfile] [-h]
> 
>   -t [CC:DD:EE:..]  output all IPv4 address/masklen pairs belonging to 
> the listed countries, given by 2 letter
> capital country codes, separated by colon. An empty 
> CC list means any country code.
>   -n table number   the ipfw table number between 0 and 65534 [default: 
> 0].
>   -v table valuethe 32-bit unsigned value of the ipfw table entry 
> [default: 0].
> 
> valid arguments in both usage forms:
> 
>   -r bstfilethe path to the binary file with the consolidated IP 
> ranges that has been.
> generated by the 'ipdb' tool [default: 
> /usr/local/etc/ipdb/IPRanges/ipcc.bst].
>   -hshow these usage instructions.
> __
> 
> With that, the ipfw configuration script may contain something alike:
> 
> …
> # allow only web access from DE, BR, US:
> /usr/local/bin/geoip -t DE:BR:US -n 7 | /sbin/ipfw -q /dev/stdin
> /sbin/ipfw -q add 70 deny tcp from not table\(7\) to any 80,443 in recv 
> WAN_if setup
> …
> 
> OR, the other way around:
> …
> # deny web access from certain disgraceful regions:
> /usr/local/bin/geoip -t KO:TR:SA:RU:GB -n 66 | /sbin/ipfw -q /dev/stdin
> /sbin/ipfw -q add 70 allow tcp from not table\(66\) to any 80,443 in recv 
> WAN_if setup
> …
> 
> 
> 
> Best regards
> 
> Rolf

Nice work :)

Now it is also possible to use geoip to create files usable for pf.
(just pipe the output through sed -e 's/table 0 add //')

Perhaps the following diff for Makefile is useful.
- use PREFIX instead hard coded path
- use "install -s" instead "strip -x -o"
- use "install -m" instead "cp ; chmod"


--- Makefile.orig   2016-07-27 21:44:25.617707000 +0200
+++ Makefile2016-07-27 21:47:18.341092000 +0200
@@ -43,6 +43,7 @@
 CC= clang
 CFLAGS= $(CDEFS) -DSVNREV=\"$(REVNUM)\" -std=c11 -g0 -Ofast -mssse3 
-Wno-parentheses -Wno-empty-body
 LDFLAGS   = -lm
+PREFIX?= /usr/local

 HEADER= store.h
 SOURCES   = store.c ipdb.c geoip.c geod.c
@@ -71,9 +72,8 @@
 update: clean all

 install: ipdb geoip geod
-   strip -x -o /usr/local/bin/ipdb ipdb
-   strip -x -o /usr/local/bin/geoip geoip
-   strip -x -o /usr/local/bin/geod geod
-   cp geod.rc /usr/local/etc/rc.d/geod
-   chmod 555 /usr/local/etc/rc.d/geod
-   cp ipdb-update.sh /usr/local/bin/ipdb-update.sh
+   install -s -m 555 ipdb ${PREFIX}/bin/ipdb
+   install -s -m 555 geoip ${PREFIX}/bin/geoip
+   install -s -m 555 geod ${PREFIX}/bin/geod
+   install -m 555 geod.rc ${PREFIX}/etc/rc.d/geod
+   install -m 555 ipdb-update.sh ${PREFIX}/bin/ipdb-update.sh


-- 
olli
___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: ipfw divert filter for IPv4 geo-blocking

[Julian Elischer]

trimming

On 27/07/2016 11:51 PM, Ian Smith wrote:

On Wed, 27 Jul 2016 10:03:01 +0800, Julian Elischer wrote:


[...]



  > country without changing everything else.
  > (the downside is that dynamic skipto's are not very efficient as they do a
  > linear search of the rules, where static skiptos cache the location of the
  > rule to skip to. it's not a terrible cost but it needs to be  kept in mind.
  > (but faster than a divert socket)

I forget .. is that linear search from the beginning, or from the
position of the rule querying the table?  Just thnking about grouping
skipto target rules to minimise traversal.  These targets in turn could
use static skiptos that will be cached.

it starts searching forwards from the current location, to stop loops.
(though it turns out you CAN make loops using some arcane sequences 
that I will not make public).


However divert reinjection searches from the start to get to the place 
you want to restart processing.

(but it's a very small loop) so put the diverts near the front if you can.



  > your application becomes an application for configuring the firewall.
  > (which you do by feeding commands down a pipe to ipfw, which is started as
  > 'ipfw -q /dev/stdin')

I went looking though ports for ipfw-classifyd, which attracted my
interest in 2008, but seems never to have made it to ports.  Written by
Mike Makonnen  (cc'd), it uses divert sockets with the
linux- based 'l7' filters for detecting traffic from a wide array of UDP
and TCP protocols, with the primary intent then of detecting various P2P
traffic and shunting it through dummynet pipes for bandwidth limiting.

I vaguely remember it.


Interesting discussion, and thanks for info on geoip tables etc.

cheers, Ian



___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: ipfw divert filter for IPv4 geo-blocking

[Ian Smith]

On Wed, 27 Jul 2016 10:03:01 +0800, Julian Elischer wrote:
 > On 27/07/2016 3:06 AM, Dr. Rolf Jansen wrote:
 > > > Am 26.07.2016 um 13:23 schrieb Julian Elischer :
 > > > On 26/07/2016 1:41 AM, Dr. Rolf Jansen wrote:
 > > > > Once a week, the IP ranges are compiled from original sources into a
 > > > > binary sorted table, containing as of today 83162 consolidated range/cc
 > > > > pairs. On starting-up, the divert daemon reads the binary file in one
 > > > > block and stores the ranges into a totally balanced binary search tree.
 > > > > Looking-up a country code for a given IPv4 address in the BST takes on
 > > > > average 20 nanoseconds on an AWS-EC2 micro instance. I don't know the
 > > > > overhead of diverting, though. I guess this may be one or two orders of
 > > > > magnitudes higher. Even though, I won't see any performance issues.

 > > > yes the diversion to user space is not a fast operation. When we wrote
 > > > it, fast was 10Mbits/sec.
 > > > The firewall tables use a radix tree (*) and might be slower than what
 > > > you have, but possibly it might be made up for by not having to do the
 > > > divert logic. it's not entorely clear from your description why you look
 > > > up  a country rather than just a pass/block result, but maybe different
 > > > sources can access different countries?.

 > > The basic idea was to develop a facility for ipfw for filtering IPv4
 > > packets by country code - see: https://github.com/cyclaero/ipdb
 > > 
 > > I simply put into /etc/rc.conf:
 > > 
 > > geod_enable="YES"
 > > geod_flags="-a DE:BR:US"
 > > 
 > > The -a flag tells, that source IP addresses only from these countries are
 > > allowed (i.e. passed through the filter). I added also a -d flag, which
 > > means deny (i.e. drop packets) from the given list of countries.
 > > 
 > > With that in place, I need to add a respective divert rule to the ipfw
 > > ruleset (the divert port of the geod daemon is 8669, remembering that 8668
 > > is the port of the natd daemon):
 > > 
 > >  ipfw -q add 70 divert 8669 tcp from any to any 80,443 in recv WAN_if
 > > setup
 > > 
 > > > I did similar once using ipfw tables but couldn't find a reliable source
 > > > of data.

 > > The IP/CC database is compiled from downloads of the daily published
 > > delegation statistics files of the 5 RIR's. I consider the RIR's being the
 > > authoritative source. Anyway, on my systems the IP/CC-database is updated
 > > only weekly, although, daily updating would be possible. I wrote a shell
 > > script for this, that can be executed by a cron job.
 > > 
 > >  https://github.com/cyclaero/ipdb/blob/master/ipdb-update.sh
 > > 
 > > 
 > > There is another tool called geoip , that I uploaded to GitHub, and that I
 > > use for looking up country codes by IP addresses on the command line.
 > > 
 > >  https://github.com/cyclaero/ipdb/blob/master/geoip.c
 > > 
 > > This one could easily be extended to produce sorted IP ranges per CC that
 > > could be fed into tables of ipfw. I am thinking of adding a command line
 > > option for specifying CC's for which the IP ranges should be exported,
 > > something like:
 > > 
 > > geoip -e DE:BR:US:IT:FR:ES
 > > 
 > > And this could print sorted IP-Ranges belonging to the listed countries.
 > > For this purpose, what would be the ideal format for directly feeding the
 > > produced output into ipfw tables?

 > The format for using tables directly is the same as that used for routing
 > tables.
 > so imagine that you had to generate a routing table that sent packets to two
 > different routers depending on their source.
 > 
 > here's a simple rule set that filters web traffic by such a 'routing table'
 > except it's routing to two different rules. It also sorts OUTGOING web
 > traffic to the same rules.
 > 
 > ipfw -q /dev/stdin <<-DONE
 > # we hate this guy
 > table 5 add 1.1.1.0/32 1000

Yeah, I've had trouble with him too ..

 > # but all ow our people to visit everyone else in that subnet
 > table 5 add 1.1.0.0/24 2000
 > # we block 1.1.2.0 through 1.1.3.255
 > table 5 add 1.1.2.0/23 1000
 > # but we allow 1.1.4.0 through to 1.1.7.255
 > table 5 add 1.1.4.0/22 2000
 > # etc
 > table 5 add 1.1.8.0/21 1000
 > table 5 add 1.2.0.0/16 1000

 > table 5 add 0.0.0.0/0 2000 # default

Now this was news to me.  It's obvious, especially knowing it uses the 
same radix table method as does routing, but never occurred to me.  Ta!

 > check-state  # If we already decided what to do,  do it
 > # select out only external traffic, into direction specific rules.
 > add 400 skipto 500 ip from any to any in recv WAN_if
 > add 410 skipto 700 ip from any to any out xmit WAN_If
 > add 320 skipto 1 # a 420 moment?
 > # Incoming packets
 > add 500 skipto tablearg tcp from table(5) to any 80,443 setup keep-state  #
 >  sort tcp setup packets between rules 1000 and 2000
 > add 600 skipto 1
 > # 

Re: ipfw divert filter for IPv4 geo-blocking

[Julian Elischer]

On 27/07/2016 9:36 PM, Dr. Rolf Jansen wrote:

Am 26.07.2016 um 23:03 schrieb Julian Elischer :
On 27/07/2016 3:06 AM, Dr. Rolf Jansen wrote:

There is another tool called geoip , that I uploaded to GitHub, and that I use 
for looking up country codes by IP addresses on the command line.

 https://github.com/cyclaero/ipdb/blob/master/geoip.c

This one could easily be extended to produce sorted IP ranges per CC that could 
be fed into tables of ipfw. I am thinking of adding a command line option for 
specifying CC's for which the IP ranges should be exported, something like:

geoip -e DE:BR:US:IT:FR:ES

And this could print sorted IP-Ranges belonging to the listed countries. For 
this purpose, what would be the ideal format for directly feeding the produced 
output into ipfw tables?

The format for using tables directly is the same as that used for routing 
tables.
…
table 5 add 1.1.1.0/32 1000
…
your application becomes an application for configuring the firewall.
(which you do by feeding commands down a pipe to ipfw, which is started as 
'ipfw -q /dev/stdin')

I finished adding a second usage form for the geoip tool, namely generation of 
ipfw table construction directives filtered by country codes.

wow, wonderful!

with that tool, and ipfw tables we have a fully functional geo 
blocking/munging solution in about 4 lines of shell script.




__
$ geoip -h
geoip v1.0.1 (16), Copyright © 2016 Dr. Rolf Jansen

Usage:

1) look-up the country code belonging to an IPv4 address given by the last 
command line argument:

geoip [-r bstfile] [-h] 
   a dotted IPv4 address to be looked-up.

2) generate a sorted list of IPv4 address/masklen pairs per country code, 
formatted as ipfw table construction directives:

geoip -t [CC:DD:EE:..] [-n table number] [-v table value] [-r bstfile] [-h]

   -t [CC:DD:EE:..]  output all IPv4 address/masklen pairs belonging to the 
listed countries, given by 2 letter
 capital country codes, separated by colon. An empty CC 
list means any country code.
   -n table number   the ipfw table number between 0 and 65534 [default: 0].
   -v table valuethe 32-bit unsigned value of the ipfw table entry 
[default: 0].

valid arguments in both usage forms:

   -r bstfilethe path to the binary file with the consolidated IP 
ranges that has been.
 generated by the 'ipdb' tool [default: 
/usr/local/etc/ipdb/IPRanges/ipcc.bst].
   -hshow these usage instructions.
__

With that, the ipfw configuration script may contain something alike:

 …
 # allow only web access from DE, BR, US:
 /usr/local/bin/geoip -t DE:BR:US -n 7 | /sbin/ipfw -q /dev/stdin
 /sbin/ipfw -q add 70 deny tcp from not table\(7\) to any 80,443 in recv 
WAN_if setup
 …

OR, the other way around:
 …
 # deny web access from certain disgraceful regions:
 /usr/local/bin/geoip -t KO:TR:SA:RU:GB -n 66 | /sbin/ipfw -q /dev/stdin
 /sbin/ipfw -q add 70 allow tcp from not table\(66\) to any 80,443 in recv 
WAN_if setup
 …



Best regards

Rolf






___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: ipfw divert filter for IPv4 geo-blocking

[Dr. Rolf Jansen]

> Am 26.07.2016 um 23:03 schrieb Julian Elischer :
> On 27/07/2016 3:06 AM, Dr. Rolf Jansen wrote:
>> There is another tool called geoip , that I uploaded to GitHub, and that I 
>> use for looking up country codes by IP addresses on the command line.
>> 
>> https://github.com/cyclaero/ipdb/blob/master/geoip.c
>> 
>> This one could easily be extended to produce sorted IP ranges per CC that 
>> could be fed into tables of ipfw. I am thinking of adding a command line 
>> option for specifying CC's for which the IP ranges should be exported, 
>> something like:
>> 
>>geoip -e DE:BR:US:IT:FR:ES
>> 
>> And this could print sorted IP-Ranges belonging to the listed countries. For 
>> this purpose, what would be the ideal format for directly feeding the 
>> produced output into ipfw tables?
> The format for using tables directly is the same as that used for routing 
> tables.
> …
> table 5 add 1.1.1.0/32 1000
> …
> your application becomes an application for configuring the firewall.
> (which you do by feeding commands down a pipe to ipfw, which is started as 
> 'ipfw -q /dev/stdin')

I finished adding a second usage form for the geoip tool, namely generation of 
ipfw table construction directives filtered by country codes.

__
$ geoip -h
geoip v1.0.1 (16), Copyright © 2016 Dr. Rolf Jansen

Usage:

1) look-up the country code belonging to an IPv4 address given by the last 
command line argument:

   geoip [-r bstfile] [-h] 
  a dotted IPv4 address to be looked-up.

2) generate a sorted list of IPv4 address/masklen pairs per country code, 
formatted as ipfw table construction directives:

   geoip -t [CC:DD:EE:..] [-n table number] [-v table value] [-r bstfile] [-h]

  -t [CC:DD:EE:..]  output all IPv4 address/masklen pairs belonging to the 
listed countries, given by 2 letter
capital country codes, separated by colon. An empty CC 
list means any country code.
  -n table number   the ipfw table number between 0 and 65534 [default: 0].
  -v table valuethe 32-bit unsigned value of the ipfw table entry 
[default: 0].

valid arguments in both usage forms:

  -r bstfilethe path to the binary file with the consolidated IP 
ranges that has been.
generated by the 'ipdb' tool [default: 
/usr/local/etc/ipdb/IPRanges/ipcc.bst].
  -hshow these usage instructions.
__

With that, the ipfw configuration script may contain something alike:

…
# allow only web access from DE, BR, US:
/usr/local/bin/geoip -t DE:BR:US -n 7 | /sbin/ipfw -q /dev/stdin
/sbin/ipfw -q add 70 deny tcp from not table\(7\) to any 80,443 in recv 
WAN_if setup
…

OR, the other way around:
…
# deny web access from certain disgraceful regions:
/usr/local/bin/geoip -t KO:TR:SA:RU:GB -n 66 | /sbin/ipfw -q /dev/stdin
/sbin/ipfw -q add 70 allow tcp from not table\(66\) to any 80,443 in recv 
WAN_if setup
…



Best regards

Rolf


___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: ipfw divert filter for IPv4 geo-blocking

[Julian Elischer]

On 27/07/2016 3:06 AM, Dr. Rolf Jansen wrote:

Am 26.07.2016 um 13:23 schrieb Julian Elischer :
On 26/07/2016 1:41 AM, Dr. Rolf Jansen wrote:

Once a week, the IP ranges are compiled from original sources into a binary 
sorted table, containing as of today 83162 consolidated range/cc pairs. On 
starting-up, the divert daemon reads the binary file in one block and stores 
the ranges into a totally balanced binary search tree. Looking-up a country 
code for a given IPv4 address in the BST takes on average 20 nanoseconds on an 
AWS-EC2 micro instance. I don't know the overhead of diverting, though. I guess 
this may be one or two orders of magnitudes higher. Even though, I won't see 
any performance issues.

yes the diversion to user space is not a fast operation. When we wrote it, fast 
was 10Mbits/sec.
The firewall tables use a radix tree (*) and might be slower than what you 
have, but possibly it might be made up for by not having to do the divert 
logic. it's not entorely clear from your description why you look up  a country 
rather than just a pass/block result, but maybe different sources can access 
different countries?.

The basic idea was to develop a facility for ipfw for filtering IPv4 packets by 
country code - see: https://github.com/cyclaero/ipdb

I simply put into /etc/rc.conf:

geod_enable="YES"
geod_flags="-a DE:BR:US"

The -a flag tells, that source IP addresses only from these countries are 
allowed (i.e. passed through the filter). I added also a -d flag, which means 
deny (i.e. drop packets) from the given list of countries.

With that in place, I need to add a respective divert rule to the ipfw ruleset 
(the divert port of the geod daemon is 8669, remembering that 8668 is the port 
of the natd daemon):

 ipfw -q add 70 divert 8669 tcp from any to any 80,443 in recv WAN_if setup


I did similar once using ipfw tables but couldn't find a reliable source of 
data.

The IP/CC database is compiled from downloads of the daily published delegation 
statistics files of the 5 RIR's. I consider the RIR's being the authoritative 
source. Anyway, on my systems the IP/CC-database is updated only weekly, 
although, daily updating would be possible. I wrote a shell script for this, 
that can be executed by a cron job.

 https://github.com/cyclaero/ipdb/blob/master/ipdb-update.sh 


There is another tool called geoip , that I uploaded to GitHub, and that I use 
for looking up country codes by IP addresses on the command line.

 https://github.com/cyclaero/ipdb/blob/master/geoip.c

This one could easily be extended to produce sorted IP ranges per CC that could 
be fed into tables of ipfw. I am thinking of adding a command line option for 
specifying CC's for which the IP ranges should be exported, something like:

geoip -e DE:BR:US:IT:FR:ES

And this could print sorted IP-Ranges belonging to the listed countries. For 
this purpose, what would be the ideal format for directly feeding the produced 
output into ipfw tables?
The format for using tables directly is the same as that used for 
routing tables.
so imagine that you had to generate a routing table that sent packets 
to two different routers depending on their source.


here's a simple rule set that filters web traffic by such a 'routing 
table'
except it's routing to two different rules. It also sorts OUTGOING web 
traffic to the same rules.


ipfw -q /dev/stdin <<-DONE
# we hate this guy
table 5 add 1.1.1.0/32 1000
# but all ow our people to visit everyone else in that subnet
table 5 add 1.1.0.0/24 2000
# we block 1.1.2.0 through 1.1.3.255
table 5 add 1.1.2.0/23 1000
# but we allow 1.1.4.0 through to 1.1.7.255
table 5 add 1.1.4.0/22 2000
# etc
table 5 add 1.1.8.0/21 1000
table 5 add 1.2.0.0/16 1000
table 5 add 0.0.0.0/0 2000 # default
check-state  # If we already decided what to do,  do it
# select out only external traffic, into direction specific rules.
add 400 skipto 500 ip from any to any in recv WAN_if
add 410 skipto 700 ip from any to any out xmit WAN_If
add 320 skipto 1
# Incoming packets
add 500 skipto tablearg tcp from table(5) to any 80,443 setup 
keep-state  # sort tcp setup packets between rules 1000 and 2000

add 600 skipto 1
# outgoing packets
add 700 skipto tablearg tcp from any  to table(5) 80,443 setup 
keep-state  # sort tcp setup packets between rules 100 and 2000

add 800 skipto 1
add 1000 drop ip from any to any
add 2000 allow ip from any to any
# further processing
add 1 .. # further processing for non tcp
DONE

for full configurability you could have a rule for each country, and a 
number for it in the table:

table 5 add 150.101.0.0/16 10610 # Australia
[...]
add 10610 block tcp from any to any 445  # only allow non encrypted 
web to those Aussie scum.

add 10611 allow ip from any to any

then by changing the rules at that location you could change the 
policy for a country without 

Re: ipfw divert filter for IPv4 geo-blocking

[Dr. Rolf Jansen]

> Am 26.07.2016 um 13:23 schrieb Julian Elischer :
> On 26/07/2016 1:41 AM, Dr. Rolf Jansen wrote:
>> Once a week, the IP ranges are compiled from original sources into a binary 
>> sorted table, containing as of today 83162 consolidated range/cc pairs. On 
>> starting-up, the divert daemon reads the binary file in one block and stores 
>> the ranges into a totally balanced binary search tree. Looking-up a country 
>> code for a given IPv4 address in the BST takes on average 20 nanoseconds on 
>> an AWS-EC2 micro instance. I don't know the overhead of diverting, though. I 
>> guess this may be one or two orders of magnitudes higher. Even though, I 
>> won't see any performance issues.
> 
> yes the diversion to user space is not a fast operation. When we wrote it, 
> fast was 10Mbits/sec.
> The firewall tables use a radix tree (*) and might be slower than what you 
> have, but possibly it might be made up for by not having to do the divert 
> logic. it's not entorely clear from your description why you look up  a 
> country rather than just a pass/block result, but maybe different sources can 
> access different countries?.

The basic idea was to develop a facility for ipfw for filtering IPv4 packets by 
country code - see: https://github.com/cyclaero/ipdb

I simply put into /etc/rc.conf:

   geod_enable="YES"
   geod_flags="-a DE:BR:US"

The -a flag tells, that source IP addresses only from these countries are 
allowed (i.e. passed through the filter). I added also a -d flag, which means 
deny (i.e. drop packets) from the given list of countries.

With that in place, I need to add a respective divert rule to the ipfw ruleset 
(the divert port of the geod daemon is 8669, remembering that 8668 is the port 
of the natd daemon):

ipfw -q add 70 divert 8669 tcp from any to any 80,443 in recv WAN_if setup

> I did similar once using ipfw tables but couldn't find a reliable source of 
> data.

The IP/CC database is compiled from downloads of the daily published delegation 
statistics files of the 5 RIR's. I consider the RIR's being the authoritative 
source. Anyway, on my systems the IP/CC-database is updated only weekly, 
although, daily updating would be possible. I wrote a shell script for this, 
that can be executed by a cron job.

https://github.com/cyclaero/ipdb/blob/master/ipdb-update.sh 


There is another tool called geoip , that I uploaded to GitHub, and that I use 
for looking up country codes by IP addresses on the command line.

https://github.com/cyclaero/ipdb/blob/master/geoip.c

This one could easily be extended to produce sorted IP ranges per CC that could 
be fed into tables of ipfw. I am thinking of adding a command line option for 
specifying CC's for which the IP ranges should be exported, something like:

   geoip -e DE:BR:US:IT:FR:ES 

And this could print sorted IP-Ranges belonging to the listed countries. For 
this purpose, what would be the ideal format for directly feeding the produced 
output into ipfw tables?

>> Independent from the actual usage case (geo-blocking), let's talk about 
>> divert filtering in general. The original question which is still unanswered 
>> can be generalized to, whether "dropping/denying" a package simply means 
>> 'forget about it' or whether the divert filter is required to do something 
>> more involved, e.g. communicate the situation somehow to ipfw.
> 
> there is no residual information about the packet in the kernel once it has 
> been passed to the  user process.
> so just "forgetting to hand it back" is sufficient to drop it.

OK, many thanks, that just answers my original doubt. At least technically, my 
daemon handles package dropping correctly, although, more elegant ways can be 
imagined to do the same thing.

Best regards

Rolf 
___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: ipfw divert filter for IPv4 geo-blocking

[Michael Sierchio]

On Tue, Jul 26, 2016 at 9:26 AM, Julian Elischer  wrote:

table 1 { DE, NL } -> 1,
>> { US, UK } -> 10100
>> table 2 { CN, KO, TR } -> 2
>>
> why multiple tables?
> if you load the table at once you can assign a country code as the
> tablearg for every run of addresses. all in one table.


I mentioned that in my earlier response - but if the point is to block
entire countries (or any collection of CIDR blocks, for that matter), it's
sufficient to have a whitelist table and a blacklist table. The table arg
could also be a skipto rule number, right? And you can do policy-based
routing, with the table arg as a FIB number.

Passing the packet to userland via divert sockets was a brilliant idea in
2003. natd was pretty much the first NAT mechanism to properly handle ICMP
error responses, too.

-- 
"Well," Brahma said, "even after ten thousand explanations, a fool is no
wiser, but an intelligent man requires only two thousand five hundred."

- The Mahābhārata
___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: ipfw divert filter for IPv4 geo-blocking

[Julian Elischer]

On 26/07/2016 1:01 AM, Jan Bramkamp wrote:



On 25/07/16 16:28, Dr. Rolf Jansen wrote:
I have written a ipfw divert filter daemon for IPv4 geo-blocking. 
It is working flawlessly on two server installations since a week.


Anyway, I am still in doubt whether I do the blocking in the 
correct way. Once the filter receives a packet from the respective 
divert socket it looks up the country code of the source IP in the 
IP-Ranges database, and if the country code shall be allowed then 
it returns the unaltered packet via said socket, otherwise, the 
filter does no further processing, so the packet is effectively 
gone, lost, dropped, discarded, or whatever would be the correct 
terminology. Is this the really the correct way of denying a 
packet, or is it necessary to inform ipfw somehow about the 
circumstances, so it can run a proper dropping procedure?


I uploaded the filter + accompanying tools to GitHub

   https://github.com/cyclaero/ipdb

Many thnaks for any advices in advance.


I would use a set of IPFW tables with skipto/call tablearg rules 
instead. 


Use the daemon to maintain the IPFW tables. I assume your database 
is a list of of (CIDR, country code) pairs. In that case the daemon 
config should probably map from sets of country codes to table 
values e.g.


table 1 { DE, NL } -> 1,
{ US, UK } -> 10100
table 2 { CN, KO, TR } -> 2

why multiple tables?
if you load the table at once you can assign a country code as the 
tablearg for every run of addresses. all in one table.




Next the daemon would calculate the minimal set of table entries to 
match these policies exactly and patch the kernel table contents if 
the database changes.


This design avoids the userspace<->kernel copies without losing 
flexibility.

___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"



___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: ipfw divert filter for IPv4 geo-blocking

[Julian Elischer]

On 26/07/2016 1:41 AM, Dr. Rolf Jansen wrote:

Am 25.07.2016 um 12:47 schrieb Michael Sierchio :

Writing a divert daemon is a praiseworthy project, but I think you could do
this without sending packets to user land.

You could use tables - …



Am 25.07.2016 um 14:01 schrieb Jan Bramkamp :

I would use a set of IPFW tables with skipto/call tablearg rules instead …

Michael and Jan, many thanks for your suggestions.

As everybody knows, 'Many roads lead to Rome.', and I am already there. I don't 
feel alike going all the way back only for the sake of trying out other routes.

and I personally am responsible for at least parts of several of them ;-)
(parts of ipdivert, netgraph, and various ipfw bits).




Once a week, the IP ranges are compiled from original sources into a binary 
sorted table, containing as of today 83162 consolidated range/cc pairs. On 
starting-up, the divert daemon reads the binary file in one block and stores 
the ranges into a totally balanced binary search tree. Looking-up a country 
code for a given IPv4 address in the BST takes on average 20 nanoseconds on an 
AWS-EC2 micro instance. I don't know the overhead of diverting, though. I guess 
this may be one or two orders of magnitudes higher. Even though, I won't see 
any performance issues.


yes the diversion to user space is not a fast operation. When we wrote 
it, fast was 10Mbits/sec.
The firewall tables use a radix tree (*) and might be slower than what 
you have, but possibly it might be made up for by not having to do the 
divert logic. it's not entorely clear from your description why you 
look up  a country rather than just a pass/block result, but maybe 
different sources can access different countries?.


I did similar once using ipfw tables but couldn't find a reliable 
source of data.




Independent from the actual usage case (geo-blocking), let's talk about divert filtering 
in general. The original question which is still unanswered can be generalized to, 
whether "dropping/denying" a package simply means 'forget about it' or whether 
the divert filter is required to do something more involved, e.g. communicate the 
situation somehow to ipfw.


there is no residual information about the packet in the kernel once 
it has been passed to the  user process.

so just "forgetting to hand it back" is sufficient to drop it.



Best regards

Rolf
___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"




___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: ipfw divert filter for IPv4 geo-blocking

[Dr. Rolf Jansen]

> Am 25.07.2016 um 12:47 schrieb Michael Sierchio :
> 
> Writing a divert daemon is a praiseworthy project, but I think you could do
> this without sending packets to user land.
> 
> You could use tables - …


> Am 25.07.2016 um 14:01 schrieb Jan Bramkamp :
> 
> I would use a set of IPFW tables with skipto/call tablearg rules instead …

Michael and Jan, many thanks for your suggestions.

As everybody knows, 'Many roads lead to Rome.', and I am already there. I don't 
feel alike going all the way back only for the sake of trying out other routes.

Once a week, the IP ranges are compiled from original sources into a binary 
sorted table, containing as of today 83162 consolidated range/cc pairs. On 
starting-up, the divert daemon reads the binary file in one block and stores 
the ranges into a totally balanced binary search tree. Looking-up a country 
code for a given IPv4 address in the BST takes on average 20 nanoseconds on an 
AWS-EC2 micro instance. I don't know the overhead of diverting, though. I guess 
this may be one or two orders of magnitudes higher. Even though, I won't see 
any performance issues.

Independent from the actual usage case (geo-blocking), let's talk about divert 
filtering in general. The original question which is still unanswered can be 
generalized to, whether "dropping/denying" a package simply means 'forget about 
it' or whether the divert filter is required to do something more involved, 
e.g. communicate the situation somehow to ipfw.

Best regards

Rolf
___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: ipfw divert filter for IPv4 geo-blocking

[Jan Bramkamp]

On 25/07/16 16:28, Dr. Rolf Jansen wrote:

I have written a ipfw divert filter daemon for IPv4 geo-blocking. It is working 
flawlessly on two server installations since a week.

Anyway, I am still in doubt whether I do the blocking in the correct way. Once 
the filter receives a packet from the respective divert socket it looks up the 
country code of the source IP in the IP-Ranges database, and if the country 
code shall be allowed then it returns the unaltered packet via said socket, 
otherwise, the filter does no further processing, so the packet is effectively 
gone, lost, dropped, discarded, or whatever would be the correct terminology. 
Is this the really the correct way of denying a packet, or is it necessary to 
inform ipfw somehow about the circumstances, so it can run a proper dropping 
procedure?

I uploaded the filter + accompanying tools to GitHub

   https://github.com/cyclaero/ipdb

Many thnaks for any advices in advance.


I would use a set of IPFW tables with skipto/call tablearg rules 
instead. Use the daemon to maintain the IPFW tables. I assume your 
database is a list of of (CIDR, country code) pairs. In that case the 
daemon config should probably map from sets of country codes to table 
values e.g.


table 1 { DE, NL } -> 1,
{ US, UK } -> 10100
table 2 { CN, KO, TR } -> 2

Next the daemon would calculate the minimal set of table entries to 
match these policies exactly and patch the kernel table contents if the 
database changes.


This design avoids the userspace<->kernel copies without losing flexibility.
___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: ipfw divert filter for IPv4 geo-blocking

[Michael Sierchio]

Writing a divert daemon is a praiseworthy project, but I think you could do
this without sending packets to user land.

You could use tables - in fact, a single table of consolidated nets by
country, in which the table entry is a CIDR block and the table arg is a
country code - and you match on table arg. You could simply put nets you
want to block in a table, and dispense with table args. That is how I do
it.

In order to do changes atomically, you need a pair of tables and a pair of
rulesets, and you can swap rulesets when you have built the new table.

On Jul 25, 2016 07:29, "Dr. Rolf Jansen" <r...@cyclaero.com> wrote:

> I have written a ipfw divert filter daemon for IPv4 geo-blocking. It is
> working flawlessly on two server installations since a week.
>
> Anyway, I am still in doubt whether I do the blocking in the correct way.
> Once the filter receives a packet from the respective divert socket it
> looks up the country code of the source IP in the IP-Ranges database, and
> if the country code shall be allowed then it returns the unaltered packet
> via said socket, otherwise, the filter does no further processing, so the
> packet is effectively gone, lost, dropped, discarded, or whatever would be
> the correct terminology. Is this the really the correct way of denying a
> packet, or is it necessary to inform ipfw somehow about the circumstances,
> so it can run a proper dropping procedure?
>
> I uploaded the filter + accompanying tools to GitHub
>
>https://github.com/cyclaero/ipdb
>
> Many thnaks for any advices in advance.
>
> Best regards
>
> Rolf
>
>
> ___
> freebsd-ipfw@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
> To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"
>
___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

ipfw divert filter for IPv4 geo-blocking

[Dr. Rolf Jansen]

I have written a ipfw divert filter daemon for IPv4 geo-blocking. It is working 
flawlessly on two server installations since a week.

Anyway, I am still in doubt whether I do the blocking in the correct way. Once 
the filter receives a packet from the respective divert socket it looks up the 
country code of the source IP in the IP-Ranges database, and if the country 
code shall be allowed then it returns the unaltered packet via said socket, 
otherwise, the filter does no further processing, so the packet is effectively 
gone, lost, dropped, discarded, or whatever would be the correct terminology. 
Is this the really the correct way of denying a packet, or is it necessary to 
inform ipfw somehow about the circumstances, so it can run a proper dropping 
procedure?

I uploaded the filter + accompanying tools to GitHub

   https://github.com/cyclaero/ipdb

Many thnaks for any advices in advance.

Best regards

Rolf

 
___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"