Re: simply copy mapped value into acl

[Andrew Hayworth]

On Mon, Nov 23, 2015 at 10:52 AM, Dennis Jacobfeuerborn
 wrote:
> Hm, I wasn't aware of the -M flag, thanks!
>
> However in you example you again you have to do multiple lookups even
> though that shouldn't be necessary. I can make decisions based on the
> fact that the IP is present in the map but what I really want to do is
> make a decision based on what the actual value for that IP in the map is
> i.e. if the value is "de" then I want to do one thing and if it is "at"
> then I want to do something else.
>
> Regards,
>   Dennis
>

I see. I believe you could accomplish what you want with something like

http-request set-var(req.ip_lookup) src,map_ip()
http-request set-header X-Test wasxx if { var(req.ip_lookup) -m str xx }
http-request set-header X-Test wasyy if { var(req.ip_lookup) -m str yy }

http-request set-var is new in HAProxy 1.6. Also note the implicit ACL
definition with '{}' - that's syntactic sugar so you don't have to
declare them before (which would be very verbose).

That said, the difference between one and two map lookups is
negligible, so I don't think you're saving much this way and it'll
make configuration harder in some ways if you keep adding IPs. I can
vouch from personal experience in production that we run 2+ map
lookups on every request at Braintree (hundreds of req/s) and it adds
no noticeable latency. Food for thought. :)

http://cbonte.github.io/haproxy-dconv/configuration-1.6.html#http-request
(especially note the section about variable scope).

-- 
- Andrew Hayworth

Re: simply copy mapped value into acl

[Dennis Jacobfeuerborn]

On 16.11.2015 15:25, Dennis Jacobfeuerborn wrote:
> Hi,
> I'm trying to figure out the best way to match a source ip against an ip
> mapping file and make decisions based on that. What I'm now doing is this:
> 
> acl acl_is_xx src,map_ip() -m str xx
> acl acl_is_yy src,map_ip() -m str yy
> 
> http-request set-header X-Test wasxx if acl_is_xx ...
> http-request set-header X-Test wasyy if acl_is_yy ...
> 
> While this works my problem is that this requires two map look-ups. What
> i would really like to do is this (pseudo code):
> 
> acl acl_value src,map_ip() -m copy
> http-request set-header X-Test wasxx if acl_value==xx
> http-request set-header X-Test wasyy if acl_value==yy
> 
> That way you only would have to do one look-up in the map and then
> determine the the different cases based on simple string matches.
> 
> As far as I can tell though ACLs only allow for matching and not for a
> straight forward copy like I tried to express with the "-m copy" above.
> 
> Is there an alternative way to express something like this?

Does nobody have any idea how to accomplish this?
This is happening in a GeoIP context and I'm now planning to make
distinct lookups for four different countries which seems pretty wasteful.

Regards,
  Dennis

Re: simply copy mapped value into acl

[Willy Tarreau]

On Tue, Nov 24, 2015 at 01:44:07AM +0100, Dennis Jacobfeuerborn wrote:
> On 23.11.2015 22:38, Willy Tarreau wrote:
> > Hi Andrew,
> > 
> > On Mon, Nov 23, 2015 at 12:19:22PM -0600, Andrew Hayworth wrote:
> >> That said, the difference between one and two map lookups is
> >> negligible, so I don't think you're saving much this way and it'll
> >> make configuration harder in some ways if you keep adding IPs. I can
> >> vouch from personal experience in production that we run 2+ map
> >> lookups on every request at Braintree (hundreds of req/s) and it adds
> >> no noticeable latency. Food for thought. :)
> > 
> > I agree, depending on the map size, you'll see something between 1 and 10
> > million lookups per second, it's quite cheap. I do abuse them as well and
> > I don't bother about the number of lookups. Sure if I have to match a
> > country among a few tens, I'll certainly use a variable first, but otherwise
> > I don't care.
> 
> I'm going to use multiple lookups until I can upgrade to 1.6 because I
> agree that it's not going to cause any major issues in our current setup.
> It's just that when I'm looking the same value up multiple times in a
> map of about 800.000 values my coding sensibilities immediately go for
> storing the result in a variable instead.

Just FWIW in 1.5 there's a tricky alternative to variables. You can perform
a sample capture in TCP rules, and reuse this capture later :

tcp-request content capture src,map(ip-to-country) len 2
...
http-request add-header x-country %[capture.req.hdr(0)]
...
use_backend country_XXX if { capture.req.hdr(0) XXX }

It will effectively work as a variable, though this variable will be dumped
into the logs. Since it's the source address you're interested in, you can
even do that in "tcp-request connection" rules, and it will be performed
only once per connection, which is the most optimal situation. But as you
can see it's a trick and it is limited because you can't do much using HTTP
inputs for example.

Regards,
Willy

Re: simply copy mapped value into acl

[Dennis Jacobfeuerborn]

On 23.11.2015 19:19, Andrew Hayworth wrote:
> On Mon, Nov 23, 2015 at 10:52 AM, Dennis Jacobfeuerborn
>  wrote:
>> Hm, I wasn't aware of the -M flag, thanks!
>>
>> However in you example you again you have to do multiple lookups even
>> though that shouldn't be necessary. I can make decisions based on the
>> fact that the IP is present in the map but what I really want to do is
>> make a decision based on what the actual value for that IP in the map is
>> i.e. if the value is "de" then I want to do one thing and if it is "at"
>> then I want to do something else.
>>
>> Regards,
>>   Dennis
>>
> 
> I see. I believe you could accomplish what you want with something like
> 
> http-request set-var(req.ip_lookup) src,map_ip()
> http-request set-header X-Test wasxx if { var(req.ip_lookup) -m str xx }
> http-request set-header X-Test wasyy if { var(req.ip_lookup) -m str yy }
> 
> http-request set-var is new in HAProxy 1.6. Also note the implicit ACL
> definition with '{}' - that's syntactic sugar so you don't have to
> declare them before (which would be very verbose).
> 
> That said, the difference between one and two map lookups is
> negligible, so I don't think you're saving much this way and it'll
> make configuration harder in some ways if you keep adding IPs. I can
> vouch from personal experience in production that we run 2+ map
> lookups on every request at Braintree (hundreds of req/s) and it adds
> no noticeable latency. Food for thought. :)
> 
> http://cbonte.github.io/haproxy-dconv/configuration-1.6.html#http-request
> (especially note the section about variable scope).

set-var is precisely the thing I was looking for, thanks!

Since I need to update to 1.6 for this but am really busy over the next
two week at least I'll probably use your first suggestion for now and
then look into the set-var version in a few weeks.
In general the ability to set a variable from a map and then later use
that for further processing is a nice feature to have,

Thanks for pointing me in the right direction!

Regards,
  Dennis

Re: simply copy mapped value into acl

[Dennis Jacobfeuerborn]

On 23.11.2015 22:38, Willy Tarreau wrote:
> Hi Andrew,
> 
> On Mon, Nov 23, 2015 at 12:19:22PM -0600, Andrew Hayworth wrote:
>> That said, the difference between one and two map lookups is
>> negligible, so I don't think you're saving much this way and it'll
>> make configuration harder in some ways if you keep adding IPs. I can
>> vouch from personal experience in production that we run 2+ map
>> lookups on every request at Braintree (hundreds of req/s) and it adds
>> no noticeable latency. Food for thought. :)
> 
> I agree, depending on the map size, you'll see something between 1 and 10
> million lookups per second, it's quite cheap. I do abuse them as well and
> I don't bother about the number of lookups. Sure if I have to match a
> country among a few tens, I'll certainly use a variable first, but otherwise
> I don't care.

I'm going to use multiple lookups until I can upgrade to 1.6 because I
agree that it's not going to cause any major issues in our current setup.
It's just that when I'm looking the same value up multiple times in a
map of about 800.000 values my coding sensibilities immediately go for
storing the result in a variable instead.

Regards,
  Dennis

Re: simply copy mapped value into acl

[Andrew Hayworth]

On Mon, Nov 23, 2015 at 6:26 AM, Dennis Jacobfeuerborn
 wrote:
>
> On 16.11.2015 15:25, Dennis Jacobfeuerborn wrote:
> > Hi,
> > I'm trying to figure out the best way to match a source ip against an ip
> > mapping file and make decisions based on that. What I'm now doing is this:
> >
> > acl acl_is_xx src,map_ip() -m str xx
> > acl acl_is_yy src,map_ip() -m str yy
> >
> > acl acl_value src,map_ip() -m copy
> > http-request set-header X-Test wasxx if acl_value==xx
> > http-request set-header X-Test wasyy if acl_value==yy
> >

> > Is there an alternative way to express something like this?
>
> Does nobody have any idea how to accomplish this?
> This is happening in a GeoIP context and I'm now planning to make
> distinct lookups for four different countries which seems pretty wasteful.

Hi Dennis -

I think this configuration would do what you want:

acl has_ip_map src -M -f 
http-request set-header X-Test %[src,map_ip()] if has_ip_map

The idea is that map-file is a two-column file like so:

# ip_addr name
1.2.3.4 wasxx
5.6.7.8 wasyy

This works by treating  as both a map AND and acl (the '-M'
flag does that). When you treat it as an ACL, it only evaluates the
first column. This lets you test the ip address, to see if it's one
you care about.

Then, you use the  as a map, and you look up the value for
your header based on the ip address - but only if the prior acl
evaluated to true.

An added benefit of this is that you can scale out easily to many
values in the map/acl file without polluting your configuration.
Additionally, you can use the socket commands to dynamically add
things to the map/acl without reloading haproxy, if you wanted
(something like 'add map   ').

Hope that helps!

- Andrew Hayworth

Re: simply copy mapped value into acl

[Dennis Jacobfeuerborn]

On 23.11.2015 17:04, Andrew Hayworth wrote:
> On Mon, Nov 23, 2015 at 6:26 AM, Dennis Jacobfeuerborn
>  wrote:
>>
>> On 16.11.2015 15:25, Dennis Jacobfeuerborn wrote:
>>> Hi,
>>> I'm trying to figure out the best way to match a source ip against an ip
>>> mapping file and make decisions based on that. What I'm now doing is this:
>>>
>>> acl acl_is_xx src,map_ip() -m str xx
>>> acl acl_is_yy src,map_ip() -m str yy
>>>
>>> acl acl_value src,map_ip() -m copy
>>> http-request set-header X-Test wasxx if acl_value==xx
>>> http-request set-header X-Test wasyy if acl_value==yy
>>>
> 
>>> Is there an alternative way to express something like this?
>>
>> Does nobody have any idea how to accomplish this?
>> This is happening in a GeoIP context and I'm now planning to make
>> distinct lookups for four different countries which seems pretty wasteful.
> 
> Hi Dennis -
> 
> I think this configuration would do what you want:
> 
> acl has_ip_map src -M -f 
> http-request set-header X-Test %[src,map_ip()] if has_ip_map
> 
> The idea is that map-file is a two-column file like so:
> 
> # ip_addr name
> 1.2.3.4 wasxx
> 5.6.7.8 wasyy
> 
> This works by treating  as both a map AND and acl (the '-M'
> flag does that). When you treat it as an ACL, it only evaluates the
> first column. This lets you test the ip address, to see if it's one
> you care about.
> 
> Then, you use the  as a map, and you look up the value for
> your header based on the ip address - but only if the prior acl
> evaluated to true.
> 
> An added benefit of this is that you can scale out easily to many
> values in the map/acl file without polluting your configuration.
> Additionally, you can use the socket commands to dynamically add
> things to the map/acl without reloading haproxy, if you wanted
> (something like 'add map   ').

Hm, I wasn't aware of the -M flag, thanks!

However in you example you again you have to do multiple lookups even
though that shouldn't be necessary. I can make decisions based on the
fact that the IP is present in the map but what I really want to do is
make a decision based on what the actual value for that IP in the map is
i.e. if the value is "de" then I want to do one thing and if it is "at"
then I want to do something else.

Regards,
  Dennis

Re: simply copy mapped value into acl

[Willy Tarreau]

Hi Andrew,

On Mon, Nov 23, 2015 at 12:19:22PM -0600, Andrew Hayworth wrote:
> That said, the difference between one and two map lookups is
> negligible, so I don't think you're saving much this way and it'll
> make configuration harder in some ways if you keep adding IPs. I can
> vouch from personal experience in production that we run 2+ map
> lookups on every request at Braintree (hundreds of req/s) and it adds
> no noticeable latency. Food for thought. :)

I agree, depending on the map size, you'll see something between 1 and 10
million lookups per second, it's quite cheap. I do abuse them as well and
I don't bother about the number of lookups. Sure if I have to match a
country among a few tens, I'll certainly use a variable first, but otherwise
I don't care.

Willy

simply copy mapped value into acl

[Dennis Jacobfeuerborn]

Hi,
I'm trying to figure out the best way to match a source ip against an ip
mapping file and make decisions based on that. What I'm now doing is this:

acl acl_is_xx src,map_ip() -m str xx
acl acl_is_yy src,map_ip() -m str yy

http-request set-header X-Test wasxx if acl_is_xx ...
http-request set-header X-Test wasyy if acl_is_yy ...

While this works my problem is that this requires two map look-ups. What
i would really like to do is this (pseudo code):

acl acl_value src,map_ip() -m copy
http-request set-header X-Test wasxx if acl_value==xx
http-request set-header X-Test wasyy if acl_value==yy

That way you only would have to do one look-up in the map and then
determine the the different cases based on simple string matches.

As far as I can tell though ACLs only allow for matching and not for a
straight forward copy like I tried to express with the "-m copy" above.

Is there an alternative way to express something like this?

Regards,
  Dennis