Re: haproxy fails to properly direct connection to correct back end.

[p s]

After sending to this list I got spammed by random people telling me that they 
are on vacation. WTF.

It's so much better to report issues on github and not to mess with emails and 
these random replies.



From: p s 
Sent: July 29, 2017 4:05 PM
To: haproxy@formilux.org
Subject: haproxy fails to properly direct connection to correct back end.


I had default version of haproxy from ubuntu, which was 1.5.x, so I decided to 
update to latest stable first before I report the bug.

After I update to latest 1.7.x haproxy stops working with my config. When I was 
trying to start I was getting:


Job for haproxy.service failed because the control process exited with error 
code. See "systemctl status haproxy.service". systemctl status haproxy.service 
wouldn't show any info about why haproxy fails. FYI, with nginx it clearly 
points to what nginx doesn't like, and there is no need to guess. Can this be 
fixed to properly show errors?


So, back to my original issue.

Basically, I added only this to the default config:


backend nodejs
timeout server 1h
timeout connect 1s
option httpclose
option forwardfor
server server1 127.0.0.1:80 #check

backend nodejs_test
timeout server 1h
timeout connect 1s
option httpclose
option forwardfor
server server2 127.0.0.1:3000
option http-no-delay

frontend all
bind *:8090
timeout client 1h
option http-no-delay
acl is_test path_beg /test
use_backend nodejs_test if is_test
default_backend nodejs
mode http

I do not run anything on port 80, now I make a request to example.com/test and 
haproxy totally fails. 1.5.x would fail with "ERROR 503: Service Unavailable." 
on first request and would work on second one, 1.7.x doesn't fail that 
miserably, it just doesn't work at all, and my wget auto-retries 3 times and 
then it works.
I inspected with wireshark what's going on and I see that haproxy when handling 
my request tries to connect to default backend on port 80 (and I don't run 
anything on port 80 atm). It's clearly a bug, it shouldn't even touch port 80, 
and should directly connect to nodejs_test on port 3000.


Another point: why don't you host your project on github, it will be way more 
popular, it would be much easier for anybody to contribute etc. Now, it does 
feel like haproxy should be avoided, as there is no bugtracker, mailing list 
that feels broken, etc... feels like it's dying.

haproxy fails to properly direct connection to correct back end.

[p s]

I had default version of haproxy from ubuntu, which was 1.5.x, so I decided to 
update to latest stable first before I report the bug.

After I update to latest 1.7.x haproxy stops working with my config. When I was 
trying to start I was getting:


Job for haproxy.service failed because the control process exited with error 
code. See "systemctl status haproxy.service". systemctl status haproxy.service 
wouldn't show any info about why haproxy fails. FYI, with nginx it clearly 
points to what nginx doesn't like, and there is no need to guess. Can this be 
fixed to properly show errors?


So, back to my original issue.

Basically, I added only this to the default config:


backend nodejs
timeout server 1h
timeout connect 1s
option httpclose
option forwardfor
server server1 127.0.0.1:80 #check

backend nodejs_test
timeout server 1h
timeout connect 1s
option httpclose
option forwardfor
server server2 127.0.0.1:3000
option http-no-delay

frontend all
bind *:8090
timeout client 1h
option http-no-delay
acl is_test path_beg /test
use_backend nodejs_test if is_test
default_backend nodejs
mode http

I do not run anything on port 80, now I make a request to example.com/test and 
haproxy totally fails. 1.5.x would fail with "ERROR 503: Service Unavailable." 
on first request and would work on second one, 1.7.x doesn't fail that 
miserably, it just doesn't work at all, and my wget auto-retries 3 times and 
then it works.
I inspected with wireshark what's going on and I see that haproxy when handling 
my request tries to connect to default backend on port 80 (and I don't run 
anything on port 80 atm). It's clearly a bug, it shouldn't even touch port 80, 
and should directly connect to nodejs_test on port 3000.


Another point: why don't you host your project on github, it will be way more 
popular, it would be much easier for anybody to contribute etc. Now, it does 
feel like haproxy should be avoided, as there is no bugtracker, mailing list 
that feels broken, etc... feels like it's dying.

Re: tcp-response content tarpit if hdr(X-Tarpit-This)

[Charlie Elgholm]

Nice, thanks. I'll upgrade our dev-machine the coming week!

Frontend has nothing to do with what I want to do, I want to
drop/tarpit/reject after the response.

Den 29 juli 2017 5:40 em skrev "bjun...@gmail.com" :

> 2017-07-29 16:57 GMT+02:00 Charlie Elgholm :
> > Ok, but anyhow, this actually means that I can use http-response to do
> > something on the response. That's good. I'll play with it for a while on
> my
> > dev-server. Nice!
> >
> > Version can be upgraded, of course, if I can just motivate it! :)
> >
> > Den 29 juli 2017 12:44 em skrev "Igor Cicimov"
> > :
> >>
> >>
> >>
> >> On Fri, Jul 28, 2017 at 10:00 PM, Charlie Elgholm 
> >> wrote:
> >>>
> >>> Ok, I'm on the 1.5.x bransch unfortunately, due to Oracle Linux issues.
> >>> Can install manually, but that might raise some eyebrows.
> >>>
> >>> But what you're telling me is that I can route the request to another
> >>> backend (or drop it) in haproxy based on something I received from one
> >>> backend??
> >>>
> >>> Den 28 juli 2017 1:40 em skrev "Igor Cicimov"
> >>> :
> >>>
> >>>
> >>>
> >>> On Fri, Jul 28, 2017 at 6:03 PM, Charlie Elgholm 
> >>> wrote:
> 
>  Thanks!
> 
>  I was really hoping for acl-validation on the basis of the response
> from
>  the backend server, and not on the incoming request at the frontend.
>  And, as much as I really like lua as a language, I'd rather keep my
>  haproxy with as small footprint as possible. =)
> 
>  Really nice example about all the possibilities though, thanks!
> 
>  This is how all examples I find operate:
>  incoming request => haproxy => frontend => acl based on what's known
>  about the incoming requests => A or B
>  A: backend => stream backend response to client
>  B: tarpit / reject
> 
>  I would like this:
>  incoming request => haproxy => frontend => backend => acl based on
>  what's known about the response from the backend => A or B
>  A: stream backend response to client
>  B: tarpit / reject
> 
> 
>  2017-07-28 9:52 GMT+02:00 Igor Cicimov  com>:
> >
> >
> >
> > On 28 Jul 2017 5:41 pm, "Charlie Elgholm" 
> wrote:
> >
> > Hi Folks,
> >
> > Either I'm too stupid, or it's because it's Friday
> >
> > Can you tarpit/reject (or other action) based on a response from the
> > backend?
> > You should be able to, right?
> >
> > Like this:
> > tcp-response content tarpit/reject if res.hdr(X-Tarpit-This)
> >
> > Can someone explain this to me? (Free beer.)
> >
> > I have a fairly complex ruleset on my backend server, written in
> Oracle
> > PL/SQL, which monitors Hack- or DoS-attempts, and I would love to
> tarpit
> > some requests on the frontend (by haproxy) based on something that
> happens
> > on my backend.
> >
> > As I do now I return a 503 response from the server, and
> iptable-block
> > those addresses for a while. But since they see the 503 response
> they'll
> > return at a later date and try again. I would like the connection to
> just
> > die (drop, no response at all) or tarpit (long timeout, so they give
> up). I
> > suppose/hope they'll eventually remove my IP from their databases.
> >
> > I'm guessing a tarpit is smarter than a reject, since the reject will
> > indicate to the attacker that somethings exist behind the server IP.
> > An iptable "drop" would be preferable, but I guess that's a little
> late
> > since haproxy has already acknowledged the connection to the
> attacker.
> >
> > --
> > Regards
> > Charlie Elgholm
> > Brightly AB
> >
> > Good example of delay with lua:
> > http://godevops.net/2015/06/24/adding-random-delay-
> specific-http-requests-haproxy-lua/
> 
> 
> 
> 
>  --
>  Regards
>  Charlie Elgholm
>  Brightly AB
> >>>
> >>>
> >>> Well the idea is to redirect the response on the backend (based on some
> >>> condition) to a local frontend where you can use the tarpit on the
> request.
> >>>
> >>> You cam also try:
> >>>
> >>> http-response silent-drop if { status 503 }
> >>>
> >>> that you can use in the backed (at least in 1.7.8, not sure for other
> >>> versions)
> >>>
> >>>
> >>>
> >>
> >> I was thinking of something along these lines:
> >>
> >> frontend ft_tarpit
> >>   mode http
> >>   bind 127.0.0.1:
> >>   default_backend bk_tarpit
> >>
> >> backend bk_tarpit
> >>   mode http
> >>   timeout tarpit 3600s
> >>   http-request tarpit
> >>
> >> backend bk_main
> >>   mode http
> >>   http-response redirect 127.0.0.1: if { status 503 }
> >>
> >> but you are out of luck again since "http-response redirect" was
> >> introduced in 1.6
> >>
> >
>
> Hi Charlie,
>
> many 

Re: tcp-response content tarpit if hdr(X-Tarpit-This)

[bjun...@gmail.com]

2017-07-29 16:57 GMT+02:00 Charlie Elgholm :
> Ok, but anyhow, this actually means that I can use http-response to do
> something on the response. That's good. I'll play with it for a while on my
> dev-server. Nice!
>
> Version can be upgraded, of course, if I can just motivate it! :)
>
> Den 29 juli 2017 12:44 em skrev "Igor Cicimov"
> :
>>
>>
>>
>> On Fri, Jul 28, 2017 at 10:00 PM, Charlie Elgholm 
>> wrote:
>>>
>>> Ok, I'm on the 1.5.x bransch unfortunately, due to Oracle Linux issues.
>>> Can install manually, but that might raise some eyebrows.
>>>
>>> But what you're telling me is that I can route the request to another
>>> backend (or drop it) in haproxy based on something I received from one
>>> backend??
>>>
>>> Den 28 juli 2017 1:40 em skrev "Igor Cicimov"
>>> :
>>>
>>>
>>>
>>> On Fri, Jul 28, 2017 at 6:03 PM, Charlie Elgholm 
>>> wrote:

 Thanks!

 I was really hoping for acl-validation on the basis of the response from
 the backend server, and not on the incoming request at the frontend.
 And, as much as I really like lua as a language, I'd rather keep my
 haproxy with as small footprint as possible. =)

 Really nice example about all the possibilities though, thanks!

 This is how all examples I find operate:
 incoming request => haproxy => frontend => acl based on what's known
 about the incoming requests => A or B
 A: backend => stream backend response to client
 B: tarpit / reject

 I would like this:
 incoming request => haproxy => frontend => backend => acl based on
 what's known about the response from the backend => A or B
 A: stream backend response to client
 B: tarpit / reject


 2017-07-28 9:52 GMT+02:00 Igor Cicimov :
>
>
>
> On 28 Jul 2017 5:41 pm, "Charlie Elgholm"  wrote:
>
> Hi Folks,
>
> Either I'm too stupid, or it's because it's Friday
>
> Can you tarpit/reject (or other action) based on a response from the
> backend?
> You should be able to, right?
>
> Like this:
> tcp-response content tarpit/reject if res.hdr(X-Tarpit-This)
>
> Can someone explain this to me? (Free beer.)
>
> I have a fairly complex ruleset on my backend server, written in Oracle
> PL/SQL, which monitors Hack- or DoS-attempts, and I would love to tarpit
> some requests on the frontend (by haproxy) based on something that happens
> on my backend.
>
> As I do now I return a 503 response from the server, and iptable-block
> those addresses for a while. But since they see the 503 response they'll
> return at a later date and try again. I would like the connection to just
> die (drop, no response at all) or tarpit (long timeout, so they give up). 
> I
> suppose/hope they'll eventually remove my IP from their databases.
>
> I'm guessing a tarpit is smarter than a reject, since the reject will
> indicate to the attacker that somethings exist behind the server IP.
> An iptable "drop" would be preferable, but I guess that's a little late
> since haproxy has already acknowledged the connection to the attacker.
>
> --
> Regards
> Charlie Elgholm
> Brightly AB
>
> Good example of delay with lua:
> http://godevops.net/2015/06/24/adding-random-delay-specific-http-requests-haproxy-lua/




 --
 Regards
 Charlie Elgholm
 Brightly AB
>>>
>>>
>>> Well the idea is to redirect the response on the backend (based on some
>>> condition) to a local frontend where you can use the tarpit on the request.
>>>
>>> You cam also try:
>>>
>>> http-response silent-drop if { status 503 }
>>>
>>> that you can use in the backed (at least in 1.7.8, not sure for other
>>> versions)
>>>
>>>
>>>
>>
>> I was thinking of something along these lines:
>>
>> frontend ft_tarpit
>>   mode http
>>   bind 127.0.0.1:
>>   default_backend bk_tarpit
>>
>> backend bk_tarpit
>>   mode http
>>   timeout tarpit 3600s
>>   http-request tarpit
>>
>> backend bk_main
>>   mode http
>>   http-response redirect 127.0.0.1: if { status 503 }
>>
>> but you are out of luck again since "http-response redirect" was
>> introduced in 1.6
>>
>

Hi Charlie,

many ideas come to my mind to solve your requirement, but all of these
require HAProxy 1.6 (or better: HAProxy 1.7)

Some keywords:

"http-response silent-drop"
"http-response track-sc" + checking stick-table entry with acl in frontend
"http-response" + (random) delay with lua (my preferred solution)


---
Best Regards

Bjoern

Re: tcp-response content tarpit if hdr(X-Tarpit-This)

[Charlie Elgholm]

Ok, but anyhow, this actually means that I can use http-response to do
something on the response. That's good. I'll play with it for a while on my
dev-server. Nice!

Version can be upgraded, of course, if I can just motivate it! :)

Den 29 juli 2017 12:44 em skrev "Igor Cicimov" <
ig...@encompasscorporation.com>:

>
>
> On Fri, Jul 28, 2017 at 10:00 PM, Charlie Elgholm 
> wrote:
>
>> Ok, I'm on the 1.5.x bransch unfortunately, due to Oracle Linux issues.
>> Can install manually, but that might raise some eyebrows.
>>
>> But what you're telling me is that I can route the request to another
>> backend (or drop it) in haproxy based on something I received from one
>> backend??
>>
>> Den 28 juli 2017 1:40 em skrev "Igor Cicimov" <
>> ig...@encompasscorporation.com>:
>>
>>
>>
>> On Fri, Jul 28, 2017 at 6:03 PM, Charlie Elgholm 
>> wrote:
>>
>>> Thanks!
>>>
>>> I was really hoping for acl-validation on the basis of the response from
>>> the backend server, and not on the incoming request at the frontend.
>>> And, as much as I really like lua as a language, I'd rather keep my
>>> haproxy with as small footprint as possible. =)
>>>
>>> Really nice example about all the possibilities though, thanks!
>>>
>>> This is how all examples I find operate:
>>> incoming request => haproxy => frontend => acl based on what's known
>>> about the incoming requests => A or B
>>> A: backend => stream backend response to client
>>> B: tarpit / reject
>>>
>>> I would like this:
>>> incoming request => haproxy => frontend => backend => acl based on
>>> what's known about the response from the backend => A or B
>>> A: stream backend response to client
>>> B: tarpit / reject
>>>
>>>
>>> 2017-07-28 9:52 GMT+02:00 Igor Cicimov :
>>>


 On 28 Jul 2017 5:41 pm, "Charlie Elgholm"  wrote:

 Hi Folks,

 Either I'm too stupid, or it's because it's Friday

 Can you tarpit/reject (or other action) based on a response from the
 backend?
 You should be able to, right?

 Like this:
 tcp-response content tarpit/reject if res.hdr(X-Tarpit-This)

 Can someone explain this to me? (Free beer.)

 I have a fairly complex ruleset on my backend server, written in Oracle
 PL/SQL, which monitors Hack- or DoS-attempts, and I would love to tarpit
 some requests on the frontend (by haproxy) based on something that happens
 on my backend.

 As I do now I return a 503 response from the server, and iptable-block
 those addresses for a while. But since they see the 503 response they'll
 return at a later date and try again. I would like the connection to just
 die (drop, no response at all) or tarpit (long timeout, so they give up). I
 suppose/hope they'll eventually remove my IP from their databases.

 I'm guessing a tarpit is smarter than a reject, since the reject will
 indicate to the attacker that somethings exist behind the server IP.
 An iptable "drop" would be preferable, but I guess that's a little late
 since haproxy has already acknowledged the connection to the attacker.

 --
 Regards
 Charlie Elgholm
 Brightly AB

 Good example of delay with lua: http://godevops.net/2015/
 06/24/adding-random-delay-specific-http-requests-haproxy-lua/

>>>
>>>
>>>
>>> --
>>> Regards
>>> Charlie Elgholm
>>> Brightly AB
>>>
>>
>> Well the idea is to redirect the response on the backend (based on some
>> condition) to a local frontend​ where you can use the tarpit on the request.
>>
>> You cam also try:
>>
>> http-response silent-drop if { status 503 }
>>
>> that you can use in the backed (at least in 1.7.8, not sure for other
>> versions)
>>
>>
>>
>>
> ​I was thinking of something along these lines:​
>
> ​frontend ft_tarpit
>   mode http
>   bind 127.0.0.1:
>   default_backend bk_tarpit
>
> backend bk_tarpit
>   mode http
>   timeout tarpit 3600s
>   http-request tarpit
>
> backend bk_main
>   mode http
>   http-response redirect 127.0.0.1: if { status 503 }​
>
> but you are out of luck again since "http-response redirect" was
> introduced in 1.6
>
>

Re: tcp-response content tarpit if hdr(X-Tarpit-This)

[Igor Cicimov]

On Fri, Jul 28, 2017 at 10:00 PM, Charlie Elgholm 
wrote:

> Ok, I'm on the 1.5.x bransch unfortunately, due to Oracle Linux issues.
> Can install manually, but that might raise some eyebrows.
>
> But what you're telling me is that I can route the request to another
> backend (or drop it) in haproxy based on something I received from one
> backend??
>
> Den 28 juli 2017 1:40 em skrev "Igor Cicimov"  com>:
>
>
>
> On Fri, Jul 28, 2017 at 6:03 PM, Charlie Elgholm 
> wrote:
>
>> Thanks!
>>
>> I was really hoping for acl-validation on the basis of the response from
>> the backend server, and not on the incoming request at the frontend.
>> And, as much as I really like lua as a language, I'd rather keep my
>> haproxy with as small footprint as possible. =)
>>
>> Really nice example about all the possibilities though, thanks!
>>
>> This is how all examples I find operate:
>> incoming request => haproxy => frontend => acl based on what's known
>> about the incoming requests => A or B
>> A: backend => stream backend response to client
>> B: tarpit / reject
>>
>> I would like this:
>> incoming request => haproxy => frontend => backend => acl based on what's
>> known about the response from the backend => A or B
>> A: stream backend response to client
>> B: tarpit / reject
>>
>>
>> 2017-07-28 9:52 GMT+02:00 Igor Cicimov :
>>
>>>
>>>
>>> On 28 Jul 2017 5:41 pm, "Charlie Elgholm"  wrote:
>>>
>>> Hi Folks,
>>>
>>> Either I'm too stupid, or it's because it's Friday
>>>
>>> Can you tarpit/reject (or other action) based on a response from the
>>> backend?
>>> You should be able to, right?
>>>
>>> Like this:
>>> tcp-response content tarpit/reject if res.hdr(X-Tarpit-This)
>>>
>>> Can someone explain this to me? (Free beer.)
>>>
>>> I have a fairly complex ruleset on my backend server, written in Oracle
>>> PL/SQL, which monitors Hack- or DoS-attempts, and I would love to tarpit
>>> some requests on the frontend (by haproxy) based on something that happens
>>> on my backend.
>>>
>>> As I do now I return a 503 response from the server, and iptable-block
>>> those addresses for a while. But since they see the 503 response they'll
>>> return at a later date and try again. I would like the connection to just
>>> die (drop, no response at all) or tarpit (long timeout, so they give up). I
>>> suppose/hope they'll eventually remove my IP from their databases.
>>>
>>> I'm guessing a tarpit is smarter than a reject, since the reject will
>>> indicate to the attacker that somethings exist behind the server IP.
>>> An iptable "drop" would be preferable, but I guess that's a little late
>>> since haproxy has already acknowledged the connection to the attacker.
>>>
>>> --
>>> Regards
>>> Charlie Elgholm
>>> Brightly AB
>>>
>>> Good example of delay with lua: http://godevops.net/2015/
>>> 06/24/adding-random-delay-specific-http-requests-haproxy-lua/
>>>
>>
>>
>>
>> --
>> Regards
>> Charlie Elgholm
>> Brightly AB
>>
>
> Well the idea is to redirect the response on the backend (based on some
> condition) to a local frontend​ where you can use the tarpit on the request.
>
> You cam also try:
>
> http-response silent-drop if { status 503 }
>
> that you can use in the backed (at least in 1.7.8, not sure for other
> versions)
>
>
>
>
​I was thinking of something along these lines:​

​frontend ft_tarpit
  mode http
  bind 127.0.0.1:
  default_backend bk_tarpit

backend bk_tarpit
  mode http
  timeout tarpit 3600s
  http-request tarpit

backend bk_main
  mode http
  http-response redirect 127.0.0.1: if { status 503 }​

but you are out of luck again since "http-response redirect" was introduced
in 1.6