Re: Haproxy 1. 5.14 + Tomcat 8 giving random 502 errors

[Igor Cicimov]

On 01/03/2016 9:57 PM, "Zoltan Lorincz"  wrote:
>
> Hi all,
>
> i am very new to haproxy. Read trough all the docs but i think something
is wrong with my configuration, because if we connect directly to tomcat we
don't get any 502 errors.
>
> The errors from haproxy look like this.
>
> Mar  1 11:41:37 www1 haproxy[15362]: xx.xx.xx.xx:56387
[01/Mar/2016:11:41:35.480] https-in~ servers/www1a 1987/0/0/-1/2029 502
8878 - - PH-- 1764/1758/46/26/0 0/0 "POST
/abc/test/b25766378a05446496645649e2ddaf7a/poll HTTP/1.1"
>
>
>
> Tomcat connector config:
>
---
>  URIEncoding = "UTF-8"
> port = "8080"
> protocol = "HTTP/1.1"
> maxThreads = "1850"
> connectionTimeout = "90"
> keepAliveTimeout = "90"
> maxKeepAliveRequests = "-1"
> redirectPort = "8443" />
>
>
---
>
I have tomcat8 running behind ssl terminating haproxy but my connector is
configured like this:



which is common way to tell tomcat that although receiving plain traffic
the response urls need to be https.

>
> Haproxy config:
>
---
> global
> log /dev/log local0
> log /dev/log local1 notice
> chroot /var/lib/haproxy
> stats socket /run/haproxy/admin.sock mode 777 level admin
> stats timeout 30s
> user haproxy
> group haproxy
> daemon
>
> # Per process limit: The default is 2000, too small for us
> maxconn 18000
> # Increase the cache from 2 (default), higher values reduce CPU usage
> tune.ssl.cachesize 6
>
> # Default SSL material locations
> ca-base /etc/ssl/certs
> crt-base /etc/ssl/private
>
> # Default ciphers to use on SSL-enabled listening sockets.
> # For more information, see ciphers(1SSL).
> ssl-default-bind-ciphers
kEECDH+aRSA+AES:kRSA+AES:+AES256:!kEDH:!LOW:!EXP:!MD5:!aNULL:!eNULL
> ssl-default-bind-options no-sslv3 no-tls-tickets
>
> defaults
> log global
> mode http
> option httplog
> option  http-server-close
> option  forwardfor
> option dontlognull
> # Set the listen limit: The default is 2000, too small for us
> maxconn 9000
>
> # we should fix this
> option accept-invalid-http-response
> option accept-invalid-http-request
> no option checkcache
>
> timeout connect 8
> timeout client  90
> timeout server  50
>
> errorfile 400 /etc/haproxy/errors/400.http
> errorfile 403 /etc/haproxy/errors/403.http
> errorfile 408 /etc/haproxy/errors/408.http
> errorfile 500 /etc/haproxy/errors/500.http
> errorfile 502 /etc/haproxy/errors/502.http
> errorfile 503 /etc/haproxy/errors/503.http
> errorfile 504 /etc/haproxy/errors/504.http
>
> frontend http-in
> bind *:80
>
>
> # Skip the message broker from redirection
> acl skip_pages   path_reg ^/([\w]{2}/)?(message|yrf-laps)/(.*)
>
> # Redirect all subdomains to www.
> redirect prefix https://www.example.com code 301 if !{ hdr_beg(host) -i
www. }
>
> # Redirect all trafic to https
> redirect scheme https if !skip_pages !{ ssl_fc }
> default_backend servers
>
> frontend https-in
> # add no-tlsv10 for disabling tls 1.0
> bind *:443 ssl  crt /etc/ssl/private/www_example_com.pem
>
> default_backend servers
> # Redirect all subdomains to www.
> redirect prefix https://www.example.com code 301 if !{ hdr_beg(host) -i
www. }
> backend servers
>
> # Skip the cre redirect
> acl stage_cre_redirect shdr_beg(Location)   http://stage.cre.com
> acl cre_redirect shdr_beg(Location)   http://www.cre.com
>
> # Skip the blog.example.com redirect
> acl blog_redirect shdr_beg(Location) http://blog.example.com
>
> # Rewrite the response location (for redirect cases)
> rspirep ^Location:\ http://(.*)  Location:\ https://\1  if  !cre_redirect
!stage_cre_redirect !blog_redirect { ssl_fc }
> # Every connection is closed and opened to the server
> option http-server-close
>
> # Recommended to enable
> option http-pretend-keepalive
> # The url to check the backend servers health
> option httpchk GET /srvstatus.htm
>
> # Balancing
> balance roundrobin
> appsession JSESSIONID len 52 timeout 3h request-learn prefix
> stick-table type string len 32 size 1M expire 3h
> # We have 3 backend servers, one is for backup
> server www1a 127.0.0.1:8080 check
> server www2a xx.xx.xx.xx:8080 check
> server www1b 127.0.0.1:8081 check  backup
>
--
>
> Sorry about the long haproxy config file. I was not sure which part is
relevant to this error.
> I would appreciate any pointers you could give me.
>
> Thank you,
> Zoltan.
>

Re: how to check if peering truly works

[Florin Andrei]

I've found why peering was not syncing up tables - I must indicate the 
peers by IP address, not by hostname. If I list peers by hostname, 
HAproxy will try to connect to peers, but it refuses any peering 
requests sent to itself (does not even listen to the peering TCP port).


So I made it work by changing it to this:

peers api
  peer haproxy-app01-staging-uswest2-aws.foobar.staging A.B.C.D:8989
  peer haproxy-app02-staging-uswest2-aws.foobar.staging X.Y.Z.K:8989

But my original question stands: is there a way to diagnose the state of 
peering just by querying the stats socket?


I also looked in the logs, but there's no mention of peering events 
there either. The statistics report page makes no mention either.


--
Florin Andrei
http://florin.myip.org/

how to check if peering truly works

[Florin Andrei]

Current config:

###
backend app_api
  mode http
  balance hdr(X-Forwarded-For)
  hash-type consistent
  stick on hdr(X-Forwarded-For)
  stick-table type string size 5M peers api
  option httpchk GET /status
  default-server inter 1 rise 10 fall 2
  server app01-staging-uswest2-aws 
app01-staging-uswest2-aws.foobar.staging:8080 check
  server app02-staging-uswest2-aws 
app02-staging-uswest2-aws.foobar.staging:8080 check



peers api
  peer haproxy-app01-staging-uswest2-aws.foobar.staging 
haproxy-app01-staging-uswest2-aws.foobar.staging:8989
  peer haproxy-app02-staging-uswest2-aws.foobar.staging 
haproxy-app02-staging-uswest2-aws.foobar.staging:8989

###

I'm checking the contents of the app_api table on both HAproxy 
instances, and they are for the most part the same, but a few entries 
are different - same key uses different server_id values on different 
HAproxy instances. I thought peering was supposed to get rid of 
inconsistencies like this.


How do I check if peering is actually enabled and passing data between 
proxies? I'm looking at the commands I could send to the stats socket, 
and there doesn't seem to be anything related to peering there.


--
Florin Andrei
http://florin.myip.org/

peers section with multiple backends

[Florin Andrei]

Configuration now:


frontend app_api_front
  bind *:80
  mode http
  default_backend app_api

backend app_api
  mode http
  balance hdr(X-Forwarded-For)
  hash-type consistent
  stick on hdr(X-Forwarded-For)
  stick-table type string size 5M peers api
  option httpchk GET /status
  default-server inter 1 rise 10 fall 2
  server app01-staging-uswest2-aws 
app01-staging-uswest2-aws.foobar.staging:8080 check
  server app02-staging-uswest2-aws 
app02-staging-uswest2-aws.foobar.staging:8080 check


peers api
  peer haproxy-app01-staging-uswest2-aws.foobar.staging 
haproxy-app01-staging-uswest2-aws.foobar.staging:8989
  peer haproxy-app02-staging-uswest2-aws.foobar.staging 
haproxy-app02-staging-uswest2-aws.foobar.staging:8989



Let's say I add a new backend, using completely different backend 
servers, and a new frontend pointing at the new backend. Think of it as 
a completely different website, but serviced by the same HAproxy 
cluster. The new backend uses the same stickiness criteria like the old 
one.


Can I re-use the same peers list for the new backend? Would each backend 
maintain a separate stick table automatically?


Or do I have to create a new peers list for each backend? In that case, 
can I use the same host:port for each peer member, or do I have to use 
different ports for each peer list?


I other words, can I reuse the existing peers for the new backend, or do 
I have to do this?


###
peers api
  peer one one:8989
  peer two two:8989

peers api2
  peer one one:8989
  peer two two:8989
###

Or do I have to do this?

###
peers api
  peer one one:8989
  peer two two:8989

peers api2
  peer one one:8990
  peer two two:8990
###


--
Florin Andrei
http://florin.myip.org/

Re: Only using map file when an entry exists

[Neil - HAProxy List]

Thanks Conrad,

That sort of thing looks better that what I had, and I'll give it a go.

I still think this is a bit long winded syntax for something that probably
quite a common things to want to do?  A map_contains type boolean function
still seems like a good to have?

Thanks

Neil

On 3 March 2016 at 13:05, Conrad Hoffmann  wrote:

> If you are using haproxy >=1.6, you might be able to do something like
> this:
>
> acl no_redir %[req.redir] -m str NO_REDIR
> http-request set-var(req.redir) \
> %[hdr(host),map(/etc/haproxy/redirect_host.map,NO_REDIR)]
> http-request redirect location %[req.redir] code 301 if !no_redir
>
> This is completely made up and untested, but I hope you get the idea.
> Avoids a second map lookup altogether, but also map lookups are quite fast,
> so unless you map is huge you don't really need to worry about this. Also,
> double negation, but this is just to give you some idea
>
> Cheers,
> Conrad
> --
> Conrad Hoffmann
> Traffic Engineer
>
> SoundCloud Ltd. | Rheinsberger Str. 76/77, 10115 Berlin, Germany
>
> Managing Director: Alexander Ljung | Incorporated in England & Wales
> with Company No. 6343600 | Local Branch Office | AG Charlottenburg |
> HRB 110657B
>

Re: Only using map file when an entry exists

[Conrad Hoffmann]

If you are using haproxy >=1.6, you might be able to do something like this:

acl no_redir %[req.redir] -m str NO_REDIR
http-request set-var(req.redir) \
%[hdr(host),map(/etc/haproxy/redirect_host.map,NO_REDIR)]
http-request redirect location %[req.redir] code 301 if !no_redir

This is completely made up and untested, but I hope you get the idea.
Avoids a second map lookup altogether, but also map lookups are quite fast,
so unless you map is huge you don't really need to worry about this. Also,
double negation, but this is just to give you some idea

Cheers,
Conrad
-- 
Conrad Hoffmann
Traffic Engineer

SoundCloud Ltd. | Rheinsberger Str. 76/77, 10115 Berlin, Germany

Managing Director: Alexander Ljung | Incorporated in England & Wales
with Company No. 6343600 | Local Branch Office | AG Charlottenburg |
HRB 110657B

Re: Only using map file when an entry exists

[Nenad Merdanovic]

Hello

On 3/3/2016 1:40 PM, Neil - HAProxy List wrote:
> Hello
> This works but is yuck (I'd have to automate generating the acl file
> from the map - not hard but not clean). Ideally I'd like a way to only
> redirect when a value is in the map what would be fine is if there were
> a contained_in_map function that I could use something like
> 
>   http-request redirect location
> %[hdr(host),map(/etc/haproxy/redirect_host.map)] code 301 if
> %[hdr(host),contained_in_map(/etc/haproxy/redirect_host.map)]
> 

Try something like:
http-request redirect location
%[hdr(host),map(/etc/haproxy/redirect_host.map)] code 301 if {
hdr(host),map(/etc/haproxy/redirect_host.map) -m found }

Regards,
Nenad

Only using map file when an entry exists

[Neil - HAProxy List]

Hello

HA-Proxy version 1.5.15 2015/11/01

I've got a service with some redirects for old virtual hosts to new
locations on main website that I want to store in a map file
/etc/haproxy/redirect_host.map with lines like
www.oldname.com http://www.shiny.net/collections/oldname

My issue is I don't want a redirect to occur when there is no entry in the
map

I started with

  http-request redirect location
%[hdr(host),map(/etc/haproxy/redirect_host.map)] code 301

This would take out the whole site as a request to http://www.shiny.net
gets a redirect with a blank location. (and so does
http://www.shiny.net/collections/oldname) - this is because they are all in
the same frontend

so as a hack around I've taken the first column to another file and gone
with

  acl isRedirectHost hdr(host) -i -f /etc/haproxy/acl_isRedirectHost.txt
  http-request redirect location
%[hdr(host),map(/etc/haproxy/redirect_host.map)] code 301 if isRedirectHost

This works but is yuck (I'd have to automate generating the acl file from
the map - not hard but not clean). Ideally I'd like a way to only redirect
when a value is in the map what would be fine is if there were a
contained_in_map function that I could use something like

  http-request redirect location
%[hdr(host),map(/etc/haproxy/redirect_host.map)] code 301 if
%[hdr(host),contained_in_map(/etc/haproxy/redirect_host.map)]

All other suggestions very welcome too

Thank you,

Neil