Re: Haproxy 1. 5.14 + Tomcat 8 giving random 502 errors
Jeff, thanks again for the detailed answer. I've already tested a configuration without various ACL's. When i check the errors logs. I got this: [02/Mar/2016:07:38:34.834] backend servers (#4): invalid response frontend https-in (#3), server www1a (#1), event #9270 src 46.140.96.254:61409, session #20091598, session flags 0x00cf HTTP msg state 26, msg flags 0x, tx flags 0xa8200060 HTTP chunk len 0 bytes, HTTP body len 0 bytes buffer flags 0x00048002, out 0 bytes, total 5 bytes pending 5 bytes, wrapping at 16384, error at position 2: 0 20c\r\n Do you have any idea what could cause this? Thank you, Zoltan. On Tue, Mar 1, 2016 at 6:11 PM, Jeff Palmer wrote: > Zoltan, > > > the "HP--" in the log means: > > P : the session was prematurely aborted by the proxy, because of a > connection limit enforcement, because a DENY filter was > matched, > because of a security check which detected and blocked a > dangerous > error in server response which might have caused information > leak > (eg: cacheable cookie). > > H : the proxy was waiting for complete, valid response HEADERS from the > server (HTTP only). > > > Since you are saying that hitting the tomcat backend directly works, > I would suggest trying to remove all the extra acls and matches. > Try getting the basic frontend/backend part working, then add your > ACL's and matches one at a time. > > The "P" in the log unfortunately leaves a lot of potential reasons, > whcih you are likely going to have to track down via trial and error. > If removing the additional ACL's doesn't cause it to start working, > I'd look at cookies and the "dangerous error" aspects. > > > > > On Tue, Mar 1, 2016 at 7:36 AM, Zoltan Lorincz wrote: > > Dear Jeff, > > > > thank you very much for your answer! > > The Tomcat connector doesn't have HTTPS enabled. > > I forgot to remove the old "redirectPort" from tomcat's connector > setting, > > i did remove it now and restarted servers but the error still persist. > > > > Thank you, > > Zoltan. > > > > > > On Tue, Mar 1, 2016 at 2:19 PM, Jeff Palmer wrote: > >> > >> You have tomcat on 8443 which is usually an SSL enabled port, but none > of > >> your backend server definitions enable SSL. > >> > >> In the 3 'server' lines towards the end of your config, add 'ssl' at the > >> end. > >> > >> Let us know! > >> > >> On Mar 1, 2016 5:57 AM, "Zoltan Lorincz" wrote: > >>> > >>> Hi all, > >>> > >>> i am very new to haproxy. Read trough all the docs but i think > something > >>> is wrong with my configuration, because if we connect directly to > tomcat we > >>> don't get any 502 errors. > >>> > >>> The errors from haproxy look like this. > >>> > >>> Mar 1 11:41:37 www1 haproxy[15362]: xx.xx.xx.xx:56387 > >>> [01/Mar/2016:11:41:35.480] https-in~ servers/www1a 1987/0/0/-1/2029 > 502 8878 > >>> - - PH-- 1764/1758/46/26/0 0/0 "POST > >>> /abc/test/b25766378a05446496645649e2ddaf7a/poll HTTP/1.1" > >>> > >>> > >>> > >>> Tomcat connector config: > >>> > >>> > --- > >>> >>> URIEncoding = "UTF-8" > >>> port = "8080" > >>> protocol = "HTTP/1.1" > >>> maxThreads = "1850" > >>> connectionTimeout = "90" > >>> keepAliveTimeout = "90" > >>> maxKeepAliveRequests = "-1" > >>> redirectPort = "8443" /> > >>> > >>> > >>> > --- > >>> > >>> > >>> Haproxy config: > >>> > >>> > --- > >>> global > >>> log /dev/log local0 > >>> log /dev/log local1 notice > >>> chroot /var/lib/haproxy > >>> stats socket /run/haproxy/admin.sock mode 777 level admin > >>> stats timeout 30s > >>> user haproxy > >>> group haproxy > >>> daemon > >>> > >>>
Re: Haproxy 1. 5.14 + Tomcat 8 giving random 502 errors
Dear Jeff,
thank you very much for your answer!
The Tomcat connector doesn't have HTTPS enabled.
I forgot to remove the old "redirectPort" from tomcat's connector setting,
i did remove it now and restarted servers but the error still persist.
Thank you,
Zoltan.
On Tue, Mar 1, 2016 at 2:19 PM, Jeff Palmer wrote:
> You have tomcat on 8443 which is usually an SSL enabled port, but none of
> your backend server definitions enable SSL.
>
> In the 3 'server' lines towards the end of your config, add 'ssl' at the
> end.
>
> Let us know!
> On Mar 1, 2016 5:57 AM, "Zoltan Lorincz" wrote:
>
>> Hi all,
>>
>> i am very new to haproxy. Read trough all the docs but i think something
>> is wrong with my configuration, because if we connect directly to tomcat we
>> don't get any 502 errors.
>>
>> The errors from haproxy look like this.
>>
>> Mar 1 11:41:37 www1 haproxy[15362]: xx.xx.xx.xx:56387
>> [01/Mar/2016:11:41:35.480] https-in~ servers/www1a 1987/0/0/-1/2029 502
>> 8878 - - PH-- 1764/1758/46/26/0 0/0 "POST
>> /abc/test/b25766378a05446496645649e2ddaf7a/poll HTTP/1.1"
>>
>>
>>
>> Tomcat connector config:
>>
>> ---
>> > URIEncoding = "UTF-8"
>> port = "8080"
>> protocol = "HTTP/1.1"
>> maxThreads = "1850"
>> connectionTimeout = "90"
>> keepAliveTimeout = "90"
>> maxKeepAliveRequests = "-1"
>> redirectPort = "8443" />
>>
>>
>> ---
>>
>>
>> Haproxy config:
>>
>> ---
>> global
>> log /dev/log local0
>> log /dev/log local1 notice
>> chroot /var/lib/haproxy
>> stats socket /run/haproxy/admin.sock mode 777 level admin
>> stats timeout 30s
>> user haproxy
>> group haproxy
>> daemon
>>
>> # Per process limit: The default is 2000, too small for us
>> maxconn 18000
>> # Increase the cache from 2 (default), higher values reduce CPU usage
>> tune.ssl.cachesize 6
>>
>> # Default SSL material locations
>> ca-base /etc/ssl/certs
>> crt-base /etc/ssl/private
>>
>> # Default ciphers to use on SSL-enabled listening sockets.
>> # For more information, see ciphers(1SSL).
>> ssl-default-bind-ciphers
>> kEECDH+aRSA+AES:kRSA+AES:+AES256:!kEDH:!LOW:!EXP:!MD5:!aNULL:!eNULL
>> ssl-default-bind-options no-sslv3 no-tls-tickets
>>
>> defaults
>> log global
>> mode http
>> option httplog
>> option http-server-close
>> option forwardfor
>> option dontlognull
>> # Set the listen limit: The default is 2000, too small for us
>> maxconn 9000
>>
>> # we should fix this
>> option accept-invalid-http-response
>> option accept-invalid-http-request
>> no option checkcache
>>
>> timeout connect 8
>> timeout client 90
>> timeout server 50
>>
>> errorfile 400 /etc/haproxy/errors/400.http
>> errorfile 403 /etc/haproxy/errors/403.http
>> errorfile 408 /etc/haproxy/errors/408.http
>> errorfile 500 /etc/haproxy/errors/500.http
>> errorfile 502 /etc/haproxy/errors/502.http
>> errorfile 503 /etc/haproxy/errors/503.http
>> errorfile 504 /etc/haproxy/errors/504.http
>>
>> frontend http-in
>> bind *:80
>>
>>
>> # Skip the message broker from redirection
>> acl skip_pages path_reg ^/([\w]{2}/)?(message|yrf-laps)/(.*)
>>
>> # Redirect all subdomains to www.
>> redirect prefix https://www.example.com code 301 if !{ hdr_beg(host) -i
>> www. }
>>
>> # Redirect all trafic to https
>> redirect scheme https if !skip_pages !{ ssl_fc }
>> default_backend servers
>>
>> frontend https-in
>> # add no-tlsv10 for disabling tls 1.0
>> bind *:443 ssl crt /etc/ssl/private/www_example_com.pem
>>
>> default_backend servers
>> # Redirect all subdomains to www.
>> redirect prefix https://www.example.com code 301 if !{ hdr_beg(host) -i
>> www. }
>> backend servers
>>
>> # Skip the cre redirect
>> acl stage_cre_redirect shdr_beg(Location) http://stage.cre.com
>> acl cre_redirect shdr_beg(Location) http://www.cre.com
&g
Haproxy 1. 5.14 + Tomcat 8 giving random 502 errors
Hi all,
i am very new to haproxy. Read trough all the docs but i think something is
wrong with my configuration, because if we connect directly to tomcat we
don't get any 502 errors.
The errors from haproxy look like this.
Mar 1 11:41:37 www1 haproxy[15362]: xx.xx.xx.xx:56387
[01/Mar/2016:11:41:35.480] https-in~ servers/www1a 1987/0/0/-1/2029 502
8878 - - PH-- 1764/1758/46/26/0 0/0 "POST
/abc/test/b25766378a05446496645649e2ddaf7a/poll HTTP/1.1"
Tomcat connector config:
---
---
Haproxy config:
---
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 777 level admin
stats timeout 30s
user haproxy
group haproxy
daemon
# Per process limit: The default is 2000, too small for us
maxconn 18000
# Increase the cache from 2 (default), higher values reduce CPU usage
tune.ssl.cachesize 6
# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
# Default ciphers to use on SSL-enabled listening sockets.
# For more information, see ciphers(1SSL).
ssl-default-bind-ciphers
kEECDH+aRSA+AES:kRSA+AES:+AES256:!kEDH:!LOW:!EXP:!MD5:!aNULL:!eNULL
ssl-default-bind-options no-sslv3 no-tls-tickets
defaults
log global
mode http
option httplog
option http-server-close
option forwardfor
option dontlognull
# Set the listen limit: The default is 2000, too small for us
maxconn 9000
# we should fix this
option accept-invalid-http-response
option accept-invalid-http-request
no option checkcache
timeout connect 8
timeout client 90
timeout server 50
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
frontend http-in
bind *:80
# Skip the message broker from redirection
acl skip_pages path_reg ^/([\w]{2}/)?(message|yrf-laps)/(.*)
# Redirect all subdomains to www.
redirect prefix https://www.example.com code 301 if !{ hdr_beg(host) -i
www. }
# Redirect all trafic to https
redirect scheme https if !skip_pages !{ ssl_fc }
default_backend servers
frontend https-in
# add no-tlsv10 for disabling tls 1.0
bind *:443 ssl crt /etc/ssl/private/www_example_com.pem
default_backend servers
# Redirect all subdomains to www.
redirect prefix https://www.example.com code 301 if !{ hdr_beg(host) -i
www. }
backend servers
# Skip the cre redirect
acl stage_cre_redirect shdr_beg(Location) http://stage.cre.com
acl cre_redirect shdr_beg(Location) http://www.cre.com
# Skip the blog.example.com redirect
acl blog_redirect shdr_beg(Location) http://blog.example.com
# Rewrite the response location (for redirect cases)
rspirep ^Location:\ http://(.*) Location:\ https://\1 if !cre_redirect
!stage_cre_redirect !blog_redirect { ssl_fc }
# Every connection is closed and opened to the server
option http-server-close
# Recommended to enable
option http-pretend-keepalive
# The url to check the backend servers health
option httpchk GET /srvstatus.htm
# Balancing
balance roundrobin
appsession JSESSIONID len 52 timeout 3h request-learn prefix
stick-table type string len 32 size 1M expire 3h
# We have 3 backend servers, one is for backup
server www1a 127.0.0.1:8080 check
server www2a xx.xx.xx.xx:8080 check
server www1b 127.0.0.1:8081 check backup
--
Sorry about the long haproxy config file. I was not sure which part is
relevant to this error.
I would appreciate any pointers you could give me.
Thank you,
Zoltan.
