> Hi,
>
> On Sat, Oct 11, Kari Mattsson wrote:
> > this got repeated for 50+ times when refreshing on Chrome browser.
> > Then to Firefox..
> > Oct 11 20:25:17 localhost haproxy[5179]: 10.6.159.238:4248
> > [11/Oct/2014:20:25:14.300] service_1_outside_80
> > service_1_inside/App_101 3264/0/0/1/+3265 200 +275 - - --NI
> > 1/1/1/1/0 0/0 {service1.example.com} {7|} "GET / HTTP/1.1"
> > Oct 11 20:25:22 localhost haproxy[5179]: 10.6.159.238:4252
> > [11/Oct/2014:20:25:22.854] service_2_outside_80
> > service_2_inside/App_142 0/0/0/1/+1 200 +275 - - --NI 1/1/1/1/0
> > 0/0 {service1.example.com} {5|} "GET / HTTP/1.1"
>
> --NI = client provided no cookie, proxy inserted one
Above, Firefox was accessing service1. Got web page from it correctly (first
line above). The 8 seconds later when F5 was pressed again on Firefox, it
landed on service2 backend server erroneously and proxy inserted cookie from
there.
Problem is, the second line is totally in error. Firefox had service1 URL on
address line, and it was showing web page from service2.
If you want, I can provide you the full config and real URLs.
> > Oct 11 20:25:27 localhost haproxy[5179]: 10.6.159.238:4254
> > [11/Oct/2014:20:25:27.914] service_2_outside_80
> > service_2_inside/App_142 0/0/0/1/+1 304 +120 SERVICE_2=app142 -
> > --VN 1/1/1/1/0 0/0 {service1.example.com} {|} "GET / HTTP/1.1"
> > Oct 11 20:27:31 localhost haproxy[5179]: 10.6.159.238:4283
> > [11/Oct/2014:20:27:31.947] service_1_outside_80
> > service_1_inside/App_101 0/0/0/1/+1 200 +237 SERVICE_1=app101 -
> > --VN 1/1/1/1/0 0/0 {service1.example.com} {7|} "GET / HTTP/1.1"
> >
> > Looks like browser will not receive a cookie for the first 2 page
> > loads.
> > On third it received... but a wrong cookie.
>
> --VN = client provided valid cookie, proxy didn't set cookie
> On the third log line what was supposed to happen ?
> Looks like haproxy received SERVICE_2=app142 cookie and the
> connection
> was send to service_2_inside/App_142
Above, on first line browser was accessing service1, and HAProxy provided
content from service2 backend servers. Frontend and backend got mixed again.
> > After 2 minutes fourth reload, and if will receive the right
> > cookie.
> > Reloading page from this on, keeps it on the browser right
> > frontend/backend.
> > Weird.
>
> What are those {service1.example.com} {7|} in logs ?
I really don't know :-/ Before it is the FQDN what browser is trying to
access. First line above is mixed/error, second line is correct (access
service1, receive data from service1).
> I'm assuming that
> SERVICE_1=/SERVICE_2=... is capture cookie SERVICE_1
> or capture cookie SERVICE_2 ?
>
> > Now back to Chrome again for one more page reload:
> > Oct 11 20:29:28 localhost haproxy[5179]: 10.6.159.238:4311
> > [11/Oct/2014:20:29:28.561] service_2_outside_80
> > service_2_inside/App_141 0/0/1/0/+1 200 +237 SERVICE_2=app141 -
> > --VN 1/1/1/1/0 0/0 {service1.example.com} {5|} "GET / HTTP/1.1"
> >
> > Damn. Chrome falls to wrong frontend/backend.
>
> Where the connection from chrome should have gone ?
Shoud have gone to: service1.example.com
It really went to: service_2_outside_80 and from there to
service_2_inside/App_141
> > One more. Fireforx, 2 page re-loads for service1.example.com:
> > Oct 11 20:31:52 localhost haproxy[5179]: 10.6.159.238:4350
> > [11/Oct/2014:20:31:52.023] service_2_outside_80
> > service_2_inside/App_142 0/0/0/1/+1 200 +237 SERVICE_2=app142 -
> > --VN 1/1/1/1/0 0/0 {service1.example.com} {5|} "GET / HTTP/1.1"
> > Oct 11 20:31:55 localhost haproxy[5179]: 10.6.159.238:4352
> > [11/Oct/2014:20:31:55.419] service_1_outside_80
> > service_1_inside/App_101 0/0/0/1/+1 200 +237 SERVICE_1=app101 -
> > --VN 1/1/1/1/0 0/0 {service1.example.com} {7|} "GET / HTTP/1.1"
> >
> > ...first wrong, then right. So, it is flip-floping.
>
> Is the {service1.example.com} captured host header ?
Yes as shown on haproxy.log. This is where browser wants to go.
> The connection
> goes
> to two different frontends (first goes to service_2_outside_80 and
> second goes to service_1_outside_80). Should it have gone to the same
> frontend ?
Correct.
For service1.example.com there is frontend service_1_outside_80
For service2.example.com there is frontend service_2_outside_80
...and for
service_1_outside_80 there is backend service_1_inside
service_2_outside_80 there is backend service_2_inside
and...
service_1_inside has real web servers App_101 + App_102
service_2_inside has real web servers App_141 + App_142
> Do you have multiple ip addresses for service1.example.com in
> /etc/hosts or dns ? (one address for service_1_outside_80 and on one
> for
> service_2_outside_80 ?)
For frontend:
service1.example.com has A + PTR records properly defined and just single IP,
and
service2.example.com has A + PTR records properly defined and just single IP.
Both server1 and server2 are in different public subnets.
For backend:
service1 backend is NATed. The App_101 and App_102 have private IP numbers.
service2 backend is public. The App_141 and App_142 have public IP numbers.
For service2 further..
service2.example.com IP number: a.b.c.140/27
App_141 IP number: a.b.c.141/27
App_142 IP number: a.b.c.142/27
So these are in same subnet.
service1 is like
http://blog.haproxy.com/2011/07/22/layer-4-load-balancing-nat-mode/
service2 is like
http://blog.haproxy.com/2011/07/29/layer-4-load-balancing-direct-server-return-mode/
..I guess..
Apparently they do not work at the same time?
> > > - you could also use tcpdump to see what cookies firefox <->
> > > haproxy
> > > send/receive ?
>
> Sorry, what I had in mind was to use tcpdump/wireshark to see what
> cookies the client(browser) receives/sends to haproy (for example
> follow
> tcp stream in wireshark).
> You can probably use chrome developer tools (ctrl+shift+i) (network)
> to see the request/response headers. (Or firebug with firefox).
I'll dig into this..
> > Changing from cookie stickiness to source ip...
> > stick-table type ip
> > stick on src
> > ...also makes no difference. Same errorneous behaviour.
>
> Do you see any entries in the stick table ? Something like
> echo "show table service_2_inside" | nc -U /path/to/stats.socket
> (or with socat instead of nc -U).
z1 ~ $ echo "show table service_1_inside" | nc -U /var/lib/haproxy/stats
# table: service_1_inside, type: string, size:1048576, used:0
z1 ~ $ echo "show table service_2_inside" | nc -U /var/lib/haproxy/stats
# table: service_2_inside, type: string, size:1048576, used:0
I'll keep experimenting...
Thanks, Kari
> -Jarno
>
> --
> Jarno Huuskonen
>
--
Kari Mattsson | +358.5069000 | https://mysync.fi | https://mysync-dm.com
Trivore Corp., Old Mill, Ruukinkatu 2-4, FI-20540 Turku, Finland
