Re: HAProxy stops working all of a sudden

2012-08-17 Thread Rahul Nair
Thanks a lot Willy,

That was a great catch.
Till now no issues are seen, seems that the issue is fixed.
I will let the group know if I see any issues further.

-Rahul N.


On Thu, Aug 16, 2012 at 4:42 PM, Rahul Nair rahul.n...@finicity.com wrote:

 Willy,

 I have set send_redirects to zero in net.ipv4.conf.*. on HAProxy server.
 I have tested this for more than 1 hour, the issue is not yet observed.
 Few more tests are yet to be done, I will update you as soon as the
 testing is done.

 Baptiste,

 Thanks for the help.
 I will try using 2 Physical NICs after this testing is done.

 -Rahul N.


 On Wed, Aug 15, 2012 at 2:15 PM, Willy Tarreau w...@1wt.eu wrote:

 Hi,

 On Wed, Aug 15, 2012 at 10:33:18AM +0200, Baptiste wrote:
  On Mon, Aug 13, 2012 at 9:11 AM, Rahul Nair rahul.n...@finicity.com
 wrote:
   Hi,
  
   I am using single NIC card and IPs of both the network (VIP  Real
 servers
   network) are configured on virtual ethernet adapters (eth0:0 
  eth0:1).
   Ip_forward is enabled on the HAProxy server.
  
   Thanks
   Rahul N.
 
  Hi,
 
  So could you try with two physical NICs please?

 Before this it would be nice to disable ICMP redirects. I suspect that the
 system is regularly sending ICMP redirects to the real servers for the
 return
 traffic when it has to route late packets that don't belong to an existing
 connection anymore.

 You can do that by setting send_redirects to zero in net.ipv4.conf.*.

 Regards,
 Willy




Re: HAProxy stops working all of a sudden

2012-08-16 Thread Rahul Nair
Willy,

I have set send_redirects to zero in net.ipv4.conf.*. on HAProxy server.
I have tested this for more than 1 hour, the issue is not yet observed.
Few more tests are yet to be done, I will update you as soon as the testing
is done.

Baptiste,

Thanks for the help.
I will try using 2 Physical NICs after this testing is done.

-Rahul N.

On Wed, Aug 15, 2012 at 2:15 PM, Willy Tarreau w...@1wt.eu wrote:

 Hi,

 On Wed, Aug 15, 2012 at 10:33:18AM +0200, Baptiste wrote:
  On Mon, Aug 13, 2012 at 9:11 AM, Rahul Nair rahul.n...@finicity.com
 wrote:
   Hi,
  
   I am using single NIC card and IPs of both the network (VIP  Real
 servers
   network) are configured on virtual ethernet adapters (eth0:0 
  eth0:1).
   Ip_forward is enabled on the HAProxy server.
  
   Thanks
   Rahul N.
 
  Hi,
 
  So could you try with two physical NICs please?

 Before this it would be nice to disable ICMP redirects. I suspect that the
 system is regularly sending ICMP redirects to the real servers for the
 return
 traffic when it has to route late packets that don't belong to an existing
 connection anymore.

 You can do that by setting send_redirects to zero in net.ipv4.conf.*.

 Regards,
 Willy




-- 
-Rahul N.
IT Department
In2M Technologies Pvt Ltd. (Finicity)
Website: www.finicity.com/india


Re: HAProxy stops working all of a sudden

2012-08-15 Thread Baptiste
On Mon, Aug 13, 2012 at 9:11 AM, Rahul Nair rahul.n...@finicity.com wrote:
 Hi,

 I am using single NIC card and IPs of both the network (VIP  Real servers
 network) are configured on virtual ethernet adapters (eth0:0   eth0:1).
 Ip_forward is enabled on the HAProxy server.

 Thanks
 Rahul N.

Hi,

So could you try with two physical NICs please?

cheers



Re: HAProxy stops working all of a sudden

2012-08-15 Thread Willy Tarreau
Hi,

On Wed, Aug 15, 2012 at 10:33:18AM +0200, Baptiste wrote:
 On Mon, Aug 13, 2012 at 9:11 AM, Rahul Nair rahul.n...@finicity.com wrote:
  Hi,
 
  I am using single NIC card and IPs of both the network (VIP  Real servers
  network) are configured on virtual ethernet adapters (eth0:0   eth0:1).
  Ip_forward is enabled on the HAProxy server.
 
  Thanks
  Rahul N.
 
 Hi,
 
 So could you try with two physical NICs please?

Before this it would be nice to disable ICMP redirects. I suspect that the
system is regularly sending ICMP redirects to the real servers for the return
traffic when it has to route late packets that don't belong to an existing
connection anymore.

You can do that by setting send_redirects to zero in net.ipv4.conf.*.

Regards,
Willy




Re: HAProxy stops working all of a sudden

2012-08-13 Thread Baptiste
Hi,

Are you using 2 NICs or a single one?
Have you enable ip_forward on the HAProxy box?

cheers



HAProxy stops working all of a sudden

2012-08-13 Thread Rahul Nair
Hi,

I am using single NIC card and IPs of both the network (VIP  Real servers
network) are configured on virtual ethernet adapters (eth0:0   eth0:1).
Ip_forward is enabled on the HAProxy server.

Thanks
Rahul N.

On Monday, August 13, 2012, Baptiste bed...@gmail.com wrote:
 Hi,

 Are you using 2 NICs or a single one?
 Have you enable ip_forward on the HAProxy box?

 cheers


-- 
-Rahul N.
IT Department
In2M Technologies Pvt Ltd. (Finicity)
Website: www.finicity.com/india


Re: HAProxy stops working all of a sudden

2012-08-12 Thread Rahul Nair
Group,

I investigated further on this.
The default value of net.ipv4.conf.eth0.rp_filter is 1
Which means that Reverse Path filter is in Strict mode i.e Each incoming
packet is tested against the FIB and if the interface  is not the best
reverse path the packet check will fail.By default failed packets are
discarded.
Since I have only one ethernet interface and the same interface has the
information of reverse path to external WAN network and Both the internal
networks (Real server networks and VIP network) I believe that there are
very less chance that this could be the issue.
Please guide me on which direction I need to investigate further it will be
a great help.

Thanks,
-Rahul N.


On Sat, Aug 11, 2012 at 9:10 PM, Rahul Nair rahul.n...@finicity.com wrote:

 Group,

 Any advise on this?

 -Rahul N.


 On Saturday, August 11, 2012, Rahul Nair rahul.n...@finicity.com wrote:
  By default net.ipv4.conf.eth0.rp_filter is 1 which means  IP spoofing
 protection is enabled (source route verification is turned on)
  As far as I understand, TPROXY spoofs outgoing connections using the
 client's IP address.
  But since all the IPs are configured on same physical adapter,  not sure
 if setting net.ipv4.conf.eth0.rp_filter to 0 will make any difference.
  Also the issue is intermittent and not Boolean in nature.
  -Rahul N.
  On Sat, Aug 11, 2012 at 12:01 AM, Rahul Nair rahul.n...@finicity.com
 wrote:
 
  Willy,
  Following are the information I could gather:
  As per this link we need to add following sysctl parameters.
  #Reverse Path Filtering: Basically, if the reply to a packet wouldn't go
 out the interface this packet came in, then this is a bogus packet and
 should be ignored.
  net.ipv4.conf.default.rp_filter = 2
  net.ipv4.conf.all.rp_filter = 2
  #Disable Reverse Path Filtering
  net.ipv4.conf.eth0.rp_filter = 0
  I am currently using a single physical ethernet interface for both
 the networks (realservers network and VIPs network)
  Are there any chances that this could create any issues?
  Regarding kernel bugs:
 
  2.6.28 to 2.6.32 have different rp_filter configuration. The rp_filter
 settings (0 or 1) for these kernels will silently block TPROXY if used on
 newer kernels.
  2.6.28 to 2.6.36 are known to have ICMP and TIME_WAIT issues.
  2.6.32 to 2.6.34 have bridging issues on some systems
 
  I am now using kernel version: 2.6.38-15-server
  Thanks
  Rahul N.
 
  On Fri, Aug 10, 2012 at 1:30 PM, Rahul Nair rahul.n...@finicity.com
 wrote:
 
  Just FYI...
  Following are the config parameters that I use:
  global
  log 127.0.0.1 local0 info
  ulimit-n 80038
  chroot /var/lib/haproxy
  daemon
  defaults
  log global
  mode http
  option dontlognull
  retries 3
  option redispatch
  maxconn 2000
  timeout connect 5000
  timeout client 30
  timeout server 30
 
  listen httpsite 10.1.16.15:80
  mode http
  balance leastconn
  cookie PHPSESSID prefix
  option httpclose
  server web01 10.1.1.20 cookie web01 check
  server web02 10.1.1.30 cookie web02 check
  listen httpssite 10.1.16.15:443
  mode tcp
  source 0.0.0.0 usesrc clientip
  balance source
  option ssl-hello-chk
  server web01 10.1.1.20 check
  server web02 10.1.1.30 check
  Thanks
  Rahul N.
  On Fri, Aug 10, 2012 at 11:20 AM, Rahul Nair rahul.n...@finicity.com
 wrote:
 
  Willy,
 
  The issue still persists.
  Not sure what am I missing.
 
  -Rahul N.
 
  On Friday, August 10, 2012, Rahul Nair rahul.n...@finicity.com wrote:
  Willy,
  I have  upgraded the Linux kernel to and haproxy to 1.4.18 and kernel
 to 2.6.38-15-server
  Will monitor it for few days and will let you know the updates.
  -Rahul N.
 
  On Fri, Aug 10, 2012 at 2:04 AM, Willy Tarreau w...@1wt.eu wrote:
 
  On Thu, Aug 09, 2012 at 11:54:08PM +0530, Rahul Nair wrote:
   Willy,
  
   From your description, it could be an issue with some connection
   tracking somewhere caused by excess of source addr:ports.
  
   Ohh ok..
   Also I just found that as per the documentation in this link , it
 says that
   it can cause problems when IP connection tracking is enabled on the
   machine, because a same connection may be seen twice with different
 states.
   Does this mean that I need to disable the  nf_conntrack module by
 adding
   net.netfilter.nf_conntrack_acct = 0  t




Re: HAProxy stops working all of a sudden

2012-08-11 Thread Rahul Nair
Group,

Any advise on this?

-Rahul N.

On Saturday, August 11, 2012, Rahul Nair rahul.n...@finicity.com wrote:
 By default net.ipv4.conf.eth0.rp_filter is 1 which means  IP spoofing
protection is enabled (source route verification is turned on)
 As far as I understand, TPROXY spoofs outgoing connections using the
client's IP address.
 But since all the IPs are configured on same physical adapter,  not sure
if setting net.ipv4.conf.eth0.rp_filter to 0 will make any difference.
 Also the issue is intermittent and not Boolean in nature.
 -Rahul N.
 On Sat, Aug 11, 2012 at 12:01 AM, Rahul Nair rahul.n...@finicity.com
wrote:

 Willy,
 Following are the information I could gather:
 As per this link we need to add following sysctl parameters.
 #Reverse Path Filtering: Basically, if the reply to a packet wouldn't go
out the interface this packet came in, then this is a bogus packet and
should be ignored.
 net.ipv4.conf.default.rp_filter = 2
 net.ipv4.conf.all.rp_filter = 2
 #Disable Reverse Path Filtering
 net.ipv4.conf.eth0.rp_filter = 0
 I am currently using a single physical ethernet interface for both
the networks (realservers network and VIPs network)
 Are there any chances that this could create any issues?
 Regarding kernel bugs:

 2.6.28 to 2.6.32 have different rp_filter configuration. The rp_filter
settings (0 or 1) for these kernels will silently block TPROXY if used on
newer kernels.
 2.6.28 to 2.6.36 are known to have ICMP and TIME_WAIT issues.
 2.6.32 to 2.6.34 have bridging issues on some systems

 I am now using kernel version: 2.6.38-15-server
 Thanks
 Rahul N.

 On Fri, Aug 10, 2012 at 1:30 PM, Rahul Nair rahul.n...@finicity.com
wrote:

 Just FYI...
 Following are the config parameters that I use:
 global
 log 127.0.0.1 local0 info
 ulimit-n 80038
 chroot /var/lib/haproxy
 daemon
 defaults
 log global
 mode http
 option dontlognull
 retries 3
 option redispatch
 maxconn 2000
 timeout connect 5000
 timeout client 30
 timeout server 30

 listen httpsite 10.1.16.15:80
 mode http
 balance leastconn
 cookie PHPSESSID prefix
 option httpclose
 server web01 10.1.1.20 cookie web01 check
 server web02 10.1.1.30 cookie web02 check
 listen httpssite 10.1.16.15:443
 mode tcp
 source 0.0.0.0 usesrc clientip
 balance source
 option ssl-hello-chk
 server web01 10.1.1.20 check
 server web02 10.1.1.30 check
 Thanks
 Rahul N.
 On Fri, Aug 10, 2012 at 11:20 AM, Rahul Nair rahul.n...@finicity.com
wrote:

 Willy,

 The issue still persists.
 Not sure what am I missing.

 -Rahul N.

 On Friday, August 10, 2012, Rahul Nair rahul.n...@finicity.com wrote:
 Willy,
 I have  upgraded the Linux kernel to and haproxy to 1.4.18 and kernel
to 2.6.38-15-server
 Will monitor it for few days and will let you know the updates.
 -Rahul N.

 On Fri, Aug 10, 2012 at 2:04 AM, Willy Tarreau w...@1wt.eu wrote:

 On Thu, Aug 09, 2012 at 11:54:08PM +0530, Rahul Nair wrote:
  Willy,
 
  From your description, it could be an issue with some connection
  tracking somewhere caused by excess of source addr:ports.
 
  Ohh ok..
  Also I just found that as per the documentation in this link , it
says that
  it can cause problems when IP connection tracking is enabled on the
  machine, because a same connection may be seen twice with different
states.
  Does this mean that I need to disable the  nf_conntrack module by
adding
  net.netfilter.nf_conntrack_acct = 0  t

-- 
-Rahul N.
IT Department
In2M Technologies Pvt Ltd. (Finicity)
Website: www.finicity.com/india


Re: HAProxy stops working all of a sudden

2012-08-10 Thread Rahul Nair
Willy,

Following are the information I could gather:

As per this link http://www.snapt-ui.com/haproxy/snapt-haproxy-and-tproxy/ we
need to add following sysctl parameters.

#Reverse Path Filtering: Basically, if the reply to a packet wouldn't go
out the interface this packet came in, then this is a bogus packet and
should be ignored.
net.ipv4.conf.default.rp_filter = 2
net.ipv4.conf.all.rp_filter = 2

#Disable Reverse Path Filtering
net.ipv4.conf.eth0.rp_filter = 0

I am currently using a single physical ethernet interface for both
the networks (realservers network and VIPs network)
Are there any chances that this could create any issues?

Regarding kernel bugs:

   - 2.6.28 to 2.6.32 have different rp_filter configuration. The rp_filter
   settings (0 or 1) for these kernels will silently block TPROXY if used on
   newer kernels.
   - 2.6.28 to 2.6.36 are known to have ICMP and TIME_WAIT issues.
   - 2.6.32 to 2.6.34 have bridging issues on some systems

I am now using kernel version: 2.6.38-15-server

Thanks
Rahul N.

On Fri, Aug 10, 2012 at 1:30 PM, Rahul Nair rahul.n...@finicity.com wrote:

 Just FYI...
 Following are the config parameters that I use:

 global
 log 127.0.0.1 local0 info
 ulimit-n 80038
 chroot /var/lib/haproxy
 daemon

 defaults
 log global
 mode http
 option dontlognull
 retries 3
 option redispatch
 maxconn 2000
 timeout connect 5000
 timeout client 30
 timeout server 30


 listen httpsite 10.1.16.15:80
 mode http
 balance leastconn
 cookie PHPSESSID prefix
 option httpclose
 server web01 10.1.1.20 cookie web01 check
 server web02 10.1.1.30 cookie web02 check

 listen httpssite 10.1.16.15:443
 mode tcp
 source 0.0.0.0 usesrc clientip
 balance source
 option ssl-hello-chk
 server web01 10.1.1.20 check
 server web02 10.1.1.30 check

 Thanks
 Rahul N.

 On Fri, Aug 10, 2012 at 11:20 AM, Rahul Nair rahul.n...@finicity.comwrote:

 Willy,

 The issue still persists.
 Not sure what am I missing.

 -Rahul N.


 On Friday, August 10, 2012, Rahul Nair rahul.n...@finicity.com wrote:
  Willy,
  I have  upgraded the Linux kernel to and haproxy to 1.4.18 and kernel
 to 2.6.38-15-server
  Will monitor it for few days and will let you know the updates.
  -Rahul N.
 
  On Fri, Aug 10, 2012 at 2:04 AM, Willy Tarreau w...@1wt.eu wrote:
 
  On Thu, Aug 09, 2012 at 11:54:08PM +0530, Rahul Nair wrote:
   Willy,
  
   From your description, it could be an issue with some connection
   tracking somewhere caused by excess of source addr:ports.
  
   Ohh ok..
   Also I just found that as per the documentation in this link , it
 says that
   it can cause problems when IP connection tracking is enabled on the
   machine, because a same connection may be seen twice with different
 states.
   Does this mean that I need to disable the  nf_conntrack module by
 adding
   net.netfilter.nf_conntrack_acct = 0  to /etc/sysctl.conf ?
 
  You can't disable nf_conntrack using a sysctl. You need to unload the
  module itself. It's not nf_conntrack_acct but nf_conntrack.
 
   Bu default this module seems to be enabled.
cat /proc/sys/net/netfilter/nf_conntrack_acct
   1
  
   Following are the answers to your questions:
  
   What's your haproxy version and kernel version ?
  
  - HA-Proxy version: 1.4.8 2010/06/16
 
  Be careful, this is quite outdated ! 2 years of fixes have been merged
  since :
   $ git log --pretty=oneline v1.4.8..|grep -c BUG
   72
 
  = Your version has 72 bugs that have already been fixed now.
 I don't remember of any affecting transparent proxying though, but
 when you fix the issue you'd be advised to update it.
 
  - Kernel Version: 2.6.32-24-server
  - OS: Ubuntu 10.04
 
  You should also check that your kernel is up to date, as what you're
  observing might as well simply be a kernel bug.
 
   Are you sure all your servers route back through your haproxy box ?
  
  - Yes the default gateway of all the real servers is HAProxy
 server.
  - On real servers I have multiple IPs of two different networks
 - One which we use for communication between HAproxy server
 and Real
 servers.
 - And One which is used by the real servers to communicate
 with our
 internal application servers
 
  OK.
 
   Did you test only from one source machine or did you have many
 clients ?
  
  - This issue occurs intermittently from one or two different
 source IPs
  - At the same time when I check the functionality from another
 source
  IP, it works fine.
 
  Fine, then it really makes me think about a conntrack issue. Also, you
  should ensure that your client never directly talks to the server
 without
  passing via haproxy (which I can imagine you do during your tests when
  observing the issue). It only makes the problem worse with conntrack.
 
  Regards,
  Willy
 
 
 
 
  --
  -Rahul N.
  IT Department
  In2M Technologies Pvt Ltd. (Finicity)
  Website: www.finicity.com/india
 

 --
 -Rahul N.
 IT Department
 In2M 

Re: HAProxy stops working all of a sudden

2012-08-10 Thread Rahul Nair
By default net.ipv4.conf.eth0.rp_filter is 1 which means  IP spoofing
protection is enabled (source route verification is turned on)
As far as I understand, TPROXY spoofs outgoing connections using the
client's IP address.
But since all the IPs are configured on same physical adapter,  not sure if
setting net.ipv4.conf.eth0.rp_filter to 0 will make any difference.
Also the issue is intermittent and not Boolean in nature.

-Rahul N.

On Sat, Aug 11, 2012 at 12:01 AM, Rahul Nair rahul.n...@finicity.comwrote:

 Willy,

 Following are the information I could gather:

 As per this linkhttp://www.snapt-ui.com/haproxy/snapt-haproxy-and-tproxy/ we
 need to add following sysctl parameters.

 #Reverse Path Filtering: Basically, if the reply to a packet wouldn't go
 out the interface this packet came in, then this is a bogus packet and
 should be ignored.
 net.ipv4.conf.default.rp_filter = 2
 net.ipv4.conf.all.rp_filter = 2

 #Disable Reverse Path Filtering
 net.ipv4.conf.eth0.rp_filter = 0

 I am currently using a single physical ethernet interface for both
 the networks (realservers network and VIPs network)
 Are there any chances that this could create any issues?

 Regarding kernel bugs:

- 2.6.28 to 2.6.32 have different rp_filter configuration. The
rp_filter settings (0 or 1) for these kernels will silently block TPROXY if
used on newer kernels.
- 2.6.28 to 2.6.36 are known to have ICMP and TIME_WAIT issues.
- 2.6.32 to 2.6.34 have bridging issues on some systems

 I am now using kernel version: 2.6.38-15-server

 Thanks
 Rahul N.


 On Fri, Aug 10, 2012 at 1:30 PM, Rahul Nair rahul.n...@finicity.comwrote:

 Just FYI...
 Following are the config parameters that I use:

 global
 log 127.0.0.1 local0 info
 ulimit-n 80038
 chroot /var/lib/haproxy
 daemon

 defaults
 log global
 mode http
 option dontlognull
 retries 3
 option redispatch
 maxconn 2000
 timeout connect 5000
 timeout client 30
 timeout server 30


 listen httpsite 10.1.16.15:80
 mode http
 balance leastconn
 cookie PHPSESSID prefix
  option httpclose
 server web01 10.1.1.20 cookie web01 check
 server web02 10.1.1.30 cookie web02 check

 listen httpssite 10.1.16.15:443
  mode tcp
 source 0.0.0.0 usesrc clientip
 balance source
 option ssl-hello-chk
 server web01 10.1.1.20 check
 server web02 10.1.1.30 check

 Thanks
 Rahul N.

 On Fri, Aug 10, 2012 at 11:20 AM, Rahul Nair rahul.n...@finicity.comwrote:

 Willy,

 The issue still persists.
 Not sure what am I missing.

 -Rahul N.


 On Friday, August 10, 2012, Rahul Nair rahul.n...@finicity.com wrote:
  Willy,
  I have  upgraded the Linux kernel to and haproxy to 1.4.18 and kernel
 to 2.6.38-15-server
  Will monitor it for few days and will let you know the updates.
  -Rahul N.
 
  On Fri, Aug 10, 2012 at 2:04 AM, Willy Tarreau w...@1wt.eu wrote:
 
  On Thu, Aug 09, 2012 at 11:54:08PM +0530, Rahul Nair wrote:
   Willy,
  
   From your description, it could be an issue with some connection
   tracking somewhere caused by excess of source addr:ports.
  
   Ohh ok..
   Also I just found that as per the documentation in this link , it
 says that
   it can cause problems when IP connection tracking is enabled on the
   machine, because a same connection may be seen twice with different
 states.
   Does this mean that I need to disable the  nf_conntrack module by
 adding
   net.netfilter.nf_conntrack_acct = 0  to /etc/sysctl.conf ?
 
  You can't disable nf_conntrack using a sysctl. You need to unload the
  module itself. It's not nf_conntrack_acct but nf_conntrack.
 
   Bu default this module seems to be enabled.
cat /proc/sys/net/netfilter/nf_conntrack_acct
   1
  
   Following are the answers to your questions:
  
   What's your haproxy version and kernel version ?
  
  - HA-Proxy version: 1.4.8 2010/06/16
 
  Be careful, this is quite outdated ! 2 years of fixes have been merged
  since :
   $ git log --pretty=oneline v1.4.8..|grep -c BUG
   72
 
  = Your version has 72 bugs that have already been fixed now.
 I don't remember of any affecting transparent proxying though, but
 when you fix the issue you'd be advised to update it.
 
  - Kernel Version: 2.6.32-24-server
  - OS: Ubuntu 10.04
 
  You should also check that your kernel is up to date, as what you're
  observing might as well simply be a kernel bug.
 
   Are you sure all your servers route back through your haproxy box ?
  
  - Yes the default gateway of all the real servers is HAProxy
 server.
  - On real servers I have multiple IPs of two different networks
 - One which we use for communication between HAproxy server
 and Real
 servers.
 - And One which is used by the real servers to communicate
 with our
 internal application servers
 
  OK.
 
   Did you test only from one source machine or did you have many
 clients ?
  
  - This issue occurs intermittently from one or two different
 source IPs
  - At the same time when I check the 

Re: HAProxy stops working all of a sudden

2012-08-09 Thread Rahul Nair
Group,

Any clues on this issue..?

Thanks
Rahul N.

On Thursday, August 9, 2012, Rahul Nair rahul.n...@finicity.com wrote:
 Hello All,
 Please help me on this issue.
 Thanks,
 Rahul N.

 On Thu, Aug 9, 2012 at 12:13 AM, Rahul Nair rahul.n...@finicity.com
wrote:

 Guys,
 I am in process of implementing HAProxy with TPROXY in our setup for
mode tcp.
 All of a sudden the website stops working and gives out error in
browser: Error 107 (net::ERR_SSL_PROTOCOL_ERROR): SSL protocol error.
 When I remove/comment  source 0.0.0.0 usesrc clientip the website
starts working fine.
 And later on when I again enable source 0.0.0.0 usesrc clientip it
starts working fine, It seems that the issue is intermittent.
 Please help me understand what exactly the problem could be.
 Hardware configuration of HAProxy server:
 RAM:256MB
 Processor:Single core
 Thanks,
 Rahul N.




 --
 -Rahul N.
 IT Department
 In2M Technologies Pvt Ltd. (Finicity)
 Website: www.finicity.com/india


-- 
Sent from Gmail Mobile


Re: HAProxy stops working all of a sudden

2012-08-09 Thread Willy Tarreau
Hello Rahul,

On Thu, Aug 9, 2012 at 12:13 AM, Rahul Nair rahul.n...@finicity.com wrote:
 Guys,
 I am in process of implementing HAProxy with TPROXY in our setup for mode 
 tcp.
 All of a sudden the website stops working and gives out error in browser: 
 Error 107 (net::ERR_SSL_PROTOCOL_ERROR): SSL protocol error.
 When I remove/comment  source 0.0.0.0 usesrc clientip the website starts 
 working fine.
 And later on when I again enable source 0.0.0.0 usesrc clientip it starts 
 working fine, It seems that the issue is intermittent.
 Please help me understand what exactly the problem could be.
 Hardware configuration of HAProxy server:
 RAM:256MB
 Processor:Single core
 Thanks,
 Rahul N.

From your description, it could be an issue with some connection tracking
somewhere caused by excess of source addr:ports. But it could be many things.
What's your haproxy version and kernel version ? Are you sure all your
servers route back through your haproxy box ? Did you test only from one
source machine or did you have many clients ?

Willy




Re: HAProxy stops working all of a sudden

2012-08-09 Thread Rahul Nair
Willy,

From your description, it could be an issue with some connection
tracking somewhere caused by excess of source addr:ports.

Ohh ok..
Also I just found that as per the documentation in this link , it says that
it can cause problems when IP connection tracking is enabled on the
machine, because a same connection may be seen twice with different states.
Does this mean that I need to disable the  nf_conntrack module by adding
net.netfilter.nf_conntrack_acct = 0  to /etc/sysctl.conf ?

Bu default this module seems to be enabled.
 cat /proc/sys/net/netfilter/nf_conntrack_acct
1

Following are the answers to your questions:

What's your haproxy version and kernel version ?

   - HA-Proxy version: 1.4.8 2010/06/16
   - Kernel Version: 2.6.32-24-server
   - OS: Ubuntu 10.04


Are you sure all your servers route back through your haproxy box ?

   - Yes the default gateway of all the real servers is HAProxy server.
   - On real servers I have multiple IPs of two different networks
  - One which we use for communication between HAproxy server and Real
  servers.
  - And One which is used by the real servers to communicate with our
  internal application servers

Did you test only from one source machine or did you have many clients ?

   - This issue occurs intermittently from one or two different source IPs
   - At the same time when I check the functionality from another source
   IP, it works fine.

Thanks
Rahul N.

On Thu, Aug 9, 2012 at 10:56 PM, Willy Tarreau w...@1wt.eu wrote:

 Hello Rahul,

 On Thu, Aug 9, 2012 at 12:13 AM, Rahul Nair rahul.n...@finicity.com
 wrote:
  Guys,
  I am in process of implementing HAProxy with TPROXY in our setup for
 mode tcp.
  All of a sudden the website stops working and gives out error in
 browser: Error 107 (net::ERR_SSL_PROTOCOL_ERROR): SSL protocol error.
  When I remove/comment  source 0.0.0.0 usesrc clientip the website
 starts working fine.
  And later on when I again enable source 0.0.0.0 usesrc clientip it
 starts working fine, It seems that the issue is intermittent.
  Please help me understand what exactly the problem could be.
  Hardware configuration of HAProxy server:
  RAM:256MB
  Processor:Single core
  Thanks,
  Rahul N.

 From your description, it could be an issue with some connection tracking
 somewhere caused by excess of source addr:ports. But it could be many
 things.
 What's your haproxy version and kernel version ? Are you sure all your
 servers route back through your haproxy box ? Did you test only from one
 source machine or did you have many clients ?

 Willy




-- 
-Rahul N.
IT Department
In2M Technologies Pvt Ltd. (Finicity)
Website: www.finicity.com/india


Re: HAProxy stops working all of a sudden

2012-08-09 Thread Rahul Nair
Willy,

I have  upgraded the Linux kernel to and haproxy to 1.4.18 and kernel
to 2.6.38-15-server
Will monitor it for few days and will let you know the updates.

-Rahul N.


On Fri, Aug 10, 2012 at 2:04 AM, Willy Tarreau w...@1wt.eu wrote:

 On Thu, Aug 09, 2012 at 11:54:08PM +0530, Rahul Nair wrote:
  Willy,
 
  From your description, it could be an issue with some connection
  tracking somewhere caused by excess of source addr:ports.
 
  Ohh ok..
  Also I just found that as per the documentation in this link , it says
 that
  it can cause problems when IP connection tracking is enabled on the
  machine, because a same connection may be seen twice with different
 states.
  Does this mean that I need to disable the  nf_conntrack module by adding
  net.netfilter.nf_conntrack_acct = 0  to /etc/sysctl.conf ?

 You can't disable nf_conntrack using a sysctl. You need to unload the
 module itself. It's not nf_conntrack_acct but nf_conntrack.

  Bu default this module seems to be enabled.
   cat /proc/sys/net/netfilter/nf_conntrack_acct
  1
 
  Following are the answers to your questions:
 
  What's your haproxy version and kernel version ?
 
 - HA-Proxy version: 1.4.8 2010/06/16

 Be careful, this is quite outdated ! 2 years of fixes have been merged
 since :
  $ git log --pretty=oneline v1.4.8..|grep -c BUG
  72

 = Your version has 72 bugs that have already been fixed now.
I don't remember of any affecting transparent proxying though, but
when you fix the issue you'd be advised to update it.

 - Kernel Version: 2.6.32-24-server
 - OS: Ubuntu 10.04

 You should also check that your kernel is up to date, as what you're
 observing might as well simply be a kernel bug.

  Are you sure all your servers route back through your haproxy box ?
 
 - Yes the default gateway of all the real servers is HAProxy server.
 - On real servers I have multiple IPs of two different networks
- One which we use for communication between HAproxy server and
 Real
servers.
- And One which is used by the real servers to communicate with our
internal application servers

 OK.

  Did you test only from one source machine or did you have many clients ?
 
 - This issue occurs intermittently from one or two different source
 IPs
 - At the same time when I check the functionality from another source
 IP, it works fine.

 Fine, then it really makes me think about a conntrack issue. Also, you
 should ensure that your client never directly talks to the server without
 passing via haproxy (which I can imagine you do during your tests when
 observing the issue). It only makes the problem worse with conntrack.

 Regards,
 Willy




-- 
-Rahul N.
IT Department
In2M Technologies Pvt Ltd. (Finicity)
Website: www.finicity.com/india


Re: HAProxy stops working all of a sudden

2012-08-09 Thread Rahul Nair
Willy,

The issue still persists.
Not sure what am I missing.

-Rahul N.

On Friday, August 10, 2012, Rahul Nair rahul.n...@finicity.com wrote:
 Willy,
 I have  upgraded the Linux kernel to and haproxy to 1.4.18 and kernel
to 2.6.38-15-server
 Will monitor it for few days and will let you know the updates.
 -Rahul N.

 On Fri, Aug 10, 2012 at 2:04 AM, Willy Tarreau w...@1wt.eu wrote:

 On Thu, Aug 09, 2012 at 11:54:08PM +0530, Rahul Nair wrote:
  Willy,
 
  From your description, it could be an issue with some connection
  tracking somewhere caused by excess of source addr:ports.
 
  Ohh ok..
  Also I just found that as per the documentation in this link , it says
that
  it can cause problems when IP connection tracking is enabled on the
  machine, because a same connection may be seen twice with different
states.
  Does this mean that I need to disable the  nf_conntrack module by
adding
  net.netfilter.nf_conntrack_acct = 0  to /etc/sysctl.conf ?

 You can't disable nf_conntrack using a sysctl. You need to unload the
 module itself. It's not nf_conntrack_acct but nf_conntrack.

  Bu default this module seems to be enabled.
   cat /proc/sys/net/netfilter/nf_conntrack_acct
  1
 
  Following are the answers to your questions:
 
  What's your haproxy version and kernel version ?
 
 - HA-Proxy version: 1.4.8 2010/06/16

 Be careful, this is quite outdated ! 2 years of fixes have been merged
 since :
  $ git log --pretty=oneline v1.4.8..|grep -c BUG
  72

 = Your version has 72 bugs that have already been fixed now.
I don't remember of any affecting transparent proxying though, but
when you fix the issue you'd be advised to update it.

 - Kernel Version: 2.6.32-24-server
 - OS: Ubuntu 10.04

 You should also check that your kernel is up to date, as what you're
 observing might as well simply be a kernel bug.

  Are you sure all your servers route back through your haproxy box ?
 
 - Yes the default gateway of all the real servers is HAProxy server.
 - On real servers I have multiple IPs of two different networks
- One which we use for communication between HAproxy server and
Real
servers.
- And One which is used by the real servers to communicate with
our
internal application servers

 OK.

  Did you test only from one source machine or did you have many clients
?
 
 - This issue occurs intermittently from one or two different source
IPs
 - At the same time when I check the functionality from another
source
 IP, it works fine.

 Fine, then it really makes me think about a conntrack issue. Also, you
 should ensure that your client never directly talks to the server without
 passing via haproxy (which I can imagine you do during your tests when
 observing the issue). It only makes the problem worse with conntrack.

 Regards,
 Willy




 --
 -Rahul N.
 IT Department
 In2M Technologies Pvt Ltd. (Finicity)
 Website: www.finicity.com/india


-- 
-Rahul N.
IT Department
In2M Technologies Pvt Ltd. (Finicity)
Website: www.finicity.com/india