RE: tproxy bug in haproxy-1.5.10
Hi, Portforwarding to a different IP on the same haproxy-box causes haproxy behave buggy. This error happens when one uses the TPROXY target for portforwarding, ie. like this: Let's say the IP of the main interface is 192.168.100.100, and traffic from outside to port 1234 shall be forwarded to a virtual IP 192.168.100.101 and port 5678 on the same host, ie: I don't understand what you are trying to do and why. Can you elaborate what you are trying to do (in production, not in a lab) and can you also share the haproxy configuration? TPROXY is used to intercept TCP connections of non-local IPs. Using TPROXY IPs that configured on your local machine makes no sense. Lukas
Re: Mobile World Congress
Hello, Wishing you all the best for Mobile World Congress Mar-2 -5 2015. I understand that you are one of the exhibitors in Mobile World Congress Mar-2 -5 2015. Would you be interested in acquiring an attendee list (or) sending a pre-conference invitation to attendees? We can also provide list of buyers attendees who are mostly like to attend the trade shows, Conference, Exhibition etc Job Title Scope: Reach top-level executives like CEOs, CFOs, COOs, Presidents, Chairman's, GMs, Mid level Managers, and all key decision makers. Details Included: Company Name, Contact Name, Title, Address, Web Site, Phone Number, Fax Number, Verified Email Address, Employee Size, and Revenue size, SIC Code, Industry Type, NAICS Code, Email verification result. You can send Pre-Show Invitation mailers using these lists or we can help you schedule email campaigns to this list. We can even build any custom database that you might be looking for and that too based on your targeted demographics like Industry, SIC code, Employee Size etc. to make your event a grand success. If you are interested in any of our services. Kindly let me know your criteria with the geography so that I can derive few free samples, counts, quotes, accuracy and validity of the list. Target Industry: ___ (Any Industry - We have it all) Target Geography: ___ (All Over the World) Let me know your interest, so that we could discuss further Regards, Angela Jones | Inside Sales
Re: Round Robin not very random
We use leastconn to work arond Java apps having to GC (so GCing machine gets less connections while full GC runs). The problem with using it for HTTP is that it can be pretty uneven with a lot of short-lived connections, but so far that was not a problem for us, we usually use leastconn on app backends and round-robin on static/varnish backends On Thu, 15 Jan 2015 22:14:18 +0800, Alexey Zilber alexeyzil...@gmail.com wrote: Hi Vivek, You're correct. I think the situation was that there was a huge influx of traffic, and some servers went over their tipping point of how much they can handle quickly. This caused connections to stack up as some servers choked. Would leastconn give the same perfornance as roundrobin? I noticed in the haproxy docs that it's not recommended for http, which is what we're using. Would it be an issue to use leastconn for http? -Alex On Thu, Jan 15, 2015 at 9:41 PM, Vivek Malik vivek.ma...@gmail.com wrote: I see roubdrobin working perfectly over here. Look at sessions total and see how they are same for every server. It seems that all your requests are not the same workload. Some servers or some requests are taking longer to fulfill and increasing load on servers. Have you tried using leastconn instead of round robin? That might give more fair distribution of load in this situation. Regards, Vivek On Jan 14, 2015 11:45 PM, Alexey Zilber alexeyzil...@gmail.com wrote: Hi All, We got hit with a bit of traffic and we saw haproxy dump most of the traffic to 3-4 app servers, sometimes even just one and driving load on there to 90. We were running 1.5.9, I upgraded to 1.5.10 and the same problem remained. Currently traffic is low so everything is load balanced evenly, but we expect another spike in a few hours and I expect the issue to return. Here's what haproxy-status looked like: Do I need to switch to maybe add a weight and tracking? We have 12 frontend appservers load balancing to 28. All run haproxy and the app server software. Thanks! Alex -- Mariusz Gronczewski, Administrator Efigence S. A. ul. Wołoska 9a, 02-583 Warszawa T: [+48] 22 380 13 13 F: [+48] 22 380 13 14 E: mariusz.gronczew...@efigence.com mailto:mariusz.gronczew...@efigence.com pgpmG2cNzUliT.pgp Description: OpenPGP digital signature
RE: No TCP RST on tcp-request connection reject
Hi! just a thought... wouldn't it make sense to add an option to tcp-request connection reject to disable the actual TCP RST? I don't see how. The socket is immediately close()'ed when it hits tcp-request connection reject, this is as cheap as it gets. So, an attacker tries to (keep) open a lot of ports: There are no open ports, tcp-request connection reject closes sockets. a) HAProxy (configured with rate limiting etc.) does a tcp-request connection reject which ends up as a TCP RST. The attacker gets the RST and immediately again Are you saying that an attacker retransmits faster because of the RST? Thats nonsense, an attacker doesn't care about the RST at all. b) the same as a) but the socket will be closed on the server side but no RST, nothing will be sent back to the remote side. The connections on the remote side will be kept open until timeout. An attacker doesn't keeps states on his local machine if his intention is to SYN flood you. When we close the connection on our local side but don't notify the remote side it will probably exhaust the attacker No, really, its not. Haproxy cannot decide whether the TCP stack sends a RST after a close() or not, this is the kernels job. Lukas
Re: tproxy bug in haproxy-1.5.10
Lukas Tribus wrote, On 01/15/2015 09:06 PM: Imagine the 192.168.100.100 is a public IP (for example 1.2.3.4), and the others are private IPs as they indeed are (192.168.*). The reason for me is to use internally (ie. between the proxy server and the backend server) only the private IPs. The remote IP that your backend sees will be the client ip anyway, not your internal IP, so whats the point? True, but that's not the issue here. As said the idea and the setup works, but haproxy has a problem with it, and therefore there is a bug in haproxy. Yet you did not explain what the actual bug is. You keep saying there is a bug, but you don't explain it. I think I already explained it completely. If you still have a question on this specific setup scenario let me know.
Server persistence with cookies - a question
Let's say that I have a server named 'friday' in my backend, and it has cookie friday on the server line. Let's also say that the web server sets the JSESSIONID with that server name in it, like this: JSESSIONID=6ED8B9D4544820B2C073579E17BF3A67.friday If I use cookie JSESSIONID in the backend config, will that see the cookie value in the cookie set by the server and send additional requests to that specific server? If that will work right, I was thinking of using cookie JSESSIONID indirect preserve ... would that be a reasonable config? I'm having a hard time figuring out from the documentation exactly what happens when the various options are combined. Thanks, Shawn
connection is rejected when using ipad with send-proxy option
We enable send-proxy for ssl connections, and have the patched apache module to deal with proxyprotocol. From Mac OS, we see it works as designed. But when we repeat the same test using ipad, then we the connection rejected. iPad cannot establish the connection to haproxy over ssl. What should we debug further? Alex
RE: tproxy bug in haproxy-1.5.10
Imagine the 192.168.100.100 is a public IP (for example 1.2.3.4), and the others are private IPs as they indeed are (192.168.*). The reason for me is to use internally (ie. between the proxy server and the backend server) only the private IPs. The remote IP that your backend sees will be the client ip anyway, not your internal IP, so whats the point? As said the idea and the setup works, but haproxy has a problem with it, and therefore there is a bug in haproxy. Yet you did not explain what the actual bug is. You keep saying there is a bug, but you don't explain it.
RE: tproxy bug in haproxy-1.5.10
As said in the inital posting, the IP of the proxy server is 192.168.100.100 and public port 1234; it needs to be forwarded to the 2nd IP 192.168.100.101 port 5678, and from there to the backend server 192.168.100.102:. The key question is: what is the reason you don't bind to 192.168.100.100:1234?
Re: tproxy bug in haproxy-1.5.10
Lukas Tribus wrote, On 01/15/2015 08:31 PM: As said in the inital posting, the IP of the proxy server is 192.168.100.100 and public port 1234; it needs to be forwarded to the 2nd IP 192.168.100.101 port 5678, and from there to the backend server 192.168.100.102:. The key question is: what is the reason you don't bind to 192.168.100.100:1234? Imagine the 192.168.100.100 is a public IP (for example 1.2.3.4), and the others are private IPs as they indeed are (192.168.*). The reason for me is to use internally (ie. between the proxy server and the backend server) only the private IPs. As said the idea and the setup works, but haproxy has a problem with it, and therefore there is a bug in haproxy. I unfortunately lost much time for finding the reason for this mysterious behavior of haproxy, so I just wanted to inform you of the bug. The bug is localized as having to do with the TPROXY forwarding even before haproxy gets the packet. As said, with an other transparent proxy (test-) program this scenario works fine.
Re: Server persistence with cookies - a question
On 1/15/2015 1:30 PM, Shawn Heisey wrote: Let's say that I have a server named 'friday' in my backend, and it has cookie friday on the server line. Let's also say that the web server sets the JSESSIONID with that server name in it, like this: JSESSIONID=6ED8B9D4544820B2C073579E17BF3A67.friday If I use cookie JSESSIONID in the backend config, will that see the cookie value in the cookie set by the server and send additional requests to that specific server? If that will work right, I was thinking of using cookie JSESSIONID indirect preserve ... would that be a reasonable config? I'm having a hard time figuring out from the documentation exactly what happens when the various options are combined. From what I can see in the log when I tried this configs on a staging backend that only has one server, having .friday at the end of the cookie does NOT work -- the third flag character is I which the docs say means that an invalid cookie was found. Having a config option that would allow this to work would be really nice. With just cookie JSESSIONID all the flags are . With cookie JSESSIONID prefix I get the expected --VN flags -- but it is haproxy that decides which server to use, not the actual server. I'm using a completely different cookie (one that is not present in the response from the application) for the production backend, and that seems to be working really well: cookie SRV insert indirect postonly nocache The only problem I can foresee with that config is that haproxy's persistence with the SRV cookie might send the request to friday, but the JSESSIONID would cause the Apache cluster to then forward the request to tomcat running on tanstaafl instead, and I'd like to avoid that. If I could have haproxy learn what server needs to see the request from the JSESSIONID cookie that tomcat sets (all servers run both Apache and tomcat), inter-server traffic from user requests would be kept to a minimum. If haproxy already has a set of config options I can set that will have it look for the server cookie value at the end of JSESSIONID, please let me know what those options are. Thanks, Shawn
Re: Round Robin not very random
Hi Vivek, You're correct. I think the situation was that there was a huge influx of traffic, and some servers went over their tipping point of how much they can handle quickly. This caused connections to stack up as some servers choked. Would leastconn give the same perfornance as roundrobin? I noticed in the haproxy docs that it's not recommended for http, which is what we're using. Would it be an issue to use leastconn for http? -Alex On Thu, Jan 15, 2015 at 9:41 PM, Vivek Malik vivek.ma...@gmail.com wrote: I see roubdrobin working perfectly over here. Look at sessions total and see how they are same for every server. It seems that all your requests are not the same workload. Some servers or some requests are taking longer to fulfill and increasing load on servers. Have you tried using leastconn instead of round robin? That might give more fair distribution of load in this situation. Regards, Vivek On Jan 14, 2015 11:45 PM, Alexey Zilber alexeyzil...@gmail.com wrote: Hi All, We got hit with a bit of traffic and we saw haproxy dump most of the traffic to 3-4 app servers, sometimes even just one and driving load on there to 90. We were running 1.5.9, I upgraded to 1.5.10 and the same problem remained. Currently traffic is low so everything is load balanced evenly, but we expect another spike in a few hours and I expect the issue to return. Here's what haproxy-status looked like: Do I need to switch to maybe add a weight and tracking? We have 12 frontend appservers load balancing to 28. All run haproxy and the app server software. Thanks! Alex
Re: Round Robin not very random
I see roubdrobin working perfectly over here. Look at sessions total and see how they are same for every server. It seems that all your requests are not the same workload. Some servers or some requests are taking longer to fulfill and increasing load on servers. Have you tried using leastconn instead of round robin? That might give more fair distribution of load in this situation. Regards, Vivek On Jan 14, 2015 11:45 PM, Alexey Zilber alexeyzil...@gmail.com wrote: Hi All, We got hit with a bit of traffic and we saw haproxy dump most of the traffic to 3-4 app servers, sometimes even just one and driving load on there to 90. We were running 1.5.9, I upgraded to 1.5.10 and the same problem remained. Currently traffic is low so everything is load balanced evenly, but we expect another spike in a few hours and I expect the issue to return. Here's what haproxy-status looked like: Do I need to switch to maybe add a weight and tracking? We have 12 frontend appservers load balancing to 28. All run haproxy and the app server software. Thanks! Alex
Re: rate-limiting by checking both src-ipaddress path-requested at the same time
Hi, I have no words to thank you. I hope to show a example config (relevant lines) for example below. stick mysrcandpathcombo stick-table type binary 8 size 10m expire 3m store gpc0_rate stick on base32+src I still don't understand what to use in a acl. Regards, Long Wu Yuan 龙 武 缘 Sr. Linux Engineer 高级工程师 ChinaNetCloud 云络网络科技(上海)有限公司 | www.ChinaNetCloud.com1238 Xietu Lu, X2 Space 1-601, Shanghai, China | 中国上海市徐汇区斜土路1238号X2空 间1-601室 24x7 Support Hotline: +86-400-618-0024 | Office Tel: +86-(21)-6422-1946 We are hiring! http://careers.chinanetcloud.com | Customer Portal - https://customer-portal.service.chinanetcloud.com/ On Thu, Jan 15, 2015 at 9:01 PM, Willy Tarreau w...@1wt.eu wrote: Hi Yuan, On Wed, Jan 14, 2015 at 08:00:36PM +0800, Yuan Long wrote: Hi Willy, You answered a question, in record time, a short while ago, for a colleague, related to the MSIE encoding workaround option accept-invalid-http-request. No pb. Next time you'd better post the question on the mailing list, because it's much more responsive. You were lucky that I saw the e-mail in fact, I miss most of them. Inspired me to sort out a question, badly need confirmation on. I posted to mailing list last week as well. OK. Can we rate-limit by checking both the src-ipaddress the path-requested at the same time. Yes you can. It's a bit tricky but there are two methods. The first one consists in using the base32+src fetch. This one hashes the URL to a 32-bit value, then concatenates the IP address to form a 64-bit value that can be sticked on. It's quite efficient and due to the very low collision rate, it's enough to consider that it corresponds to src+path. The downside is that when dumping the table's entries from the CLI, you cannot tell what URL corresponds to the key you're seeing. The second method consists in building a new header from the concatenation of the src and the path, then track this header. It requires version 1.6 or the commercial version HAPEE which includes this backport. It allows you to keep a string in memory which is more handy to see the related URLs. The downside is that it consumes much more memory since you store a complete string and not just 32-bit. So for a rate limiting (eg: against attacks), I'd still recommend using only base32+src. It was made exactly for this and is efficient enough. Hoping this helps, Willy
