On 9 Jul 2017 12:20 pm, "Norman Branitsky" <[email protected]> wrote:
Thanks for the responses. shouldbe q931, in a private email to me, and Baptiste, both suggested I not use DNS. Baptiste suggested VRRP and shouldbe q931 suggested something similar using keepalived. I replied to shouldbe q931 thus: As far as I know, keepalived requires a separate network interface connecting the 2 servers to manage the heartbeat connection. In my case, the 2 HAProxy servers are in different Amazon AWS Availability Zones (Data Centers) with different network subnets. I don’t think I can make keepalived work in this configuration. Of course it can work see https://icicimov.github.io/blog/high-availability/Keepalived-in-Amazon-VPC-across-availability-zones/ Igor, you suggested 30 seconds was too long for a health check failover. Unfortunately, that is the minimum setting that NS1 supports. If your servers are in aws why are you not using Route53 then? It allows for low dns ttl even lower then 10 sec. It has many advanced options of load balancing and health checking for sure it is superior compared to NS1. The reason the NS1 Filter only returns the first healthy HAProxy is that this configuration has been in place for quite some time while my HAProxy servers were running version 1.5. I’m in the process of upgrading all my HAProxy servers to version 1.7.8 with a peers section defined. If your app does not need sticky sessions you dont need peers setup and you can start using both haproxies right away. You can also set haproxy to insert its own cookies and mantain the stickinnes in that way. I've been using it that way with active-active haproxy servers for ages in AWS with Route53 dns health checks (min hc interval is 10 seconds though and additonal costs apply) for apps that need sticky sessions. So I believe I can now safely change the NS1 Filter to round-robin mode. You’re suggestion to reduce the JVM ttl value sounds interesting. I’m guessing you think this will force IE to refresh its DNS cache. Something interesting appeared in my testing. With haproxy1 and haproxy2 running, I connected to my app using IE. I then shutdown haproxy1. After 30 seconds, NS1 performed a DNS failover to haproxy2 and the IE client complained about no connection. (In a 2nd tab it connects immediately as usual.) After about 20 minutes the first tab reconnected. I then restarted haproxy1. After 30 seconds, NS1 performed a DNS switch back to haproxy1. Now the IE client continued to operate correctly! It’s as if it had seen haproxy1 before so it didn’t complain on the switch back?!? Just confirming that returning multiple records to the clients might fix ie10 issue. According to this https://blogs.msdn.microsoft.com/ieinternals/2012/09/26/braindump-dns/ ie10 will cache up to 256 records up to 30min and wil not respect ttl. You need to change this setting in the windows registry which you can't expect your costumers to do. *From:* Igor Cicimov [mailto:[email protected]] *Sent:* July-08-17 9:14 AM *To:* Norman Branitsky <[email protected]> *Cc:* HAProxy <[email protected]> *Subject:* RE: HAProxy failover - DNS change cached by IE for a long time On 8 Jul 2017 2:58 am, "Norman Branitsky" <[email protected]> wrote: I changed the TTL on my application’s DNS entry, to no avail. Try tuning these parameters in jvm, assuming Sun oracle jdk here: -Dsun.net.inetaddr.ttl=value -Dsun.net.inetaddr.negative.ttl=value If security manager is installled System wide, by adding a line containing networkaddress.cache.ttl=value in $JAVA_HOME/jre/lib/security/java.security JDK 1.6, 1.7 & 1.8 default cache setting: 30 secs (When a security manager is not set) -1 (When a security manager is set) * DNS Cache is refreshed every 30 seconds So adjust the value to some low value of 10 sec say. Once the DNS entry updates to point to the 2nd HAProxy server, IE displays it’s dnserror.htm page: “This page can’t be displayed”. Copy/Paste the URL into a new tab and the page renders immediately. The original tab continues to display the dnserror page – probably for 20 minutes. *From:* Norman Branitsky [mailto:[email protected]] *Sent:* June-27-17 10:44 AM *To:* [email protected] *Subject:* HAProxy failover - DNS change cached by IE for a long time This sender failed our fraud detection checks and may not be who they appear to be. Learn about spoofing <http://aka.ms/LearnAboutSpoofing> Feedback <http://aka.ms/SafetyTipsFeedback> Using the NS1 managed DNS service, I monitor the health of 2 HAProxy 1.7.7 servers defined as peers. NS1 checks the health of the HAProxy servers every 30 seconds. If haproxy1 fails to respond, NS1 changes the DNS response to point to haproxy2. When haproxy1 comes back online, NS1 reverts the DNS response to haproxy1. NS1 checks the health of my Java application server every 60 seconds. NS1 DNS records looks like this: haproxy1 A record haproxy2 A record tm1 CNAME record “Dynamic” – NS1 “filter” returns the first in the list of all health haproxy servers vr CNAME record pointing to tm1 – name of the Java application server If I connect to my Java application with Chrome or Firefox, I often don’t notice the haproxy DNS failover. If I do get a connection error, it almost always reconnects within seconds. I don’t lose my session. If I connect to my Java application with IE (only tested IE10 mode so far), the haproxy DNS failover cause a DNS error. This error won’t clear for at least 20 minutes. If I open a new tab I connect instantly. Since the JSESSIONID cookie is still available, I’m still logged in but obviously not on the same data entry page. What can I do to kick IE in the head and cause it to refresh its DNS cache? It doesn’t seem to respect the TTL value.
