More info on the over-quick+newly-noisy resolves that were triggering this ...
We've been running 1.8.19 (https://packages.debian.org/stretch-backports/haproxy) with 'hold valid 60s' configured, which was acting-ish like 'timeout resolve 60s' (which was *not* configured). So when we moved to current 2.0 , with the fix for /issues/345 , resolutions which had been happening every 60s now happened every 1s (the default?), with each IP change now making noise it had not made before => ergo, logs/disk filled. Adding 'timeout resolve 60s' reduces the noise by a factor of 60. The duplication of logging the new(?) 'changed its IP' messages to daemon.log (when only local* facilities are configured) is still getting root-cause analysis. === https://github.com/haproxy/haproxy/issues/345 https://github.com/haproxy/haproxy/commit/f50e1ac4442be41ed8b9b7372310d1d068b85b33 http://www.haproxy.org/download/1.8/src/CHANGELOG * 2019/11/25 : 1.8.23 * BUG: dns: timeout resolve not applied for valid resolutions On Thu, Apr 15, 2021 at 1:43 AM Jim Freeman <[email protected]> wrote: > > Migrating from stock stretch-backports+1.8 to Debian_10/Buster+2.0 (to > purge 'reqrep' en route to 2.2), /var/log/ is suddenly filling disk > with messages about changed IPs : > === > 2021-04-14T01:08:40.123303+00:00 ip-10-36-217-169 haproxy[569]: > my_web_service changed its IP from 52.208.198.117 to 34.251.174.55 by > DNS cache. > === > daemon.log and syslog (plus the usual haproxy.log) all get hammered. > > Many of the backends (200+) are of the flavor : > server-template my_web_service 8 my_web_service.herokuapp.com:443 ... > resolvers fs_resolvers resolve-prefer ipv4 > > The herokuapp.com addresses change constantly, but this has not been a > problem heretofore. > > This is puzzling, since haproxy.cfg directs all logs to local* > After some investigation, it turns out that the daemon.log and syslog > entries arrive via facility.level=daemon.info. I've made rsyslog cfg > changes that now stop the haproxy msgs from overrunning daemon.log and > syslog (and allow only a representative fraction to hit haproxy.log). > > Two questions : > 1) What is different about 2.0 that "changed its IP" entries are so > voluminous ? > 2) Why is daemon.info involved in the logging, when the haproxy.cfg > settings only designate local* facilities ? > > Thanks for any insights (and for stupendous software) ! > ==== > Running HAProxy 2.0 from : > https://haproxy.debian.net/#?distribution=Debian&release=buster&version=2.0 > > on stock Buster AWS AMI : > https://wiki.debian.org/Cloud/AmazonEC2Image > https://wiki.debian.org/Cloud/AmazonEC2Image
