Re: Haproxy custom log - apache replacement
Hello ! thanks for your answer, I will have a closer look about this configuration. I have seen it in the documentation before but never tried it Regards, Benoit
FOB HK FOB sz
大陆至深圳港口运输,报关,中港线 联系 邮 箱:nanfeng...@163.com 一、大陆至香港散货 免费上门提货 0.68元/公斤; 泡货120元/立方; 近期经常外出,有订单当天收货的请给我打电话。 祝你工作顺利、开心快乐! 东莞市南丰物流(香港)有限公司 联 系 人:13922925659/ {Mobile}13662814456 咨询接单QQ:654340993 联系 邮 箱:nanfeng...@163.com运 输 地址:东莞东城牛山景观路1-3栋(市委党校对面)>服务只有起点,满意满意终点<*为了保证你能继续收到我们的优质信息,请将我们加入你的联系人列表。如本类邮件打扰您的工作,请备注邮箱账号回复到本邮箱,我司将不再发送。谢
Re: Haproxy custom log - apache replacement
Hi. Am 08-09-2016 21:16, schrieb Benoit GEORGELIN - Association Web4all: Hello members of the list, I was wondering if Haproxy custom log could log the same things as apache log format does ? Well due to the fact that we don't know your apache log format I assume you mean CLF (common log format) ;-). http://httpd.apache.org/docs/2.4/mod/mod_log_config.html I also not know which version of haproxy you use so I refer to the latest stable one. There is a predefined crf format in haproxy which is documented here http://cbonte.github.io/haproxy-dconv/1.6/configuration.html#8.2 http://cbonte.github.io/haproxy-dconv/1.6/configuration.html#8.2.4 Haproxy is on the top for HTTP servers farms and I would like to stop logging on the HTTP servers side (apache or ngnix) and consume the log made by Haproxy with everything I get from a typical Apache log I think, at that time, Haproxy does not log everything like apache will do, but maybe it's something than can be changed ? HAProxy is able to log much more the apache ;-). You will need to add some lines like these to get the additional information from the request. ## # log the beginning of the referrer capture request header Referer len 20 capture request header User-agent len 20 ## As described here. http://cbonte.github.io/haproxy-dconv/1.6/configuration.html#8.8 That would be nice and will save a lot of time/ressources In my case, this is what i'm doing : - Haproxy Loadblance to 5 HTTP servers Apache2 custom log with X-Forwarder-for - Apache2 Each HTTP serveur runs apache2 and log http requests locally to access_domain_log Every night, merging of the log and then process the full log with Http log analyser like Awstats Before you can feed Awstats with the new logs you will need to pre-process it. Maybe the halog tool ( http://git.haproxy.org/?p=haproxy-1.6.git;a=tree;f=contrib/halog;h=189001c583ddcf3e35060171a58c39fd5115b9c2;hb=HEAD ) can help. There is also a doc for this tool. https://www.haproxy.com/static/media/uploads/eng/resources/appnotes_0054_analyze_haproxy_logs_with_halog_en.pdf Store the logs to a dedicated storage for archiving If I could get the same HTTP information directly from Haproxy log , this will same us a lot of time an ressource usage. Haproxy logs -> log shipper -> elasticsearch /hadoop -> kibana or something else to display log information Thanks for your help Cordialement, Benoît G Best regards Aleks
[PATCH] New DNS parser
Hi all, Please find in attachment 10 patches to cover the following new topic in HAProxy: 1. a new DNS parser, which stores the DNS response into a DNS structure, instead of manipulating a buffer. => it doesn't add any feature by itself, but it will make DNS consumer tasks much easier when using DNS responses 2. when the DNS response finishes with a CNAME, now HAProxy sends a new query, changing the query type (from to A or A to ) I heavily tested the code, but I'd like more people to test it in their own environment. We can now move forward on the next big development: filling servers in a backend based on records read in a DNS responses. Conrad: I have a quick and dirty and not finished patch to read and store SRV records. If you want to use it for your own dev, please let me know. Baptiste From 2d196c70952be351508e3ee154d6c57d5cefee2e Mon Sep 17 00:00:00 2001 From: Baptiste Assmann Date: Mon, 18 Apr 2016 19:42:57 +0200 Subject: [PATCH 01/11] CLEANUP/MINOR dns: comment do not follow up code update The loop comment is not appropriate anymore and needed to be updated according to the code. backport: no --- src/dns.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/dns.c b/src/dns.c index c76637f..b9dce6b 100644 --- a/src/dns.c +++ b/src/dns.c @@ -725,7 +725,7 @@ int dns_get_ip_from_response(unsigned char *resp, unsigned char *resp_end, /* move forward data_len for analyzing next record in the response */ reader += data_len; } /* switch (record type) */ - } /* for i 0 to ancount */ + } /* list for each record entries */ /* Select an IP regarding configuration preference. * Top priority is the prefered network ip version, -- 1.9.1 From 83e6c3f60ade30a175b40b17f312fbf1e5b5aae2 Mon Sep 17 00:00:00 2001 From: Baptiste Assmann Date: Sat, 26 Mar 2016 14:12:50 +0100 Subject: [PATCH 02/11] MINOR: chunk: new strncat function Purpose of this function is to append data to the end of a chunk when we know only the pointer to the beginning of the string and the string length. --- include/common/chunk.h | 13 + 1 file changed, 13 insertions(+) diff --git a/include/common/chunk.h b/include/common/chunk.h index aac5282..205523c 100644 --- a/include/common/chunk.h +++ b/include/common/chunk.h @@ -120,6 +120,19 @@ static inline int chunk_strcat(struct chunk *chk, const char *str) return 1; } +/* appends characters from str after . + * Returns 0 in case of failure. + */ +static inline int chunk_strncat(struct chunk *chk, const char *str, int nb) +{ + if (unlikely(chk->len < 0 || chk->len + nb >= chk->size)) + return 0; + + memcpy(chk->str + chk->len, str, nb); + chk->len += nb; + return 1; +} + /* Adds a trailing zero to the current chunk and returns the pointer to the * following part. The purpose is to be able to use a chunk as a series of * short independant strings with chunk_* functions, which do not need to be -- 1.9.1 From 1a8a094fcdc40bc10200ed4d036a5e553904b4f3 Mon Sep 17 00:00:00 2001 From: Baptiste Assmann Date: Sat, 26 Mar 2016 15:09:48 +0100 Subject: [PATCH 03/11] MINOR: dns: wrong DNS_MAX_UDP_MESSAGE value Current implementation of HAProxy's DNS resolution expect only 512 bytes of data in the response. Update DNS_MAX_UDP_MESSAGE to match this. Backport: can be backported to 1.6 --- include/types/dns.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/types/dns.h b/include/types/dns.h index 1b240fa..19c6edc 100644 --- a/include/types/dns.h +++ b/include/types/dns.h @@ -30,7 +30,7 @@ */ #define DNS_MAX_LABEL_SIZE 63 #define DNS_MAX_NAME_SIZE 255 -#define DNS_MAX_UDP_MESSAGE 4096 +#define DNS_MAX_UDP_MESSAGE 512 /* DNS error messages */ #define DNS_TOO_LONG_FQDN "hostname too long" -- 1.9.1 From adac8d19d888cf63315290f71ccfe0dcfd05b80d Mon Sep 17 00:00:00 2001 From: Baptiste Assmann Date: Wed, 9 Dec 2015 14:02:01 +0100 Subject: [PATCH 04/11] MINOR: dns: new MAX values DNS_MIN_RECORD_SIZE: minimal size of a DNS record DNS_MAX_QUERY_RECORDS: maximum number of query records we allow. For now, we send one DNS query per request. DNS_MAX_ANSWER_RECORDS: maximum number of records we may found in a response WIP dns: new MAX values --- include/types/dns.h | 10 ++ 1 file changed, 10 insertions(+) diff --git a/include/types/dns.h b/include/types/dns.h index 19c6edc..01d42f9 100644 --- a/include/types/dns.h +++ b/include/types/dns.h @@ -32,6 +32,16 @@ #define DNS_MAX_NAME_SIZE 255 #define DNS_MAX_UDP_MESSAGE 512 +/* DNS minimun record size: 1 char + 1 NULL + type + class */ +#define DNS_MIN_RECORD_SIZE ( 1 + 1 + 2 + 2 ) + +/* maximum number of query records in a DNS response + * For now, we allow only one */ +#define DNS_MAX_QUERY_RECORDS 1 + +/* maximum number of answer record in a DNS response */ +#define DNS_MAX_ANSWER_RECORDS ((DNS_MAX_UDP_MESSAGE - DNS_HEADER_SIZE) / DNS_MIN_RECORD_SIZE) + /* DNS error messages */ #define DNS_TOO_LONG_FQDN "hostname too long" #def
Haproxy custom log - apache replacement
Hello members of the list, I was wondering if Haproxy custom log could log the same things as apache log format does ? Haproxy is on the top for HTTP servers farms and I would like to stop logging on the HTTP servers side (apache or ngnix) and consume the log made by Haproxy with everything I get from a typical Apache log I think, at that time, Haproxy does not log everything like apache will do, but maybe it's something than can be changed ? That would be nice and will save a lot of time/ressources In my case, this is what i'm doing : - Haproxy Loadblance to 5 HTTP servers Apache2 custom log with X-Forwarder-for - Apache2 Each HTTP serveur runs apache2 and log http requests locally to access_domain_log Every night, merging of the log and then process the full log with Http log analyser like Awstats Store the logs to a dedicated storage for archiving If I could get the same HTTP information directly from Haproxy log , this will same us a lot of time an ressource usage. Haproxy logs -> log shipper -> elasticsearch /hadoop -> kibana or something else to display log information Thanks for your help Cordialement, Benoît G
Re: Bytes in / out counters for TCP Keepalive Sessions
Adding the list back. Sorry for dropping it earlier. > On 8 Sep 2016, at 19:56, PiBa-NL wrote: > > Hi, > Op 8-9-2016 om 15:43 schreef Daniel Schneller: >>> http://cbonte.github.io/haproxy-dconv/1.7/snapshot/configuration.html#4.2-option%20contstats >> Indeed, that sounds like it. So, 1.6 would not have helped me here ;) >> But good to know that this is the expected behavior. > Just for clarity.. despite my link pointing to a 1.7 manual page, 1.4 already > have that same contstat option available for you to use. > http://cbonte.github.io/haproxy-dconv/1.4/configuration.html#option contstats > Regards, > PiBa-NL Damn. Thanks for pointing that out again, I did not even think to search for it in older doc releases! Very cool. :) Daniel
Re: Haproxy 1.6.9 failed to compile regex
Hello Veiko, Am 08.09.2016 um 13:11 schrieb Veiko Kukk: Yes, turned out, build box had newer pcre installed (7 vs 8 major version). Compiling pcre statically solved that error/problem. Good to hear. If you link against a shared library, you will have to make sure they match in major versions (same OS), otherwise you will have major problems with it (OpenSSL major release differences also lead to haproxy crashes). If you link statically to a library, you have to remember to upgrade the library and recompile haproxy, if the library has a vulnerability. Means that haproxy -vv reported "Built with PCRE version" version wrong previously. That confused me. This returns the output of pcre_version() and yes, the text should be renamed to "Running with PCRE version", because the runtime library returns the version in this case. BTW: openssl version is also reported wrong by haproxy -vv. OpenSSL informations on the other hand should be correct. "Built with" returns the build time OPENSSL_VERSION_TEXT and "Running on" makes the library return the version ("SSLeay_version(SSLEAY_VERSION)"). There is really no interpretation going on at haproxy level. Not sure why you would see different "Running on" informations on the same box, pointing to the same libssl and libcrypto library in the ldd output. That doesn't make sense to me. Lukas
Re: Haproxy 1.6.9 failed to compile regex
On 07/09/16 18:33, Lukas Tribus wrote: Do you compile on the same box were the executable runs? No. Looks like you have a mess with your system libraries. Yes, turned out, build box had newer pcre installed (7 vs 8 major version). Compiling pcre statically solved that error/problem. Means that haproxy -vv reported "Built with PCRE version" version wrong previously. That confused me. Compare "ldd haproxy" (working and non-working executable). You will probably see that it points to a different path (both pcre and openssl). No, both pointed to proper installed libraries. I checked it before writing to list. BTW: openssl version is also reported wrong by haproxy -vv. Veiko
Re: Incorrect counters in stats interface
> On 07 Sep 2016, at 23:12, David Birdsong wrote: > > Queue Cur is a gauge and so not representative of historical values. > > Queue Max of zero is telling though. > > In addition to queue timeout, there are other ways haproxy can synthesize an > http response on behalf of the backend server. Check for connection errors. > Also, awk '{if ($11 >= 500) print};' gives me no matches from haproxy.log, but counter for backend's Total Sessions 5xx respones does increase.