Re: Segfault on basic auth?
Ahh, yes okay I saw the thread but didn't imagine it was related. I'll be able to test the newest tomorrow, and I'll close the loop here. Thanks for all you do, Phillip On Wed, Sep 23, 2015 at 3:40 AM, Willy Tarreau <w...@1wt.eu> wrote: > Hi Phillip, > > On Wed, Sep 23, 2015 at 12:59:05PM +0930, Phillip Decker wrote: > > Hi guys, > > > > Not to pile on, but when running with 1.6-dev4, I've noticed a segfault > > that doesn't happen if you substitute 1.5.9 back in. I can post more > here > > later in the day, but I have a meeting to run to shortly- > > (...) > > I introduced a bug with the http-reuse which causes statuses 401 and 407 > on http-server-close connections to dereference a NULL. It was fixed in > dev5. I'm pretty sure that's what you're seeing. Just update do dev5 or > better, latest snapshot, to confirm you don't see it anymore. > > thanks, > Willy > >
Segfault on basic auth?
Hi guys, Not to pile on, but when running with 1.6-dev4, I've noticed a segfault that doesn't happen if you substitute 1.5.9 back in. I can post more here later in the day, but I have a meeting to run to shortly- In short, here are the conditions: - haproxy as an SSL enpoint in front of Nagios , which is itself expecting https. - haproxy listening to browsers on the regular 443, with "ssl verify required" - haproxy connecting to Nagios with "ssl verify none" Everything starts normally, and the main nagios page opens normally with HTTPS enabled. When you click a link to hosts or any of the links for server statuses, it pops up what looks like a basic authentication on the browser, and haproxy 1.6-dev4 segfaults. Changing nothing about the setup except the symlink to haproxy to 1.5.9, everything works as it should. Current options 1.6 d4 was built with: TARGET=linux2628 USE_ZLIB=1 USE_OPENSSL=1, USE_PCRE=1. Is this happening to anyone else? If not, I'll try and put together a test case I can post up here. Thanks! Phillip
Next dev version?
Hi guys, I know we're all busy, but I just had a quick question - do we have a ballpark idea when the next dev tag will be set? (ie. v1.6dev4 ?) Or even further, when (tentatively) the code freeze for 1.6 will happen? Thanks! Phillip
Re: [PATCH] Add log-format variable %HQ, to log HTTP query strings
Andrew-
This patch works well for my use case. Leaving the question mark in
mirrors the behavior of %q in httpd, fwiw.
Should it also print a hyphen in case the field is empty and the quotation
mode is not on? %HQ instead of %{+Q}HQ?
Phillip
On Fri, Jul 31, 2015 at 12:21 PM, Andrew Hayworth
andrew.haywo...@getbraintree.com wrote:
Since this came up in another thread, it seems reasonable to add a
patch that implements %HQ as a log-format variable to record the HTTP
query string. Leaving the initial '?' is intentional, but I don't feel
strongly one way or another.
--
- Andrew Hayworth
From b87770d5e513fc923d0d94d2b1d0de00d88acb98 Mon Sep 17 00:00:00 2001
From: Andrew Hayworth andrew.haywo...@getbraintree.com
Date: Fri, 31 Jul 2015 16:14:16 +
Subject: [PATCH 1/1] Add log-format variable %HQ, to log HTTP query strings
Since sample fetches are not always available in the response phase,
this patch implements %HQ such that:
GET /foo?bar=baz HTTP/1.0
...would be logged as:
?bar=baz
---
doc/configuration.txt | 1 +
include/types/log.h | 1 +
src/log.c | 38 ++
3 files changed, 40 insertions(+)
diff --git a/doc/configuration.txt b/doc/configuration.txt
index db97cc7..b3ba8a0 100644
--- a/doc/configuration.txt
+++ b/doc/configuration.txt
@@ -13987,6 +13987,7 @@ Please refer to the table below for currently
defined variables :
| | %H | hostname | string
|
| H | %HM | HTTP method (ex: POST)| string
|
| H | %HP | HTTP request URI without query string (path) | string
|
+ | H | %HQ | HTTP request URI query string (ex: ?bar=baz) | string
|
| H | %HU | HTTP request URI (ex: /foo?bar=baz) | string
|
| H | %HV | HTTP version (ex: HTTP/1.0) | string
|
| | %ID | unique-id | string
|
diff --git a/include/types/log.h b/include/types/log.h
index bbfe020..d0fb966 100644
--- a/include/types/log.h
+++ b/include/types/log.h
@@ -96,6 +96,7 @@ enum {
LOG_FMT_HTTP_METHOD,
LOG_FMT_HTTP_URI,
LOG_FMT_HTTP_PATH,
+ LOG_FMT_HTTP_QUERY,
LOG_FMT_HTTP_VERSION,
LOG_FMT_HOSTNAME,
LOG_FMT_UNIQUEID,
diff --git a/src/log.c b/src/log.c
index ffd8f10..1112f8a 100644
--- a/src/log.c
+++ b/src/log.c
@@ -111,6 +111,7 @@ static const struct logformat_type
logformat_keywords[] = {
{ hsl, LOG_FMT_HDRRESPONSLIST, PR_MODE_TCP, LW_RSPHDR, NULL }, /*
header response list */
{ HM, LOG_FMT_HTTP_METHOD, PR_MODE_HTTP, LW_REQ, NULL }, /* HTTP
method */
{ HP, LOG_FMT_HTTP_PATH, PR_MODE_HTTP, LW_REQ, NULL }, /* HTTP path */
+ { HQ, LOG_FMT_HTTP_QUERY, PR_MODE_HTTP, LW_REQ, NULL }, /* HTTP query
*/
{ HU, LOG_FMT_HTTP_URI, PR_MODE_HTTP, LW_REQ, NULL }, /* HTTP full
URI */
{ HV, LOG_FMT_HTTP_VERSION, PR_MODE_HTTP, LW_REQ, NULL }, /* HTTP
version */
{ lc, LOG_FMT_LOGCNT, PR_MODE_TCP, LW_INIT, NULL }, /* log counter */
@@ -937,6 +938,7 @@ int build_logline(struct stream *s, char *dst,
size_t maxsize, struct list *list
struct chunk chunk;
char *uri;
char *spc;
+ char *qmark;
char *end;
struct tm tm;
int t_request;
@@ -1578,6 +1580,42 @@ int build_logline(struct stream *s, char *dst,
size_t maxsize, struct list *list
last_isspace = 0;
break;
+ case LOG_FMT_HTTP_QUERY: // %HQ
+ uri = txn-uri ? txn-uri : BADREQ;
+
+ if (tmp-options LOG_OPT_QUOTE)
+ LOGCHAR('');
+
+ end = uri + strlen(uri);
+ // look for the first question mark
+ while (uri end *uri != '?')
+ uri++;
+
+ qmark = uri;
+
+ // look for first space or question mark after url
+ while (uri end !HTTP_IS_SPHT(*uri))
+ uri++;
+
+ if (!txn-uri) {
+ chunk.str = BADREQ;
+ chunk.len = strlen(BADREQ);
+ } else {
+ chunk.str = qmark;
+ chunk.len = uri - qmark;
+ }
+
+ ret = encode_chunk(tmplog, dst + maxsize, '#', url_encode_map,
chunk);
+ if (ret == NULL || *ret != '\0')
+ goto out;
+
+ tmplog = ret;
+ if (tmp-options LOG_OPT_QUOTE)
+ LOGCHAR('');
+
+ last_isspace = 0;
+ break;
+
case LOG_FMT_HTTP_URI: // %HU
uri = txn-uri ? txn-uri : BADREQ;
--
2.1.3
Re: Capture sequencing in logs
One other log question in this same vein -
I'm trying to duplicate the functionality of the %q flag in Apache, and I
don't see a way in the documentation to print _only_ the query string, that
is, the information after the question mark in a URI. I see the URI
without the query (path), the full URI, and looking up specific parameters
in the URI... am I missing an obvious flag somewhere?
Thanks in advance for any assistance,
Phillip
On Mon, Jul 6, 2015 at 5:31 PM, Phillip Decker pdecker999+hapr...@gmail.com
wrote:
Nenad, thank you so much for your help; I was able to run that
successfully on a test instance, I think that may fix the majority of my
issue!
Thanks again,
Phillip
On Mon, Jul 6, 2015 at 4:55 PM, Nenad Merdanovic ni...@nimzo.info wrote:
Hello Phillip,
First, let me apologize for misspelling your name the first time.
On 7/6/2015 10:46 PM, Phillip Decker wrote:
Hi Nenad, thanks for the reply!
Okay, insofar as the first link - I can see where I can retrieve
captures for use in acls or elsewhere, but is there a way to use these
directly in the custom log format (
http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#8.2.4 )?
Being able to intersperse captures and the usual custom log format
entries on the same log line is my main problem.
Yes, I was excited to use the new declared captures, but it came down to
the same question as above; is there a way to use a declared capture in
a custom log format line? If I can, it solves my problems...
http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#8.2.4
As you can see here:
If a variable is named between square brackets ('[' .. ']') then it is
used as a sample expression rule (see section 7.3). This it useful to
add some less common information such as the client's SSL certificate's
DN, or to log the key that would be used to store an entry into a stick
table.
So essentially what you have to do is add something like
%[capture.req.hdr(0)] to your log-format string. And then just change
the 0 to whatever capture you want depending on the capture ordering.
Hope this helps.
Nenda
In log.c I see that %hr is doing a for loop on the session's front end
proxy struct's list of capture fields, but doesn't look like it can take
any arguments or anything for declared captures. Looking through the
diffs on that file back two years, I see a reference in April to moving
the captures from http_txn to stream, but a look at the diffs show that
doesn't change this issue.
I'm looking through
log.c branch 1.6 : void parse_logformat_string(const char *fmt, struct
proxy *curproxy, struct list *list_format, int options, int cap, const
char *file, int line)
for an indication of a way to define this, and I don't see it so far...
am I missing it?
Thanks again for any help!
Phillip
On Mon, Jul 6, 2015 at 4:10 PM, Nenad Merdanovic ni...@nimzo.info
mailto:ni...@nimzo.info wrote:
Hello Philip
On 7/6/2015 9:47 PM, Phillip Decker wrote:
Hello again all,
I have a configuration problem, and I have a workaround that I
don't
like, so I'm hoping someone here might have a better solution -
I have a number of capture fields such as ssl_c_s_dn, request
header
referrer, request header X-Forwarded-For, etc that get logged
using %hr,
among another 12 or so fields such as %t, %H, %B, %ci and so on.
The problem is the client has an established log format which
goes to an
enterprise centralized log db. I'd love to be able to write out
this
format directly with haproxy, but since all the captured fields in
haproxy get appended together by %hr and %hs, I don't know of a
way to
put one captured field in spot #4 of a log line, and then another
captured field in spot #7, etc.
In order to work around this, I have haproxy instead outputting
all this
data to an intermediate log format, and then have another
process
running which picks these up and rearranges them into the
arbitrary log
format that the enterprise accepts.
Well, it seems like you are looking to reorder the captures, so
take a
look at:
http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#7.3.6-capture.req.hdr
You can use this in your log format to specify a different ordering
than
the one how the captures are sorted in the config.
In 1.6 you could also use:
http://cbonte.github.io/haproxy-dconv/configuration-1.6.html#declare%20capture
Let me know if this helps.
Regards,
Nenad
Re: Capture sequencing in logs
And it only kinda works because when there is no question mark then the field will have the uri instead of being empty... On Thu, Jul 30, 2015 at 7:12 PM, Phillip Decker pdecker999+hapr...@gmail.com wrote: Funny, yeah I was just playing with it and couldn't get that to work, so I just did another git pull thinking maybe I just wasn't updated, then came back to my email and saw your second reply. Hrm. Well, something that seems to sorta work is this (in the log-format line): %[capture.req.uri,regsub(^.*\?,)] So, grabbing the full uri and then regex replace everything up to the '?' with nothing, but I don't know what kind of underlying impacts that approach might have, if any... Phillip On Thu, Jul 30, 2015 at 6:25 PM, Cyril Bonté cyril.bo...@free.fr wrote: On 31/07/2015 00:14, Cyril Bonté wrote: Hi Phillip, On 31/07/2015 00:05, Phillip Decker wrote: One other log question in this same vein - I'm trying to duplicate the functionality of the %q flag in Apache, and I don't see a way in the documentation to print _only_ the query string, that is, the information after the question mark in a URI. I see the URI without the query (path), the full URI, and looking up specific parameters in the URI... am I missing an obvious flag somewhere? This is only available in 1.6 development branch : http://cbonte.github.io/haproxy-dconv/configuration-1.6.html#query Oops, I replied too quickly, as such HTTP sample fetches are not available in log-format. Maybe we can discuss adding a %HQ (or %HQS) log variable in the future ? -- Cyril Bonté
Re: substring matching backend names...
Tested, and very nice; that works! Thanks so much! Phillip On Sat, Jul 11, 2015 at 2:20 AM, Willy Tarreau w...@1wt.eu wrote: Hi, On Fri, Jul 10, 2015 at 04:32:38PM -0400, Phillip Decker wrote: Hello again, I was migrating a setup from the older style acl-based host to backend mapping, to the newer map-based approach, e.g. use_backend %[req.hdr(host),lower,map_beg(mapping.conf,default)] and that works fine. But now I'm tempted to simply name the backends after each servername, (eg. 'mail' from mail.google.com, or 'maps' from maps.google.com). Which in my imagination would look something like: use_backend %[req.hdr_beg(host),lower] But is there a way to substring or regex off the front of the host string, and pass that to the use_backend call? In other words, continuing the google.com example, if people are hitting the server with host: mail.google.com, and maps.google.com, and just for completeness, notes.books.google.com, I'd like to have backends: backend mail mode http foo backend maps mode http bar ... backend notes.books mode http baz .. Is there a way to accomplish this without building them into a mapped file in 1.5.x? (1.6 solutions are okay too, but won't be able to move to that until it's in stable.) In 1.6 there's the word converter which stops at the first delimiter, so that if you use word(1,.) on notes.books.domain.com, it will return notes. From what I understand, in your specific case it does not perfectly respond to your needs but for many other cases that could be fine. Also it makes sense to use it to cut the hostname only and ignore the port. Otherwise there's the regsub converter, to which you pass a regex, a substitute string, and optionally some flags. There are some limitations such as the fact that the comma character cannot be used there due to a limitation in the config language which uses it as an argument delimiter, but for a domain it's not a problem. Thus a fairly complete solution to your need above could look like this : use_backend %[req.hdr(host),word(1,:),lower,regsub(\.domain\.com$,)] By replacing .domain.com at the end of the name with an empty string, it will then return maps, mail, notes.book in your example above. Regards, Willy
substring matching backend names...
Hello again, I was migrating a setup from the older style acl-based host to backend mapping, to the newer map-based approach, e.g. use_backend %[req.hdr(host),lower,map_beg(mapping.conf,default)] and that works fine. But now I'm tempted to simply name the backends after each servername, (eg. 'mail' from mail.google.com, or 'maps' from maps.google.com). Which in my imagination would look something like: use_backend %[req.hdr_beg(host),lower] But is there a way to substring or regex off the front of the host string, and pass that to the use_backend call? In other words, continuing the google.com example, if people are hitting the server with host: mail.google.com, and maps.google.com, and just for completeness, notes.books.google.com, I'd like to have backends: backend mail mode http foo backend maps mode http bar ... backend notes.books mode http baz .. Is there a way to accomplish this without building them into a mapped file in 1.5.x? (1.6 solutions are okay too, but won't be able to move to that until it's in stable.) Phillip
Re: Capture sequencing in logs
Hi Nenad, thanks for the reply! Okay, insofar as the first link - I can see where I can retrieve captures for use in acls or elsewhere, but is there a way to use these directly in the custom log format ( http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#8.2.4 )? Being able to intersperse captures and the usual custom log format entries on the same log line is my main problem. Yes, I was excited to use the new declared captures, but it came down to the same question as above; is there a way to use a declared capture in a custom log format line? If I can, it solves my problems... In log.c I see that %hr is doing a for loop on the session's front end proxy struct's list of capture fields, but doesn't look like it can take any arguments or anything for declared captures. Looking through the diffs on that file back two years, I see a reference in April to moving the captures from http_txn to stream, but a look at the diffs show that doesn't change this issue. I'm looking through log.c branch 1.6 : void parse_logformat_string(const char *fmt, struct proxy *curproxy, struct list *list_format, int options, int cap, const char *file, int line) for an indication of a way to define this, and I don't see it so far... am I missing it? Thanks again for any help! Phillip On Mon, Jul 6, 2015 at 4:10 PM, Nenad Merdanovic ni...@nimzo.info wrote: Hello Philip On 7/6/2015 9:47 PM, Phillip Decker wrote: Hello again all, I have a configuration problem, and I have a workaround that I don't like, so I'm hoping someone here might have a better solution - I have a number of capture fields such as ssl_c_s_dn, request header referrer, request header X-Forwarded-For, etc that get logged using %hr, among another 12 or so fields such as %t, %H, %B, %ci and so on. The problem is the client has an established log format which goes to an enterprise centralized log db. I'd love to be able to write out this format directly with haproxy, but since all the captured fields in haproxy get appended together by %hr and %hs, I don't know of a way to put one captured field in spot #4 of a log line, and then another captured field in spot #7, etc. In order to work around this, I have haproxy instead outputting all this data to an intermediate log format, and then have another process running which picks these up and rearranges them into the arbitrary log format that the enterprise accepts. Well, it seems like you are looking to reorder the captures, so take a look at: http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#7.3.6-capture.req.hdr You can use this in your log format to specify a different ordering than the one how the captures are sorted in the config. In 1.6 you could also use: http://cbonte.github.io/haproxy-dconv/configuration-1.6.html#declare%20capture Let me know if this helps. Regards, Nenad
Re: Capture sequencing in logs
Nenad, thank you so much for your help; I was able to run that successfully
on a test instance, I think that may fix the majority of my issue!
Thanks again,
Phillip
On Mon, Jul 6, 2015 at 4:55 PM, Nenad Merdanovic ni...@nimzo.info wrote:
Hello Phillip,
First, let me apologize for misspelling your name the first time.
On 7/6/2015 10:46 PM, Phillip Decker wrote:
Hi Nenad, thanks for the reply!
Okay, insofar as the first link - I can see where I can retrieve
captures for use in acls or elsewhere, but is there a way to use these
directly in the custom log format (
http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#8.2.4 )?
Being able to intersperse captures and the usual custom log format
entries on the same log line is my main problem.
Yes, I was excited to use the new declared captures, but it came down to
the same question as above; is there a way to use a declared capture in
a custom log format line? If I can, it solves my problems...
http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#8.2.4
As you can see here:
If a variable is named between square brackets ('[' .. ']') then it is
used as a sample expression rule (see section 7.3). This it useful to
add some less common information such as the client's SSL certificate's
DN, or to log the key that would be used to store an entry into a stick
table.
So essentially what you have to do is add something like
%[capture.req.hdr(0)] to your log-format string. And then just change
the 0 to whatever capture you want depending on the capture ordering.
Hope this helps.
Nenda
In log.c I see that %hr is doing a for loop on the session's front end
proxy struct's list of capture fields, but doesn't look like it can take
any arguments or anything for declared captures. Looking through the
diffs on that file back two years, I see a reference in April to moving
the captures from http_txn to stream, but a look at the diffs show that
doesn't change this issue.
I'm looking through
log.c branch 1.6 : void parse_logformat_string(const char *fmt, struct
proxy *curproxy, struct list *list_format, int options, int cap, const
char *file, int line)
for an indication of a way to define this, and I don't see it so far...
am I missing it?
Thanks again for any help!
Phillip
On Mon, Jul 6, 2015 at 4:10 PM, Nenad Merdanovic ni...@nimzo.info
mailto:ni...@nimzo.info wrote:
Hello Philip
On 7/6/2015 9:47 PM, Phillip Decker wrote:
Hello again all,
I have a configuration problem, and I have a workaround that I
don't
like, so I'm hoping someone here might have a better solution -
I have a number of capture fields such as ssl_c_s_dn, request
header
referrer, request header X-Forwarded-For, etc that get logged
using %hr,
among another 12 or so fields such as %t, %H, %B, %ci and so on.
The problem is the client has an established log format which goes
to an
enterprise centralized log db. I'd love to be able to write out
this
format directly with haproxy, but since all the captured fields in
haproxy get appended together by %hr and %hs, I don't know of a
way to
put one captured field in spot #4 of a log line, and then another
captured field in spot #7, etc.
In order to work around this, I have haproxy instead outputting
all this
data to an intermediate log format, and then have another process
running which picks these up and rearranges them into the
arbitrary log
format that the enterprise accepts.
Well, it seems like you are looking to reorder the captures, so take
a
look at:
http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#7.3.6-capture.req.hdr
You can use this in your log format to specify a different ordering
than
the one how the captures are sorted in the config.
In 1.6 you could also use:
http://cbonte.github.io/haproxy-dconv/configuration-1.6.html#declare%20capture
Let me know if this helps.
Regards,
Nenad
Capture sequencing in logs
Hello again all, I have a configuration problem, and I have a workaround that I don't like, so I'm hoping someone here might have a better solution - I have a number of capture fields such as ssl_c_s_dn, request header referrer, request header X-Forwarded-For, etc that get logged using %hr, among another 12 or so fields such as %t, %H, %B, %ci and so on. The problem is the client has an established log format which goes to an enterprise centralized log db. I'd love to be able to write out this format directly with haproxy, but since all the captured fields in haproxy get appended together by %hr and %hs, I don't know of a way to put one captured field in spot #4 of a log line, and then another captured field in spot #7, etc. In order to work around this, I have haproxy instead outputting all this data to an intermediate log format, and then have another process running which picks these up and rearranges them into the arbitrary log format that the enterprise accepts. I'd love to get rid of having to maintain this separate process long term. When I first saw the discussion of defined captures, I thought maybe there'd be a way to use those definitions in the log format directly, so you could have a log line that was in a certain sequence where the captures were mixed in: time, client ip, client subject dn, HTTP request uri, http status code, X-Forwarded-For IP, bytes transferred, etc (not the actual format, but you get the point). However, I don't see a way to do that. Then I saw the LUA logging calls, and so I was reading the API, trying to figure out if I could write the whole log statement using LUA, but without having tried it yet, is it a) possible to do so for every request, and b ) is there a performance penalty for doing so? Is there another way to handle this? How have other people dealt with this problem? Thanks! Phillip PS - That was an amazing find last week on the pipelining bug; thanks for your hard work! It's appreciated.
DN Single line representation delimiter
Hello, I'm writing with regard to the current delimiter used when haproxy passes the DN as one line from an SSL cert along in the headers to backend servers- Right now, that seems to be hardcoded to a '/' in the ssl_sock.c file, in function ssl_sock_get_dn_oneline(...) on approximately file line 2545: *(p++)='/'; Would anyone mind if we made that a configurable value? We have multiple servers which are expecting the DN entries to be delimited with a comma, ',' as per RFC 1779 (part 2.2 - page two). I'm willing to take a stab at it and submit the diffs, but I acknowledge that I'm new to this community, and I'm not sure the customs/convention. Have a great afternoon everyone! Phillip
