Hi Willy,
On 17 May 2017, at 20:54, Willy Tarreau wrote:
Hi Stéphane,
On Wed, May 17, 2017 at 02:03:26AM +0200, Stéphane Cottin wrote:
Thanks, this is the right answer, recompile with
DEFINE=-DREQURI_LEN=4096
fixes this issue.
that's what I was going to say :-)
This limit should be raised at runtime according to the log "len"
option, if
you want long log lines, you want them complete :)
In fact no, but we should have a global tunable for it now. The reason
why
we recommend to always keep the URI last is to ensure logs can safely
be
truncated at the least interesting place, because for most users, the
long
trail of query string arguments are mostly useless, while for
captures,
states and timers there is no compromise.
If you have the time to provide a patch doing this, that's clearly
something
we could merge. It's not too hard, REQURI_LEN is used at only a few
places.
You could have global.requri_len which would be set using
tune.http.log_uri_len
for example.
Best regards,
Willy
patch attached.
Stéphane
diff --git a/CHANGELOG b/CHANGELOG
index 40d24d44..0db5e7ae 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -2,6 +2,7 @@ ChangeLog :
===========
2017/04/03 : 1.8-dev1
+ - MINOR: Add tune.http.logurilen
- BUG/MEDIUM: proxy: return "none" and "unknown" for unknown LB algos
- BUG/MINOR: stats: make field_str() return an empty string on NULL
- DOC: Spelling fixes
diff --git a/doc/configuration.txt b/doc/configuration.txt
index edd9e79b..cff91c59 100644
--- a/doc/configuration.txt
+++ b/doc/configuration.txt
@@ -597,6 +597,7 @@ The following keywords are supported in the "global"
section :
- tune.chksize
- tune.comp.maxlevel
- tune.http.cookielen
+ - tune.http.logurilen
- tune.http.maxhdr
- tune.idletimer
- tune.lua.forced-yield
@@ -758,7 +759,8 @@ log <address> [len <length>] [format <format>] <facility>
[max level [min level]
truncate them before sending them. Accepted values are 80 to 65535
inclusive. The default value of 1024 is generally fine for all
standard usages. Some specific cases of long captures or
- JSON-formated logs may require larger values.
+ JSON-formated logs may require larger values. You may also need to
+ increase "tune.http.logurilen" if your request uris are truncated.
<format> is the log format used when generating syslog messages. It may be
one of the following :
@@ -1324,6 +1326,14 @@ tune.http.cookielen <number>
When not specified, the limit is set to 63 characters. It is recommended not
to change this value.
+tune.http.logurilen <number>
+ Sets the maximum length of request uri in logs. This prevent to truncate long
+ requests uris with valuable query strings in log lines. This is not related
+ to syslog limits. If you increase this limit, you may also increase the
+ 'log ... len yyyy' parameter. Your syslog deamon may also need specific
+ configuration directives too.
+ The default value is 1024.
+
tune.http.maxhdr <number>
Sets the maximum number of headers in a request. When a request comes with a
number of headers greater than this value (including the first line), it is
diff --git a/include/common/defaults.h b/include/common/defaults.h
index 81dc3b16..f53c611e 100644
--- a/include/common/defaults.h
+++ b/include/common/defaults.h
@@ -273,7 +273,7 @@
#define STREAM_MAX_COST (sizeof(struct stream) + \
2 * sizeof(struct channel) + \
2 * sizeof(struct connection) + \
- REQURI_LEN + \
+ global.tune.requri_len + \
2 * global.tune.cookie_len)
#endif
diff --git a/include/types/global.h b/include/types/global.h
index 57b969dd..b35b4f1f 100644
--- a/include/types/global.h
+++ b/include/types/global.h
@@ -139,6 +139,7 @@ struct global {
int chksize; /* check buffer size in bytes, defaults to
BUFSIZE */
int pipesize; /* pipe size in bytes, system defaults if
zero */
int max_http_hdr; /* max number of HTTP headers, use
MAX_HTTP_HDR if zero */
+ int requri_len; /* max len of request URI, use REQURI_LEN if
zero */
int cookie_len; /* max length of cookie captures */
int pattern_cache; /* max number of entries in the pattern
cache. */
int sslcachesize; /* SSL cache size in session, defaults to
20000 */
diff --git a/src/cfgparse.c b/src/cfgparse.c
index f1673219..40cd606d 100644
--- a/src/cfgparse.c
+++ b/src/cfgparse.c
@@ -888,6 +888,16 @@ int cfg_parse_global(const char *file, int linenum, char
**args, int kwm)
}
global.tune.cookie_len = atol(args[1]) + 1;
}
+ else if (!strcmp(args[0], "tune.http.logurilen")) {
+ if (alertif_too_many_args(1, file, linenum, args, &err_code))
+ goto out;
+ if (*(args[1]) == 0) {
+ Alert("parsing [%s:%d] : '%s' expects an integer
argument.\n", file, linenum, args[0]);
+ err_code |= ERR_ALERT | ERR_FATAL;
+ goto out;
+ }
+ global.tune.requri_len = atol(args[1]) + 1;
+ }
else if (!strcmp(args[0], "tune.http.maxhdr")) {
if (alertif_too_many_args(1, file, linenum, args, &err_code))
goto out;
@@ -7361,6 +7371,9 @@ int check_config_validity()
if (!global.tune.cookie_len)
global.tune.cookie_len = CAPTURE_LEN;
+ if (!global.tune.requri_len)
+ global.tune.requri_len = REQURI_LEN;
+
pool2_capture = create_pool("capture", global.tune.cookie_len,
MEM_F_SHARED);
/* Post initialisation of the users and groups lists. */
diff --git a/src/proto_http.c b/src/proto_http.c
index 53bfd8b6..357401fe 100644
--- a/src/proto_http.c
+++ b/src/proto_http.c
@@ -459,7 +459,7 @@ void init_proto_http()
/* memory allocations */
pool2_http_txn = create_pool("http_txn", sizeof(struct http_txn),
MEM_F_SHARED);
- pool2_requri = create_pool("requri", REQURI_LEN, MEM_F_SHARED);
+ pool2_requri = create_pool("requri", global.tune.requri_len ,
MEM_F_SHARED);
pool2_uniqueid = create_pool("uniqueid", UNIQUEID_LEN, MEM_F_SHARED);
}
@@ -2959,8 +2959,8 @@ int http_wait_for_request(struct stream *s, struct
channel *req, int an_bit)
if ((txn->uri = pool_alloc2(pool2_requri)) != NULL) {
int urilen = msg->sl.rq.l;
- if (urilen >= REQURI_LEN)
- urilen = REQURI_LEN - 1;
+ if (urilen >= global.tune.requri_len )
+ urilen = global.tune.requri_len - 1;
memcpy(txn->uri, req->buf->p, urilen);
txn->uri[urilen] = 0;
