Re: [PR] updating req.body_param doc with chunk-encoding limitation
Hello, On Tue, Oct 22, 2019 at 10:23:05AM -0400, PR Bot wrote: > Patch title(s): >updating req.body_param doc with chunk-encoding limitation > > Link: >https://github.com/haproxy/haproxy/pull/333 diff --git a/doc/configuration.txt b/doc/configuration.txt index b79795c68..fc988b320 100644 --- a/doc/configuration.txt +++ b/doc/configuration.txt @@ -15823,7 +15823,8 @@ req.body_param([) : string parameter as presented in the request body (no URL decoding is performed). Note that the ACL version of this fetch iterates over multiple parameters and will iteratively report all parameters values if no name is - given. + given. Also, only the first chunk is currently analyzed in case of + chunked-encoded body. Actually this was true before the introduction of HTX in 1.9. When you use HTX mode (using option http-use-htx in 1.9, by default in 2.0, and only one supported in 2.1), you don't have this limitation anymore. Thus I'd propose that instead we don't change the doc. Thanks! Willy
[PR] updating req.body_param doc with chunk-encoding limitation
Dear list! Author: fclerg <29798784+fcl...@users.noreply.github.com> Number of patches: 1 This is an automated relay of the Github pull request: updating req.body_param doc with chunk-encoding limitation Patch title(s): updating req.body_param doc with chunk-encoding limitation Link: https://github.com/haproxy/haproxy/pull/333 Edit locally: wget https://github.com/haproxy/haproxy/pull/333.patch && vi 333.patch Apply locally: curl https://github.com/haproxy/haproxy/pull/333.patch | git am - Description: Instructions: This github pull request will be closed automatically; patch should be reviewed on the haproxy mailing list (haproxy@formilux.org). Everyone is invited to comment, even the patch's author. Please keep the author and list CCed in replies. Please note that in absence of any response this pull request will be lost.
Re: req.body_param([])
Dear Jarno,
Thanks for the feedback on this.
I investigated further to find the solution to this issue. It turned out
that not all PUT requests had a problem with filtering out the
customerId parameter we're looking for.
Some requests seem to encode the HTTP PUT body somehow or have set some
HTTP header which prevents haproxy from extracting the parameter.
I'm still trying to find out what exactly is different.
Best
Simon
Am 14.05.2018 um 14:40 schrieb Jarno Huuskonen:
Hi Simon,
On Mon, May 14, Simon Schabel wrote:
HA-Proxy version 1.7.5-2~bpo8+1 2017/05/27
The setting for the logging was done in the /default /section as:
log-format %Ci:%Cp\ [%t]\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %st\
%B\ %cc\ %cs\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %hr\ %hs\
%[capture.req.hdr(0)]\ %{+Q}r
option log-separate-errors
option log-health-checks
and in the /http /and /https /section the body parameter capturing
is activated as:
# enable HTTP body logging
option http-buffer-request
declare capture request len 4
http-request capture req.body_param(customerId) id 0
As my haproxy version differs from yours I'm unsure where I might
made a configuration error.
I tested with 1.8.8 and 1.7.5 and with both versions I managed to
log customerId (with simple curl -X PUT/POST).
Are the POST/PUT requests large, is it possible that the customerId doesn't
fit in haproxy buffer (default 16k (I think)) ?
Can you test with curl to see if customerId is logged then:
curl -v -X PUT -d'customerId=911' http://yourhost.yourdomain/yourpath
# bigfile is some random file much larger than 16k
and curl -v -X PUT -d@bigfile -d'customerId=912'
http://yourhost.yourdomain/yourpath
-Jarno
--
SEMKNOX GmbH
Simon Schabel
Webergasse 1, Haus B/1 · 01067 Dresden
simon.scha...@semknox.com · +49-351-32 123 102 · www.semknox.com
Re: req.body_param([])
Hi Simon,
On Mon, May 14, Simon Schabel wrote:
> HA-Proxy version 1.7.5-2~bpo8+1 2017/05/27
>
> The setting for the logging was done in the /default /section as:
>
> log-format %Ci:%Cp\ [%t]\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %st\
> %B\ %cc\ %cs\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %hr\ %hs\
> %[capture.req.hdr(0)]\ %{+Q}r
> option log-separate-errors
> option log-health-checks
>
> and in the /http /and /https /section the body parameter capturing
> is activated as:
>
> # enable HTTP body logging
> option http-buffer-request
> declare capture request len 4
> http-request capture req.body_param(customerId) id 0
>
> As my haproxy version differs from yours I'm unsure where I might
> made a configuration error.
I tested with 1.8.8 and 1.7.5 and with both versions I managed to
log customerId (with simple curl -X PUT/POST).
Are the POST/PUT requests large, is it possible that the customerId doesn't
fit in haproxy buffer (default 16k (I think)) ?
Can you test with curl to see if customerId is logged then:
curl -v -X PUT -d'customerId=911' http://yourhost.yourdomain/yourpath
# bigfile is some random file much larger than 16k
and curl -v -X PUT -d@bigfile -d'customerId=912'
http://yourhost.yourdomain/yourpath
-Jarno
--
Jarno Huuskonen
Re: req.body_param([])
Dear Jarno,
Thanks for your message.
Ok, my /haproxy -vv/ output is as follows:
HA-Proxy version 1.7.5-2~bpo8+1 2017/05/27
Copyright 2000-2017 Willy Tarreau
Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -g -O2 -fPIE -fstack-protector-strong -Wformat
-Werror=format-security -D_FORTIFY_SOURCE=2
OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1
USE_LUA=1 USE_PCRE=1 USE_NS=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with OpenSSL version : OpenSSL 1.0.2k 26 Jan 2017
Running on OpenSSL version : OpenSSL 1.0.2l 25 May 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.35 2014-04-04
Running on PCRE version : 8.35 2014-04-04
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with Lua version : Lua 5.3.1
Built with transparent proxy support using: IP_TRANSPARENT
IPV6_TRANSPARENT IP_FREEBIND
Built with network namespace support
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available filters :
[COMP] compression
[TRACE] trace
[SPOE] spoe
The setting for the logging was done in the /default /section as:
log-format %Ci:%Cp\ [%t]\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %st\ %B\
%cc\ %cs\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %hr\ %hs\
%[capture.req.hdr(0)]\ %{+Q}r
option log-separate-errors
option log-health-checks
and in the /http /and /https /section the body parameter capturing is
activated as:
# enable HTTP body logging
option http-buffer-request
declare capture request len 4
http-request capture req.body_param(customerId) id 0
As my haproxy version differs from yours I'm unsure where I might made a
configuration error.
Thanks for your help,
best
Simon
Am 11.05.2018 um 12:43 schrieb Jarno Huuskonen:
Hi,
On Wed, May 09, Simon Schabel wrote:
We use the req.body_param([]) setting to retrieve body
parameter from the incoming HTTP queries and place them into the
logs.
Unfortunately this only works with HTTP POST requests. In our case
we need to extract the parameter from PUT requests as well.
Would it be an option to use req.body_param([]) on any HTTP
method type instead of restricting it to HTTP POST?
Can you share your haproxy -vv and the logging config ?
I just tested with haproxy-ss-20180507 and this minimal test
seems to get req.body_param(log) to stick table on both POST/PUT requests
(tested w/curl -X PUT|POST):
...
frontend test
option http-buffer-request
bind ipv4@127.0.0.1:8080
http-request track-sc2 req.body_param(log),lower table test_be if
METH_POST || METH_PUT
default_backend test_be
backend test_be
stick-table type string len 48 size 64 expire 240s store http_req_cnt
server wp2 some.ip.add.ress:80 id 2
...
curl -X PUT -d'log=user1' http://127.0.0.1:8080/
curl -X POST -d'log=user2' http://127.0.0.1:8080/
-Jarno
--
SEMKNOX GmbH
Simon Schabel
Webergasse 1, Haus B/1 · 01067 Dresden
simon.scha...@semknox.com · +49-351-32 123 102 · www.semknox.com
Re: req.body_param([])
Hi, On Wed, May 09, Simon Schabel wrote: > We use the req.body_param([]) setting to retrieve body > parameter from the incoming HTTP queries and place them into the > logs. > > Unfortunately this only works with HTTP POST requests. In our case > we need to extract the parameter from PUT requests as well. > > Would it be an option to use req.body_param([]) on any HTTP > method type instead of restricting it to HTTP POST? Can you share your haproxy -vv and the logging config ? I just tested with haproxy-ss-20180507 and this minimal test seems to get req.body_param(log) to stick table on both POST/PUT requests (tested w/curl -X PUT|POST): ... frontend test option http-buffer-request bind ipv4@127.0.0.1:8080 http-request track-sc2 req.body_param(log),lower table test_be if METH_POST || METH_PUT default_backend test_be backend test_be stick-table type string len 48 size 64 expire 240s store http_req_cnt server wp2 some.ip.add.ress:80 id 2 ... curl -X PUT -d'log=user1' http://127.0.0.1:8080/ curl -X POST -d'log=user2' http://127.0.0.1:8080/ -Jarno -- Jarno Huuskonen
req.body_param([])
Hello, We use the req.body_param([]) setting to retrieve body parameter from the incoming HTTP queries and place them into the logs. Unfortunately this only works with HTTP POST requests. In our case we need to extract the parameter from PUT requests as well. Would it be an option to use req.body_param([]) on any HTTP method type instead of restricting it to HTTP POST? Best Simon -- SEMKNOX GmbH Simon Schabel Webergasse 1, Haus B/1 · 01067 Dresden simon.scha...@semknox.com · +49-351-32 123 102 · www.semknox.com
