Re: htx with compression issue, "Gunzip error: Body lacks gzip magics"
Hi guys, On Wed, Jan 02, 2019 at 07:42:37PM +0100, PiBa-NL wrote: > The patch fixes the reg-test for me as well, I guess its good to go :). Great, thanks for letting me know, now merged! Willy
Re: htx with compression issue, "Gunzip error: Body lacks gzip magics"
Hi Christopher, Willy, Op 2-1-2019 om 15:37 schreef Christopher Faulet: Le 29/12/2018 à 01:29, PiBa-NL a écrit : compression with htx, and a slightly delayed body content it will prefix some rubbish and corrupt the gzip header.. Hi Pieter, In fact, It is not a bug related to the compression. But a pure HTX one, about the defragmentation when we need space to store data. Here is a patch. It fixes the problem for me. Okay so the compression somehow 'triggers' this defragmentation to happen, are there simpler ways to make that happen 'on demand' ? Willy, if it is ok for you, I can merge it in upstream and backport it in 1.9. -- Christopher Faulet The patch fixes the reg-test for me as well, I guess its good to go :). Thanks. Regards, PiBa-NL (Pieter)
Re: htx with compression issue, "Gunzip error: Body lacks gzip magics"
On Wed, Jan 02, 2019 at 03:37:54PM +0100, Christopher Faulet wrote: > In fact, It is not a bug related to the compression. But a pure HTX one, > about the defragmentation when we need space to store data. Here is a patch. > It fixes the problem for me. > > Willy, if it is ok for you, I can merge it in upstream and backport it in > 1.9. I'm always OK, especially for bugs I don't understand :-) Willy
Re: htx with compression issue, "Gunzip error: Body lacks gzip magics"
Le 29/12/2018 à 01:29, PiBa-NL a écrit : Hi List, When using compression with htx, and a slightly delayed body content it will prefix some rubbish and corrupt the gzip header.. Below output i get with attached test.. Removing http-use-htx 'fixes' the test. This happens with both 1.9.0 and todays commit a2dbeb2, not sure if this ever worked before.. c1 0.1 len|1A\r c1 0.1 chunk|\222\7\0\0\0\377\377\213\10\0\0\0\0\0\4\3JLJN\1\0\0\0\377\377 c1 0.1 len|0\r c1 0.1 bodylen = 26 ** c1 0.1 === expect resp.status == 200 c1 0.1 EXPECT resp.status (200) == "200" match ** c1 0.1 === expect resp.http.content-encoding == "gzip" c1 0.1 EXPECT resp.http.content-encoding (gzip) == "gzip" match ** c1 0.1 === gunzip c1 0.1 Gunzip error: Body lacks gzip magics Can someone take a look? Thanks in advance. Hi Pieter, In fact, It is not a bug related to the compression. But a pure HTX one, about the defragmentation when we need space to store data. Here is a patch. It fixes the problem for me. Willy, if it is ok for you, I can merge it in upstream and backport it in 1.9. -- Christopher Faulet >From a6e9d6be951b8724921d1582a2cddea81b5b6a6a Mon Sep 17 00:00:00 2001 From: Christopher Faulet Date: Wed, 2 Jan 2019 11:23:44 +0100 Subject: [PATCH] BUG/MAJOR: htx: Return the good block address after a defrag When an HTX structure is defragmented, it is possible to retrieve the new block corresponding to an old one. This is useful to do a defrag during a loop on blocks, to be sure to continue looping on the good block. But, instead of returning the address of the new block in the HTX structure, the one in the temporary structure used to do the defrag was returned, leading to unexpected behaviours. This patch must be backported to 1.9. --- src/htx.c | 13 - 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/src/htx.c b/src/htx.c index bda293b43..83243e060 100644 --- a/src/htx.c +++ b/src/htx.c @@ -26,13 +26,15 @@ struct htx_blk *htx_defrag(struct htx *htx, struct htx_blk *blk) struct buffer *chunk = get_trash_chunk(); struct htx *tmp = htxbuf(chunk); struct htx_blk *newblk, *oldblk; - uint32_t new, old; + uint32_t new, old, blkpos; uint32_t addr, blksz; int32_t sl_off = -1; if (!htx->used) return NULL; + blkpos = -1; + new = 0; addr = 0; tmp->size = htx->size; @@ -54,13 +56,14 @@ struct htx_blk *htx_defrag(struct htx *htx, struct htx_blk *blk) if (htx->sl_off == oldblk->addr) sl_off = addr; + /* if is defined, set its new position */ + if (blk != NULL && blk == oldblk) + blkpos = new; + memcpy((void *)tmp->blocks + addr, htx_get_blk_ptr(htx, oldblk), blksz); new++; addr += blksz; - /* if is defined, set its new location */ - if (blk != NULL && blk == oldblk) - blk = newblk; } while (new < htx->used); htx->sl_off = sl_off; @@ -68,7 +71,7 @@ struct htx_blk *htx_defrag(struct htx *htx, struct htx_blk *blk) htx->front = htx->tail = new - 1; memcpy((void *)htx->blocks, (void *)tmp->blocks, htx->size); - return blk; + return ((blkpos == -1) ? NULL : htx_get_blk(htx, blkpos)); } /* Reserves a new block in the HTTP message with a content of -- 2.19.2
htx with compression issue, "Gunzip error: Body lacks gzip magics"
Hi List,
When using compression with htx, and a slightly delayed body content it
will prefix some rubbish and corrupt the gzip header..
Below output i get with attached test.. Removing http-use-htx 'fixes'
the test.
This happens with both 1.9.0 and todays commit a2dbeb2, not sure if this
ever worked before..
c1 0.1 len|1A\r
c1 0.1
chunk|\222\7\0\0\0\377\377\213\10\0\0\0\0\0\4\3JLJN\1\0\0\0\377\377
c1 0.1 len|0\r
c1 0.1 bodylen = 26
** c1 0.1 === expect resp.status == 200
c1 0.1 EXPECT resp.status (200) == "200" match
** c1 0.1 === expect resp.http.content-encoding == "gzip"
c1 0.1 EXPECT resp.http.content-encoding (gzip) == "gzip" match
** c1 0.1 === gunzip
c1 0.1 Gunzip error: Body lacks gzip magics
Can someone take a look? Thanks in advance.
Regards,
PiBa-NL (Pieter)
# Checks htx with compression and a short delay between headers and data send
by the server
varnishtest "Connection counters check"
feature ignore_unknown_macro
server s1 {
rxreq
txresp -nolen -hdr "Content-Length: 4"
delay 0.05
send "abcd"
} -start
haproxy h1 -conf {
global
stats socket /tmp/haproxy.socket level admin
defaults
mode http
option http-use-htx
frontend fe1
bind "fd@${fe1}"
compression algo gzip
#filter trace name BEFORE-HTTP-COMP
#filter compression
#filter trace name AFTER-HTTP-COMP
default_backend b1
backend b1
server srv1 ${s1_addr}:${s1_port}
} -start
# configure port for lua to call fe4
client c1 -connect ${h1_fe1_sock} {
txreq -url "/" -hdr "Accept-Encoding: gzip"
rxresp
expect resp.status == 200
expect resp.http.content-encoding == "gzip"
gunzip
expect resp.body == "abcd"
} -run
