Re: Haproxy :: compression issue for IPv6

2022-10-24 Thread Christopher Faulet 

Le 10/21/22 à 11:12, Narendra Patel a écrit :

Hi Christopher,

This did help. The buffer seemed to have been exhausted.
Increasing it by 100 bytes to 1124 got the issue resolved.

Thank you so much for your response. It was really helpful :)



FYI, I pushed a fix in 2.7-DEV. It will be backported ASAP. Thanks !

--
Christopher Faulet

Re: Haproxy :: compression issue for IPv6

2022-10-21 Thread Narendra Patel 
Hi Christopher,

This did help. The buffer seemed to have been exhausted.
Increasing it by 100 bytes to 1124 got the issue resolved.

Thank you so much for your response. It was really helpful :)

Thanks!

Regards,
Narendra Patel

On Fri, Oct 21, 2022 at 12:48 PM Christopher Faulet 
wrote:

> Le 10/21/22 à 08:16, Narendra Patel a écrit :
> > Hi Team,
> >
> > We are having one issue with routing traffic through HAProxy.
> >
> > Have configured IPv6 for our Google load balancer backed by HAProxy. The
> > architecture looks like below:
> >
> > GLB --> L1 HAProxy --> Kong Gateway --> L2 HAProxy --> App node
> >
> > L1 HAProxy has the logic for compression.
> >
> > # zlib Compression
> >  compression algo gzip
> >  compression type text/html text/tab-separated-values text/plain
> text/xml
> > text/csv text/css text/javascript application/xml
> application/x-javascript
> > application/javascript
> >
> > But while evaluating IPv6, we are seeing one peculiar error. For IPV6
> request
> > for a css file, we see that HAProxy injects a "Content-Encoding: gzip"
> Header,
> > but does not really compress the file thus corrupting the response.
> >
> > We tried doing a tcpdump on L1 HAProxy and Kong to check.
> > Besides the `x-forwarded-for` header, HAProxy sends the same request to
> Kong for
> > both IPv4 and IPv6.
> > Also Kong responds with the same header values in response to HAProxy
> for both
> > IPv4 and IPv6.
> > Content length also is the same in Kong's response to HAProxy for both
> IPv4 and
> > IPv6. So Kong is not doing any compression.
> >
> > HAProxy version is : HAProxy version 2.6.4-2a2078c 2022/08/22
> >
> > Attaching the tcpdump details for HAProxy and Kong for both IPv4 and
> IPv6. Have
> > masked a few values for compliance.
> >
> > P.S: We are on GCP and the IPv6 setup looks like this :
> >
> https://cloud.google.com/load-balancing/docs/ipv6#ipv6_termination_and_proxy
> > <
> https://cloud.google.com/load-balancing/docs/ipv6#ipv6_termination_and_proxy
> >
> >
> > Any directions on this would be really appreciated.
>
> Hi,
>
> In the HAProxy IPv6 tcpdump, there is no "Vary" header. It means HAProxy
> fails
> to add it. When this happens, an error is triggered and the compression is
> not
> performed. So here, there is an issue because "Content-Encoding" header is
> not
> removed while it should be. As a workaround, you may try to increase
> "tune.maxrewrite" value in the global section. It seems you add several
> headers
> in your L1 HAProxy, exceeding the default reserve (1024 bytes). If the
> response
> payload is received with the headers, the buffer may be full, preventing
> some
> headers additions.
>
> Note that it may also happen on any headers added via http-response rules.
> In
> this case, this will generate an internal error, and a 500 response will
> be
> returned to the client. So, we should fix the compression related error.
> But, it
> remains a good idea to increase "tune.maxrewrite" value in your case.
>
> Regards,
> --
> Christopher Faulet
>
>

Re: Haproxy :: compression issue for IPv6

2022-10-21 Thread Christopher Faulet 

Le 10/21/22 à 08:16, Narendra Patel a écrit :

Hi Team,

We are having one issue with routing traffic through HAProxy.

Have configured IPv6 for our Google load balancer backed by HAProxy. The 
architecture looks like below:


GLB --> L1 HAProxy --> Kong Gateway --> L2 HAProxy --> App node

L1 HAProxy has the logic for compression.

# zlib Compression
     compression algo gzip
     compression type text/html text/tab-separated-values text/plain text/xml 
text/csv text/css text/javascript application/xml application/x-javascript 
application/javascript


But while evaluating IPv6, we are seeing one peculiar error. For IPV6 request 
for a css file, we see that HAProxy injects a "Content-Encoding: gzip" Header, 
but does not really compress the file thus corrupting the response.


We tried doing a tcpdump on L1 HAProxy and Kong to check.
Besides the `x-forwarded-for` header, HAProxy sends the same request to Kong for 
both IPv4 and IPv6.
Also Kong responds with the same header values in response to HAProxy for both 
IPv4 and IPv6.
Content length also is the same in Kong's response to HAProxy for both IPv4 and 
IPv6. So Kong is not doing any compression.


HAProxy version is : HAProxy version 2.6.4-2a2078c 2022/08/22

Attaching the tcpdump details for HAProxy and Kong for both IPv4 and IPv6. Have 
masked a few values for compliance.


P.S: We are on GCP and the IPv6 setup looks like this : 
https://cloud.google.com/load-balancing/docs/ipv6#ipv6_termination_and_proxy 



Any directions on this would be really appreciated.


Hi,

In the HAProxy IPv6 tcpdump, there is no "Vary" header. It means HAProxy fails 
to add it. When this happens, an error is triggered and the compression is not 
performed. So here, there is an issue because "Content-Encoding" header is not 
removed while it should be. As a workaround, you may try to increase 
"tune.maxrewrite" value in the global section. It seems you add several headers 
in your L1 HAProxy, exceeding the default reserve (1024 bytes). If the response 
payload is received with the headers, the buffer may be full, preventing some 
headers additions.


Note that it may also happen on any headers added via http-response rules. In 
this case, this will generate an internal error, and a 500 response will be 
returned to the client. So, we should fix the compression related error. But, it 
remains a good idea to increase "tune.maxrewrite" value in your case.


Regards,
--
Christopher Faulet

Haproxy :: compression issue for IPv6

2022-10-20 Thread Narendra Patel 
Hi Team,

We are having one issue with routing traffic through HAProxy.

Have configured IPv6 for our Google load balancer backed by HAProxy. The
architecture looks like below:

GLB --> L1 HAProxy --> Kong Gateway --> L2 HAProxy --> App node

L1 HAProxy has the logic for compression.

# zlib Compression
compression algo gzip
compression type text/html text/tab-separated-values text/plain
text/xml text/csv text/css text/javascript application/xml
application/x-javascript application/javascript

But while evaluating IPv6, we are seeing one peculiar error. For IPV6
request for a css file, we see that HAProxy injects a "Content-Encoding:
gzip" Header, but does not really compress the file thus corrupting the
response.

We tried doing a tcpdump on L1 HAProxy and Kong to check.
Besides the `x-forwarded-for` header, HAProxy sends the same request to
Kong for both IPv4 and IPv6.
Also Kong responds with the same header values in response to HAProxy for
both IPv4 and IPv6.
Content length also is the same in Kong's response to HAProxy for both IPv4
and IPv6. So Kong is not doing any compression.

HAProxy version is : HAProxy version 2.6.4-2a2078c 2022/08/22

Attaching the tcpdump details for HAProxy and Kong for both IPv4 and IPv6.
Have masked a few values for compliance.

P.S: We are on GCP and the IPv6 setup looks like this :
https://cloud.google.com/load-balancing/docs/ipv6#ipv6_termination_and_proxy

Any directions on this would be really appreciated.

Thanks!

Regards,
Narendra Patel
!.0.1..#PROXY TCP4 106.194.224.166 130.211.46.71 5358 443
GET /css/font-awesome/css/font-awesome.min.css HTTP/1.1
Host: dsautomationqa.qa-test.com
User-Agent: curl/7.84.0
Accept: */*
Accept-Encoding: gzip


05:35:43.662182 IP 10.29.144.30.80 > 35.191.3.208.56684: Flags [.], seq 1:4225, ack 208, win 229, options [nop,nop,TS val 825101877 ecr 553922719], length 4224: HTTP: HTTP/1.1 200 
E]@.@.|.
...#P.lM..y.p.
1..5!.0.HTTP/1.1 200 
Content-Type: text/css;charset=UTF-8
test-pod: 3000
x-test-update: faf454c28f889eace83180d6e0c08ec742113b9d
test-node: test
x-robots-tag: none
accept-ranges: bytes
etag: W/"26711-1666321007000"
last-modified: Fri, 21 Oct 2022 02:56:47 GMT
date: Fri, 21 Oct 2022 05:35:42 GMT
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
strict-transport-security: max-age=31536000; includeSubDomains
content-security-policy-report-only: default-src 'self' 'unsafe-eval'; script-src 'report-sample' 'self' 'unsafe-eval' 'unsafe-inline' https://content.p-a.test.com/ https://data.p-a.test.com/ https://ssl.google-analytics.com/ga.js https://www.gstatic.com/ https://www.recaptcha.net; style-src 'report-sample' 'self' 'unsafe-inline' blob: https://content.p-a.test.com; object-src 'none'; base-uri 'self'; connect-src 'self' https://api-js.mixpanel.com; font-src 'self' https://fonts.gstatic.com data: ; frame-src 'self'; img-src 'self' https://content.p-a.test.com https://data.p-a.test.com https://test-www-static.test.com https://ssl.google-analytics.com data: ; manifest-src 'self'; media-src 'self'; report-uri https://csp-reporter.qa-test.com/ ;
x-test-request-id: 6AC2E0A6:14EE_82D32E47:01BB_--_410D|qa-gcp-l1webuitest01+http_l1_webui
cache-control: public,max-age=2592000
x-test-override-cache: true
content-encoding: gzip
transfer-encoding: chunked
vary: Accept-Encoding
.|.a1+..PROXY TCP6 2401:4900:56f0:a175:9541:3ef8:2fae:2c62 2600:1901:0:174e:: 41730 443
GET /css/font-awesome/css/font-awesome.min.css HTTP/1.1
Host: dsautomationqa.qa-test.com
User-Agent: curl/7.84.0
Accept: */*
Accept-Encoding: gzip


05:33:28.168939 IP 10.29.144.30.80 > 35.191.14.7.62628: Flags [.], seq 1:7041, ack 238, win 229, options [nop,nop,TS val 824966384 ecr 394037345], length 7040: HTTP: HTTP/1.1 200 
E.@.@.=.
...#P..8..[.,Y
1+...|.aHTTP/1.1 200 
Content-Type: text/css;charset=UTF-8
test-pod: 3000
x-test-update: faf454c28f889eace83180d6e0c08ec742113b9d
test-node: test
x-robots-tag: none
accept-ranges: bytes
etag: W/"26711-1666321007000"
last-modified: Fri, 21 Oct 2022 02:56:47 GMT
date: Fri, 21 Oct 2022 05:33:27 GMT
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
strict-transport-security: max-age=31536000; includeSubDomains
content-security-policy-report-only: default-src 'self' 'unsafe-eval'; script-src 'report-sample' 'self' 'unsafe-eval' 'unsafe-inline' https://content.p-a.test.com/ https://data.p-a.test.com/ https://ssl.google-analytics.com/ga.js https://www.gstatic.com/ https://www.recaptcha.net; style-src 'report-sample' 'self' 'unsafe-inline' blob: https://content.p-a.test.com; object-src 'none'; base-uri 'self'; connect-src 'self' https://api-js.mixpanel.com; font-src 'self' https://fonts.gstatic.com data: ; frame-src 'self'; img-src 'self' https://content.p-a.test.com https://data.p-a.test.com https://test-www-static.test.com https://ssl.google-analytics.com data: ; manifest-src 'self'; media-src 'self'; report-uri https://csp-reporter.qa-test.com/ ;
x-test-request-id: