Re: [PATCH] Add the possibility to compress requests

[Olivier Houchard]

Hi,

On Tue, Apr 04, 2023 at 09:45:24PM +0200, Willy Tarreau wrote:
> Hi Olivier,
> 
> On Tue, Apr 04, 2023 at 12:29:15AM +0200, Olivier Houchard wrote:
> > Hi,
> > 
> > The attached patchset is a first attempt at adding the possibility to
> > compress requests, as well as responses.
> 
> This is pretty cool, I definitely see how this can be super useful :-)
 
:)

> > It adds a new keyword for compression, "tocompress" (any better
> > alternative name would be appreciated).
> 
> I would call it "direction" which I find more natural and understandable.
 
I like that a lot more. Thank you!

> > The valid values are "request",
> > if you want to compress requests, "response" if you want to compress
> > responses or "both", if you want to compress both.
> > The default is to compress responses only.
> 
> I have a few below (besides renaming "tocompress").
 
[...]

> > diff --git a/include/haproxy/compression-t.h 
> > b/include/haproxy/compression-t.h
> > index cdbf0d14e..d15d15ee3 100644
> > --- a/include/haproxy/compression-t.h
> > +++ b/include/haproxy/compression-t.h
> > @@ -37,6 +37,9 @@
> >  /* Compression flags */
> >  
> >  #define COMP_FL_OFFLOAD0x0001 /* Compression offload */
> > +#define COMP_FL_COMPRESS_REQ   0x0002 /* Compress requests */
> > +#define COMP_FL_COMPRESS_RES   0x0004 /* Compress responses */
> 
> So these ones would be COMP_FL_DIR_{REQ,RES}.
 
Done!

> >  struct comp {
> > struct comp_algo *algos;
> > struct comp_type *types;
> > diff --git a/src/flt_http_comp.c b/src/flt_http_comp.c
> > index 507009692..7680e241f 100644
> > --- a/src/flt_http_comp.c
> > +++ b/src/flt_http_comp.c
> > @@ -33,7 +33,9 @@ struct flt_ops comp_ops;
> >  
> >  struct comp_state {
> > struct comp_ctx  *comp_ctx;   /* compression context */
> > +   struct comp_ctx  *comp_ctx_req; /* compression context for request */
> > struct comp_algo *comp_algo;  /* compression algorithm if not NULL */
> > +   struct comp_algo *comp_algo_req; /* Algo to be used for request */
> > unsigned int  flags;  /* COMP_STATE_* */
> 
> I find it confusing to have comp_ctx and comp_ctx_req (and same for algo).
> I'd rather see a preliminary patch renaming comp_ctx to comp_ctx_res and
> comp_algo to comp_algo_res (just perform a mechanical rename). Or you could
> also have an array [2] for each, which will even remove some ifs in the
> code.
 
Ok I made that an array.

> Also I don't understand how you can have different algos for each direction
> since the config language allows you to define "compression algo" and
> "compression tocompress" so you cannot (apparently) have a situation where
> the two algos differ. I think it can make sense to use different algos for
> each direction (at least compression settings, e.g. expensive compression
> for uploads, relayed over expensive links, little compression for downloads
> that possibly comes from a cache).
 
The idea was that while only one algo could be used for requests, the
response algorithm would depend on what was supported by the client.

> Thus maybe an approach could be to use the algo to indicate the direction
> at the same time:
> 
>compression algo foobar   => sets response only, for compatibility
>compression algo-res foobar   => sets response only, explicit syntax
>compression algo-req foobar   => sets request only, explicit syntax
> 
> Then there's probably no need to have a keyword to set it at once in
> both directions since both actions are totally independent and it's
> probably not desired to encourage users to conflate the two directions
> in a single keyword.
> 
> You might need the same for "compression types" I think, so that you
> don't try to compress regular forms or login:password that will confuse
> applications, but only large data.
 
But I did that, so now there's even more reason to have separate
algorithms.

> > -static int set_compression_response_header(struct comp_state *st,
> > -  struct stream *s,
> > -  struct http_msg *msg);
> > +static int set_compression_header(struct comp_state *st,
> > + struct stream *s,
> > + struct http_msg *msg,
> > + int for_response);
>   
> For this one as well I would call it "direction" (or "dir"), as we
> already do in samples and a few other parts that work in both
> directions.
 
Ok I was about to do that, but somebody from the haproxy team whom I
won't name told me it was even better to just deduce the direction from
the http_msg, so I did just that.

> > +static void 
> > +comp_prepare_compress_request(struct comp_state *st, struct stream *s, 
> > struct http_msg *msg)
> > +{
> > +   struct htx *htx = htxbuf(>chn->buf);
> > +   struct http_txn *txn = s->txn;
> > +   struct http_hdr_ctx ctx;
> > +
> > +   ctx.blk = NULL;
> > +   /* 

RE: Interest in HA Proxy from Sonicwall

[Kenny Lederman]

Hi Team,

You can disregard as Jonathan Purcell 
jpurc...@haproxy.com has already reached out to me 
on this request.

Thank you.

Kenny Lederman
Enterprise Account Manager
(206) 455-6488 - Office
(847) 932-9771 - Cell
kenny.leder...@softchoice.com
[cid:image001.gif@01D967B5.F0910E40]
[Softchoice]
415 1st Avenue North, Suite 300
Seattle, WA  98109


From: Илья Шипицин 
Sent: Wednesday, April 5, 2023 11:56 AM
To: Aleksandar Lazic 
Cc: Kenny Lederman ; haproxy@formilux.org
Subject: Re: Interest in HA Proxy from Sonicwall

External message. Do not click links or open attachments unless you recognize 
the source. Message externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce 
jointe à moins d’en connaître la provenance.


ср, 5 апр. 2023 г. в 20:18, Aleksandar Lazic 
mailto:al-hapr...@none.at>>:
Hi Kenny.

On 05.04.23 20:04, Kenny Lederman wrote:
> Hi team,
>
> Do you have an account rep assigned to Sonicwall that could help me with
> getting a POC set up?

This is the Open Source Mailing list, if you want to get in touch with
the Company behind HAProxy please use this.

original intention was not clear :) maybe Kenny is looking for open source 
individuals to hire them in purpose.

otherwise, yes, https://www.haproxy.com is proper way 
to contact sales/commercial/whatever.


https://www.haproxy.com/contact-us/

Of course can you setup the Open Source HAProxy by your team, the
documentation is hosted at this URL.

http://docs.haproxy.org/

> Thank you,
>
> Kenny Lederman

Best Regards
Alex

> Enterprise Account Manager
>
> (206) 455-6488 - Office
>
> (847) 932-9771 - Cell
>
> kenny.leder...@softchoice.com 
> >
>
> 
>
>
>
> Softchoice 
>
>
>
> 415 1st Avenue North, Suite 300
> Seattle, WA  98109
>
>
>
> Manage Subscription
> Unsubscribe
> Privacy
> 
>

Manage Subscription   
Unsubscribe  
Privacy

Re: Interest in HA Proxy from Sonicwall

[Илья Шипицин]

ср, 5 апр. 2023 г. в 20:18, Aleksandar Lazic :

> Hi Kenny.
>
> On 05.04.23 20:04, Kenny Lederman wrote:
> > Hi team,
> >
> > Do you have an account rep assigned to Sonicwall that could help me with
> > getting a POC set up?
>
> This is the Open Source Mailing list, if you want to get in touch with
> the Company behind HAProxy please use this.
>

original intention was not clear :) maybe Kenny is looking for open source
individuals to hire them in purpose.

otherwise, yes, https://www.haproxy.com is proper way to contact
sales/commercial/whatever.


>
> https://www.haproxy.com/contact-us/
>
> Of course can you setup the Open Source HAProxy by your team, the
> documentation is hosted at this URL.
>
> http://docs.haproxy.org/
>
> > Thank you,
> >
> > Kenny Lederman
>
> Best Regards
> Alex
>
> > Enterprise Account Manager
> >
> > (206) 455-6488 - Office
> >
> > (847) 932-9771 - Cell
> >
> > kenny.leder...@softchoice.com 
> >
> > 
> >
> >
> >
> > Softchoice 
> >
> >
> >
> > 415 1st Avenue North, Suite 300
> > Seattle, WA  98109
> >
> >
> >
> > Manage Subscription
> > Unsubscribe
> > Privacy
> > 
> >
>
>

Re: Interest in HA Proxy from Sonicwall

[Aleksandar Lazic]

Hi Kenny.

On 05.04.23 20:04, Kenny Lederman wrote:

Hi team,

Do you have an account rep assigned to Sonicwall that could help me with 
getting a POC set up?


This is the Open Source Mailing list, if you want to get in touch with 
the Company behind HAProxy please use this.


https://www.haproxy.com/contact-us/

Of course can you setup the Open Source HAProxy by your team, the 
documentation is hosted at this URL.


http://docs.haproxy.org/


Thank you,

Kenny Lederman


Best Regards
Alex


Enterprise Account Manager

(206) 455-6488 - Office

(847) 932-9771 - Cell

kenny.leder...@softchoice.com 





Softchoice 



415 1st Avenue North, Suite 300
Seattle, WA  98109



Manage Subscription 
Unsubscribe 
Privacy 

Interest in HA Proxy from Sonicwall

[Kenny Lederman]

Hi team,

Do you have an account rep assigned to Sonicwall that could help me with 
getting a POC set up?

Thank you,

Kenny Lederman
Enterprise Account Manager
(206) 455-6488 - Office
(847) 932-9771 - Cell
kenny.leder...@softchoice.com
[cid:image001.gif@01D967AE.74FCB830]
[Softchoice]
415 1st Avenue North, Suite 300
Seattle, WA  98109



Manage Subscription   
Unsubscribe  
Privacy

Re: log settings have no effect in case of SSL handshake problems

[Remi Tricot-Le Breton]

Hello,

On 05/04/2023 09:39, Scharfenberg, Carsten wrote:


Hello,

I’m trying to setup haproxy RFC5424 logging to localhost and 
forwarding to a central log aggregator with rsyslog.


Although this setup sounds quite straight forward and common to me, 
it’s really hard to setup due to weak documentation of both – haproxy 
and rsyslog – in this context and a lack of examples.


Nevertheless I’ve succeeded after some hours of trial-and-error…

Only my settings do not work in case of SSL handshake problems. In 
this case I still get standard log messages from haproxy. Is it 
possible to setup RFC5424 also for this case?


These are my settings:

global

  log localhost:1514 format rfc5424 local0

log-send-hostname

[…]

defaults

  log global

  log-format-sd %{+E}o[my_sdid@12345\ client_ip=\"%ci\"\ 
client_port=\"%cp\"\ haproxy_frontend=\"%ft\"\ haproxy_backend=\"%b\"\ 
haproxy_server=\"%s\"\ haproxy_time_receive=\"%TR\"\ 
haproxy_time_queue=\"%Tc\"\ haproxy_time_response=\"%Tr\"\ 
haproxy_time_total=\"%Ta\"\ http_status_code=\"%ST\"\ 
bytes_read=\"%B\"\ haproxy_termination_state=\"%ts\"\ 
haproxy_total_connections=\"%ac\"\ 
haproxy_frontend_connections=\"%fc\"\ 
haproxy_backend_connections=\"%bc\"\ 
haproxy_server_connections=\"%sc\"\ haproxy_server_retries=\"%rc\"\ 
haproxy_server_queue=\"%sq\"\ haproxy_backend_queue=\"%bq\"\ 
http_request_headers=\"%hr\"\ http_response_headers=\"%hs\"\ 
http_request_method=\"%HM\"\ http_version=\"%HV\"\ 
http_request_path=\"%HPO\"\ http_request_query=\"%HQ\"]


  option httplog

[…]

frontend my_frontend

  mode http

  bind 1.2.3.4:443 ssl […]

[…]

backend my_backend

[…]

A “normal” log message looks like this:

<134>1 2023-04-05T09:00:14.893116+02:00 my_host haproxy 94107 - 
[my_sdid@12345 client_ip="4.3.2.1" client_port="65344" 
haproxy_frontend="my_frontend~" haproxy_backend="my_backend" 
haproxy_server="my_server01" haproxy_time_receive="0" 
haproxy_time_queue="1" haproxy_time_response="4" 
haproxy_time_total="5" http_status_code="200" bytes_read="168" 
haproxy_termination_state="--" haproxy_total_connections="1" 
haproxy_frontend_connections="1" haproxy_backend_connections="0" 
haproxy_server_connections="0" haproxy_server_retries="0" 
haproxy_server_queue="0" haproxy_backend_queue="0" 
http_request_headers="{my_user_agent}" http_response_headers="" 
http_request_method="GET" http_version="HTTP/1.1" 
http_request_path="/path" http_request_query="?query=foo"] 
4.3.2.1:65344 [05/Apr/2023:09:00:14.887] my_frontend~ 
my_backend/my_server01 0/0/1/4/5 200 168 - -  1/1/0/0/0 0/0 
{my_user_agent} "GET /path?query=foo HTTP/1.1"


In case the SSL handshake fails (e.g. because of a simple TCP 
connection check):


<134>1 2023-04-05T09:00:14.047002+02:00 my_host haproxy 94107 - - 
4.3.2.1:65341 [05/Apr/2023:09:00:13.996] my_frontend/1: Connection 
closed during SSL handshake




You might want to have a look at the "error-log-format" option that 
would allow you to define a dedicated log-format in case of SSL 
handshake errors (among others). The log line you have corresponds 
precisely to the legacy error log format that is used if no specific 
'error-log-format' is defined.


Regards
Rémi Le Breton

log settings have no effect in case of SSL handshake problems

[Scharfenberg, Carsten]

Hello,

I'm trying to setup haproxy RFC5424 logging to localhost and forwarding to a 
central log aggregator with rsyslog.
Although this setup sounds quite straight forward and common to me, it's really 
hard to setup due to weak documentation of both - haproxy and rsyslog - in this 
context and a lack of examples.
Nevertheless I've succeeded after some hours of trial-and-error...
Only my settings do not work in case of SSL handshake problems. In this case I 
still get standard log messages from haproxy. Is it possible to setup RFC5424 
also for this case?

These are my settings:

global
  log localhost:1514 format rfc5424 local0
  log-send-hostname
[...]

defaults
  log global
  log-format-sd %{+E}o[my_sdid@12345\ client_ip=\"%ci\"\ client_port=\"%cp\"\ 
haproxy_frontend=\"%ft\"\ haproxy_backend=\"%b\"\ haproxy_server=\"%s\"\ 
haproxy_time_receive=\"%TR\"\ haproxy_time_queue=\"%Tc\"\ 
haproxy_time_response=\"%Tr\"\ haproxy_time_total=\"%Ta\"\ 
http_status_code=\"%ST\"\ bytes_read=\"%B\"\ haproxy_termination_state=\"%ts\"\ 
haproxy_total_connections=\"%ac\"\ haproxy_frontend_connections=\"%fc\"\ 
haproxy_backend_connections=\"%bc\"\ haproxy_server_connections=\"%sc\"\ 
haproxy_server_retries=\"%rc\"\ haproxy_server_queue=\"%sq\"\ 
haproxy_backend_queue=\"%bq\"\ http_request_headers=\"%hr\"\ 
http_response_headers=\"%hs\"\ http_request_method=\"%HM\"\ 
http_version=\"%HV\"\ http_request_path=\"%HPO\"\ http_request_query=\"%HQ\"]
  option httplog
[...]

frontend my_frontend
  mode http
  bind 1.2.3.4:443 ssl [...]
[...]

backend my_backend
[...]

A "normal" log message looks like this:
<134>1 2023-04-05T09:00:14.893116+02:00 my_host haproxy 94107 - [my_sdid@12345 
client_ip="4.3.2.1" client_port="65344" haproxy_frontend="my_frontend~" 
haproxy_backend="my_backend" haproxy_server="my_server01" 
haproxy_time_receive="0" haproxy_time_queue="1" haproxy_time_response="4" 
haproxy_time_total="5" http_status_code="200" bytes_read="168" 
haproxy_termination_state="--" haproxy_total_connections="1" 
haproxy_frontend_connections="1" haproxy_backend_connections="0" 
haproxy_server_connections="0" haproxy_server_retries="0" 
haproxy_server_queue="0" haproxy_backend_queue="0" 
http_request_headers="{my_user_agent}" http_response_headers="" 
http_request_method="GET" http_version="HTTP/1.1" http_request_path="/path" 
http_request_query="?query=foo"] 4.3.2.1:65344 [05/Apr/2023:09:00:14.887] 
my_frontend~ my_backend/my_server01 0/0/1/4/5 200 168 - -  1/1/0/0/0 0/0 
{my_user_agent} "GET /path?query=foo HTTP/1.1"

In case the SSL handshake fails (e.g. because of a simple TCP connection check):
<134>1 2023-04-05T09:00:14.047002+02:00 my_host haproxy 94107 - - 4.3.2.1:65341 
[05/Apr/2023:09:00:13.996] my_frontend/1: Connection closed during SSL handshake

Thanks and Regards,
Carsten