Question on deleting cookies from an HTTP request

[Lokesh Jindal]

Hey folks

I have found that there is no operator "del-cookie" in HAProxy to delete
cookies from the request. (HAProxy does support the operator "del-header").

Can you explain why such an operator is not supported? Is it due to
complexity? Due to performance? It will be great if you can share details
behind this design choice.

We have use cases where we want to delete cookies from the request. Not
having this support in HAProxy also makes me question if one should be
deleting request cookies in the reverse proxy layer.

Thanks
Lokesh

Re: About the SPOE

[Lokesh Jindal]

Hey Willy

Resending this email in case you missed the last one. Let me know if you had 
any follow up questions/comments.
I saw https://github.com/haproxy/haproxy/issues/2502 created by Christopher - 
looking forward to discussion on that issue.


- Lokesh

From: Lokesh Jindal 
Date: Monday, March 18, 2024 at 4:55 PM
To: Willy Tarreau 
Cc: Abhijeet Rastogi , Christopher Faulet 
, haproxy@formilux.org , Aleksandar 
Lazic 
Subject: Re: About the SPOE
Hey Willy

Please see my response inline below.

- Lokesh

From: Willy Tarreau 
Date: Monday, March 18, 2024 at 4:08 AM
To: Lokesh Jindal 
Cc: Abhijeet Rastogi , Christopher Faulet 
, haproxy@formilux.org , Aleksandar 
Lazic 
Subject: Re: About the SPOE
Hi Lokesh, Abhijeet, Alex,

First, thanks for jumping into this thread, the purpose of the
deprecation is in a big part to try to collect the requirements
of possibly existing users. Mind you that the rare times we hear
about SPOE is only because of problems, so it's difficult to figure
what to keep and what to cut from the existing design.

More on that below:

On Fri, Mar 15, 2024 at 05:15:14PM +0000, Lokesh Jindal wrote:
> Hey Christopher
>
> Adding to what my colleague, Abhijeet, said.
>
>
>   1.  We plan to ramp traffic to HAProxy very soon where we will heavily rely
>   on SPOA. In our testing, we are satisfied with SPOE in terms of
>   performance. The flexibility to write SPOA in any language not only allows
>   us to handle "complex use cases" like connecting to non-http downstreams,
>   but also helps in observability - metrics and logging.

Interesting, the initial internal e-mail in 2016 that ignited the SPOE
design was driven from the same observations: in web environments it's
quite rare to find developers who are at ease with system-level languages,
yet they are the ones most likely to request to extend the proxy. For
this reason we wanted to offer the possibility to call code written in
other languages.


In addition it was estimated that the ability to connect
to the agent over the network and using secure connections was absolutely
essential. It brings the ability to scale the processing engine without
adding more LB nodes, and even to offload that to another DC, infrastructure
or even to delegate it to a 3rd party.
[Lokesh]:
One of our major flows requires us to connect to couchbase for every request.
Today, we are running one HAProxy process + one golang SPOA process on every 
host of the reverse proxy layer.
This keeps the setup simple and avoids making a network call for HAProxy-SPOA 
communication, reducing latency.

We have a couple of other flows where the SPOA connects to HTTP services for 
things like SSO, some type of cookie renewal, etc.
These are not as latency sensitive as the flow mentioned above.

Further, we anticipate the interaction of SPOA with these external services 
will change often –
how we parse the http response from the services, how we make the request to 
these services,
addition of new calls to services, etc.
In these cases, the SPOA will give us ability to develop features fast.

Among the use cases which immediately came to mind were authentication,
database accesses, name resolution, "remote maps", request classification,
IP reputation, etc. In addition we thought that limiting ourselves to short
request/responses like this was probably limiting and that it would have
been useful to support streaming so that we could implement image
recompression, caching, WAFs etc.

The first PoC implementation that was merged in version 1.7 lacked the
streaming abilities, and it's still the current implementation. It took
a while before we received feedback on it, since then caching was
implemented, the demand for image recompression is close to non-existing,
and WAFs users have well accommodated to dealing with extra layers by now
it seems.


So basically we're left with something ultra-complex that deals
with short request-responses most of the time, and that suffers from the
initial design which was way bigger than the use cases.
[Lokesh]: I think this applies to us, as explained in my comment above.


>   2.  What is the best alternative to SPOE?

I don't know yet, and the purpose of this deprecation precisely is to
engage discussion with users. One could think about various HTTP-based
protocols for which it is easier to implement a server, some gRPC maybe,
possibly even a stripped-down version of the SPOP protocol if we figure
that everyone is running on a reasonable subset that's much easier to
deal with than the whole stuff we have now.

> Two options that we are aware of
>   - Write fetchers/converters in lua or write filters in other languages
>   using the Lua API. In your experience, how do they compare to SPOE in terms
>   of:
>  *   Performance
>  *   Fault isolation

The benefits of SPOE as you've found, clearly are in terms of flexibility
as i

Re: About the SPOE

[Lokesh Jindal]

Hey Willy

Please see my response inline below.

- Lokesh

From: Willy Tarreau 
Date: Monday, March 18, 2024 at 4:08 AM
To: Lokesh Jindal 
Cc: Abhijeet Rastogi , Christopher Faulet 
, haproxy@formilux.org , Aleksandar 
Lazic 
Subject: Re: About the SPOE
Hi Lokesh, Abhijeet, Alex,

First, thanks for jumping into this thread, the purpose of the
deprecation is in a big part to try to collect the requirements
of possibly existing users. Mind you that the rare times we hear
about SPOE is only because of problems, so it's difficult to figure
what to keep and what to cut from the existing design.

More on that below:

On Fri, Mar 15, 2024 at 05:15:14PM +0000, Lokesh Jindal wrote:
> Hey Christopher
>
> Adding to what my colleague, Abhijeet, said.
>
>
>   1.  We plan to ramp traffic to HAProxy very soon where we will heavily rely
>   on SPOA. In our testing, we are satisfied with SPOE in terms of
>   performance. The flexibility to write SPOA in any language not only allows
>   us to handle "complex use cases" like connecting to non-http downstreams,
>   but also helps in observability - metrics and logging.

Interesting, the initial internal e-mail in 2016 that ignited the SPOE
design was driven from the same observations: in web environments it's
quite rare to find developers who are at ease with system-level languages,
yet they are the ones most likely to request to extend the proxy. For
this reason we wanted to offer the possibility to call code written in
other languages.


In addition it was estimated that the ability to connect
to the agent over the network and using secure connections was absolutely
essential. It brings the ability to scale the processing engine without
adding more LB nodes, and even to offload that to another DC, infrastructure
or even to delegate it to a 3rd party.
[Lokesh]:
One of our major flows requires us to connect to couchbase for every request.
Today, we are running one HAProxy process + one golang SPOA process on every 
host of the reverse proxy layer.
This keeps the setup simple and avoids making a network call for HAProxy-SPOA 
communication, reducing latency.

We have a couple of other flows where the SPOA connects to HTTP services for 
things like SSO, some type of cookie renewal, etc.
These are not as latency sensitive as the flow mentioned above.

Further, we anticipate the interaction of SPOA with these external services 
will change often –
how we parse the http response from the services, how we make the request to 
these services,
addition of new calls to services, etc.
In these cases, the SPOA will give us ability to develop features fast.

Among the use cases which immediately came to mind were authentication,
database accesses, name resolution, "remote maps", request classification,
IP reputation, etc. In addition we thought that limiting ourselves to short
request/responses like this was probably limiting and that it would have
been useful to support streaming so that we could implement image
recompression, caching, WAFs etc.

The first PoC implementation that was merged in version 1.7 lacked the
streaming abilities, and it's still the current implementation. It took
a while before we received feedback on it, since then caching was
implemented, the demand for image recompression is close to non-existing,
and WAFs users have well accommodated to dealing with extra layers by now
it seems.


So basically we're left with something ultra-complex that deals
with short request-responses most of the time, and that suffers from the
initial design which was way bigger than the use cases.
[Lokesh]: I think this applies to us, as explained in my comment above.


>   2.  What is the best alternative to SPOE?

I don't know yet, and the purpose of this deprecation precisely is to
engage discussion with users. One could think about various HTTP-based
protocols for which it is easier to implement a server, some gRPC maybe,
possibly even a stripped-down version of the SPOP protocol if we figure
that everyone is running on a reasonable subset that's much easier to
deal with than the whole stuff we have now.

> Two options that we are aware of
>   - Write fetchers/converters in lua or write filters in other languages
>   using the Lua API. In your experience, how do they compare to SPOE in terms
>   of:
>  *   Performance
>  *   Fault isolation

The benefits of SPOE as you've found, clearly are in terms of flexibility
as it allows to scale the number of analysers independently on the number
of LBs, and it almost makes problems almost unnoticeable. For example, if
your code relied on unstable 3rd party libraries, crashing your SPOA doesn't
bring down the whole proxy. Similarly in terms of added latency, all the
latency is in the external component, the rest of the traffic is not
affected as it would be by placing some heavy processing directly inside
the haproxy proces

Re: About the SPOE

[Lokesh Jindal]

Hey Christopher

Adding to what my colleague, Abhijeet, said.


  1.  We plan to ramp traffic to HAProxy very soon where we will heavily rely 
on SPOA. In our testing, we are satisfied with SPOE in terms of performance. 
The flexibility to write SPOA in any language not only allows us to handle 
“complex use cases” like connecting to non-http downstreams, but also helps in 
observability – metrics and logging.
  2.  What is the best alternative to SPOE? Two options that we are aware of – 
Write fetchers/converters in lua or write filters in other languages using the 
Lua API. In your experience, how do they compare to SPOE in terms of:
 *   Performance
 *   Fault isolation
  3.  As Abhijeet said, can you share a list of issues with SPOE that make it 
hard to maintain?

Be it SPOE or an alternate solution that allows us to handle complex use cases 
with good performance, fault isolation (as much as possible) and observability, 
we will be happy to help develop/maintain it.

- Lokesh

From: Abhijeet Rastogi 
Date: Friday, March 15, 2024 at 8:23 AM
To: Christopher Faulet 
Cc: haproxy@formilux.org 
Subject: Re: About the SPOE
Hi Christopher,

Thank you for starting the discussion here.

>Worst, other parts of HAProxy evolved,
especially applets part, making maintenance ever more expensive.

Can we elaborate on what makes maintenance expensive?

>We must be realistic on the subject, there was no real adoption on the SPOE and
this partly explains why no time was invest on it.

My company primarily uses SPOE in cases where we've to connect with
non-http downstreams, for example, connecting to the data layer
(Couchbase etc) that makes the proxy stateless. Other uses-cases are
SSO implementation etc.
We are a new user to HAproxy, with multi-million QPS on HAproxy
already, and we've plans to ramp multiples of that in the coming year
or two.

>Now we are open to discussion on this subject. Let us know your feeling and if
you have any suggestion, we will be happy to talk about it.

Our goal at my company has been to not modify HAproxy core and still
add features to it without running in the same memory space as HAproxy
for safety. Of course, we use config DSL and lua as much as possible,
but it falls short in terms of writing complex business logic.
(non-http downstreams etc).
If it's not SPOE, what is a true replacement? With my limited
knowledge, not a true replacement though, maybe lua filter API and
then write these complex in-flight operations in Rust?
With the learnings from today, what would be the ideal way to replace
it, considering we had engineers to maintain it?

Slightly off-topic, SPOE has its flaws too though. One of our
use-cases is to stream (kafka) HTTP traffic information (cookies,
request data etc) in an efficient fashion and we actually did not use
SPOE as the cost to exchange information was too high. In this
particular case, we ended up streaming data via txn/req variables to
logs instead (ring buffer).

Thanks,
Abhijeet


On Fri, Mar 15, 2024 at 7:10 AM Christopher Faulet  wrote:
>
> Hi all,
>
> It was evoked on the ML by Willy and mentioned in few issues on GH. It is now
> official. The SPOE was marked as deprecated for the 3.0. It is not a pleasant
> announce because it is always an admission of failure to remove a feature.
> Sadly, this filter should be refactored to work properly. It was implemented 
> as
> a functional PoC for the 1.7 and since then, no time was invest to improve it
> and make it truly maintainable in time. Worst, other parts of HAProxy evolved,
> especially applets part, making maintenance ever more expensive.
>
> We must be realistic on the subject, there was no real adoption on the SPOE 
> and
> this partly explains why no time was invest on it. So we are really sorry for
> users relying on it. But we cannot continue in this direction.
>
> The 3.0 is be an LTS version. It means the SPOE will still be maintained on 
> this
> version and lower ones for 5 years. On the 3.1, it will be marked as
> unmaintained and possibly removed if an alternative solution is implemented.
>
> It remains few months before the 3.0 release to change our mind. Maybe this
> announce will be an electroshock to give it a new lease of life. Otherwise it 
> is
> time to find an alternative solution based on an existing protocol.
>
> For all 3.0 users, there is now a warning if a SPOE filter is configured. But
> there is also a global option to silent it. To do so,
> "expose-deprecated-directives" must be added in the global section.
>
> Now we are open to discussion on this subject. Let us know your feeling and if
> you have any suggestion, we will be happy to talk about it.
>
> Regards,
> --
> Christopher Faulet
>


--
Cheers,
Abhijeet 
(https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fabhi.host%2F&data=05%7C02%7Cljindal%40linkedin.com%7C9a34d3ec1e5e4df77e6e08dc4503e41e%7C72f988bf86f141af91ab2d7cd011db47%7C0%7C0%7C638461130234909653%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAw

Re: [PATCH] MEDIUM: sample: Enhances converter "bytes" to take variable names as arguments

[Lokesh Jindal]

Hello Willy

Thanks for the detailed explanations and the feedback.
PFA the two patches based on your feedback. My apologies for the delay.

Truncation can be useful but I could understand that when extracting bytes
> one would want this exact number of bytes to continue the processing. I
> just wanted to mention this to make sure this is what you had in mind as
> well

Yes, this is what I was thinking.

Here I'm wondering what we should do if the user asks for more bytes than
> available. I think we have 3 options:
>   - return 0 so that the sample's flags are consulted by the caller (e.g.
> SMP_F_MAY_CHANGE will allow the caller to come back later trying to
> get more bytes, or fail if not enough)


>   - truncate to what is available


>   - maybe a combination of both, i.e. return 0 as long as the caller does
> not pass SMP_OPT_FINAL, or return a truncated response.


Sounds good. The revised patch uses option 3 above.


> >   /* Try to decode a variable. */
> > - if (vars_check_arg(&args[0], NULL))
> > + if (vars_check_arg(&args[0], err))
> >   return 1;


> I can't say if this change was intentional or not, nor if the NULL had a
> particular reason or not, but at least this is unrelated to your patch
> and would need to be in its own patch, so that in case of regression we
> can decide whether to fix or revert it. And if it really fixes an issue
> maybe we'll want to backport it.

This was intentional. I thought it was a "miss" in the existing code. This
change is now part of the second patch.

I have also fixed indentation/tabs, column width, etc. based on your
feedback.


I have a Q for you. Pasting code from the revised patch.
+ // arg1 value is greater than the remaining length
+ if (smp->opt & SMP_OPT_FINAL) {
+   // truncate to remaining length
+   length = smp->data.u.str.data - start_idx;
+ } else {
+   smp->data.u.str.data = 0;
+   return 0;
+ }

I tried to write a test in the vtc file for the "else" clause above, but
could not figure out when we will hit that condition. Here is what I tried.
Added following the config section.
+   http-response set-header hdr_bytes_0_7
 "%[res.hdr(input),bytes(0,7)]"
+   http-response set-header hdr_bytes_0_7_concat
"%[res.hdr(input),bytes(0,7),concat()]"

Added following in the client section
+   # since "bytes" is the non-final operator, response would be empty
+   expect resp.http.hdr_bytes_0_7_concat == ""

But the test failed:
 c2EXPECT resp.http.hdr_bytes_0_7_concat (012345) == "" failed

Anyway, I can add this such a test in another patch if needed and time
permits. Let me know what you think.

Thanks
Lokesh

On Fri, Sep 8, 2023 at 10:05 AM Willy Tarreau  wrote:

> On Thu, Sep 07, 2023 at 10:45:58AM -0700, Lokesh Jindal wrote:
> > Resending the patch with the right email subject. Previos email
> > discussion - here
> > <https://www.mail-archive.com/haproxy@formilux.org/msg43973.html>.
>
> I saw your previous mail, just didn't have time yet to review it.
>
> > PFA the patch file.
>
> Thanks!
>
> > I have two questions in general:
> > 1. Can you explain what this method does and why is it needed? (I used it
> > in my patch following the pattern in the converter "sub")
> > *smp_set_owner(&smp_arg0, smp->px, smp->sess, smp->strm, smp->opt);**
>
> It just makes sure that the newly created sample properly references
> the context from which it was created. I remember that *some* functions
> retrieve some elements there, maybe ACLs or Lua actions that need to
> yield, etc.
>
> > 2. In my patch, *sample_conv_bytes* returns 0 in case of an invalid arg
> > e.g. negative offset. Can you explain when a converter should return 0 V
> 1?
> > I am not sure how that impacts the haproxy behavior.
>
> Normally when evaluating an expression, 0 is returned to indicate
> "nothing for now" but this doesn't preclude that it will not be possible
> to have more info later (depending on extra flags added such as
> SMP_F_MAY_CHANGE). 1 normally indicates that we have a result and that
> the sample is correctly filled. If a zero is returned without any flag,
> it will usually abort the evaluation of the expression (e.g. it cannot
> be converted thus processed). ACL evaluation will not go further with
> that sample for example. If 1 is returned but the sample type is not
> filled, implicitly the caller will see that SMP_T_ANY is returned and
> not compatible with the next expected type so it will stop there as well.
> Thus I think that returning 0 or 1 in case of failure to process args 

[PATCH] MEDIUM: sample: Enhances converter "bytes" to take variable names as arguments

[Lokesh Jindal]

Resending the patch with the right email subject. Previos email
discussion - here
.

PFA the patch file.

I have two questions in general:
1. Can you explain what this method does and why is it needed? (I used it
in my patch following the pattern in the converter "sub")
*smp_set_owner(&smp_arg0, smp->px, smp->sess, smp->strm, smp->opt);**

2. In my patch, *sample_conv_bytes* returns 0 in case of an invalid arg
e.g. negative offset. Can you explain when a converter should return 0 V 1?
I am not sure how that impacts the haproxy behavior.

Thanks
Lokesh


0001-MEDIUM-sample-Enhances-converter-bytes-to-take-varia.patch
Description: Binary data

Re: Request for feedback: Add support for txn args as arguments in converter "bytes"

[Lokesh Jindal]

PFA the patch.

I have two questions:
1. Can you explain what this method does and why is it needed? (I used it
in my patch following the pattern in the converter "sub") -
*smp_set_owner(&smp_arg0,
smp->px, smp->sess, smp->strm, smp->opt);*
2. In my patch, *sample_conv_bytes* returns 0 in case of an invalid arg
e.g. negative offset. Can you explain when a converter should return 0 V 1?
I am not sure how that impacts the haproxy behavior.

On Mon, Aug 28, 2023 at 9:56 AM Lokesh Jindal <15ljin...@gmail.com> wrote:

> Thanks for the response and the corrections, Willy.
>
> *We need to decide what to do when the variable does not*
>
> *exist or is empty. We can't make converters fail for now, so most
> likelyit will have to end up as value zero for offset and/or length*.
>
> Here is the implementation today - link
> <https://github.com/haproxy/haproxy/blob/e7d9082315e33e41347b2a7db949e995f5b7/src/sample.c#L2706>.
> We set the length of the output to 0 in case of an invalid input (e.g. arg0
> value >= length of the bytes in the input to the converter).
> So, for all other invalid inputs (e.g. variable in arg[0] does not exist),
> we can do the same.
>
> We can discuss more after I share the patch.
>
> Thanks
> Lokesh
>
> On Mon, Aug 28, 2023 at 2:53 AM Willy Tarreau  wrote:
>
>> Hi Lokesh,
>>
>> On Fri, Aug 25, 2023 at 01:44:48PM -0700, Lokesh Jindal wrote:
>> > Hey folks
>> >
>> > I am writing to gather feedback on an idea before doing the
>> implementation.
>> > Per the documentation, converter "bytes" accepts integer values as
>> > arguments, but not txn args.
>> > i.e. ,bytes(2,8) will work
>> > but ,bytes(txn.start_idx,txn.length) will not work.
>> >
>> > For our use case, we need to parse some binary data (a cookie) to
>> extract
>> > some info in haproxy. However, the bytes that need to be extracted are
>> not
>> > fixed and will depend on the request. We could use simple arithmetic to
>> > determine the index/length for bytes to be extracted and store them in
>> txn
>> > args. These txn args can then be used with converter "bytes".
>> >
>> > I can see that the converter "sub" already supports txn args as
>> arguments.
>> > I have successfully validated the proposed idea with an implementation
>> > following the same pattern as "sub".
>>
>> In fact it's not "txn", it's variables in general. Most of the arithmetic
>> functions support a variable name as an alternative to a numeric value. I
>> tend to think it would indeed make sense to support both integers and
>> variables for bytes(), it could even be the same for a few other ones
>> (maybe field(), word(), ltrim(), rtrim() etc).
>>
>> > Let me know what you think. If there are no concerns, I can send a
>> patch.
>>
>> I'm all for it. We need to decide what to do when the variable does not
>> exist or is empty. We can't make converters fail for now, so most likely
>> it will have to end up as value zero for offset and/or length.
>>
>> Thanks,
>> Willy
>>
>


0001-MEDIUM-sample-Enhances-converter-bytes-to-take-varia.patch
Description: Binary data

Re: Request for feedback: Add support for txn args as arguments in converter "bytes"

[Lokesh Jindal]

Thanks for the response and the corrections, Willy.

*We need to decide what to do when the variable does not*

*exist or is empty. We can't make converters fail for now, so most likelyit
will have to end up as value zero for offset and/or length*.

Here is the implementation today - link
<https://github.com/haproxy/haproxy/blob/e7d9082315e33e41347b2a7db949e995f5b7/src/sample.c#L2706>.
We set the length of the output to 0 in case of an invalid input (e.g. arg0
value >= length of the bytes in the input to the converter).
So, for all other invalid inputs (e.g. variable in arg[0] does not exist),
we can do the same.

We can discuss more after I share the patch.

Thanks
Lokesh

On Mon, Aug 28, 2023 at 2:53 AM Willy Tarreau  wrote:

> Hi Lokesh,
>
> On Fri, Aug 25, 2023 at 01:44:48PM -0700, Lokesh Jindal wrote:
> > Hey folks
> >
> > I am writing to gather feedback on an idea before doing the
> implementation.
> > Per the documentation, converter "bytes" accepts integer values as
> > arguments, but not txn args.
> > i.e. ,bytes(2,8) will work
> > but ,bytes(txn.start_idx,txn.length) will not work.
> >
> > For our use case, we need to parse some binary data (a cookie) to extract
> > some info in haproxy. However, the bytes that need to be extracted are
> not
> > fixed and will depend on the request. We could use simple arithmetic to
> > determine the index/length for bytes to be extracted and store them in
> txn
> > args. These txn args can then be used with converter "bytes".
> >
> > I can see that the converter "sub" already supports txn args as
> arguments.
> > I have successfully validated the proposed idea with an implementation
> > following the same pattern as "sub".
>
> In fact it's not "txn", it's variables in general. Most of the arithmetic
> functions support a variable name as an alternative to a numeric value. I
> tend to think it would indeed make sense to support both integers and
> variables for bytes(), it could even be the same for a few other ones
> (maybe field(), word(), ltrim(), rtrim() etc).
>
> > Let me know what you think. If there are no concerns, I can send a patch.
>
> I'm all for it. We need to decide what to do when the variable does not
> exist or is empty. We can't make converters fail for now, so most likely
> it will have to end up as value zero for offset and/or length.
>
> Thanks,
> Willy
>

Request for feedback: Add support for txn args as arguments in converter "bytes"

[Lokesh Jindal]

Hey folks

I am writing to gather feedback on an idea before doing the implementation.
Per the documentation, converter "bytes" accepts integer values as
arguments, but not txn args.
i.e. ,bytes(2,8) will work
but ,bytes(txn.start_idx,txn.length) will not work.

For our use case, we need to parse some binary data (a cookie) to extract
some info in haproxy. However, the bytes that need to be extracted are not
fixed and will depend on the request. We could use simple arithmetic to
determine the index/length for bytes to be extracted and store them in txn
args. These txn args can then be used with converter "bytes".

I can see that the converter "sub" already supports txn args as arguments.
I have successfully validated the proposed idea with an implementation
following the same pattern as "sub".

Let me know what you think. If there are no concerns, I can send a patch.

Thanks
Lokesh