Re: About the SPOE
Hey Willy Please see my response inline below. - Lokesh From: Willy Tarreau Date: Monday, March 18, 2024 at 4:08 AM To: Lokesh Jindal Cc: Abhijeet Rastogi , Christopher Faulet , haproxy@formilux.org , Aleksandar Lazic Subject: Re: About the SPOE Hi Lokesh, Abhijeet, Alex, First, thanks for jumping into this thread, the purpose of the deprecation is in a big part to try to collect the requirements of possibly existing users. Mind you that the rare times we hear about SPOE is only because of problems, so it's difficult to figure what to keep and what to cut from the existing design. More on that below: On Fri, Mar 15, 2024 at 05:15:14PM +, Lokesh Jindal wrote: > Hey Christopher > > Adding to what my colleague, Abhijeet, said. > > > 1. We plan to ramp traffic to HAProxy very soon where we will heavily rely > on SPOA. In our testing, we are satisfied with SPOE in terms of > performance. The flexibility to write SPOA in any language not only allows > us to handle "complex use cases" like connecting to non-http downstreams, > but also helps in observability - metrics and logging. Interesting, the initial internal e-mail in 2016 that ignited the SPOE design was driven from the same observations: in web environments it's quite rare to find developers who are at ease with system-level languages, yet they are the ones most likely to request to extend the proxy. For this reason we wanted to offer the possibility to call code written in other languages. In addition it was estimated that the ability to connect to the agent over the network and using secure connections was absolutely essential. It brings the ability to scale the processing engine without adding more LB nodes, and even to offload that to another DC, infrastructure or even to delegate it to a 3rd party. [Lokesh]: One of our major flows requires us to connect to couchbase for every request. Today, we are running one HAProxy process + one golang SPOA process on every host of the reverse proxy layer. This keeps the setup simple and avoids making a network call for HAProxy-SPOA communication, reducing latency. We have a couple of other flows where the SPOA connects to HTTP services for things like SSO, some type of cookie renewal, etc. These are not as latency sensitive as the flow mentioned above. Further, we anticipate the interaction of SPOA with these external services will change often – how we parse the http response from the services, how we make the request to these services, addition of new calls to services, etc. In these cases, the SPOA will give us ability to develop features fast. Among the use cases which immediately came to mind were authentication, database accesses, name resolution, "remote maps", request classification, IP reputation, etc. In addition we thought that limiting ourselves to short request/responses like this was probably limiting and that it would have been useful to support streaming so that we could implement image recompression, caching, WAFs etc. The first PoC implementation that was merged in version 1.7 lacked the streaming abilities, and it's still the current implementation. It took a while before we received feedback on it, since then caching was implemented, the demand for image recompression is close to non-existing, and WAFs users have well accommodated to dealing with extra layers by now it seems. So basically we're left with something ultra-complex that deals with short request-responses most of the time, and that suffers from the initial design which was way bigger than the use cases. [Lokesh]: I think this applies to us, as explained in my comment above. > 2. What is the best alternative to SPOE? I don't know yet, and the purpose of this deprecation precisely is to engage discussion with users. One could think about various HTTP-based protocols for which it is easier to implement a server, some gRPC maybe, possibly even a stripped-down version of the SPOP protocol if we figure that everyone is running on a reasonable subset that's much easier to deal with than the whole stuff we have now. > Two options that we are aware of > - Write fetchers/converters in lua or write filters in other languages > using the Lua API. In your experience, how do they compare to SPOE in terms > of: > * Performance > * Fault isolation The benefits of SPOE as you've found, clearly are in terms of flexibility as it allows to scale the number of analysers independently on the number of LBs, and it almost makes problems almost unnoticeable. For example, if your code relied on unstable 3rd party libraries, crashing your SPOA doesn't bring down the whole proxy. Similarly in terms of added latency, all the latency is in the external component, the rest of the traffic is not affected as it would be by placing some heavy processing directly inside the haproxy process. [Lokesh]: Yes, fault isolation is a major incentive for us to use SPOA. Flexibility to use any language for
RE: Attendee List - RSA Conference 2024
Hi, May I go ahead and send you the contacts and pricing details for your review? Looking forward to your response... Best Regards, Marla Dicandia _ From: Marla Dicandia Sent: Wednesday, March 6, 2024 9:33 AM To: haproxy@formilux.org Subject: RE: Attendee List - RSA Conference 2024 Hi - hope all is well, If you're interested to purchase the list, reply with "send counts and cost". Regards, Marla Dicandia _ From: Marla Dicandia Sent: Tuesday, March 5, 2024 4:43 PM To: haproxy@formilux.org Subject: Attendee List - RSA Conference 2024 Hi, I'm curious to know if would you be interested in The RSA Conference 2024 Attendees contacts? Attendees are - Security professionals, C-suite executives, CEOs, CISOs, CIOs, Investors, Channel partners, Security vendors, Industry analysts, Researchers and academics, & others. If you would like to know more details about this reply as, "Send counts and cost details". The information in the list includes the Company Profile, Email Address, Company Website, confirmed phone number, and so on. Awaiting your reply. Cheers, Marla Dicandia Event Attendees Specialist If you are not interested in my emails get back to me with "Takeout"
Re: [PATCH 0/2] CI entropy adjust (clang asan fix) and spell fixes
On Sun, Mar 17, 2024 at 05:01:37PM +0100, Ilia Shipitsin wrote: > couple of patches > 1) spell fixes > 2) CI sysctl to make new ubuntu kernels and asan friends again Now merged, thanks for dealing with this Ilya. I understood from the GH issue that we can hope to remove it by the end of this week, but at least if it can help avoid triggering dummy bugs, that will be a progress! Willy
RE: RSA Conference Attendees Data List 2024
Hi, ꓣ ꓣWould you be interested in acquiring RSA Conference Attendees Data List 2024 ꓣ ꓣNumber of Contacts :-30,852 ꓣCost :- $1,752 ꓣ ꓣInterested? Let me know your thoughts and advice on the next steps. ꓣKind Regards, Alice Sol
Re: Dataplane exits at haproxytech/haproxy-ubuntu:2.9 in Containers
On Sun, Mar 17, 2024 at 07:53:17PM +0100, Aleksandar Lazic wrote: > Hi. > > Looks like there was a similar question in the forum > https://discourse.haproxy.org/t/trouble-with-starting-the-data-plane-api/9200 > > Any idea how to fix this? > Honestly no idea, you should try an issue there: https://github.com/haproxytech/dataplaneapi/issues -- William Lallemand
Re: About the SPOE
Hi Lokesh, Abhijeet, Alex, First, thanks for jumping into this thread, the purpose of the deprecation is in a big part to try to collect the requirements of possibly existing users. Mind you that the rare times we hear about SPOE is only because of problems, so it's difficult to figure what to keep and what to cut from the existing design. More on that below: On Fri, Mar 15, 2024 at 05:15:14PM +, Lokesh Jindal wrote: > Hey Christopher > > Adding to what my colleague, Abhijeet, said. > > > 1. We plan to ramp traffic to HAProxy very soon where we will heavily rely > on SPOA. In our testing, we are satisfied with SPOE in terms of > performance. The flexibility to write SPOA in any language not only allows > us to handle "complex use cases" like connecting to non-http downstreams, > but also helps in observability - metrics and logging. Interesting, the initial internal e-mail in 2016 that ignited the SPOE design was driven from the same observations: in web environments it's quite rare to find developers who are at ease with system-level languages, yet they are the ones most likely to request to extend the proxy. For this reason we wanted to offer the possibility to call code written in other languages. In addition it was estimated that the ability to connect to the agent over the network and using secure connections was absolutely essential. It brings the ability to scale the processing engine without adding more LB nodes, and even to offload that to another DC, infrastructure or even to delegate it to a 3rd party. Among the use cases which immediately came to mind were authentication, database accesses, name resolution, "remote maps", request classification, IP reputation, etc. In addition we thought that limiting ourselves to short request/responses like this was probably limiting and that it would have been useful to support streaming so that we could implement image recompression, caching, WAFs etc. The first PoC implementation that was merged in version 1.7 lacked the streaming abilities, and it's still the current implementation. It took a while before we received feedback on it, since then caching was implemented, the demand for image recompression is close to non-existing, and WAFs users have well accommodated to dealing with extra layers by now it seems. So basically we're left with something ultra-complex that deals with short request-responses most of the time, and that suffers from the initial design which was way bigger than the use cases. > 2. What is the best alternative to SPOE? I don't know yet, and the purpose of this deprecation precisely is to engage discussion with users. One could think about various HTTP-based protocols for which it is easier to implement a server, some gRPC maybe, possibly even a stripped-down version of the SPOP protocol if we figure that everyone is running on a reasonable subset that's much easier to deal with than the whole stuff we have now. > Two options that we are aware of > - Write fetchers/converters in lua or write filters in other languages > using the Lua API. In your experience, how do they compare to SPOE in terms > of: > * Performance > * Fault isolation The benefits of SPOE as you've found, clearly are in terms of flexibility as it allows to scale the number of analysers independently on the number of LBs, and it almost makes problems almost unnoticeable. For example, if your code relied on unstable 3rd party libraries, crashing your SPOA doesn't bring down the whole proxy. Similarly in terms of added latency, all the latency is in the external component, the rest of the traffic is not affected as it would be by placing some heavy processing directly inside the haproxy process. > 3. As Abhijeet said, can you share a list of issues with SPOE that make it > hard to maintain? On the top of my head I can enumerate a few: - the load balancing is integraly reimplemented at the SPOE level between applets - the load balancing is affected by the support of response-less messages, which prevent haproxy from having an estimate of the server's load, which means that an algo such as leastconn would not make sense in such a condition - the idle connections management is integraly reimplemented at the SPOE level using applets as well. An idle SPOE connection is in fact a full application-layer stream with one applet on one side and a connection on the other side. These cannot be migrated between threads, which require even more complex stream-to-stream thread-safe communication mechanisms and synchronisation. - it's possible to receive a response on a different connection than the one that saw the request. This adds complexity in the matching request lookup, needs for extra synchronisation so that the other stream doesn't vanish while we're delivering the reponse, and it also makes it harder to keep track of the number of in-flight requests.
