I ended up trying to give SPOE a try, since it seemed like a possible contender for this use case.
I tried to give the LUA, Python, Rust and Go examples a go, and it seems like I need as many workers/threads on the agents, as I have concurrently processed requests in haproxy. Effectively I spawned the ps_lua.loa example: === ./spoa -f ps_lua.lua SPOA is listening on port 12345 SPOA worker 01 started SPOA worker 02 started SPOA worker 03 started SPOA worker 04 started SPOA worker 05 started (string) "Load lua message processors" (string) "Load lua message processors" (string) "Load lua message processors" (string) "Load lua message processors" (string) "Load lua message processors" === If I do an "h2load -t 1 -n 10 -c 5 http://localhost/test.txt";, all requests succeed. The average response time is 262us: === finished in 3.83ms, 2610.97 req/s, 6.87MB/s requests: 10 total, 10 started, 10 done, 10 succeeded, 0 failed, 0 errored, 0 timeout status codes: 10 2xx, 0 3xx, 0 4xx, 0 5xx traffic: 26.96KB (27610) total, 7.07KB (7240) headers (space savings 9.16%), 19.54KB (20010) data min max mean sd +/- sd time for request: 307us 699us 470us 137us 80.00% time for connect: 1.08ms 2.34ms 1.77ms 480us 60.00% time to 1st byte: 3.00ms 3.14ms 3.05ms 57us 80.00% req/s : 554.62 586.64 571.08 11.46 60.00% === Now, the issue starts if I increase clients from 5 to 10. Exactly half the requests will reach the 1000ms processing timeout configured. I'm assuming this is because when haproxy tries to call the SPOE filter with the requests, the agent (being it the LUA, Python and Go examples) are not able to accept the request the millisecond haproxy tries to send it to the agent. On spoe-agent, I have `timeout hello 5s` configured. e.g. if I simply make the ps_lua.lua output "Hello World" in the spoa.register_message function, I get exactly 5 "hello world", instead of 10, and eventually after the 5 second timeout has been reached for `hello`, I get the various "failed to read/write" frames: === /spoa -f ps_lua.lua SPOA is listening on port 12345 SPOA worker 01 started SPOA worker 02 started SPOA worker 03 started SPOA worker 04 started SPOA worker 05 started (string) "Load lua message processors" (string) "Load lua message processors" (string) "Load lua message processors" (string) "Load lua message processors" (string) "Load lua message processors" hello world hello world hello world hello world hello world 1736514434.980989 [01] Failed to read Haproxy NOTIFY frame 1736514434.980992 [03] Failed to read Haproxy NOTIFY frame 1736514434.980992 [04] Failed to read Haproxy NOTIFY frame 1736514434.980992 [02] Failed to read Haproxy NOTIFY frame 1736514434.980991 [05] Failed to read Haproxy NOTIFY frame 1736514434.981003 [01] Close the client socket because of I/O errors 1736514434.981004 [03] Close the client socket because of I/O errors 1736514434.981011 [04] Close the client socket because of I/O errors 1736514434.981015 [02] Close the client socket because of I/O errors 1736514434.981016 [05] Close the client socket because of I/O errors 1736514434.981045 [01] Failed to write Agent frame 1736514434.981046 [03] Failed to write Agent frame 1736514434.981049 [01] Close the client socket because of I/O errors 1736514434.981049 [03] Close the client socket because of I/O errors 1736514434.981058 [04] Failed to write Agent frame 1736514434.981061 [02] Failed to write Agent frame 1736514434.981061 [04] Close the client socket because of I/O errors 1736514434.981061 [05] Failed to write Agent frame 1736514434.981062 [02] Close the client socket because of I/O errors 1736514434.981066 [05] Close the client socket because of I/O errors === Is the expectation of SPOE, that the agents are fully event-driven as well, to be able to handle the AGENT-HELLO frames coming from haproxy, and then get them processed (within the processing timeout). If that's the case, that obviously complicates the usage of SPOE a lot more Best Regards, Lucas Rolff > On 10 Jan 2025, at 00:11, Lucas Rolff <lu...@lucasrolff.com> wrote: > > Hello, > > I'm looking into possibilities to implement some slightly more complex logic > into haproxy that's being used when talking to origins. > Looking through the documentation, I see I can obviously use Lua, which > offers great flexibility. > > I'm looking to implement a few additional checks and balances before I > forward requests to the origin/backend for being processed. > > Two of these things are: > - validation of bearer tokens (API tokens) on haproxy level > - S3 signing > > Bearer tokens are stored in a Redis instance, and I can obviously write Lua > that's ran for a given http-request, but from what I understand, the way the > Lua is ran, means I obviously have to establish the whole Redis > instance/connection every time the code is executed, and I can't keep a > connection open. > Would it be possible for using the SPOE feature of haproxy to essentially run > a separate program, that would bridge this gap, basically being a super fast > processing engine for tokens, where I keep this persistent Redis connection > open, or are there better ways of doing it? > > Likewise for S3 signing ( I know it's supported by haproxy enterprise, but in > this particular case, I'm not there yet, to where I can justify it, sorry!) , > there's, from what I can see, quite a few ways this can be done: > - I can use Lua to do the hash calculations (again, likely store these AWS > access key/secret in Redis) > - Possibly use SPOE (assuming this is one of the possible use-cases - it > mentions ldap and SSO for example, this is just another kind of signing) > - I can do it the ugly way, and proxy the request from haproxy to a small > Golang app or similar, and then let the Go application talk to S3 backend > directly > - Proxy from haproxy to nginx, and do a bunch of logic in nginx using Lua, > and forward onto the origin. However, introducing a nginx into the stack, is > ideally something I'd like to avoid > > Any pointers to whether SPOE would make sense here, would be great, or if you > have any other suggestions, or possibly even experience doing something > similar! > > Thanks in advance! 👌 > > Best Regards, > Lucas Rolff
