Hi,
On 4/17/25 13:05, Jacques Heunis (BLOOMBERG/ LONDON) wrote:
> Hi there,
>
> I'm trying to setup a simple TCP service in lua (via
> `core.register_service()`) to accept several lines of text at a time from an
> external process and I've recently run into issues when that lua yields
> (either forcibly or intentionally).
> More specifically it appears that whenever the service function yields, the
> data that has already been received but not yet processed is lost because the
> channel's buffer gets cleared out.
> I can reliably reproduce this on 2.8.0, 3.1.0 and master as of commit 2c3d656
> (latest at the time of writing), details below.
>
> I understand that some lua contexts are not allowed to yield but its not
> clear if this is the case for applets/services. More broadly it's not clear
> to me from the documentation what context such a service runs in.
> The documentation for `core.register_service()` does not specify the context
> in which the service function will be run and while the documentation does
> mention an "applet" context, it appears only once under `core.thread`.
> So my first question is: Is a service/applet function allowed to yield?
Yes indeed a service is allowed to yield and will yield automatically
after tune.lua.forced-yield lua instructions which defaults to 10000
>
> I assume so because `AppletTCP.getline()` itself yields internally if no data
> is immediately available.
> However, if they are allowed to yield then this seems like a bug since I
> would certainly not have expected that my TCP-based service would observe
> gaps in the data it receives.
I suspect it is a bug as well, doesn't sound right
>
> I can reliably reproduce this behaviour using the following setup:
> haproxy config:
> ```
> global
> log stdout format raw local0 info
> lua-load haproxy_yieldtest.lua
>
> defaults
> log global
> timeout connect 10s
> timeout client 1m
> timeout server 1m
>
> listen echo
> bind *:9090
> mode tcp
> tcp-request content use-service lua.print_input
> ```
>
> haproxy_yieldtest.lua:
> ```
> core.register_service("print_input", "tcp", function(applet)
> core.Info("Start printing input...")
> while true do
> local inputs = applet:getline()
> if inputs == nil or string.len(inputs) == 0 then
> core.Info("closing input connection")
> return
> end
> core.Info("Received line: "..inputs)
> core.yield()
> end
> end)
> ```
>
> and then testing it by running this simple bash script:
> ```
> #!/usr/bin/bash
> for i in $(seq 1 9999); do
> for j in $(seq 1 50); do
> echo "${i}_foo_${j}"
> done
> sleep 2
> done
> ```
>
> and running it with `./test_seq.sh | netcat localhost 9090`.
Thank you very much for this reproducer, it was very helpful.
Turns out this used to work fine in older stable releases (ie: 2.4, 2.6)
and somehow it stopped working properly near 2.8 indeed
A quick bisect based on your reproducer tells us that:
> commit 31572229ede61eb4ce1ee494f4436df353165a43
> Author: Christopher Faulet <[email protected]>
> Date: Fri Mar 31 11:13:48 2023 +0200
>
> MEDIUM: hlua/applet: Use the sedesc to report and detect end of processing
>
> There are 3 kinds of applet in lua: The co-sockets, the TCP services and
> the
> HTTP services. The three are refactored to use the SE descriptor instead
> of
> the channel to report error and end-of-stream.
>
> src/hlua.c | 108
> +++++++++++++++++++++++++++----------------------------------
> 1 file changed, 48 insertions(+), 60 deletions(-)
Is the first commit that exhibits this unexpected behavior/bug.
So indeed all versions since 2.8 appear to be affected.
We'll try to understand the regression and provide a fix shortly, thank
you very much!
Regards,
Aurelien
