race condition causing some corruption in lua socket service response

[Sachin Shetty]

Hi,

We are using a http-req lua action to dynamically set some app specific
metadata headers. The lua handler connects to a upstream memcache like
service over tcp to fetch additional metadata.

Here is a simplified config:


function get_from_gds(txn)
    local key = txn.sf:req_fhdr("host")
    local sock = core.tcp()
    -- Connect timeout after patch
    sock:settimeout(3)
    local result = DOMAIN_NOT_FOUND
    local status, error = sock:connect(gds_host, gds_port)
    if not status then
        core.Alert("Error in connecting:" .. key .. ":" .. error)
        return GDS_ERROR, "Error: " .. error
    end
    sock:settimeout(2)
    sock:send(key .. "\r\n")
    while true do
        local s, status, partial = sock:receive("*l")
        if s == "END" then break end
        result = s
    end
    sock:close()
    core.Alert("Returning from GDS:" .. key .. ":" .. result)
    return result
end

core.register_action("get_proxy", { "http-req" }, get_from_gds)

Functionally it works, I am load testing with 100 concurrent threads over
several hours, and like once in million I see result to be not same as what
was sent by the upstream service - I have added logging and confirmed that
the upstream server sends the correct response, but the result variable
somehow gets mixed up with a value from another concurrent running request.

Any idea? I have looked at both the lua code and the upstream service and
made sure all variables are local etc, but not able to spot anything.

Thanks
Sachin