hahait commented on issue #14149:
URL: https://github.com/apache/apisix/issues/14149#issuecomment-6675154928
Thanks for looking into this. To add more concrete evidence from production:
### 1. Incident duration data — this is not a transient blip, it's a
permanent stuck state until manual Pod intervention
We hit this in production on four separate occasions, on different
services/clusters:
Incident A (single stale backend IP): detected at 17:56, root-caused and
resolved by manually restarting one Pod at 23:29 — ~5.5 hours of continuous,
non-decaying error rate before the manual fix. No sign of self-recovery during
that window.
Incident B (12 stale backend IPs on one Pod, triggered by a bulk scale-down
of ~85 Pods in 17 minutes): error rate measured hourly, flat at 3.95%–4.4% for
18+ consecutive hours with zero decay trend.
### 2. Why TTL does not act as a safety net here — based on our source-code
reading, would appreciate official confirmation
`endpoint_lrucache` is created with a 300s TTL and no `invalid_stale`:
```lua
local endpoint_lrucache = core.lrucache.new({
ttl = 300,
count = 1024
})
```
`lrucache.lua`:
```lua
local function fetch_valid_cache(lru_obj, invalid_stale, item_ttl,
item_release, key, version)
local obj, stale_obj = lru_obj:get(key)
if obj and obj.ver == version then
return obj
end
if not invalid_stale and stale_obj and stale_obj.ver == version then
lru_obj:set(key, stale_obj, item_ttl) -- re-arms TTL, returns the
SAME stale object
return stale_obj
end
...
end
```
`lua-resty-lrucache`'s `get()`, on expiry, returns `obj=nil` but still hands
back the old value as `stale_obj`:
```lua
if node.expire >= 0 and node.expire < ngx_now() then
return nil, val, node.user_flags
end
```
So on TTL expiry, `fetch_valid_cache` doesn't discard the entry — it checks
`stale_obj.ver == version`, and since `version` never changes again once the
write-race has poisoned the entry, this is always true. It re-arms the same TTL
on the same stale value and `create_obj_fun` (the only path that actually
re-reads shared memory) is never called.
TTL here is a resurrection timer, not a correctness mechanism — it never
forces recomputation as long as `version `matches, no matter how small it's
set. Setting `invalid_stale = true` for this cache would bound the worst case
to roughly one TTL interval instead of indefinite.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
