janiussyafiq commented on code in PR #13632:
URL: https://github.com/apache/apisix/pull/13632#discussion_r3503219457
##########
apisix/plugins/ai-cache.lua:
##########
@@ -126,16 +159,35 @@ function _M.access(conf, ctx)
ctx.ai_cache_status = "MISS"
return
end
- release(conf, red)
-
if res ~= nil and res ~= ngx_null then
local cached = core.json.decode(res)
if cached and cached.body then
+ release(conf, red)
return serve_hit(conf, ctx, cached)
end
core.log.warn("ai-cache: discarding malformed cache entry for ",
ctx.ai_cache_key)
end
+ -- L1 miss: attempt an L2 semantic lookup when the layer is configured.
+ -- `red` is still open and reused inside lookup() for the L1 backfill SET.
+ -- pcall ensures ANY throw (e.g. FFI pack failure on malformed embedding)
stays
+ -- fail-open and never leaks the connection or 500s the request.
+ if has_layer(conf, "semantic") and conf.semantic then
+ local ok, hit = pcall(semantic.lookup, red, conf, ctx, body)
Review Comment:
Agreed, after the fix I ran some QA test and found that peak conns dropped
significantly. Good catch!
<img width="553" height="130" alt="image"
src="https://github.com/user-attachments/assets/df92ad32-e185-4c73-958a-3ad68ca659c6";
/>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
