shreemaan-abhishek opened a new pull request, #13765:
URL: https://github.com/apache/apisix/pull/13765
### Description
In `stream_check_mode = realtime`, `ai-aws-content-moderation` and
`ai-aliyun-content-moderation` build their moderation batch from
`ctx.llm_response_contents_in_chunk`, which `ai-providers/base.lua` refills
once
per **upstream** chunk. When a protocol converter is active (e.g. an
Anthropic
client over an OpenAI upstream), that upstream chunk is dispatched to the
client
as several **converted** chunks and `lua_body_filter` runs once per converted
chunk, appending the same texts every time. Consequences:
- the batch sent to Comprehend / aliyun held the text N times over,
- `stream_check_cache_size` tripped N times sooner than configured,
- moderation request volume scaled with the converter's fan-out, adding
latency
and third-party cost on cross-protocol routes.
Measured with the fixture used by the new test: one upstream chunk fans out
into
7 Anthropic events and produced 7 identical Comprehend calls instead of 1.
Without a converter (1 upstream chunk -> 1 downstream chunk) nothing was
duplicated, which is why this went unnoticed. Moderation coverage was never
affected - the duplicated batch still contained everything.
Fix: `base.lua` bumps a per-upstream-chunk counter
(`ctx.llm_response_chunk_seq`) where it already resets
`ctx.llm_response_contents_in_chunk`; each plugin records the sequence it
last
consumed and takes the chunk's text only on its first run. With the counter
absent (callers other than the streaming loop) behaviour is unchanged.
Accumulating the `body` argument instead was considered and rejected: `body`
is
the raw SSE frame of the converted chunk, so the batch would carry the JSON
envelope rather than the response text, and `stream_check_cache_size` would
end
up measuring envelope bytes.
Tests added:
- `t/plugin/ai-aws-content-moderation.t` TEST 33 (repeated filter runs for
one
upstream chunk take its text once) and TEST 34/35 (end-to-end Anthropic
client over OpenAI upstream: Comprehend is called exactly once; fails on
master with 7 identical calls).
- `t/plugin/ai-aliyun-content-moderation.t` TEST 74 and TEST 75/76, same
shape.
Behaviour and configuration are unchanged, so no documentation update is
needed.
Follow-up to #13735, raised by @AlinsRan in review
(`discussion_r3672582793`).
#### Which issue(s) this PR fixes:
No separate issue - reported in review of #13735.
### Checklist
- [x] I have explained the need for this PR and the problem it solves
- [x] I have explained the changes or the new features added to this PR
- [x] I have added tests corresponding to this change
- [x] I have updated the documentation to reflect this change (n/a - no
user-visible behaviour or schema change)
- [x] I have verified that this change is backward compatible
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
