andrewmusselman opened a new issue, #45:
URL: https://github.com/apache/tooling-gofannon/issues/45
The key facts for designing the caching patch:
* messages is passed straight into kwargs and on to
litellm.acompletion(**kwargs) untouched (line ~98). So if a caller puts
cache_control blocks into message content, litellm already forwards them to
Bedrock — no call_llm change is strictly required for caching to work, if the
message content is in the structured block form.
* But every agent currently passes messages=[{"role":"user","content": "<big
string>"}] — a plain string, which can't carry cache_control.
* litellm.drop_params = True is set, which matters: unknown params get
dropped, but cache_control inside content blocks is not a top-level param, so
it survives.
So the cleanest, lowest-risk design is: add an optional cache_system_prefix
parameter to call_llm that, when provided, restructures the first message's
content into a cacheable block + the volatile remainder. This confines all the
Bedrock cache-block knowledge to one place and lets agents opt in with a single
kwarg, rather than every agent learning the block format.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
