Copilot commented on code in PR #2113:
URL: https://github.com/apache/apisix-website/pull/2113#discussion_r3870997260
##########
website/src/pages/docs.tsx:
##########
@@ -31,7 +31,34 @@ const PageTitle = styled.h1`
`;
const PageSubtitle = styled.div`
- margin-bottom: 4rem;
+ max-width: 780px;
+ margin-bottom: 1.5rem;
+ font-size: 1.1rem;
+ line-height: 1.7;
+`;
+
+const TaskNav = styled.nav`
+ display: flex;
+ flex-wrap: wrap;
+ gap: 0.75rem;
+ margin-bottom: 1.5rem;
+`;
+
+const TaskLink = styled.a`
+ padding: 0.65rem 0.9rem;
+ border: 1px solid var(--ifm-color-emphasis-300);
+ border-radius: 999px;
+ font-weight: 600;
+
+ &:hover {
+ text-decoration: none;
+ border-color: var(--ifm-color-primary);
+ }
Review Comment:
TaskLink defines a hover style but no keyboard focus indication. Add a
visible `:focus-visible` style (e.g., outline / box-shadow and/or border-color)
so keyboard users can see which pill is focused.
##########
website/src/pages/docs.tsx:
##########
@@ -187,8 +214,28 @@ const Docs: FC = () => {
<Translate id="docs.webpage.title.Document">Documentation</Translate>
</PageTitle>
<PageSubtitle>
- <Translate id="docs.webpage.title.DocumentSubtitle">We love open
source.</Translate>
+ <Translate id="docs.webpage.title.DocumentSubtitle">
+ Choose the Apache APISIX documentation set that matches your task.
Each card opens the
+ latest published documentation for that project.
+ </Translate>
</PageSubtitle>
+ <TaskNav aria-label={translate({ id: 'docs.taskNavigation.label',
message: 'Choose documentation by task' })}>
+ <TaskLink href="#apisix">
+ <Translate id="docs.taskNavigation.gateway">Start or operate
APISIX</Translate>
+ </TaskLink>
+ <TaskLink href="#ingress-controller">
+ <Translate id="docs.taskNavigation.kubernetes">Run APISIX on
Kubernetes</Translate>
+ </TaskLink>
+ <TaskLink href="#java-plugin-runner">
+ <Translate id="docs.taskNavigation.plugins">Choose an external
plugin runner</Translate>
+ </TaskLink>
+ </TaskNav>
Review Comment:
The task navigation anchors are hard-coded (`#apisix`,
`#ingress-controller`, `#java-plugin-runner`), which can silently break if the
underlying project slugs change or if cards are renamed/reordered. Consider
deriving these anchor targets from the same source of truth used to render the
cards (e.g., docs data) or centralizing them as constants tied to the project
definitions.
##########
blog/en/blog/2025/06/18/ai-gateway-future-trend-of-ai-infrastructure.md:
##########
@@ -1,192 +1,169 @@
---
-title: "AI Gateways: The Future Trend of AI Infrastructure"
+title: "AI Gateway Infrastructure: Roles, Boundaries, and Trends"
authors:
- name: Yilia Lin
title: Technical Writer
url: https://github.com/Yilialinn
image_url: https://github.com/Yilialinn.png
keywords:
- - API gateway
- - AI middleware
- - API gateway vs AI gateway
- - AI governance
- - AI cost control
- - AI security
- - APISIX AI gateway
-description: "Explore AI Gateway infrastructure trends and how Apache APISIX
can help manage LLM traffic, model routing, token limits, and AI application
security."
+ - AI gateway infrastructure
+ - AI infrastructure gateway
+ - AI gateway trends
+ - AI gateway market
+ - LLM gateway
+ - Apache APISIX AI gateway
+description: "Understand where an AI gateway fits in AI infrastructure, which
controls belong at the gateway, what remains elsewhere, and how to assess
adoption trends."
tags: [Ecosystem]
image:
https://static.api7.ai/uploads/2025/03/07/Qs4WrU0I_apisix-ai-gateway.webp
---
-> Discover how AI gateways are revolutionizing enterprise AI infrastructure,
offering centralized control, security, cost management, and governance for AI
models and services.
-<!--truncate-->
-
-## AI Infrastructure Revolution
-
-The enterprise AI landscape has exploded into fragmented chaos. Marketing
teams deploy GPT-4 for content generation, developers fine-tune Llama 3 for
coding assistants, while legal departments rely on Claude 3 for contract
analysis. This siloed adoption creates three critical pain points:
-
-1. **Security Vulnerabilities**: 68% of enterprises report unauthorized AI
tool usage leading to PII leaks (Gartner 2025)
-2. **Cost Overruns**: Unmonitored token consumption causes 41% of companies to
exceed AI budgets by 200%+ (McKinsey)
-3. **Governance Failure**: 83% of compliance violations trace to inconsistent
AI policy enforcement (Deloitte Audit Report)
+> An AI gateway can provide a controlled network path to model providers, but
it is only one part of production AI infrastructure. Its useful scope is
traffic policy, provider access, usage controls, and gateway-level
telemetry—not model evaluation, agent orchestration, or compliance by itself.
-Enter **AI gateways**—the middleware revolution transforming enterprise AI
from experimental tools to production-grade infrastructure. These systems
consolidate fragmented AI interactions through a unified control layer, much
like Kubernetes did for container orchestration. An AI gateway is a specialized
middleware layer that manages and secures interactions between your
applications and AI models, such as **OpenAI**'s offerings. This technology,
akin to an **API gateway**, provides visibility and control over your AI
applications. The future of AI infrastructure is increasingly modular, enabling
flexible and robust machine learning teams.
-
-## What Is an AI Gateway
-
-An [AI
gateway](https://apisix.apache.org/blog/2025/03/06/what-is-an-ai-gateway/) is a
middleware platform designed to manage and facilitate the integration and
deployment of artificial intelligence models and services, such as OpenAI,
Anthropic, Gemini, etc. It acts as a bridge between AI models and the
applications that use them, simplifying integration and deployment, especially
for large language models. Essentially, an AI gateway serves as a crucial
control point for managing AI services within an organization. It also plays a
vital role in security by inspecting inbound prompts and outbound responses to
prevent data leaks and mitigate risks within the AI application workflow.
-
-
+<!--truncate-->
-## AI Gateway vs API Gateway: Critical Differences
+Organizations often begin with direct calls from an application to one model
API. As the number of applications, teams, and providers grows, that approach
can make credentials, usage policies, and operational evidence inconsistent. An
**AI gateway infrastructure** layer can provide a shared enforcement point for
traffic that already passes through it.
-While [AI gateways and API
gateways](https://apisix.apache.org/blog/2025/03/21/ai-gateway-vs-api-gateway-differences-explained/)
share some infrastructure-level similarities, they differ significantly in
purpose, functionality, and optimization.
+That does not make the gateway the center of every AI system. A production
design still needs clear owners for application authorization, retrieval, model
evaluation, workflow state, data governance, and incident response. This
article explains the gateway's practical role, its boundaries, and the adoption
signals worth evaluating without relying on market-size forecasts.
-| Feature | AI Gateway | API Gateway |
-|---------|------------|-------------|
-| Primary Use Case | Managing, securing, and optimizing traffic to AI/LLM
services (e.g., OpenAI, Anthropic, custom models) | Routing and securing
general-purpose REST/gRPC APIs for web, mobile, and microservices |
-| Request Characteristics | Often large payloads (e.g., prompts), streaming
input/output, expensive per-call | Lightweight, transactional HTTP/gRPC
requests |
-| Cost Awareness | Tracks tokens, usage costs, and budget limits per user/app
| Generally unaware of downstream compute or pricing costs |
-| Observability Needs | Input/output tracing, latency + token logging,
hallucination detection | Standard request logs, metrics (latency, throughput,
error rate) |
-| Security Features | PII redaction, prompt inspection, AI-specific abuse
filters | OAuth, JWT, IP allowlists, rate limiting |
-| Optimization Techniques | Caching AI responses, model fallback, prompt
standardization, and dynamic routing by cost or latency | Load balancing,
circuit breaking, and service discovery |
-| Plugin Support | AI-specific (e.g., pre-/post-processing, moderation,
reranking) | General plugins (e.g., auth, logging, CORS) |
-| Streaming Support | Critical: supports real-time token streaming from LLMs |
Optional: typically used for HTTP/2 or WebSocket |
-| Governance Controls | Usage quotas, cost controls, and team-level
restrictions for AI services | API-level access controls, usage policies per
role/team |
-| Integration Targets | LLM APIs (e.g., OpenAI, Anthropic, local models like
Llama), AI agents, RAG systems | Microservices, internal APIs, public-facing
APIs |
+## Key Takeaways
-**Summary of Key Distinctions**:
+- An AI gateway is a traffic intermediary for model and AI-service calls, not
an AI application runtime.
+- High-value gateway controls include client authentication, provider
credential isolation, request limits, model routing, bounded fallback, usage
accounting, and transport-level telemetry.
+- Prompt inspection and content filtering are useful policy inputs, but they
do not prove that a response is correct, safe, or compliant.
+- Provider APIs differ in request schemas, streaming behavior, token
reporting, error semantics, and pricing. A common endpoint reduces some client
coupling but does not erase those differences.
+- The right evaluation starts from explicit failure modes and responsibility
boundaries, not from a checklist that assumes every product implements the same
behavior.
-- **Focus**: AI gateways specialize in **intelligent traffic management for AI
models**, while AI gateways focus on standard API traffic orchestration.
-- **Observability**: AI gateways require **fine-grained monitoring**,
including cost and token-level visibility.
-- **Security**: AI gateways offer **general web security**, whereas AI
gateways need **content-level protections** (e.g., for prompt injection).
-- **Optimization**: AI gateways can **route based on AI-specific metrics**
(e.g., model latency, accuracy, cost), unlike traditional AI gateways.
+## Where an AI Gateway Fits
-
+An [AI
gateway](https://apisix.apache.org/blog/2025/03/06/what-is-an-ai-gateway/) sits
on the request path between authorized clients and one or more model or
AI-service endpoints. Depending on the implementation, it can apply general API
gateway policies and AI-specific processing before forwarding a request.
-## Why AI Gateways Are Essential for Enterprises?
+The traffic path and adjacent responsibilities are:
-In a world where AI adoption is accelerating, AI gateways offer a **critical
layer of control, visibility, and governance**. They enable enterprises to
confidently integrate AI into their systems securely, scalably, and sustainably.
+1. An application or agent runtime sends an authenticated model request to the
AI gateway.
+2. The gateway applies configured traffic policy and sends a provider-specific
request to a managed or private model endpoint.
+3. The gateway emits approved metrics and protected logs.
+4. Retrieval, tools, and workflow state remain connected to the application
runtime rather than moving into the gateway.
+5. Evaluation and governance systems provide reviewed policy and evidence to
the application and gateway configuration processes; they are not inline model
proxies by default.
-**You need an AI gateway when:**
+The application or agent runtime still decides why a model is called, which
tools may be used, and how results affect business state. Retrieval systems own
document selection and authorization. Evaluation systems measure quality and
safety against defined test cases. The gateway controls only the traffic and
context it can observe.
-- You're using LLMs or AI APIs in production (e.g., OpenAI, Claude, Gemini).
-- You want **centralized governance and cost control** over AI usage.
-- You need **security and content moderation** for AI prompts/responses.
-- You must **support multiple models** with fallback or dynamic routing.
+This distinction matters because many AI risks occur outside the network hop.
A gateway cannot determine whether retrieved documents were authorized
correctly, whether an agent's plan is valid, or whether a generated answer is
factually correct unless another trusted component supplies that evidence.
-Here's a breakdown of **why AI gateways are crucial** for modern enterprises:
+## Responsibilities That Fit the Gateway
-### 1. Centralized Control for AI Services
+### 1. Client Identity and Provider Credential Isolation
-Enterprises today adopt multiple AI models (e.g., OpenAI, Hugging Face,
internal LLMs) across cloud and on-prem environments. An AI gateway provides:
+The gateway can authenticate calling applications or workloads and apply
route-level authorization before a provider request is made. It can also keep
provider credentials out of distributed clients by adding the upstream
credential at the trusted gateway boundary.
-- **Routing logic** based on cost, latency, or use case.
-- **Model versioning** to avoid breaking downstream systems.
-- **Fallback mechanisms** (e.g., if GPT-4 fails, fall back to Claude).
+This design is not a substitute for business authorization. An upstream
application still has to decide whether a user may access a particular record,
tool, or action. Public browser and mobile clients should not receive a shared
provider secret.
-
+Request headers require deliberate handling. Some AI proxy implementations
forward client headers unless they are removed or overwritten. Before sending
traffic to a third-party provider, define and test an outbound header policy so
cookies, internal identity headers, and unrelated authorization values do not
cross the provider boundary.
-### 2. Security and Compliance
+### 2. Model Routing and Bounded Fallback
-AI gateways serve as security enforcement layers:
+A gateway may select an upstream by configured provider, model, priority,
weight, health signal, or another supported rule. This can centralize endpoint
changes and reduce duplicated routing code.
-- **Rate limiting and quota management** to control the usage of costly LLM
APIs.
-- **Authentication & Authorization** for internal and external consumers.
-- **PII masking and data redaction** to ensure data privacy before reaching
LLMs.
-- **Audit logs** to support compliance (e.g., GDPR, SOC 2).
+Fallback must remain bounded. Retrying a non-idempotent tool action or
replaying a large request across providers can increase cost or produce
duplicate effects. Different providers can also return materially different
answers. Define which errors are eligible, cap attempts and time, preserve an
end-to-end deadline, and expose the selected provider and fallback reason in
telemetry.
-### 3. Observability and Monitoring
+The gateway should not choose a model based on an unverified claim of answer
quality. Quality-based routing requires an evaluation method, current evidence,
and an owner outside the request proxy.
-Visibility is critical when running generative AI workloads:
+### 3. Request, Token, and Budget Controls
-- **Logging inputs/outputs and response times** for debugging.
-- **Tracing** to understand latency bottlenecks.
-- **Monitoring token usage and cost** for budget optimization.
+General request-rate and concurrency limits protect gateway and upstream
capacity. AI-aware controls can additionally use reported prompt, completion,
or total tokens when the selected integration exposes those values.
-### 4. Performance Optimization
+Token limits are not automatically financial budgets. Provider prices can vary
by model, region, cache state, batch mode, and contract. If cost allocation
matters, keep a versioned price source, record the model and usage dimensions
needed for reconciliation, and compare gateway records with provider billing
data. Do not use a best-effort in-memory counter or log queue as the financial
system of record.
-AI gateways can significantly improve efficiency:
+### 4. Gateway-Level Observability
-- **Caching responses** to avoid redundant LLM calls.
-- **Load balancing** across multiple AI model endpoints.
-- **Streaming support** for faster UX in chat applications.
+Useful gateway signals include:
-### 5. Cost Control and Governance
+- request count, status, and latency;
+- time to first token or response for streaming requests, as exposed by the
integration;
+- selected provider and model;
+- reported prompt and completion tokens;
+- retries, fallbacks, and limit rejections; and
+- connection termination or response-size limits.
-With AI APIs costing per-token or per-call, an AI gateway enables:
+Prompt and response bodies may contain personal, confidential, or regulated
data. Payload logging should be off by default unless there is a reviewed
purpose, redaction policy, access boundary, and retention period. Sampling and
redaction also need negative tests; a log statement saying that data is
protected is not evidence that secrets cannot reach a sink.
-- **Usage policies per team or app** to prevent budget overages.
-- **Token counting and cost attribution** for internal chargebacks.
-- **Auto-throttling** or alerting based on budget thresholds.
+### 5. Narrow, Testable Content Policies
-### 6. Flexibility for Hybrid/Multi-Cloud AI
+Some gateways can reject inputs using allow/deny patterns or call an external
moderation service. These controls can block known formats or policy
categories, but they have false-positive and false-negative behavior.
-AI workloads are often hybrid (cloud + on-prem) or multi-cloud. An AI gateway:
+A regular-expression prompt guard is not a semantic prompt-injection detector.
A moderation response is not proof of factual accuracy. Treat these controls as
one layer in a larger application safety design, with explicit failure behavior
when the policy service is slow or unavailable.
-- Supports **traffic routing across environments**.
-- Helps abstract away vendor-specific endpoints.
-- Allows **easy swapping of model providers** without rewriting client code.
+## What Remains Outside the Gateway
-### 7. Plugin Ecosystem for AI Use Cases
+The following responsibilities usually belong elsewhere:
-Advanced AI gateways support plugins for:
+- **Agent planning and durable workflow state:** an agent runtime or workflow
engine owns steps, approvals, compensation, and recovery.
+- **Retrieval authorization:** the application and retrieval layer decide
which documents and vector records a principal may access.
+- **Model and prompt evaluation:** an evaluation system measures quality,
robustness, and regressions using representative tests.
+- **Human approval:** business owners define which actions require review and
how an approval is recorded.
+- **Data lifecycle governance:** source systems and governance teams own
classification, residency, deletion, and legal requirements.
+- **Provider availability and billing truth:** provider APIs and billing
exports remain authoritative for their service behavior and charges.
-- **Prompt templating and standardization**
-- **Content moderation (e.g., toxicity detection)**
-- **Custom pre- and post-processing**
+An AI gateway can enforce a reviewed decision at the traffic boundary. It
should not silently become the decision maker for controls that require
business context it does not have.
-## Trends Shaping AI Gateways
+## Apache APISIX as an Implementation Example
-Here's a comprehensive look at the **trends shaping AI gateways** in 2025 and
beyond, driven by advancements in large language models (LLMs), multi-model
architectures, enterprise governance demands, and the need for scalable, secure
AI infrastructure.
+Apache APISIX combines general gateway plugins with AI-specific plugins. The
exact schema and behavior depend on the APISIX release, so verify the
documentation for the version you run.
-### 1. Multi-Model Routing and Federation
+- [`ai-proxy`](https://apisix.apache.org/docs/apisix/plugins/ai-proxy/)
converts supported request formats for documented model providers and can
expose model, token, duration, and time-to-first-token fields to access logs.
+-
[`ai-proxy-multi`](https://apisix.apache.org/docs/apisix/plugins/ai-proxy-multi/)
supports multiple configured model instances with documented load-balancing
and fallback behavior. When a failed upstream is retried, the current plugin
records that instance's error body in the error log. Treat those logs as
potentially sensitive and review their access, export, and retention before
enabling fallback.
+-
[`ai-rate-limiting`](https://apisix.apache.org/docs/apisix/plugins/ai-rate-limiting/)
can apply token-based limits using local or supported Redis policies. Counter
availability and any degradation setting are part of the enforcement decision.
+-
[`ai-prompt-guard`](https://apisix.apache.org/docs/apisix/plugins/ai-prompt-guard/)
applies configured allow and deny patterns to recognized prompt formats. Its
`fail_mode` controls unrecognized traffic and defaults to `skip`; its scope is
pattern matching, not general semantic safety classification.
-Modern AI apps increasingly call multiple models—OpenAI for coding, Claude for
summarization, open-source LLMs for privacy.
+General authentication, request transformation, traffic control, and logging
plugins can be composed with these features. Current `ai-proxy` behavior
forwards client headers other than `Host`, `Content-Length`, and
`Accept-Encoding` unless they are removed or overwritten. Strip cookies and
unrelated authorization or internal identity headers before the provider
request. Composition still requires testing of plugin order, identity
variables, outbound headers, streaming, error paths, and sensitive logs. A
plugin being available does not mean it is enabled or correctly configured on
every route.
Review Comment:
The post correctly notes behavior varies by APISIX release, but then states
'Current `ai-proxy` behavior...' without pinning the version. To avoid future
doc drift (and to align with the PR goal of verifying against APISIX 3.18.0),
update this sentence (and similar behavior claims nearby) to explicitly say 'As
of APISIX 3.18.0...' or otherwise qualify the exact version/conditions under
which the behavior was observed.
##########
blog/en/blog/2025/06/18/ai-gateway-future-trend-of-ai-infrastructure.md:
##########
@@ -1,192 +1,169 @@
---
-title: "AI Gateways: The Future Trend of AI Infrastructure"
+title: "AI Gateway Infrastructure: Roles, Boundaries, and Trends"
authors:
- name: Yilia Lin
title: Technical Writer
url: https://github.com/Yilialinn
image_url: https://github.com/Yilialinn.png
keywords:
- - API gateway
- - AI middleware
- - API gateway vs AI gateway
- - AI governance
- - AI cost control
- - AI security
- - APISIX AI gateway
-description: "Explore AI Gateway infrastructure trends and how Apache APISIX
can help manage LLM traffic, model routing, token limits, and AI application
security."
+ - AI gateway infrastructure
+ - AI infrastructure gateway
+ - AI gateway trends
+ - AI gateway market
+ - LLM gateway
+ - Apache APISIX AI gateway
+description: "Understand where an AI gateway fits in AI infrastructure, which
controls belong at the gateway, what remains elsewhere, and how to assess
adoption trends."
tags: [Ecosystem]
image:
https://static.api7.ai/uploads/2025/03/07/Qs4WrU0I_apisix-ai-gateway.webp
---
-> Discover how AI gateways are revolutionizing enterprise AI infrastructure,
offering centralized control, security, cost management, and governance for AI
models and services.
-<!--truncate-->
-
-## AI Infrastructure Revolution
-
-The enterprise AI landscape has exploded into fragmented chaos. Marketing
teams deploy GPT-4 for content generation, developers fine-tune Llama 3 for
coding assistants, while legal departments rely on Claude 3 for contract
analysis. This siloed adoption creates three critical pain points:
-
-1. **Security Vulnerabilities**: 68% of enterprises report unauthorized AI
tool usage leading to PII leaks (Gartner 2025)
-2. **Cost Overruns**: Unmonitored token consumption causes 41% of companies to
exceed AI budgets by 200%+ (McKinsey)
-3. **Governance Failure**: 83% of compliance violations trace to inconsistent
AI policy enforcement (Deloitte Audit Report)
+> An AI gateway can provide a controlled network path to model providers, but
it is only one part of production AI infrastructure. Its useful scope is
traffic policy, provider access, usage controls, and gateway-level
telemetry—not model evaluation, agent orchestration, or compliance by itself.
-Enter **AI gateways**—the middleware revolution transforming enterprise AI
from experimental tools to production-grade infrastructure. These systems
consolidate fragmented AI interactions through a unified control layer, much
like Kubernetes did for container orchestration. An AI gateway is a specialized
middleware layer that manages and secures interactions between your
applications and AI models, such as **OpenAI**'s offerings. This technology,
akin to an **API gateway**, provides visibility and control over your AI
applications. The future of AI infrastructure is increasingly modular, enabling
flexible and robust machine learning teams.
-
-## What Is an AI Gateway
-
-An [AI
gateway](https://apisix.apache.org/blog/2025/03/06/what-is-an-ai-gateway/) is a
middleware platform designed to manage and facilitate the integration and
deployment of artificial intelligence models and services, such as OpenAI,
Anthropic, Gemini, etc. It acts as a bridge between AI models and the
applications that use them, simplifying integration and deployment, especially
for large language models. Essentially, an AI gateway serves as a crucial
control point for managing AI services within an organization. It also plays a
vital role in security by inspecting inbound prompts and outbound responses to
prevent data leaks and mitigate risks within the AI application workflow.
-
-
+<!--truncate-->
-## AI Gateway vs API Gateway: Critical Differences
+Organizations often begin with direct calls from an application to one model
API. As the number of applications, teams, and providers grows, that approach
can make credentials, usage policies, and operational evidence inconsistent. An
**AI gateway infrastructure** layer can provide a shared enforcement point for
traffic that already passes through it.
-While [AI gateways and API
gateways](https://apisix.apache.org/blog/2025/03/21/ai-gateway-vs-api-gateway-differences-explained/)
share some infrastructure-level similarities, they differ significantly in
purpose, functionality, and optimization.
+That does not make the gateway the center of every AI system. A production
design still needs clear owners for application authorization, retrieval, model
evaluation, workflow state, data governance, and incident response. This
article explains the gateway's practical role, its boundaries, and the adoption
signals worth evaluating without relying on market-size forecasts.
-| Feature | AI Gateway | API Gateway |
-|---------|------------|-------------|
-| Primary Use Case | Managing, securing, and optimizing traffic to AI/LLM
services (e.g., OpenAI, Anthropic, custom models) | Routing and securing
general-purpose REST/gRPC APIs for web, mobile, and microservices |
-| Request Characteristics | Often large payloads (e.g., prompts), streaming
input/output, expensive per-call | Lightweight, transactional HTTP/gRPC
requests |
-| Cost Awareness | Tracks tokens, usage costs, and budget limits per user/app
| Generally unaware of downstream compute or pricing costs |
-| Observability Needs | Input/output tracing, latency + token logging,
hallucination detection | Standard request logs, metrics (latency, throughput,
error rate) |
-| Security Features | PII redaction, prompt inspection, AI-specific abuse
filters | OAuth, JWT, IP allowlists, rate limiting |
-| Optimization Techniques | Caching AI responses, model fallback, prompt
standardization, and dynamic routing by cost or latency | Load balancing,
circuit breaking, and service discovery |
-| Plugin Support | AI-specific (e.g., pre-/post-processing, moderation,
reranking) | General plugins (e.g., auth, logging, CORS) |
-| Streaming Support | Critical: supports real-time token streaming from LLMs |
Optional: typically used for HTTP/2 or WebSocket |
-| Governance Controls | Usage quotas, cost controls, and team-level
restrictions for AI services | API-level access controls, usage policies per
role/team |
-| Integration Targets | LLM APIs (e.g., OpenAI, Anthropic, local models like
Llama), AI agents, RAG systems | Microservices, internal APIs, public-facing
APIs |
+## Key Takeaways
-**Summary of Key Distinctions**:
+- An AI gateway is a traffic intermediary for model and AI-service calls, not
an AI application runtime.
+- High-value gateway controls include client authentication, provider
credential isolation, request limits, model routing, bounded fallback, usage
accounting, and transport-level telemetry.
+- Prompt inspection and content filtering are useful policy inputs, but they
do not prove that a response is correct, safe, or compliant.
+- Provider APIs differ in request schemas, streaming behavior, token
reporting, error semantics, and pricing. A common endpoint reduces some client
coupling but does not erase those differences.
+- The right evaluation starts from explicit failure modes and responsibility
boundaries, not from a checklist that assumes every product implements the same
behavior.
-- **Focus**: AI gateways specialize in **intelligent traffic management for AI
models**, while AI gateways focus on standard API traffic orchestration.
-- **Observability**: AI gateways require **fine-grained monitoring**,
including cost and token-level visibility.
-- **Security**: AI gateways offer **general web security**, whereas AI
gateways need **content-level protections** (e.g., for prompt injection).
-- **Optimization**: AI gateways can **route based on AI-specific metrics**
(e.g., model latency, accuracy, cost), unlike traditional AI gateways.
+## Where an AI Gateway Fits
-
+An [AI
gateway](https://apisix.apache.org/blog/2025/03/06/what-is-an-ai-gateway/) sits
on the request path between authorized clients and one or more model or
AI-service endpoints. Depending on the implementation, it can apply general API
gateway policies and AI-specific processing before forwarding a request.
-## Why AI Gateways Are Essential for Enterprises?
+The traffic path and adjacent responsibilities are:
-In a world where AI adoption is accelerating, AI gateways offer a **critical
layer of control, visibility, and governance**. They enable enterprises to
confidently integrate AI into their systems securely, scalably, and sustainably.
+1. An application or agent runtime sends an authenticated model request to the
AI gateway.
+2. The gateway applies configured traffic policy and sends a provider-specific
request to a managed or private model endpoint.
+3. The gateway emits approved metrics and protected logs.
+4. Retrieval, tools, and workflow state remain connected to the application
runtime rather than moving into the gateway.
+5. Evaluation and governance systems provide reviewed policy and evidence to
the application and gateway configuration processes; they are not inline model
proxies by default.
-**You need an AI gateway when:**
+The application or agent runtime still decides why a model is called, which
tools may be used, and how results affect business state. Retrieval systems own
document selection and authorization. Evaluation systems measure quality and
safety against defined test cases. The gateway controls only the traffic and
context it can observe.
-- You're using LLMs or AI APIs in production (e.g., OpenAI, Claude, Gemini).
-- You want **centralized governance and cost control** over AI usage.
-- You need **security and content moderation** for AI prompts/responses.
-- You must **support multiple models** with fallback or dynamic routing.
+This distinction matters because many AI risks occur outside the network hop.
A gateway cannot determine whether retrieved documents were authorized
correctly, whether an agent's plan is valid, or whether a generated answer is
factually correct unless another trusted component supplies that evidence.
-Here's a breakdown of **why AI gateways are crucial** for modern enterprises:
+## Responsibilities That Fit the Gateway
-### 1. Centralized Control for AI Services
+### 1. Client Identity and Provider Credential Isolation
-Enterprises today adopt multiple AI models (e.g., OpenAI, Hugging Face,
internal LLMs) across cloud and on-prem environments. An AI gateway provides:
+The gateway can authenticate calling applications or workloads and apply
route-level authorization before a provider request is made. It can also keep
provider credentials out of distributed clients by adding the upstream
credential at the trusted gateway boundary.
-- **Routing logic** based on cost, latency, or use case.
-- **Model versioning** to avoid breaking downstream systems.
-- **Fallback mechanisms** (e.g., if GPT-4 fails, fall back to Claude).
+This design is not a substitute for business authorization. An upstream
application still has to decide whether a user may access a particular record,
tool, or action. Public browser and mobile clients should not receive a shared
provider secret.
-
+Request headers require deliberate handling. Some AI proxy implementations
forward client headers unless they are removed or overwritten. Before sending
traffic to a third-party provider, define and test an outbound header policy so
cookies, internal identity headers, and unrelated authorization values do not
cross the provider boundary.
-### 2. Security and Compliance
+### 2. Model Routing and Bounded Fallback
-AI gateways serve as security enforcement layers:
+A gateway may select an upstream by configured provider, model, priority,
weight, health signal, or another supported rule. This can centralize endpoint
changes and reduce duplicated routing code.
-- **Rate limiting and quota management** to control the usage of costly LLM
APIs.
-- **Authentication & Authorization** for internal and external consumers.
-- **PII masking and data redaction** to ensure data privacy before reaching
LLMs.
-- **Audit logs** to support compliance (e.g., GDPR, SOC 2).
+Fallback must remain bounded. Retrying a non-idempotent tool action or
replaying a large request across providers can increase cost or produce
duplicate effects. Different providers can also return materially different
answers. Define which errors are eligible, cap attempts and time, preserve an
end-to-end deadline, and expose the selected provider and fallback reason in
telemetry.
-### 3. Observability and Monitoring
+The gateway should not choose a model based on an unverified claim of answer
quality. Quality-based routing requires an evaluation method, current evidence,
and an owner outside the request proxy.
-Visibility is critical when running generative AI workloads:
+### 3. Request, Token, and Budget Controls
-- **Logging inputs/outputs and response times** for debugging.
-- **Tracing** to understand latency bottlenecks.
-- **Monitoring token usage and cost** for budget optimization.
+General request-rate and concurrency limits protect gateway and upstream
capacity. AI-aware controls can additionally use reported prompt, completion,
or total tokens when the selected integration exposes those values.
-### 4. Performance Optimization
+Token limits are not automatically financial budgets. Provider prices can vary
by model, region, cache state, batch mode, and contract. If cost allocation
matters, keep a versioned price source, record the model and usage dimensions
needed for reconciliation, and compare gateway records with provider billing
data. Do not use a best-effort in-memory counter or log queue as the financial
system of record.
-AI gateways can significantly improve efficiency:
+### 4. Gateway-Level Observability
-- **Caching responses** to avoid redundant LLM calls.
-- **Load balancing** across multiple AI model endpoints.
-- **Streaming support** for faster UX in chat applications.
+Useful gateway signals include:
-### 5. Cost Control and Governance
+- request count, status, and latency;
+- time to first token or response for streaming requests, as exposed by the
integration;
+- selected provider and model;
+- reported prompt and completion tokens;
+- retries, fallbacks, and limit rejections; and
+- connection termination or response-size limits.
-With AI APIs costing per-token or per-call, an AI gateway enables:
+Prompt and response bodies may contain personal, confidential, or regulated
data. Payload logging should be off by default unless there is a reviewed
purpose, redaction policy, access boundary, and retention period. Sampling and
redaction also need negative tests; a log statement saying that data is
protected is not evidence that secrets cannot reach a sink.
-- **Usage policies per team or app** to prevent budget overages.
-- **Token counting and cost attribution** for internal chargebacks.
-- **Auto-throttling** or alerting based on budget thresholds.
+### 5. Narrow, Testable Content Policies
-### 6. Flexibility for Hybrid/Multi-Cloud AI
+Some gateways can reject inputs using allow/deny patterns or call an external
moderation service. These controls can block known formats or policy
categories, but they have false-positive and false-negative behavior.
-AI workloads are often hybrid (cloud + on-prem) or multi-cloud. An AI gateway:
+A regular-expression prompt guard is not a semantic prompt-injection detector.
A moderation response is not proof of factual accuracy. Treat these controls as
one layer in a larger application safety design, with explicit failure behavior
when the policy service is slow or unavailable.
-- Supports **traffic routing across environments**.
-- Helps abstract away vendor-specific endpoints.
-- Allows **easy swapping of model providers** without rewriting client code.
+## What Remains Outside the Gateway
-### 7. Plugin Ecosystem for AI Use Cases
+The following responsibilities usually belong elsewhere:
-Advanced AI gateways support plugins for:
+- **Agent planning and durable workflow state:** an agent runtime or workflow
engine owns steps, approvals, compensation, and recovery.
+- **Retrieval authorization:** the application and retrieval layer decide
which documents and vector records a principal may access.
+- **Model and prompt evaluation:** an evaluation system measures quality,
robustness, and regressions using representative tests.
+- **Human approval:** business owners define which actions require review and
how an approval is recorded.
+- **Data lifecycle governance:** source systems and governance teams own
classification, residency, deletion, and legal requirements.
+- **Provider availability and billing truth:** provider APIs and billing
exports remain authoritative for their service behavior and charges.
-- **Prompt templating and standardization**
-- **Content moderation (e.g., toxicity detection)**
-- **Custom pre- and post-processing**
+An AI gateway can enforce a reviewed decision at the traffic boundary. It
should not silently become the decision maker for controls that require
business context it does not have.
-## Trends Shaping AI Gateways
+## Apache APISIX as an Implementation Example
-Here's a comprehensive look at the **trends shaping AI gateways** in 2025 and
beyond, driven by advancements in large language models (LLMs), multi-model
architectures, enterprise governance demands, and the need for scalable, secure
AI infrastructure.
+Apache APISIX combines general gateway plugins with AI-specific plugins. The
exact schema and behavior depend on the APISIX release, so verify the
documentation for the version you run.
Review Comment:
The post correctly notes behavior varies by APISIX release, but then states
'Current `ai-proxy` behavior...' without pinning the version. To avoid future
doc drift (and to align with the PR goal of verifying against APISIX 3.18.0),
update this sentence (and similar behavior claims nearby) to explicitly say 'As
of APISIX 3.18.0...' or otherwise qualify the exact version/conditions under
which the behavior was observed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
