kiwiflyer opened a new pull request, #14733:
URL: https://github.com/apache/cloudstack/pull/14733
### Description
**This is a work in progress**
This PR adds **CloudStack AI Services**: a native API plugin
(`plugins/api/ai-services`)
and a `ui/` section that let an operator source open-weight LLMs, deploy
them on NVIDIA
GPU capacity, tune them through profiles, and sell them to tenants as
OpenAI-compatible
endpoints.
The plugin is the front door only — auth, RBAC, async jobs, events, usage
and limits. It
delegates to an external reconciler over mTLS, which is the component that
touches GPU
resources, CKS and the serving layer. **That engine is not in this PR**,
which is the main
thing I would like feedback on; see "Open questions" below.
What it contains -
- **58 new API commands** — 38 v1, registered always, and 20 v2, registered
only when the
global setting `ai.services.v2.enabled` is on (default **false**), so a GA
install does
not advertise an API for features it is not running.
- **A new UI section** of 19 screens — model catalogue, endpoints with
scoped API keys and
copy-ready snippets, a playground, generation profiles and
effective-parameter
resolution, usage and budgets, plus admin screens for setup preflight, GPU
capacity and
the model registry. Routes carry a `permission`, so CloudStack drops any
screen whose
command the caller cannot call, and the Tier 2 screens appear and
disappear with that one
setting.
- **Ten `ai.services.*` ConfigKeys**, including the engine URLs and its mTLS
material.
- **Default role access is declared on `@APICommand(authorized = …)`**, so
there is no
migration, no `commands.properties` and no operator step. Explicit
`role_permissions`
rules still win.
The change is **additive**: 158 files, 19,207 insertions and a single
deletion (a comma, to
append to `en.json`). Only four existing files are touched —
`plugins/pom.xml`,
`client/pom.xml`, `ui/src/config/router.js`, and
`ui/public/locales/en.json`, into which 150
keys are appended without reordering or reformatting anything already there.
### Where the rest of it lives
Everything this plugin talks to is Apache-2.0 and public, so the serving
layer can be built
and the design read is available here:
**https://github.com/kiwiflyer/CloudStack_AI_Plugin**
| | |
| --- | --- |
|
[`SPEC.md`](https://github.com/kiwiflyer/CloudStack_AI_Plugin/blob/main/SPEC.md)
| The technical specification. Section 9 is the command map this plugin is
generated from, section 10 the engine REST contract it calls. |
|
[`docs/installation.md`](https://github.com/kiwiflyer/CloudStack_AI_Plugin/blob/main/docs/installation.md)
| How to stand the whole thing up: the four pieces and their order, every
config key and environment variable, and an honest list of what has never been
run. |
|
[`aisvc/`](https://github.com/kiwiflyer/CloudStack_AI_Plugin/tree/main/aisvc) |
The reconciler itself — the piece that is not in this PR. `observe() → diff() →
plan() → execute()`, one leader per zone. |
|
[`gateway-hooks/`](https://github.com/kiwiflyer/CloudStack_AI_Plugin/tree/main/gateway-hooks)
| The enforcement hooks the gateway loads for keys, scopes, profiles, quotas,
budgets and guardrails. |
|
[`install.py`](https://github.com/kiwiflyer/CloudStack_AI_Plugin/blob/main/install.py)
| Scripts everything outside CloudStack — virtualenv, dependencies,
migrations, mTLS material, reachability. Standard library only. |
For the serving layer specifically, the two decisions that shape it are
[ADR-0001](https://github.com/kiwiflyer/CloudStack_AI_Plugin/blob/main/docs/adr/0001-serving-layer.md)
(plain Kubernetes `Deployment`s rendered by the engine, with blue/green
weighting at the
gateway rather than in the serving layer) and
[ADR-0002](https://github.com/kiwiflyer/CloudStack_AI_Plugin/blob/main/docs/adr/0002-gateway.md)
(LiteLLM Proxy, MIT side only). Where a CKS cluster comes from is
[ADR-0007](https://github.com/kiwiflyer/CloudStack_AI_Plugin/blob/main/docs/adr/0007-kubernetes-cluster-source.md),
and [all eight
ADRs](https://github.com/kiwiflyer/CloudStack_AI_Plugin/tree/main/docs/adr)
record what was decided and what is still open.
Also worth reading:
[why this exists and what it lets an operator
sell](https://github.com/kiwiflyer/CloudStack_AI_Plugin/blob/main/docs/why-cloudstack-ai-services.md),
the [limits-and-defaults
sheet](https://github.com/kiwiflyer/CloudStack_AI_Plugin/blob/main/docs/feature-spec-sheet.md)
(every threshold and TTL read from the code, with gaps marked **·not built**
rather than
omitted), a [walkthrough of the UI
section](https://github.com/kiwiflyer/CloudStack_AI_Plugin/blob/main/docs/ui-guide.md)
screen by screen, and the
[third-party licence
position](https://github.com/kiwiflyer/CloudStack_AI_Plugin/blob/main/docs/third-party-licenses.md)
— the dependency tree follows ASF Category A/B/X, GPL/LGPL/AGPL are excluded
including for
Python imports, and CI fails on anything it cannot classify.
<!-- Fixes: # -->
### Types of changes
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] Enhancement (improves an existing feature and functionality)
- [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
- [ ] Build/CI
- [ ] Test (unit or integration test code)
### Feature/Enhancement Scale or Bug Severity
#### Feature/Enhancement Scale
- [x] Major
- [ ] Minor
#### Bug Severity
- [ ] BLOCKER
- [ ] Critical
- [ ] Major
- [ ] Minor
- [ ] Trivial
### Screenshots (if appropriate):
<!-- Upload the images here before posting; they live in the source repo at
docs/images/ui/ — model-catalog.jpg, endpoints.jpg,
endpoint-snippets.jpg,
setup-wizard.jpg, playground-open-session.jpg,
playground-session-open.jpg.
They are also laid out in context in docs/ui-guide.md. -->
### How Has This Been Tested?
Against a live **4.23.0.0** management server (the simulator image), with
the plugin built,
installed and loaded, and the UI section compiled into that server's own
`ui/` tree.
- **481 JUnit 5 tests** in the module. `mvn -pl plugins/api/ai-services
test` is green and
checkstyle is clean against `cloud-style.xml`.
- **Command registration measured, not assumed**: `listApis` goes from 865
to 903 with the
plugin loaded — exactly the 38 v1 commands — and to 923 with
`ai.services.v2.enabled` on,
exactly the 20 v2 ones. The gate holds on a real server in both directions.
- **Marvin integration tests** (`test/integration/plugins/ai-services/`)
cover all 58
commands across three default roles, allowed and denied. They call each
command with no
arguments, because access is checked before parameters are: a role that
may call one
fails on its arguments (431) and one that may not fails on the name (432),
so nothing is
executed either way.
- **Playwright**, 40 passed / 7 skipped, against the plugin and the UI in
the same
management server: the section visible per role, every admin screen absent
for a domain
admin and a tenant, and a tenant typing an admin screen's URL getting a
404 rather than a
refusal. The 7 skips are the `ai.services.v2.enabled`-off group, which
cannot run while
the setting is on.
- **Tenant isolation measured end to end**: the same `listAiModels` returns
two models to
root admin, two to an account in `/Customers/Acme`, and none to one in
`/ROOT`.
- The UI also goes through **CloudMonkey and a Terraform provider**, so
nothing in the
section is reachable only by clicking.
#### How did you try to break this feature and the system with this change?
The failure mode I worried about most is **CloudStack dropping something in
silence**, so
most of the hardening is aimed there:
- **A plugin that fails to wire is not an optional plugin, it is a dead
management server.**
One missing bean in the Spring context took the whole `/client` context
down — no API at
all, for anything — while 451 unit tests stayed green, because they
construct the manager
directly. There is now a test that walks every `@Inject` field and fails
on any type
without a bean.
- **CloudStack ignores an unknown request parameter.** A dialog or client
can therefore
collect a value that goes nowhere — a key "restricted" to two models and a
CIDR was being
created unrestricted. Tests now compare every parameter each caller sends
against the
command that receives it, in both directions, including required
parameters never offered.
- **A response field the far side sends but the response class does not
declare is dropped
the same way.** Ten were, including a guardrail profile's checks. A test
walks each
engine payload against its response class; a field may be omitted only
with a recorded
reason.
- **Role denial, not just role visibility**: a tenant is refused all 38
commands without an
explicit grant, verified on a live server rather than in a unit test.
- **Engine failover** is exercised in all four branches, including the one
that matters:
a replica that answered is never retried, so a POST cannot be applied
twice.
- Deliberate negatives are asserted too — a refused command must surface the
engine's own
sentence rather than a bare status code, and an async command must not
report success for
work no job ever did.
**What has not been tested, and I would not want a reviewer to assume
otherwise:**
- **Nothing here has run against a live `main`/24.0.0-SNAPSHOT server.** It
compiles and its
tests pass against this tree, and none of the ~45 CloudStack APIs it
imports has moved,
but compiling is not loading — and the one defect only a real server could
show took the
management server down.
- **The GPUs are simulated.** Request signing, the API, RBAC and the schema
are real; no
physical GPU capacity has been allocated by this code.
- **Only the module itself has been built here**, with
`mvn -pl plugins/api/ai-services test`. A full reactor build has not been
run, and this
PR edits `plugins/pom.xml` and `client/pom.xml`, so CI is the first thing
that will
compile the whole tree with the module in it. I would not be surprised by
a packaging
problem there.
- The UI section has not been built with `npm run build` in this tree. A
build reporting
`Compiled successfully` says nothing about whether labels resolve anyway —
that has to be
checked on a rendered screen.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]