kiwiflyer opened a new pull request, #14733:
URL: https://github.com/apache/cloudstack/pull/14733

   ### Description
   
   **This is a work in progress**
   
   This PR adds **CloudStack AI Services**: a native API plugin 
(`plugins/api/ai-services`)
   and a `ui/` section that let an operator source open-weight LLMs, deploy 
them on NVIDIA
   GPU capacity, tune them through profiles, and sell them to tenants as 
OpenAI-compatible
   endpoints.
   
   The plugin is the front door only — auth, RBAC, async jobs, events, usage 
and limits. It
   delegates to an external reconciler over mTLS, which is the component that 
touches GPU
   resources, CKS and the serving layer. **That engine is not in this PR**, 
which is the main
   thing I would like feedback on; see "Open questions" below.
   
   What it contains - 
   
   - **58 new API commands** — 38 v1, registered always, and 20 v2, registered 
only when the
     global setting `ai.services.v2.enabled` is on (default **false**), so a GA 
install does
     not advertise an API for features it is not running.
   - **A new UI section** of 19 screens — model catalogue, endpoints with 
scoped API keys and
     copy-ready snippets, a playground, generation profiles and 
effective-parameter
     resolution, usage and budgets, plus admin screens for setup preflight, GPU 
capacity and
     the model registry. Routes carry a `permission`, so CloudStack drops any 
screen whose
     command the caller cannot call, and the Tier 2 screens appear and 
disappear with that one
     setting.
   - **Ten `ai.services.*` ConfigKeys**, including the engine URLs and its mTLS 
material.
   - **Default role access is declared on `@APICommand(authorized = …)`**, so 
there is no
     migration, no `commands.properties` and no operator step. Explicit 
`role_permissions`
     rules still win.
   
   The change is **additive**: 158 files, 19,207 insertions and a single 
deletion (a comma, to
   append to `en.json`). Only four existing files are touched — 
`plugins/pom.xml`,
   `client/pom.xml`, `ui/src/config/router.js`, and 
`ui/public/locales/en.json`, into which 150
   keys are appended without reordering or reformatting anything already there.
   
   ### Where the rest of it lives
   
   Everything this plugin talks to is Apache-2.0 and public, so the serving 
layer can be built
   and the design read is available here:
   
   **https://github.com/kiwiflyer/CloudStack_AI_Plugin**
   
   | | |
   | --- | --- |
   | 
[`SPEC.md`](https://github.com/kiwiflyer/CloudStack_AI_Plugin/blob/main/SPEC.md)
 | The technical specification. Section 9 is the command map this plugin is 
generated from, section 10 the engine REST contract it calls. |
   | 
[`docs/installation.md`](https://github.com/kiwiflyer/CloudStack_AI_Plugin/blob/main/docs/installation.md)
 | How to stand the whole thing up: the four pieces and their order, every 
config key and environment variable, and an honest list of what has never been 
run. |
   | 
[`aisvc/`](https://github.com/kiwiflyer/CloudStack_AI_Plugin/tree/main/aisvc) | 
The reconciler itself — the piece that is not in this PR. `observe() → diff() → 
plan() → execute()`, one leader per zone. |
   | 
[`gateway-hooks/`](https://github.com/kiwiflyer/CloudStack_AI_Plugin/tree/main/gateway-hooks)
 | The enforcement hooks the gateway loads for keys, scopes, profiles, quotas, 
budgets and guardrails. |
   | 
[`install.py`](https://github.com/kiwiflyer/CloudStack_AI_Plugin/blob/main/install.py)
 | Scripts everything outside CloudStack — virtualenv, dependencies, 
migrations, mTLS material, reachability. Standard library only. |
   
   For the serving layer specifically, the two decisions that shape it are
   
[ADR-0001](https://github.com/kiwiflyer/CloudStack_AI_Plugin/blob/main/docs/adr/0001-serving-layer.md)
   (plain Kubernetes `Deployment`s rendered by the engine, with blue/green 
weighting at the
   gateway rather than in the serving layer) and
   
[ADR-0002](https://github.com/kiwiflyer/CloudStack_AI_Plugin/blob/main/docs/adr/0002-gateway.md)
   (LiteLLM Proxy, MIT side only). Where a CKS cluster comes from is
   
[ADR-0007](https://github.com/kiwiflyer/CloudStack_AI_Plugin/blob/main/docs/adr/0007-kubernetes-cluster-source.md),
   and [all eight 
ADRs](https://github.com/kiwiflyer/CloudStack_AI_Plugin/tree/main/docs/adr)
   record what was decided and what is still open.
   
   Also worth reading:
   [why this exists and what it lets an operator 
sell](https://github.com/kiwiflyer/CloudStack_AI_Plugin/blob/main/docs/why-cloudstack-ai-services.md),
   the [limits-and-defaults 
sheet](https://github.com/kiwiflyer/CloudStack_AI_Plugin/blob/main/docs/feature-spec-sheet.md)
   (every threshold and TTL read from the code, with gaps marked **·not built** 
rather than
   omitted), a [walkthrough of the UI 
section](https://github.com/kiwiflyer/CloudStack_AI_Plugin/blob/main/docs/ui-guide.md)
   screen by screen, and the
   [third-party licence 
position](https://github.com/kiwiflyer/CloudStack_AI_Plugin/blob/main/docs/third-party-licenses.md)
   — the dependency tree follows ASF Category A/B/X, GPL/LGPL/AGPL are excluded 
including for
   Python imports, and CI fails on anything it cannot classify.
   
   <!-- Fixes: # -->
   
   ### Types of changes
   
   - [ ] Breaking change (fix or feature that would cause existing 
functionality to change)
   - [x] New feature (non-breaking change which adds functionality)
   - [ ] Bug fix (non-breaking change which fixes an issue)
   - [ ] Enhancement (improves an existing feature and functionality)
   - [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
   - [ ] Build/CI
   - [ ] Test (unit or integration test code)
   
   ### Feature/Enhancement Scale or Bug Severity
   
   #### Feature/Enhancement Scale
   
   - [x] Major
   - [ ] Minor
   
   #### Bug Severity
   
   - [ ] BLOCKER
   - [ ] Critical
   - [ ] Major
   - [ ] Minor
   - [ ] Trivial
   
   ### Screenshots (if appropriate):
   
   <!-- Upload the images here before posting; they live in the source repo at
        docs/images/ui/ — model-catalog.jpg, endpoints.jpg, 
endpoint-snippets.jpg,
        setup-wizard.jpg, playground-open-session.jpg, 
playground-session-open.jpg.
        They are also laid out in context in docs/ui-guide.md. -->
   
   ### How Has This Been Tested?
   
   Against a live **4.23.0.0** management server (the simulator image), with 
the plugin built,
   installed and loaded, and the UI section compiled into that server's own 
`ui/` tree.
   
   - **481 JUnit 5 tests** in the module. `mvn -pl plugins/api/ai-services 
test` is green and
     checkstyle is clean against `cloud-style.xml`.
   - **Command registration measured, not assumed**: `listApis` goes from 865 
to 903 with the
     plugin loaded — exactly the 38 v1 commands — and to 923 with 
`ai.services.v2.enabled` on,
     exactly the 20 v2 ones. The gate holds on a real server in both directions.
   - **Marvin integration tests** (`test/integration/plugins/ai-services/`) 
cover all 58
     commands across three default roles, allowed and denied. They call each 
command with no
     arguments, because access is checked before parameters are: a role that 
may call one
     fails on its arguments (431) and one that may not fails on the name (432), 
so nothing is
     executed either way.
   - **Playwright**, 40 passed / 7 skipped, against the plugin and the UI in 
the same
     management server: the section visible per role, every admin screen absent 
for a domain
     admin and a tenant, and a tenant typing an admin screen's URL getting a 
404 rather than a
     refusal. The 7 skips are the `ai.services.v2.enabled`-off group, which 
cannot run while
     the setting is on.
   - **Tenant isolation measured end to end**: the same `listAiModels` returns 
two models to
     root admin, two to an account in `/Customers/Acme`, and none to one in 
`/ROOT`.
   - The UI also goes through **CloudMonkey and a Terraform provider**, so 
nothing in the
     section is reachable only by clicking.
   
   #### How did you try to break this feature and the system with this change?
   
   The failure mode I worried about most is **CloudStack dropping something in 
silence**, so
   most of the hardening is aimed there:
   
   - **A plugin that fails to wire is not an optional plugin, it is a dead 
management server.**
     One missing bean in the Spring context took the whole `/client` context 
down — no API at
     all, for anything — while 451 unit tests stayed green, because they 
construct the manager
     directly. There is now a test that walks every `@Inject` field and fails 
on any type
     without a bean.
   - **CloudStack ignores an unknown request parameter.** A dialog or client 
can therefore
     collect a value that goes nowhere — a key "restricted" to two models and a 
CIDR was being
     created unrestricted. Tests now compare every parameter each caller sends 
against the
     command that receives it, in both directions, including required 
parameters never offered.
   - **A response field the far side sends but the response class does not 
declare is dropped
     the same way.** Ten were, including a guardrail profile's checks. A test 
walks each
     engine payload against its response class; a field may be omitted only 
with a recorded
     reason.
   - **Role denial, not just role visibility**: a tenant is refused all 38 
commands without an
     explicit grant, verified on a live server rather than in a unit test.
   - **Engine failover** is exercised in all four branches, including the one 
that matters:
     a replica that answered is never retried, so a POST cannot be applied 
twice.
   - Deliberate negatives are asserted too — a refused command must surface the 
engine's own
     sentence rather than a bare status code, and an async command must not 
report success for
     work no job ever did.
   
   **What has not been tested, and I would not want a reviewer to assume 
otherwise:**
   
   - **Nothing here has run against a live `main`/24.0.0-SNAPSHOT server.** It 
compiles and its
     tests pass against this tree, and none of the ~45 CloudStack APIs it 
imports has moved,
     but compiling is not loading — and the one defect only a real server could 
show took the
     management server down.
   - **The GPUs are simulated.** Request signing, the API, RBAC and the schema 
are real; no
     physical GPU capacity has been allocated by this code.
   - **Only the module itself has been built here**, with
     `mvn -pl plugins/api/ai-services test`. A full reactor build has not been 
run, and this
     PR edits `plugins/pom.xml` and `client/pom.xml`, so CI is the first thing 
that will
     compile the whole tree with the module in it. I would not be surprised by 
a packaging
     problem there.
   - The UI section has not been built with `npm run build` in this tree. A 
build reporting
     `Compiled successfully` says nothing about whether labels resolve anyway — 
that has to be
     checked on a rendered screen.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to