This is an automated email from the ASF dual-hosted git repository. lahirujayathilake pushed a commit to branch contributor-readiness in repository https://gitbox.apache.org/repos/asf/airavata-custos.git
commit d879c5a13667eca85fd003b838ac66283ced2ac2 Author: lahiruj <[email protected]> AuthorDate: Tue Jun 23 20:05:05 2026 -0400 Updated onboarding docs and included .env.example --- .env.example | 44 ++++++++++++++++++++++++++++++ CONTRIBUTING.md | 40 ++++++++++++++++++++++++--- INSTALL.md | 27 ++++++++++++------ README.md | 8 ++++++ connectors/ACCESS/AMIE-Processor/README.md | 30 +++----------------- 5 files changed, 110 insertions(+), 39 deletions(-) diff --git a/.env.example b/.env.example new file mode 100644 index 000000000..c66cdab96 --- /dev/null +++ b/.env.example @@ -0,0 +1,44 @@ +# Sample environment file for the Custos backend. +# Copy to .env, fill in real values, and `source .env` (or use a dotenv loader). +# The portal has its own env template at web/.env.example. + +# --- Server: required ---------------------------------------------------- + +# MySQL/MariaDB DSN. Must allow multi-statements for migrations. +DATABASE_DSN=admin:admin@tcp(localhost:3306)/custos?parseTime=true&charset=utf8mb4&multiStatements=true + +# OIDC issuer + audience, into config/custos.yaml. +# Server refuses to start without these once auth is wired. +OIDC_ISSUER_URL= +OIDC_AUDIENCE= + +# --- Server: dev convenience --------------------------------------------- + +# UUID of the compute cluster this Custos instance serves. +# Shared across connectors that need to know which cluster they belong to. +CUSTOS_CLUSTER_ID=00000000-0000-0000-0000-000000000001 + +# --- AMIE connector (ACCESS-CI) ------------------------------------------ + +AMIE_BASE_URL=http://localhost:8180 +AMIE_SITE_CODE=TESTSITE +AMIE_API_KEY=dev +AMIE_CLUSTER_ID=00000000-0000-0000-0000-000000000001 + +# --- COmanage connector -------------------------------------------------- + +COMANAGE_REGISTRY_URL= +COMANAGE_CO_ID= +COMANAGE_API_USER= +COMANAGE_API_KEY= +COMANAGE_UNIX_CLUSTER_ID= +COMANAGE_PERSON_ID_TYPE= + +# --- SLURM connector ----------------------------------------------------- + +SLURM_API= +SLURM_USER= +SLURM_TOKEN= +SLURM_API_VERSION= +# Used only by the Usage-Monitor connector. +SLURM_MONITOR_CLUSTER_ID= diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 608e1d8fa..5fcaa4e58 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -16,7 +16,7 @@ For instructions on running Custos locally against a database, see [INSTALL.md]( - `pkg/` — public packages (models, service, events) - `connectors/` — protocol/site-specific connectors (ACCESS, SLURM, …) - `extensions/` — out-of-process extensions (PAM module, SSH cert signer, …) -- `dev-ops/compose/` — local Docker Compose stack (MariaDB, Keycloak, Vault, Prometheus, Grafana) +- `dev-ops/compose/` — local Docker Compose stack (MariaDB, Adminer, Prometheus, Grafana, Vault) ## Build @@ -53,18 +53,50 @@ Every SQL file must carry the Apache 2.0 license header (see existing files for ## Tests -Run the full test suite from the repo root: +Two tiers: unit tests and integration tests, separated by a build tag. + +### Unit tests (default, no external services) ```bash go test ./... ``` -Connector packages (for example `connectors/SLURM/Association-Mapper/...`) use `httptest` and do not require external services. +Runs every package's unit tests. Handlers and stores use `httptest` and mocks. +No DB, no network. This is what should pass on every commit. + +### Integration tests (build tag, real services) + +Integration test files carry `//go:build integration` at the top, so the default +`go test ./...` skips them silently. To run them, pass the tag and provide the +required env vars: + +```bash +# General integration tests (server, core service, identity resolver). +# Reuses the dev compose stack on :3306. +export CORE_TEST_DATABASE_DSN='admin:admin@tcp(localhost:3306)/custos?parseTime=true&charset=utf8mb4&multiStatements=true' +go test -tags integration ./... +``` + +The AMIE connector ships its own integration stack because its tests mutate a +lot of state and we don't want to corrupt the dev DB. It brings up an isolated +MariaDB on `:3307` and a mock AMIE server on `:8181`, runs the suite, then +tears it down: + +```bash +make integration-test-amie +``` + +(equivalent to `scripts/run-amie-integration-tests.sh`). The `:3307` is +deliberate. It's a separate stack defined in `dev-ops/local-amie/`, independent +of the `:3306` dev compose. + +If you see a test file you expect to run but `go test ./...` reports zero tests +for the package, check the first line for `//go:build integration`. ## Submitting changes 1. Open an issue describing the change, if one does not already exist. -2. Create a topic branch off `main`. +2. Create a topic branch off `master`. 3. Make focused, well-scoped commits with clear messages. 4. Ensure `go build ./...`, `go vet ./...`, and `go test ./...` all pass. 5. Open a pull request and link the related issue. diff --git a/INSTALL.md b/INSTALL.md index 185c29294..4f0d1bc8f 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -35,7 +35,6 @@ From the repository root: ```bash export DATABASE_DSN='admin:admin@tcp(localhost:3306)/custos?parseTime=true&charset=utf8mb4&multiStatements=true' -export HTTP_ADDR=':8080' go run ./cmd/server ``` @@ -44,20 +43,30 @@ On startup, the server: 1. Opens the database connection (`internal/db`). 2. Runs the embedded migrations (`internal/db/migrations/`). 3. Wires the event bus, service layer, and HTTP router. -4. Listens on `HTTP_ADDR`. +4. Listens on the port from `config/custos.yaml` (`core.api.port`, default `8080`). + +### 3. Seed dev users (optional) + +The schema is created by the server's migrations on first run. After that, apply +the dev seed to populate sample identity data (org, users, roles, privileges): + +```bash +docker exec -i custos_db mariadb -uadmin -padmin custos \ + < dev-ops/compose/seeds/dev_users_and_roles.sql +``` + +The seed uses `INSERT IGNORE`, so re-running is safe. See the seed file for +exactly what it inserts. ### Environment variables -| Variable | Default | Description | -|----------------------|--------------------------------------------------------------------|----------------------------------------------| -| `DATABASE_DSN` | _(required)_ | Go MySQL DSN; must allow multi-statements | -| `HTTP_ADDR` | `:8080` | Listen address for the HTTP API | -| `DB_MAX_OPEN_CONNS` | `25` | Max open DB connections | -| `DB_MAX_IDLE_CONNS` | `5` | Max idle DB connections | +See [.env.example](.env.example) for the template. Copy it to `.env`, fill in +the required values (`DATABASE_DSN`, `OIDC_ISSUER_URL`, `OIDC_AUDIENCE`), and +source it before running the server. ### Optional services -The Compose file also defines Keycloak, Vault, Adminer, Prometheus, and Grafana. Start them as needed, for example: +The Compose file also defines Adminer, Prometheus, Grafana, and Vault. Start them as needed, for example: ```bash docker compose up -d adminer # DB UI at http://localhost:18080 diff --git a/README.md b/README.md index fd03a7acc..8bee61c32 100644 --- a/README.md +++ b/README.md @@ -89,6 +89,14 @@ go test ./... See each connector's and extension's README for run and configuration details. +## Documentation + +- [INSTALL.md](INSTALL.md): run the server locally against the dev compose stack +- [CONTRIBUTING.md](CONTRIBUTING.md): coding conventions, build, and test workflow +- [docs/API-Docs.md](docs/API-Docs.md): REST API reference +- [docs/Allocation-Data-Models.md](docs/Allocation-Data-Models.md): domain model overview +- [docs/ACCESS-HPC-Reference.md](docs/ACCESS-HPC-Reference.md): ACCESS-CI integration reference + ## Questions or Need Help? * Open a [GitHub issue](https://github.com/apache/airavata-custos/issues) diff --git a/connectors/ACCESS/AMIE-Processor/README.md b/connectors/ACCESS/AMIE-Processor/README.md index bd977a920..72f51c370 100644 --- a/connectors/ACCESS/AMIE-Processor/README.md +++ b/connectors/ACCESS/AMIE-Processor/README.md @@ -4,18 +4,13 @@ ACCESS-CI AMIE packet processing service for Apache Airavata Custos. This Go service polls the [ACCESS-CI](https://access-ci.org/) AMIE (Account Management Information Exchange) API for allocation packets, processes them through typed handlers, manages person/project/account lifecycle, and replies to AMIE. It is the ACCESS allocation source adapter within the Custos allocations platform. -## Prerequisites - -- Go 1.22+ -- MariaDB — use the shared dev stack: `docker compose -f compose/docker-compose.yml up db` (creates the `access_ci` database automatically via `compose/dbinit/init-db.sh`) - ## Quick Start ### 1. Start the database ```bash # From the repo root -docker compose -f compose/docker-compose.yml up db -d +docker compose -f dev-ops/compose/docker-compose.yml up db -d ``` ### 2. Configure @@ -28,7 +23,7 @@ export AMIE_API_KEY="your-api-key" The default `config.yaml` works for local development with the Docker Compose MariaDB defaults. -For local dev without a real ACCESS endpoint, point the service at the local mock AMIE server in [`devtools/amie/`](../devtools/amie/README.md) +For local dev without a real ACCESS endpoint, point the service at the local mock AMIE server in [`mock-server/`](./mock-server/README.md) ```bash export AMIE_BASE_URL="http://localhost:8180"; @@ -39,7 +34,7 @@ export AMIE_API_KEY="dev" ### 3. Build ```bash -cd allocations/access-amie +cd connectors/ACCESS/AMIE-Processor go build -o bin/access-amie . ``` @@ -135,7 +130,7 @@ For sustained traffic at a configurable rate, use k6 with `mock-server/amie-traf ## Observability -The service exports Prometheus metrics at `/metrics`. A pre-built Grafana dashboard is available at `compose/grafana/dashboards/amie-service.json` showing packet processing stats, failures, retries, and processing duration percentiles. +The service exports Prometheus metrics at `/metrics`. A pre-built Grafana dashboard is available at `dev-ops/compose/grafana/dashboards/amie-service.json` showing packet processing stats, failures, retries, and processing duration percentiles. To run the metrics stack: @@ -208,20 +203,3 @@ AMIE API ──poll──> Poller ──persist──> DB (packets + events) | Log level | `log.level` | `LOG_LEVEL` | `info` | | Log format | `log.format` | `LOG_FORMAT` | `text` | | Provisioner type | `provisioner.type` | — | `noop` | - -## Go Workspace - -This module is part of a Go workspace defined in `allocations/go.work`: - -``` -allocations/ - go.work Workspace root - provisioner/ Shared HPC provisioner interface (zero deps) - access-amie/ This module (ACCESS-CI AMIE adapter) -``` - -The `provisioner/` module defines the `Provisioner` interface for HPC cluster account and project provisioning. Currently using a no-op stub; a future SLURM provisioner will implement it. - -## License - -Licensed under the Apache License, Version 2.0. See [LICENSE](../../LICENSE) for details.
