The GitHub Actions job "Required Checks" on texera.git/gh-readonly-queue/main/pr-5932-a1a7eb031628507c30ecd0eadb9b744b14e52c71 has succeeded. Run started by GitHub user aicam (triggered by aicam).
Head commit for run: 322babf6205cf72277d13a4f2d16c5954a4dfbd9 / ali risheh <[email protected]> feat(k8s): make app-tier object storage configurable via storage.s3 (#5932) ### What changes were proposed in this PR? This makes the **application tier's** object-storage target configurable, as the first step toward supporting an external S3 store alongside the in-cluster MinIO. It is **non-breaking**: the default (on-prem / in-cluster MinIO) install renders an identical set of resources. Today `file-service` and `workflow-computing-unit-manager` hardcode the S3 endpoint and credentials to the in-cluster MinIO Service (`{{ .Release.Name }}-minio`) and its auto-generated Secret. This PR routes both through a new `storage.s3` values block: - **`values.yaml`** — new `storage.s3` block: `endpoint`, `region`, `existingSecret`, `accessKeyId`, `secretAccessKey`. All default to empty. - **`templates/base/_helpers.tpl`** (new) — helpers that resolve the S3 endpoint and the credentials Secret name/keys. When `storage.s3.endpoint` is empty they fall back to the in-cluster MinIO Service and its `{{ .Release.Name }}-minio` Secret (keys `root-user`/`root-password`), so the default render is unchanged. - **`templates/aws/s3-credentials-secret.yaml`** (new) — a `{{ .Release.Name }}-s3-credentials` Secret, rendered **only** when an external `endpoint` is set and no `existingSecret` is supplied. Renders nothing on the default install. - **`file-service` / `workflow-computing-unit-manager` deployments** — the `STORAGE_S3_*` env now comes from the helpers; `STORAGE_S3_REGION` is added only in external mode. How it behaves: - **Default (no config):** services point at the in-cluster MinIO exactly as before. - **External S3:** set `storage.s3.endpoint` + `region` + credentials (or `existingSecret`) → both services use that store; the chart materializes the credentials Secret unless you bring your own. **Out of scope (intentionally deferred to keep this PR small and atomic):** LakeFS blockstore + Lakekeeper warehouse external-S3 wiring, the `minio.enabled` switch to drop the in-cluster MinIO entirely, and a `values-aws.yaml` example overlay. Those are the LakeFS/Lakekeeper half of "make object storage pluggable" and will land as a follow-up. ### Any related issues, documentation, discussions? Closes #5931 (app-tier `storage.s3` task). Part of #5891 — unify AWS (EKS) and on-premise Kubernetes deployment under `bin/k8s` (parent feature). Follows #5757 (Helm template reorg) and the design discussion in #5641. ### How was this PR tested? Verified the default install is unchanged and the external path renders correctly: 1. **No-op render proof:** `helm template texera bin/k8s` on this branch vs on `main` renders the **same 102 resources** — identical after ignoring comments. The only textual artifact is one empty `---` document from the gated-off `s3-credentials-secret.yaml` (its `# license` header is emitted by Helm even though the `{{- if }}` body is empty); it produces no Kubernetes object. `helm lint` passes. 2. **External S3 render:** `helm template ... --set storage.s3.endpoint=https://s3.us-west-2.amazonaws.com --set storage.s3.accessKeyId=… --set storage.s3.secretAccessKey=…` renders the `*-s3-credentials` Secret, repoints both deployments' `STORAGE_S3_ENDPOINT`/credentials at it (keys `access-key-id`/`secret-access-key`), and adds `STORAGE_S3_REGION`. 3. **Bring-your-own Secret:** with `--set storage.s3.existingSecret=my-creds`, the deployments reference `my-creds` and the chart generates **no** Secret. 4. `helm lint bin/k8s` passes for both the default and the external value sets. 5. `helm install` locally with Minikube and test creating dataset and running workflows No unit tests were added — the change is limited to Helm chart values/templates, validated by the render diff and `helm lint` above. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 4.8) Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]> Report URL: https://github.com/apache/texera/actions/runs/28552761575 With regards, GitHub Actions via GitBox
