aicam opened a new pull request, #5932:
URL: https://github.com/apache/texera/pull/5932
### What changes were proposed in this PR?
This makes the **application tier's** object-storage target configurable, as
the first step toward supporting an external S3 store alongside the in-cluster
MinIO. It is **non-breaking**: the default (on-prem / in-cluster MinIO) install
renders an identical set of resources.
Today `file-service` and `workflow-computing-unit-manager` hardcode the S3
endpoint and credentials to the in-cluster MinIO Service (`{{ .Release.Name
}}-minio`) and its auto-generated Secret. This PR routes both through a new
`storage.s3` values block:
- **`values.yaml`** — new `storage.s3` block: `endpoint`, `region`,
`existingSecret`, `accessKeyId`, `secretAccessKey`. All default to empty.
- **`templates/base/_helpers.tpl`** (new) — helpers that resolve the S3
endpoint and the credentials Secret name/keys. When `storage.s3.endpoint` is
empty they fall back to the in-cluster MinIO Service and its `{{ .Release.Name
}}-minio` Secret (keys `root-user`/`root-password`), so the default render is
unchanged.
- **`templates/aws/s3-credentials-secret.yaml`** (new) — a `{{ .Release.Name
}}-s3-credentials` Secret, rendered **only** when an external `endpoint` is set
and no `existingSecret` is supplied. Renders nothing on the default install.
- **`file-service` / `workflow-computing-unit-manager` deployments** — the
`STORAGE_S3_*` env now comes from the helpers; `STORAGE_S3_REGION` is added
only in external mode.
How it behaves:
- **Default (no config):** services point at the in-cluster MinIO exactly as
before.
- **External S3:** set `storage.s3.endpoint` + `region` + credentials (or
`existingSecret`) → both services use that store; the chart materializes the
credentials Secret unless you bring your own.
**Out of scope (intentionally deferred to keep this PR small and atomic):**
LakeFS blockstore + Lakekeeper warehouse external-S3 wiring, the
`minio.enabled` switch to drop the in-cluster MinIO entirely, and a
`values-aws.yaml` example overlay. Those are the LakeFS/Lakekeeper half of
"make object storage pluggable" and will land as a follow-up.
### Any related issues, documentation, discussions?
Closes #5931 (app-tier `storage.s3` task).
Part of #5891 — unify AWS (EKS) and on-premise Kubernetes deployment under
`bin/k8s` (parent feature).
Follows #5757 (Helm template reorg) and the design discussion in #5641.
### How was this PR tested?
Verified the default install is unchanged and the external path renders
correctly:
1. **No-op render proof:** `helm template texera bin/k8s` on this branch vs
on `main` renders the **same 102 resources** — identical after ignoring
comments. The only textual artifact is one empty `---` document from the
gated-off `s3-credentials-secret.yaml` (its `# license` header is emitted by
Helm even though the `{{- if }}` body is empty); it produces no Kubernetes
object. `helm lint` passes.
2. **External S3 render:** `helm template ... --set
storage.s3.endpoint=https://s3.us-west-2.amazonaws.com --set
storage.s3.accessKeyId=… --set storage.s3.secretAccessKey=…` renders the
`*-s3-credentials` Secret, repoints both deployments'
`STORAGE_S3_ENDPOINT`/credentials at it (keys
`access-key-id`/`secret-access-key`), and adds `STORAGE_S3_REGION`.
3. **Bring-your-own Secret:** with `--set
storage.s3.existingSecret=my-creds`, the deployments reference `my-creds` and
the chart generates **no** Secret.
4. `helm lint bin/k8s` passes for both the default and the external value
sets.
No unit tests were added — the change is limited to Helm chart
values/templates, validated by the render diff and `helm lint` above.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
