aicam opened a new pull request, #6295:
URL: https://github.com/apache/texera/pull/6295
### What changes were proposed in this PR?
This completes the "make object storage pluggable" work by wiring the **data
tier** (LakeFS blockstore + Lakekeeper Iceberg warehouse) to an external S3
store and adding a single switch to drop the in-cluster MinIO entirely. #5932
did the **application tier** (`storage.s3`) and explicitly deferred this half.
It is **non-breaking**: the default (on-prem / in-cluster MinIO) install
renders the same set of resources.
- **`minio.enabled` switch** (`values.yaml`, default `true`) to turn off the
bundled MinIO subchart and point everything at external S3:
- `Chart.yaml` -- the `minio` dependency gets `condition: minio.enabled`.
- `templates/on-prem/minio-persistence.yaml` -- the MinIO PV/PVC is gated
on `minio.enabled` as well as `minio.persistence.enabled`.
- `templates/base/external-names/external-names.yaml` -- the
`<release>-minio` `ExternalName` alias (exposed to the computing-unit-pool
namespace) is only rendered when MinIO is enabled; with external S3 the CU pods
reach the store directly via `STORAGE_S3_ENDPOINT`.
- **Lakekeeper init job**
(`templates/base/lakekeeper/lakekeeper-init-job.yaml`):
- `STORAGE_S3_*` env now resolves through the shared `texera.s3.*` helpers
introduced in #5932 (endpoint + credentials Secret name/keys) instead of
hardcoding `<release>-minio`.
- New `lakekeeperInit.createBucket` (default `true`) -- when `false`, the
job skips `mc mb` and logs that it is using a pre-existing external bucket.
Needed because an external bucket already exists and its IAM principal may lack
`s3:CreateBucket`.
- New `lakekeeperInit.warehouse.flavor` -- `s3-compat` (MinIO/other; emits
`endpoint` + `path-style-access: true`) or native `aws` (real AWS S3; omits
both). New optional `lakekeeperInit.warehouse.keyPrefix` to isolate the
warehouse within a shared bucket.
- **`values.yaml`** -- documents `minio.enabled`, the LakeFS
external-blockstore override path, and the new `lakekeeperInit` keys.
- **`values-aws.yaml`** (new) -- a complete, copy-pasteable all-external-S3
overlay: MinIO off, `storage.s3` (endpoint/region/credentials or
`existingSecret`), LakeFS region-only S3 blockstore with credentials injected
via `extraEnvVars`, and Lakekeeper `flavor: aws` + `createBucket: false`.
How it behaves:
- **Default (no config):** LakeFS and Lakekeeper use the in-cluster MinIO
exactly as before; the MinIO subchart, PV/PVC, and `ExternalName` all render as
today.
- **External S3 (`-f values-aws.yaml`):** no MinIO is deployed; the app
tier, LakeFS, and Lakekeeper all target the external S3 buckets, and the
Lakekeeper warehouse is registered with the native AWS flavor against a
pre-existing bucket.
### Any related issues, documentation, discussions?
Closes #6294 (LakeFS blockstore + Lakekeeper warehouse external-S3 task).
Part of #5891 -- unify AWS (EKS) and on-premise Kubernetes deployment under
`bin/k8s` (parent feature).
Follows #5932 (app-tier `storage.s3`) and #5757 (Helm template reorg).
### How was this PR tested?
**1. Render parity (default install unchanged).** `helm template texera
bin/k8s` on this branch vs on `main` renders the same set of resources; `helm
lint bin/k8s` passes. The new `{{- if }}` guards (MinIO ExternalName,
persistence, createBucket, flavor) all evaluate to their prior behavior when
`minio.enabled`/`createBucket` default to `true`.
**2. External-S3 render.** `helm template texera bin/k8s -f
bin/k8s/values-aws.yaml` deploys no MinIO subchart / PV / PVC / ExternalName,
routes the Lakekeeper init job's `STORAGE_S3_*` at the external endpoint and
credentials Secret, skips the `mc mb` bucket-creation step, and registers the
warehouse storage profile with `"flavor": "aws"` (no `endpoint` /
`path-style-access`). `helm lint` passes for this value set.
**3. End-to-end on a live EKS cluster with real AWS S3.** Stood up an
isolated EKS cluster (`us-west-2`, its own AWS account, no MinIO) plus two
pre-created S3 buckets (LakeFS blockstore + Iceberg warehouse) and an IAM user
scoped to just those buckets with **no** `CreateBucket` permission. Built all
Texera service images from this branch, deployed the chart with an external-S3
overlay derived from `values-aws.yaml`, and exercised the full stack:
- LakeFS + Lakekeeper init jobs authenticated to external S3 and initialized
(Lakekeeper `aws` flavor, bucket-creation skipped against the pre-existing
warehouse bucket).
- Created a dataset -> objects written to the external LakeFS bucket;
dataset preview served via presigned S3 URLs (external buckets need a CORS
rule, noted below).
- Ran a workflow (CSV file scan) end to end -> **success, 100 rows
returned**, with the Iceberg result table physically materialized in the
external warehouse bucket (`metadata.json` + data objects confirmed via the S3
API). This exercises the full external-S3 path: file-service reads the CSV from
LakeFS -> S3, the computing-unit pod reaches the Lakekeeper REST catalog, and
results are written to Iceberg -> S3 and read back.
All test AWS resources (cluster, buckets, IAM user) were torn down afterward.
Operational note surfaced by the live test (not a chart defect, worth
documenting): with a real external S3 store, dataset preview requires a CORS
rule on the buckets (`GET`/`HEAD`) because the presigned URLs are served
cross-origin from `*.s3.<region>.amazonaws.com`, whereas the in-cluster MinIO
was same-origin.
No unit tests were added -- the change is limited to Helm chart
values/templates, validated by the render diff, `helm lint`, and the live
end-to-end run above.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
