aicam opened a new issue, #6294:
URL: https://github.com/apache/texera/issues/6294
### Feature Summary
PR #5932 made the **application tier's** object storage configurable via a
`storage.s3` block, but left the two **data-tier** components still hardwired
to the in-cluster MinIO:
- **LakeFS** blockstore (dataset versioning) points at `{{ .Release.Name
}}-minio`.
- **Lakekeeper** Iceberg REST catalog warehouse is initialized against
MinIO, and its init job unconditionally creates the bucket and emits an
`s3-compat` storage profile (custom endpoint + path-style access).
There is also no single switch to drop the bundled MinIO entirely, and no
worked example of an all-external-S3 deployment. As a result a fully
external-S3 install (e.g. AWS S3 on EKS) is not yet possible with the chart.
This is the LakeFS/Lakekeeper half of "make object storage pluggable" that
#5932 explicitly deferred.
### Proposed Solution or Design
Finish the external-S3 story so an operator can run Texera with **no
in-cluster MinIO at all**, backing the app tier, LakeFS, and Lakekeeper by an
external S3 store:
- **`minio.enabled` switch** (`values.yaml`, default `true`) gating the
bundled MinIO subchart (`Chart.yaml` dependency `condition`), its persistence
template, and the MinIO `ExternalName` alias exposed to the computing-unit pool
namespace. Default install renders identically.
- **LakeFS external blockstore**: keep the in-cluster MinIO blockstore as
the default; document overriding `lakefsConfig` to a region-only S3 blockstore
with credentials injected via `extraEnvVars` for external S3.
- **Lakekeeper init job**: route `STORAGE_S3_*` through the shared
`texera.s3.*` helpers (added in #5932); add `lakekeeperInit.createBucket` (skip
`mc mb` for pre-existing external buckets whose IAM principal lacks
`CreateBucket`); add `warehouse.flavor` (`s3-compat` vs native `aws`, the
latter omitting endpoint + path-style access) and an optional
`warehouse.keyPrefix` to isolate the warehouse within a shared bucket.
- **`values-aws.yaml`** example overlay: a complete, copy-pasteable
all-external-S3 configuration (MinIO off, `storage.s3`, LakeFS S3 blockstore,
Lakekeeper `flavor: aws` + `createBucket: false`).
Non-breaking: with no overrides the chart renders the same set of resources
as today.
### Affected Area
Deployment / Infrastructure, Storage / Metadata
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
