miki3421 opened a new issue, #286: URL: https://github.com/apache/openserverless/issues/286
## Problem On the 0.9.0 operator, resume-time SeaweedFS IAM reconciliation can run before the private Filer HTTP listener is available. The SeaweedFS StatefulSet currently has neither a startup probe nor a readiness probe, so Kubernetes can report Ready before the service is usable. The fresh-install path already waits for the private Filer. The existing-installation `seaweedfs_deploy.reconcile()` path immediately runs anonymous-grant reconciliation and persisted-policy verification. ## Observed evidence On an existing single-node K3s installation inside WSL2, the retained startup logs show this sequence (CET, 2026-11-08): - 11:20:33: SeaweedFS is marked Ready. - 11:20:54.743: `whisk_resume` fails in `SeaweedfsClient.verify_anonymous_access()` with `RuntimeError: Could not verify anonymous S3 web-only grants`. - 11:20:57.431: SeaweedFS starts its Filer HTTP listener on port 9090. - 11:22:04.975: the subsequent `whisk_resume` attempt succeeds. Current private Filer readback independently confirms the expected bucket-scoped anonymous grants. The original exception suppresses its underlying request error, so the precise failed request type/status cannot be recovered. This timing is strong evidence of a startup-order race for the observed episode; no controlled restart has yet been performed to reproduce it. The relevant live source files match operator commit `93fc94ac2f69a1f52bd36e09e76344023460260b` on 0.9.0. The workload uses SeaweedFS 3.97. An earlier clean laboratory installation did not reproduce this exception. ## Proposed minimum correction 1. Wait for actual private Filer HTTP availability before resume-time IAM operations, with bounded retry and recovery of a port-forward lost during startup. 2. Add suitable SeaweedFS startup/readiness probes for new installations. 3. Reconcile missing probes on existing StatefulSets without replacing their image, command, environment, storage or custom probes; wait for the resulting rollout before using the service. 4. Keep persisted IAM verification fail-closed and propagate actual policy/bootstrap failures. Do not turn an IAM mismatch into a successful readiness retry. ## Acceptance - Delayed/unavailable Filer does not trigger premature IAM changes or ingress publication. - A ready Filer allows both bootstrap and resume; an actual IAM failure remains an error. - Existing installation settings and PVCs are preserved; applying the correction again does not restart an unchanged workload. - Local regression tests, isolated 0.9.0 laboratory validation, then an operator-authorized WSL validation before a PR. This issue tracks operator startup/readiness behavior separately from the doctor/reporting issues in https://github.com/sciabarracom/easylife/issues/5. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
