miki3421 opened a new issue, #281: URL: https://github.com/apache/openserverless/issues/281
## Summary The OpenServerless operator integration suite still assumes that the SeaweedFS control API is exposed through a fixed NodePort. This assumption became invalid after `openserverless-operator` PR [#155](https://github.com/apache/openserverless-operator/pull/155) correctly changed the SeaweedFS Service from `NodePort` to `ClusterIP` and restricted its control ports. As a result, several integration scripts fail with `Connection reset by peer`. The two Milvus scripts are reported as Milvus failures even though execution stops in `seaweedfs.create()` before `milvus.create()` is reached. This issue proposes updating the integration access path while retaining the private `ClusterIP` Service and the NetworkPolicy. It does not propose restoring a NodePort. ## Affected component and branch - Component: `apache/openserverless-operator` - Branch: `0.9.0` - Confirmed affected integration scripts: - `tests/kind/milvus_standalone_test.ipy` - `tests/kind/milvus_standalone_slim_test.ipy` - `tests/kind/seaweedfs_test.ipy` - `tests/kind/seaweedfs_util_test.ipy` - `tests/kind/storage_static_test.ipy` ## Root cause The Kind configuration maps host port `9090` to node container port `32091`, which was the former SeaweedFS filer NodePort. The Milvus integration scripts still do the following before creating Milvus: ```python os.environ['SEAWEEDFS_API_HOST'] = 'localhost' os.environ['SEAWEEDFS_API_PORT'] = '9090' assert(etcd.create()) assert(seaweedfs.create()) ``` PR #155 removed NodePort `32091` and changed the Service to `ClusterIP`. Therefore, `localhost:9090` no longer reaches the SeaweedFS filer API. `seaweedfs.create()` retries the readiness request for approximately 30 seconds and logs repeated connection resets. The shared `wait_for_http()` helper then returns without raising an exception when its deadline expires, despite its docstring declaring `TimeoutError`. Execution continues until `make_public_bucket()` performs another request and raises `ConnectionError`. The resulting traceback points to the Milvus integration file, but the failing call is `seaweedfs.create()` and Milvus has not yet been created. ## Evidence - The final CI run for operator PR [#141](https://github.com/apache/openserverless-operator/pull/141), with Milvus 2.5.2 and the private Milvus tunnel changes, passed the full and slim Milvus scenarios: [run 39257637756](https://github.com/apache/openserverless-operator/actions/runs/39257637756). - Operator PR [#155](https://github.com/apache/openserverless-operator/pull/155) failed both Milvus integration scripts while still using Milvus 2.5.2: [run 40344816581](https://github.com/apache/openserverless-operator/actions/runs/40344816581). - Operator PR [#154](https://github.com/apache/openserverless-operator/pull/154) reports the same failure set after updating Milvus to 2.5.27: [run 40479000340](https://github.com/apache/openserverless-operator/actions/runs/40479000340). - The Milvus full and slim integration scripts are unchanged between the successful #141 run and the #155 failure. This shows that the failure is not caused by the Milvus 2.5.27 image update. ## Proposed solution ### 1. Keep the private service boundary Retain: - the SeaweedFS `ClusterIP` Service; - the removal of filer/master NodePorts; - the current NetworkPolicy restrictions. The tests must adapt to the private endpoint rather than reopening it. ### 2. Use a reusable private endpoint for out-of-cluster execution Use the existing loopback-only `service_endpoint()` mechanism for the SeaweedFS filer control port after the Service has been applied and before readiness/bootstrap operations run. The cleanest implementation would separate SeaweedFS creation into two internal phases: 1. render and apply the StatefulSet, Service and NetworkPolicy, then wait for the pod; 2. obtain the control endpoint and perform readiness checks and bootstrap operations. For phase 2: - in-cluster execution should continue to use `seaweedfs:9090`; - out-of-cluster integration execution should use an RBAC-authorized `kubectl port-forward` bound only to `127.0.0.1` and an ephemeral local port; - endpoint overrides should be scoped to the operation and restored afterwards. An alternative test-only post-apply hook, similar to the Milvus integration hook, could be considered if maintainers prefer not to refactor the creation flow. The important property is that the tunnel must be established only after the Service exists and before any external SeaweedFS HTTP request. ### 3. Remove fixed NodePort assumptions from the tests Update the affected integration scripts to stop relying on: - fixed host port `9090`; - fixed node port `32091`; - direct access to a private control API. The tests should consume the endpoint returned by the shared private connection helper. ### 4. Make readiness timeout behavior explicit Update `wait_for_http()` to raise `TimeoutError` after the deadline, as documented. This will make the first inaccessible endpoint fail at the readiness step instead of producing a later and misleading bootstrap traceback. Callers that intentionally tolerate an unavailable endpoint should handle the timeout explicitly. ## Acceptance criteria - SeaweedFS filer and master ports remain unavailable through NodePort. - The full and slim Milvus integration scripts pass without fixed host or node ports. - The direct SeaweedFS and static-storage integration scripts pass through the same private endpoint mechanism. - The local tunnel binds only to `127.0.0.1`, uses an ephemeral port and is always terminated during cleanup. - In-cluster operator execution continues to use the Kubernetes Service directly. - `wait_for_http()` raises `TimeoutError` when no accepted response is received before the deadline. - A SeaweedFS connectivity failure is reported as a SeaweedFS readiness/bootstrap failure, not as a Milvus image regression. - No production Service exposure or NetworkPolicy relaxation is introduced. ## Scope This proposal is limited to correcting the integration access path and timeout reporting. It does not change SeaweedFS tenant isolation, credentials, data, PVCs, or the Milvus image version. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
