miki3421 opened a new issue, #280: URL: https://github.com/apache/openserverless/issues/280
## Problem `cron-init` can keep earlier Jobs active while creating another Job every minute. The operator generates no concurrency restriction or Job deadline, while the executor has unbounded HTTP waits and pagination paths. Several dependency failures are also reported as a successful process exit. This issue separates the reproducible code/configuration defects from the initial cause of the reported production incident, which remains unproven. ## Affected version and source verification Reported image: `apache/openserverless-operator:0.1.0-incubating.2510012114`. The public tag resolves to [c562668a35c8f383c8ec6d0065da29baa39f0bcd](https://github.com/apache/openserverless-operator/commit/c562668a35c8f383c8ec6d0065da29baa39f0bcd). The five files below, plus `cron.sh` and `pyproject.toml`, were extracted from both published amd64 and arm64 images and matched byte-for-byte against that tag: 14/14 comparisons. Registry index checked on 2026-10-23: `sha256:d41201ded2a544c7affc8cbe8084d0f5ac10d6b8031897c96a383d6fd4d55847`. The deployed Pods' immutable imageIDs were not supplied, so tag reuse or a different cached image cannot be completely excluded. The same relevant behavior was reproduced in the current public `main`, pinned to [fb481d243d902e4f995a14bd6bb1137c79045b6c](https://github.com/apache/openserverless-operator/commit/fb481d243d902e4f995a14bd6bb1137c79045b6c). ## Reported incident observations On 2026-10-23: - Approximately 9,500 active `cron-init` Jobs, more than 9,000 Pending Pods and approximately 211 Running Pods. - A one-minute schedule, effective `concurrencyPolicy: Allow`, `restartPolicy: OnFailure`, no `activeDeadlineSeconds`, and no container resource requests/limits. - Scheduler events included `Too many pods`. - Some running Pods were eight days old; their last messages referred to a CouchDB `nuvolaris_whisks/_find` query. - CouchDB logged timeouts and HTTP 500 responses during the overload. - After suspending the CronJob and cleaning up accumulated workloads, the query responded in approximately 27 ms and services became Ready again. These are reported incident observations, not measurements collected by accessing the affected cluster during this investigation. ## Original code paths 1. [nuvolaris/couchdb_util.py, lines 152–170](https://github.com/apache/openserverless-operator/blob/c562668a35c8f383c8ec6d0065da29baa39f0bcd/nuvolaris/couchdb_util.py#L152-L170): `find_doc()` uses `session.post()` without a timeout. A non-200 response is logged and converted to `None`. 2. [nuvolaris/actionexecutor.py, lines 94–120](https://github.com/apache/openserverless-operator/blob/c562668a35c8f383c8ec6d0065da29baa39f0bcd/nuvolaris/actionexecutor.py#L94-L120): `find_docs()` has no page bound or bookmark progress guard. A non-empty page with a blank/repeated bookmark can repeat indefinitely; a missing bookmark raises `KeyError`. A failed later page returns the documents collected so far. 3. [nuvolaris/actionexecutor.py, lines 148–196](https://github.com/apache/openserverless-operator/blob/c562668a35c8f383c8ec6d0065da29baa39f0bcd/nuvolaris/actionexecutor.py#L148-L196): OpenWhisk POST and PUT also lack timeouts. `blocking=false` does not bound the HTTP client's wait. 4. [nuvolaris/actionexecutor.py, lines 317–396](https://github.com/apache/openserverless-operator/blob/c562668a35c8f383c8ec6d0065da29baa39f0bcd/nuvolaris/actionexecutor.py#L317-L396): failed invocation/update results are not propagated to the process exit status. Failed readiness also returns normally. 5. [nuvolaris/cronjob.py, lines 24–58](https://github.com/apache/openserverless-operator/blob/c562668a35c8f383c8ec6d0065da29baa39f0bcd/nuvolaris/cronjob.py#L24-L58), [nuvolaris/templates/cron-init.yaml](https://github.com/apache/openserverless-operator/blob/c562668a35c8f383c8ec6d0065da29baa39f0bcd/nuvolaris/templates/cron-init.yaml), and [deploy/scheduler/cron.yaml](https://github.com/apache/openserverless-operator/blob/c562668a35c8f383c8ec6d0065da29baa39f0bcd/deploy/scheduler/cron.yaml): the actual generated CronJob has the unbounded defaults described above. The actual `cronjob.create()`, template helpers and Kustomize 4.5.7 were exercised locally, replacing only the Kubernetes apply boundary with object capture. Both the original and current source generate those same missing controls. The generator correctly replaces the stale example image and empty `SCHEDULER_CONFIG` in the base manifest; those placeholders are not being assumed to be the generated production configuration. ## Local fault reproduction The actual original/current executor functions were run against loopback-only synthetic HTTP servers, with Kubernetes imports stubbed. HTTP/cron dependencies were extracted from the original image: requests 2.32.4, urllib3 2.3.0, croniter 1.4.1, python-dateutil 2.9.0.post0. The host Python was 3.12.3; this was not a Kubernetes controller test or execution of the full image. The mock CouchDB returns HTTP 200 for readiness, then supplies the selected fault on `POST /nuvolaris_whisks/_find` (or the corresponding rebranded path). Controller faults are supplied after valid synthetic action/subject pages. | Fault/control | Original and current main result | |---|---| | Empty scan / healthy action invocation and autoexec update | Exit 0 | | CouchDB responds after 0.4 seconds | Waits for the response, then exits 0 | | CouchDB accepts the connection but does not respond to _find | Still blocked when the external one-second test watchdog terminates it | | Immediate HTTP 500 from _find | No POST retry; exit 0 | | One valid page followed by HTTP 500 | Partial action list can be executed; exit 0 | | Non-empty docs with missing bookmark | KeyError; exit 1 | | Non-empty docs with blank/repeated bookmark | Does not terminate before the watchdog | | Unending sequence of advancing bookmarks | No page bound; does not terminate before the watchdog | | Controller connection refused / POST or PUT returns 500 | Helper returns false, process exits 0 | | Controller accepts the connection but POST/PUT does not respond | Does not terminate before the watchdog | | CouchDB unreachable at readiness, using the real 30-second window | Retries readiness and exits 0 after approximately 31.1 seconds | The watchdog termination is an observation limit, not an application-generated timeout or exit code. No claim is made that a one-second test reproduces eight days of production behavior. ## Diagnosis boundaries The unbounded executor plus overlapping Jobs is a demonstrated accumulation mechanism. At one Job/minute, eight days permits 11,520 executions before accounting for completed, failed or missed runs; approximately 9,500 is consistent with that order of magnitude. However: - Missing timeouts alone do **not** identify why the first request stalled. - A completed HTTP 500 by itself does **not** explain Jobs remaining active for days: the tested process exits. - A missing bookmark also exits; the non-advancing-bookmark case requires non-empty documents and has not been established from the production evidence. - The 27 ms response after cleanup supports overload feedback but does not establish or exclude an initial network, storage, CouchDB or query-planning problem. - Missing indexes/full scans have not been demonstrated: no production `_explain`, index inventory or pre-overload execution statistics were supplied. History limits only clean up finished Jobs. Default Job backoff does not terminate a live blocked process, and an incorrectly successful exit does not trigger failure retry. A Job execution deadline must be set at `jobTemplate.spec.activeDeadlineSeconds`; `startingDeadlineSeconds` is not a runtime bound. ## Existing later fix [PR #144 / 14e7a309](https://github.com/apache/openserverless-operator/commit/14e7a309c770b1f6e3bc79b60d2fdd65e711faf2) isolates malformed cron annotations and bounds their size. It is already in the main revision checked above, but does not add HTTP timeouts, pagination progress checks, Job deadlines or a concurrency policy. Operational patches on the affected cluster have not been treated as upstream fixes. ## Expected correction and regression coverage Before implementing a production fix, the intended behavior should include: - Bounded HTTP connect/read waits and explicit scan failures, including a 500 on a later page. - Pagination progress validation and a finite work bound. - A nonzero process exit for dependency failures, while preserving per-action isolation. - A concurrency policy and Job deadline that prevent unlimited overlap, plus explicit resource settings. - No blind retries of action-invocation POSTs: acceptance followed by a lost response can otherwise cause duplicate invocations. Regression coverage should include every fault above, normal multi-page scans, healthy invocation/autoexec updates, generated manifests, and real Kubernetes deadline/backoff behavior with Pending Pods. The concurrency/deadline choice also needs scheduling semantics reviewed: `Forbid` can skip ticks when a scan takes longer than the interval, and the existing executor has no durable catch-up cursor or exactly-once guarantee. No change has been deployed as part of this investigation, and no fix PR accompanies this report. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
