Deep070203 opened a new pull request, #20814:
URL: https://github.com/apache/druid/pull/20814
Fixes #20432.
### Description
`KubernetesClusterDockerTest` has intermittently failed in CI
(`docker-tests`) during class setup because K3s pods occasionally failed to
become `Ready=True` within 300 s. When the timeout occurred,
`K3sClusterResource.waitUntilPodIsReady` threw an `ISE` reporting only the pod
name without any diagnostic details (no pod phase, conditions, container
states, restart counts, or logs), making it difficult to determine why the pod
stalled.
#### Added readiness probe and tuned heap in `manifests/druid-service.yaml`
- Added an HTTP GET `/status/health` `readinessProbe` with
`initialDelaySeconds: 5`, `periodSeconds: 5`, and `failureThreshold: 30` to
`manifests/druid-service.yaml`, matching the existing pattern in
`druid-service-with-operator.yaml`. With this probe, Kubernetes sets the pod
condition `Ready=True` only after the Druid service starts and reports healthy,
and records `Unhealthy` warning events if the probe fails.
- Bumped `DRUID_XMX` from `128m` to `256m` in `manifests/druid-service.yaml`
to provide sufficient heap headroom during service initialization on
resource-constrained CI runners and prevent OOM/crash restarts.
#### Added synchronization for service pod creation in `K3sClusterResource`
- Updated `K3sClusterResource.waitUntilPodsAreReady` to wait until all
expected service pods are created by the Deployment controllers before
evaluating readiness conditions, avoiding race conditions where a subset of
pods was evaluated before all service pods appeared in the namespace.
#### Added rich diagnostics on pod readiness timeout
- Updated `K3sClusterResource.waitUntilPodIsReady` on timeout to query
Fabric8 and assemble a detailed diagnostic report in the exception message:
- Pod phase and conditions (`Ready`, `ContainersReady`, `Initialized`,
`PodScheduled`) with reasons and messages.
- Container statuses including ready state, restart count, current state
(waiting reason, terminated exit code/reason), and last terminated state (e.g.,
`OOMKilled`, exit code 137).
- Relevant Kubernetes cluster events from `client.v1().events()` for the
pod (e.g. `BackOff`, `Unhealthy`, `FailedScheduling`).
- Tail of the container log.
#### Added pod log dumping for CI artifact collection
- Added `dumpPodLogs` in `K3sClusterResource` to write full pod logs to
`druid-container-logs/<testClassName>/k3s-pods/<podName>.log` upon timeout or
health check failure, enabling GitHub Actions CI to archive and upload them as
part of the `failure-druid-container-logs` artifact.
#### Added `K3sClusterResourceTest` unit tests
- Added comprehensive unit tests in `K3sClusterResourceTest` verifying
diagnostic formatting across `Running`, `CrashLoopBackOff` (with `OOMKilled`
last state), and readiness probe failure scenarios, as well as verifying that
`manifests/druid-service.yaml` generates the expected `readinessProbe` and heap
configuration when processed by `K3sDruidService`.
#### Release note
None (internal test and CI stabilization improvement).
<hr>
##### Key changed/added classes in this PR
* `K3sClusterResource`
* `K3sClusterResourceTest`
* `manifests/druid-service.yaml`
<hr>
This PR has:
- [x] been self-reviewed.
- [ ] added documentation for new or modified features or behaviors.
- [ ] a release note entry in the PR description.
- [ ] added Javadocs for most classes and all non-trivial methods. Linked
related entities via Javadoc links.
- [ ] added or updated version, license, or notice information in
[licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
- [x] added comments explaining the "why" and the intent of the code
wherever would not be obvious for an unfamiliar reader.
- [x] added unit tests or modified existing tests to cover new code paths,
ensuring the threshold for [code
coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md)
is met.
- [ ] added integration tests.
- [ ] been tested in a test Druid cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
