FrankChen021 opened a new issue, #20312: URL: https://github.com/apache/druid/issues/20312
Triage of CI failures on the last 20 `master` commits as of 2026-09-10 (~06:00 UTC). The 5 newest commits were still running at the time of triage; of the remaining 15, 5 had at least one failed job. All 5 failures are flaky tests or CI infrastructure outages, none is caused by the commit itself: every failing test passed on the neighbouring master commits, and none of the failing runs had been retried. ### Summary | Commit | Failed job | Failure log | Root cause | Verdict | |---|---|---|---|---| | 8d9545ea58 (#20247, clone sync criteria) | docker-tests | [job 102752252468](https://github.com/apache/druid/actions/runs/34439796339/job/102752252468) | `IngestionDockerTest.test_runIndexTask_andKillData` timed out after 240s waiting for `segment/schemaCache/dataSource/removed` from the broker | Flaky | | 61ed0a389d (#20272, portable printf) | unit tests (25, R*,B*,Q*,V*) | [job 102731457738](https://github.com/apache/druid/actions/runs/34432737655/job/102731457738) | `ReferenceCountingResourceHolderTest.testResourceHandlerClearedByJVM` expected leak count 1, got 3; `QueryLaningTest.test_queryUsesLaneInQueryContext_inManualStrategy` timed out after 100s waiting for segments to become available | Flaky | | 61ed0a389d (#20272) | docker-tests | [job 102731457503](https://github.com/apache/druid/actions/runs/34432737655/job/102731457503) | `KubernetesClusterDockerTest` setup: k3s pod `druid-router-*` never became Ready | Flaky / infra | | 8ff36f7ef8 (#20292, granularity lookups) | security vulnerabilities (cron) | [job 102731264787](https://github.com/apache/druid/actions/runs/34432672790/job/102731264787) | OWASP dependency-check: `NvdApiException: NVD Returned Status Code: 503` | Infra | | 3985d927a6 (#20271, compact object headers) | unit tests (25, K*,E*,W*,Z*,Y*,X*) | [job 102727804299](https://github.com/apache/druid/actions/runs/34431507380/job/102727804299) | `KubernetesTaskRunnerTest.test_start_whenDeserializationExceptionThrown_isIgnored`: EasyMock unexpected call `peonClient.deleteCompletedPeonJobsOlderThan(...)`, failed all 4 attempts | Flaky (race) | | c7c73625a5 (#20277, jetty bump) | unit tests (25, S*) | [job 102713226586](https://github.com/apache/druid/actions/runs/34426655504/job/102713226586) | `setup_test_profiling_env.sh` exited 35 before Maven started: curl SSL connect error downloading `jfr-profiler-1.0.0.jar` from static.imply.io | Infra | ### Analysis and suggested fixes **1. `KubernetesTaskRunnerTest.test_start_whenDeserializationExceptionThrown_isIgnored`** `KubernetesTaskRunner.start()` schedules `client.deleteCompletedPeonJobsOlderThan(...)` on a real `ScheduledExecutorService` with a 1 ms initial delay. Whether it fires before the test's `verifyAll()` depends on thread scheduling, so the strict `@Mock peonClient` sometimes sees an unexpected call. It failed 4/4 attempts in one run because the retries run in the same loaded JVM. The commit only added a JVM flag to surefire and is unrelated. #20285 hit the same failure independently. Suggested fix: expect the cleanup call with `.anyTimes()` (or use a nice mock for `peonClient`); better, inject the cleanup `ScheduledExecutorService` into `KubernetesTaskRunner` so tests can supply a manual/no-op executor. **2. `ReferenceCountingResourceHolderTest.testResourceHandlerClearedByJVM`** `ReferenceCountingResourceHolder.LEAKED_RESOURCES` is a JVM-global static counter and surefire runs with `reuseForks=true`. The test's `System.gc()` loop also collects unclosed holders leaked by earlier test classes in the same fork, so the delta is >1. Suggested fix: assert `leakedResources() >= initial + 1` together with `released.get()`, or drop the counter assertion and rely only on the `released` flag which is specific to this test. **3. `QueryLaningTest` / `IngestionDockerTest.test_runIndexTask_andKillData`** Both wait on a single emitted metric with a fixed timeout in an embedded cluster on a shared GitHub runner; a slow segment load or a missed emission tick trips it. Suggested fix: poll the actual state (segment availability via `sys.segments`, or datasource absence in the broker schema) with retries instead of waiting for one metric event, as #19416 already did for the `sys.segments` part of this test. Raising the 100s timeout in `EmbeddedClusterApis.waitForAllSegmentsToBeAvailable` is a cheaper stopgap. **4. `KubernetesClusterDockerTest` (router pod not Ready)** Same image build and test passed on the very next master commit; the `printf` change in the failing commit is behaviour-preserving. Suggested fix: on timeout in `K3sClusterResource.waitUntilPodIsReady`, include `kubectl describe pod` output and container logs in the failure message so the next occurrence is diagnosable; consider a longer readiness timeout for the router, which starts last. **5. Security vulnerabilities cron job (NVD 503)** Suggested fix: cache the NVD database across runs and/or configure `nvdApiKey` + `nvdMaxRetryCount` in `dependency-check-maven`; treat `UpdateException` as a soft failure. **6. `setup_test_profiling_env.sh` exit 35** An external CDN hiccup failed a whole unit test shard before any test ran. Suggested fix: make the profiler download non-fatal, e.g. `curl --retry 3 ... || { echo "JFR_PROFILER_ARG_LINE="; exit 0; }`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
