FrankChen021 opened a new issue, #20428: URL: https://github.com/apache/druid/issues/20428
_This issue was generated automatically by Claude Code (Anthropic's AI coding agent) running a scheduled CI-triage routine on behalf of @FrankChen021. Analysis and suggested fixes are AI-produced; please verify before acting on them._ This triage covers the master commits from 2026-09-25 (local time, UTC+8). 6 of the 7 commits had at least one failed push-triggered job, and 5 of the 6 `Unit & Integration tests CI` runs had failures. Most of the failures (34 of 35 failed jobs across 4 commits) come from one CI infrastructure outage: Testcontainers could not pull `quay.io/minio/minio:latest` within its 2-minute timeout. That broke every MinIO-backed embedded test in every shard and in `docker-tests`, and pushed the `S*` shard past its 60-minute timeout. The outage was persistent: every run from `bce8f60af2` through `f7fbbe7457` failed this way, while the last master commit on 2026-09-24 (`fe9dd22977`) was green. It stopped once #20420 switched the S3 embedded tests from MinIO to RustFS. The remaining failures are `TaskQueueScaleTest.doMassLaunchAndExit`, a known snapshot-counting race that #20291 (merged 2026-09-26) fixes, and one `ConsulDiscoveryPlainDockerTest.test_streamLogs_ofCancelledTask` failure. A single Maven parent-POM resolution error also hit `QTest 2/4`. None of these failures comes from the commit it ran on, and the master runs after #20420 and #20291 (`8bdc64135f`, `c7f1f0adb3`, `0387ffa902`, `017c18ab6d`) are green. No re-runs had been attempted on any of the failed runs. ### Summary | Commit | Failed job | Failure log | Root cause | Verdict | |---|---|---|---|---| | bce8f60af2 (#20415, bump actions-timeline) | `docker-tests` | [job 107809810714](https://github.com/apache/druid/actions/runs/36052068790/job/107809810714) | `ConsulDiscovery{Plain,TLS,MTLS}DockerTest`, `IngestionDockerTest` `initializationError`: `Can't get Docker image: quay.io/minio/minio:latest` | Infra (fix in #20420) | | bce8f60af2 (#20415, bump actions-timeline) | `unit tests (25, M*,A*,V*,W*)` | [job 107809811540](https://github.com/apache/druid/actions/runs/36052068790/job/107809811540) | `MinIOStorageResourceTest.testMinIOContainerLifecycle` (timed out after 120 s); `MinIOStorageTest` `initializationError`: MinIO image pull | Infra (fix in #20420) | | bce8f60af2 (#20415, bump actions-timeline) | `unit tests (25, H*,D*,T*,O*)` | [job 107809811587](https://github.com/apache/druid/actions/runs/36052068790/job/107809811587) | `HdfsToS3ParallelIndexTest`: MinIO image pull; `TaskQueueScaleTest.doMassLaunchAndExit`: summed task count 985/992/999/997 vs 1000, failed all 4 attempts | Infra (fix in #20420); Flaky (race) (fix in #20291) | | bce8f60af2 (#20415, bump actions-timeline) | `unit tests (25, S*)` | [job 107809811638](https://github.com/apache/druid/actions/runs/36052068790/job/107809811638) | `S3ToHdfsParallelIndexTest`, `S3PerfTest$MinIO_{Crt,Netty,Sync}`, `S3ExternQueryTest`: MinIO image pull | Infra (fix in #20420) | | bce8f60af2 (#20415, bump actions-timeline) | `unit tests (25, N*,Q*)` | [job 107809811643](https://github.com/apache/druid/actions/runs/36052068790/job/107809811643) | `QueryVirtualStorageTest` `initializationError`: MinIO image pull | Infra (fix in #20420) | | bce8f60af2 (#20415, bump actions-timeline) | `unit tests (25, I*,L*,J*)` | [job 107809811746](https://github.com/apache/druid/actions/runs/36052068790/job/107809811746) | `IngestionSmokeTest`, `IngestionBackwardCompatibilityDockerTest`, `ITS3SQLBasedIngestionTest`, `ITS3ToS3ParallelIndexTest`: MinIO image pull | Infra (fix in #20420) | | bce8f60af2 (#20415, bump actions-timeline) | `unit tests (25, R*,B*,P*)` | [job 107809811796](https://github.com/apache/druid/actions/runs/36052068790/job/107809811796) | `PartialProjectionLoadRuleQueryTest`: MinIO image pull (`BalancingStrategiesTest.testClusterGetsBalancedWhenServerIsAdded` also failed once and passed on retry) | Infra (fix in #20420) | | bce8f60af2 (#20415, bump actions-timeline) | `unit tests (25, E*,G*,F*)` | [job 107809811808](https://github.com/apache/druid/actions/runs/36052068790/job/107809811808) | `EmbeddedDurableShuffleStorageTest`: MinIO image pull (`EmbeddedKafkaSupervisorTest.test_runSupervisor_withEmptyDimension` also failed once and passed on retry) | Infra (fix in #20420) | | 27180f525c (#19815, web console dependency vulnerabilities) | `docker-tests` | [job 107864720389](https://github.com/apache/druid/actions/runs/36068816270/job/107864720389) | `ConsulDiscovery{Plain,TLS,MTLS}DockerTest`, `IngestionDockerTest`: MinIO image pull | Infra (fix in #20420) | | 27180f525c (#19815, web console dependency vulnerabilities) | `unit tests (25, N*,Q*)` | [job 107864721174](https://github.com/apache/druid/actions/runs/36068816270/job/107864721174) | `QueryVirtualStorageTest`: MinIO image pull | Infra (fix in #20420) | | 27180f525c (#19815, web console dependency vulnerabilities) | `unit tests (25, H*,D*,T*,O*)` | [job 107864721217](https://github.com/apache/druid/actions/runs/36068816270/job/107864721217) | `HdfsToS3ParallelIndexTest`: MinIO image pull | Infra (fix in #20420) | | 27180f525c (#19815, web console dependency vulnerabilities) | `unit tests (25, M*,A*,V*,W*)` | [job 107864721227](https://github.com/apache/druid/actions/runs/36068816270/job/107864721227) | `MinIOStorageResourceTest`, `MinIOStorageTest`: MinIO image pull | Infra (fix in #20420) | | 27180f525c (#19815, web console dependency vulnerabilities) | `unit tests (25, E*,G*,F*)` | [job 107864721241](https://github.com/apache/druid/actions/runs/36068816270/job/107864721241) | `EmbeddedDurableShuffleStorageTest`: MinIO image pull (`EmbeddedKafkaSupervisorTest.test_runSupervisor_withEmptyDimension` also failed once and passed on retry) | Infra (fix in #20420) | | 27180f525c (#19815, web console dependency vulnerabilities) | `unit tests (25, R*,B*,P*)` | [job 107864721312](https://github.com/apache/druid/actions/runs/36068816270/job/107864721312) | `PartialProjectionLoadRuleQueryTest`: MinIO image pull | Infra (fix in #20420) | | 27180f525c (#19815, web console dependency vulnerabilities) | `unit tests (25, I*,L*,J*)` | [job 107864721340](https://github.com/apache/druid/actions/runs/36068816270/job/107864721340) | `IngestionSmokeTest`, `IngestionBackwardCompatibilityDockerTest`, `ITS3*` tests: MinIO image pull | Infra (fix in #20420) | | 27180f525c (#19815, web console dependency vulnerabilities) | `unit tests (25, S*)` | [job 107864721400](https://github.com/apache/druid/actions/runs/36068816270/job/107864721400) | `S3*` embedded tests (5): MinIO image pull | Infra (fix in #20420) | | 94e290864b (#20410, bump forbiddenapis 3.11) | `docker-tests` | [job 107908402877](https://github.com/apache/druid/actions/runs/36082874795/job/107908402877) | `ConsulDiscovery{Plain,TLS,MTLS}DockerTest`, `IngestionDockerTest`: MinIO image pull | Infra (fix in #20420) | | 94e290864b (#20410, bump forbiddenapis 3.11) | `unit tests (25, S*)` | [job 107908403099](https://github.com/apache/druid/actions/runs/36082874795/job/107908403099) | Step timed out after 60 minutes: 5 MinIO-backed `S3*` tests each spent about 7 minutes retrying the image pull | Infra (fix in #20420) | | 94e290864b (#20410, bump forbiddenapis 3.11) | `unit tests (25, N*,Q*)` | [job 107908403279](https://github.com/apache/druid/actions/runs/36082874795/job/107908403279) | `QueryVirtualStorageTest`: MinIO image pull | Infra (fix in #20420) | | 94e290864b (#20410, bump forbiddenapis 3.11) | `unit tests (25, M*,A*,V*,W*)` | [job 107908403296](https://github.com/apache/druid/actions/runs/36082874795/job/107908403296) | `MinIOStorageResourceTest`, `MinIOStorageTest`: MinIO image pull | Infra (fix in #20420) | | 94e290864b (#20410, bump forbiddenapis 3.11) | `unit tests (25, E*,G*,F*)` | [job 107908403335](https://github.com/apache/druid/actions/runs/36082874795/job/107908403335) | `EmbeddedDurableShuffleStorageTest`: MinIO image pull | Infra (fix in #20420) | | 94e290864b (#20410, bump forbiddenapis 3.11) | `unit tests (25, H*,D*,T*,O*)` | [job 107908403347](https://github.com/apache/druid/actions/runs/36082874795/job/107908403347) | `HdfsToS3ParallelIndexTest`: MinIO image pull | Infra (fix in #20420) | | 94e290864b (#20410, bump forbiddenapis 3.11) | `unit tests (25, R*,B*,P*)` | [job 107908403374](https://github.com/apache/druid/actions/runs/36082874795/job/107908403374) | `PartialProjectionLoadRuleQueryTest`: MinIO image pull | Infra (fix in #20420) | | 94e290864b (#20410, bump forbiddenapis 3.11) | `unit tests (25, I*,L*,J*)` | [job 107908403377](https://github.com/apache/druid/actions/runs/36082874795/job/107908403377) | `IngestionSmokeTest`, `IngestionBackwardCompatibilityDockerTest`, `ITS3*` tests: MinIO image pull | Infra (fix in #20420) | | f7fbbe7457 (#20411, bump hibernate-validator) | `docker-tests` | [job 107908696991](https://github.com/apache/druid/actions/runs/36082973105/job/107908696991) | `ConsulDiscovery{Plain,TLS,MTLS}DockerTest`, `IngestionDockerTest`: MinIO image pull | Infra (fix in #20420) | | f7fbbe7457 (#20411, bump hibernate-validator) | `unit tests (25, H*,D*,T*,O*)` | [job 107908697293](https://github.com/apache/druid/actions/runs/36082973105/job/107908697293) | `HdfsToS3ParallelIndexTest`: MinIO image pull (`TaskQueueScaleTest.doMassLaunchAndExit` also failed once, 984/1000, and passed on retry) | Infra (fix in #20420) | | f7fbbe7457 (#20411, bump hibernate-validator) | `unit tests (25, S*)` | [job 107908697303](https://github.com/apache/druid/actions/runs/36082973105/job/107908697303) | Step timed out after 60 minutes: 5 MinIO-backed `S3*` tests each spent about 7 minutes retrying the image pull | Infra (fix in #20420) | | f7fbbe7457 (#20411, bump hibernate-validator) | `unit tests (25, E*,G*,F*)` | [job 107908697321](https://github.com/apache/druid/actions/runs/36082973105/job/107908697321) | `EmbeddedDurableShuffleStorageTest`: MinIO image pull (`EmbeddedDimensionValueSetShardSpecTest.test_twoSupervisors_withPartitionDimensionValues_segmentsPublishAndQueriesSucceed` also failed once and passed on retry) | Infra (fix in #20420) | | f7fbbe7457 (#20411, bump hibernate-validator) | `unit tests (25, M*,A*,V*,W*)` | [job 107908697338](https://github.com/apache/druid/actions/runs/36082973105/job/107908697338) | `MinIOStorageResourceTest`, `MinIOStorageTest`: MinIO image pull (`WorkerSketchFetcherTest.test_SequentialRetryDisabled_singleFailure` also failed once and passed on retry) | Infra (fix in #20420) | | f7fbbe7457 (#20411, bump hibernate-validator) | `run-separated-tests (QTest 2/4)` | [job 107908697350](https://github.com/apache/druid/actions/runs/36082973105/job/107908697350) | Maven aborted before any test ran: `Non-resolvable parent POM ... org.apache:apache:pom:40 ... from/to ${repoOrgId} (${repoOrgUrl})` | Infra | | f7fbbe7457 (#20411, bump hibernate-validator) | `unit tests (25, R*,B*,P*)` | [job 107908697370](https://github.com/apache/druid/actions/runs/36082973105/job/107908697370) | `PartialProjectionLoadRuleQueryTest`: MinIO image pull | Infra (fix in #20420) | | f7fbbe7457 (#20411, bump hibernate-validator) | `unit tests (25, I*,L*,J*)` | [job 107908697406](https://github.com/apache/druid/actions/runs/36082973105/job/107908697406) | `IngestionSmokeTest`, `IngestionBackwardCompatibilityDockerTest`, `ITS3*` tests: MinIO image pull | Infra (fix in #20420) | | f7fbbe7457 (#20411, bump hibernate-validator) | `unit tests (25, N*,Q*)` | [job 107908697414](https://github.com/apache/druid/actions/runs/36082973105/job/107908697414) | `QueryVirtualStorageTest`: MinIO image pull | Infra (fix in #20420) | | a01e318f96 (#20420, use rustfs instead of minio for s3 embedded-tests) | `unit tests (25, H*,D*,T*,O*)` | [job 107912839489](https://github.com/apache/druid/actions/runs/36084362652/job/107912839489) | `TaskQueueScaleTest.doMassLaunchAndExit`: `running+pending+waiting` = 973/988/997/999 vs expected 1000, failed all 4 attempts | Flaky (race) (fix in #20291) | | 53e814e87c (#20425, batch deletes in DurableStorageCleaner) | `docker-tests` | [job 108039867472](https://github.com/apache/druid/actions/runs/36125282875/job/108039867472) | `ConsulDiscoveryPlainDockerTest.test_streamLogs_ofCancelledTask`: streamed task log was empty (`IngestionSmokeTest.java:398`, `assertFalse(logs.isEmpty())`); the docker job has no surefire retries; the TLS and MTLS variants passed | Flaky (race) | ### Analysis and suggested fixes **1. MinIO Testcontainers image pull (`quay.io/minio/minio:latest`), all unit-test shards and `docker-tests`** Every embedded test that starts a `MinIOStorageResource` (`HdfsToS3ParallelIndexTest`, `S3*`, `ITS3*`, `IngestionSmokeTest`, `QueryVirtualStorageTest`, `PartialProjectionLoadRuleQueryTest`, `EmbeddedDurableShuffleStorageTest`, the Consul docker tests, and others) failed in `EmbeddedClusterTestBase.setup` with `ContainerFetchException: Can't get Docker image`, caused by `ConditionTimeoutException ... RemoteDockerImage was not fulfilled within 2 minutes`. Each affected class then spent about 7 minutes in pull retries, so the `S*` shard (5 such classes) hit its 60-minute step timeout on `94e290864b` and `f7fbbe7457`. The four commits involved only touch CI metadata, web-console dependencies, and Maven dependency versions. `fe9dd22977` (2026-09-24) passed, and every run from `bce8f60af2` to `f7fbbe7457` failed in the same way. That makes this a registry-side outage that persisted until a fix landed, not a flake. The same problem was tracked earlier in #20333 (Docker Hub), which #20331 addressed by moving to Quay. Suggested fix: already done by #20420, which replaces MinIO with RustFS for the S3 embedded tests; `a01e318f96` and later runs no longer pull the MinIO image. To make the next registry outage cheaper, consider pinning the image by tag or digest instead of `:latest`, pre-pulling it once per job with a retry (for example `docker pull ... \|\| sleep 30 && docker pull ...`) so a failure is fast and explicit, and lowering the per-class pull timeout so a registry outage cannot push a shard past its 60-minute limit. **2. `TaskQueueScaleTest.doMassLaunchAndExit` (indexing-service, H\*,D\*,T\*,O\* shard)** This is the same race already triaged in #20312, #20366 and #20414. After adding 1000 `NoopTask`s, the test sums `getRunningTaskCount()`, `getPendingTaskCount()` and `getWaitingTaskCount()`. Those are three separate snapshots taken while the 1 ms management loop is moving tasks from waiting to pending to running, so a task that changes state between two snapshots is counted in neither, and the sum comes out a few short (973 to 999 here). It failed all 4 surefire attempts on `bce8f60af2` and `a01e318f96`, and failed once before passing on `f7fbbe7457`. Neither commit touches `TaskQueue`. Suggested fix: already merged in #20291 (2026-09-26), which drops the summed-snapshot assertion and keeps only the atomic `taskQueue.getTasks().size()` check. #20426 (a separate attempt) was closed. No further action is needed unless the test fails again after `0387ffa902`. **3. `ConsulDiscoveryPlainDockerTest.test_streamLogs_ofCancelledTask` (embedded-tests, `docker-tests`)** The test starts a 100 s `NoopTask`, cancels it, waits for the `task/run/time` FAILED metric, and then polls `TaskLogStreamer.streamTaskLog(taskId, 0)` only until the `Optional` is present. In the Docker/Consul setup, the peon runs in a separate container, and its log is pushed after the process exits. The stream can therefore be present but not yet complete (here it was empty) when the test reads it, which fails `assertFalse(logs.isEmpty())` at `IngestionSmokeTest.java:398`. The TLS and MTLS variants of the same test passed in the same job. #20425 only changes `DurableStorageCleaner` batching and does not affect task logs. #20291 now passes the test's `dataSource` to this `NoopTask` and drains leftover tasks in `cleanUp()`, which reduces interference from earlier tests but does not remove this read-too-early window. Suggested fix: make the poll condition content-based. Wrap the stream-and-read in the `waitForResult` supplier and wait until the returned string contains `"Running task[%s] for [%d] millis"`, rather than waiting only for `Optional::isPresent`. The existing assertions can stay as a final check. **4. `QTest 2/4`: Maven parent POM resolution (quidem-ut, run-separated-tests)** Maven failed while building the project model: `Non-resolvable parent POM for org.apache.druid:druid:39.0.0-SNAPSHOT: ... org.apache:apache:pom:40 (absent): Could not transfer artifact ... from/to ${repoOrgId} (${repoOrgUrl})`. The unexpanded placeholders mean the download of the Apache parent POM from Central failed, and Maven then tried a repository declared with unresolved properties. The same error, with `apache:pom:39`, was reported in #20414. The other 8 unit-test jobs of the same run resolved the parent POM, so this looks like a transient Central/cache miss rather than a problem in #20411. Suggested fix: make sure `org.apache:apache` is included in the restored `~/.m2` cache key (or resolve it in a warm-up `mvn -N dependency:go-offline`/`help:effective-pom` step with retries), and set `-Daether.connector.http.retryHandler.count=3` (or the resolver 2.x equivalent) in `.mvn/maven.config`. If this keeps recurring after #20386 (resolver upgrade to 2.0.23), check whether the new resolver handles the unresolved `${repoOrgUrl}` repository differently. **5. Single-attempt flakes that passed on retry (did not fail their jobs)** Each of these failed once and passed on the surefire rerun: `EmbeddedKafkaSupervisorTest.test_runSupervisor_withEmptyDimension` (twice, both at line 210, a 10 s `waitForEventAggregate` on `ingest/events/processed`), `EmbeddedDimensionValueSetShardSpecTest.test_twoSupervisors_withPartitionDimensionValues_segmentsPublishAndQueriesSucceed` (line 146, same 10 s latch timeout), `BalancingStrategiesTest.testClusterGetsBalancedWhenServerIsAdded` (line 136), and `WorkerSketchFetcherTest.test_SequentialRetryDisabled_singleFailure` (line 335, where the caught exception's message did not contain `Task fetch failed`). The two Kafka tests wait only 10 s for 100 rows on a runner that was also retrying MinIO pulls in parallel test JVMs, so they are timing-sensitive. Suggested fix: give the Kafka `waitForEventAggregate` calls an explicit timeout of 30 to 60 s. In `WorkerSketchFetcherTest`, assert on the root cause (`Throwables.getRootCause(e).getMessage()`) instead of the wrapper's message. Track `BalancingStrategiesTest` if it recurs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
