oscerd opened a new pull request, #27936: URL: https://github.com/apache/camel/pull/27936
[CAMEL-25436](https://issues.apache.org/jira/browse/CAMEL-25436): the SPIRE-backed ITs of `camel-spiffe` fail in two independent ways. Both are fixed in `camel-test-infra-spiffe`. 1. **Flaky on Jenkins: a readiness race.** The infra declared the stack ready as soon as the agent's Workload API socket existed. A SPIRE server (1.15.3, default config) only passes a new registration entry on to its agents when it next rebuilds its in-memory entry cache, [every 5 s](https://github.com/spiffe/spire/blob/v1.15.3/pkg/server/endpoints/endpoints.go#L55) ([rebuild loop](https://github.com/spiffe/spire/blob/v1.15.3/pkg/server/endpoints/entryfetcher.go#L64-L81)). The agent then picks it up on its next sync. Until then every fetch is answered `PERMISSION_DENIED: no identity issued`. 2. **Broken on rootless Podman/Docker: the wrong uid in the selector.** The entry used `unix:uid:<host uid>`. The agent's unix attestor reads the caller's uid from inside the agent container's user namespace, where rootless Podman shows the engine owner as uid 0, so the selector never matched. ## What Jenkins shows In builds #2273, #2274 and #2277 of `Camel Core (Build and test)/main`, one of the three ubuntu cells fails `SpiffeWorkloadApiIT` 4/4. The IT class order accounts for all 9 cells: | cells | first IT class | what happened | |---|---|---| | 3 failing | `SpiffeWorkloadApiIT` | its one-shot fetches land in the window: 4/4 fail. The failsafe rerun starts a fresh SPIRE stack (`SingletonService` is keyed on the JUnit root store) and fails the same way. | | 6 passing | `SpiffeMutualTlsIT` | its `X509Source` logs `no identity issued` 2-3 times, retries, then gets the SVID. That hides the window from `SpiffeWorkloadApiIT`, which runs after it. | So the race happens on every node. Failsafe's filesystem class order, which differs between nodes, decides whether it shows up. This is also why #27509 did not settle it. Its `waitForAgentHealthy()` gate cannot close the window: `spire-agent healthcheck` [only checks that the gRPC service is SERVING](https://github.com/spiffe/spire/blob/v1.15.3/cmd/spire-agent/cli/healthcheck/healthcheck.go#L89-L99), which the agent already is while it holds no entries. On Jenkins' rootful Docker its rootless detection kept the host uid, so the failures seen after it merged are consistent with this race rather than with that change. ## The fix - **Readiness gate:** after creating the entry, the infra waits (60 s max) for the agent to log `Creating X509-SVID` for `spiffe://example.org/workload`. The agent [logs that](https://github.com/spiffe/spire/blob/v1.15.3/pkg/agent/manager/sync.go#L236) once the entry is in its cache. From then on a JWT fetch matches the entry, and an X509 fetch [waits for the SVID](https://github.com/spiffe/spire/blob/v1.15.3/pkg/agent/manager/cache/lru_cache.go#L687-L707) instead of being denied. The wait reuses Testcontainers' `LogMessageWaitStrategy`, which replays the log from the start, and on timeout the error includes the agent log. - **The uid the agent sees:** the entry is now created after the agent starts. Its uid is the JVM's uid translated through the agent container's `/proc/<pid>/uid_map`, or the kernel overflow uid when the map doesn't cover it. That gives the right answer on rootful engines (unchanged), on rootless Podman/Docker (0) and under Podman `--userns=keep-id` (unchanged), where a "rootless means 0" rule would be wrong. If the map can't be read, it falls back to the host uid, the previous behaviour. - The javadocs no longer list rootless Podman as unsupported. Only test-infra and IT javadocs change. The one new dependency is `assertj-core` at test scope, for the new unit test. ## Testing Local runs on Docker 29.7.2 (rootful) and Podman 5.8.7 (rootless): | engine | code | IT order | result | |---|---|---|---| | rootful Docker | `main` | `SpiffeWorkloadApiIT` only | 2 failures + 2 errors, `no identity issued` (the Jenkins failure) | | rootful Docker | this PR | `SpiffeWorkloadApiIT` only | 4/4 pass | | rootful Docker | this PR | `SpiffeWorkloadApiIT` first (`runOrder=reversealphabetical`), 4 runs | 6/6 pass in each run | | rootful Docker | this PR | `SpiffeMutualTlsIT` first | 6/6 pass | | rootless Podman | `main` | default (`SpiffeMutualTlsIT` first) | 6/6 fail. `SpiffeWorkloadApiIT` still fails 62 s after the agent started, so the cause is the uid, not the race | | rootless Podman | this PR | default, and `SpiffeWorkloadApiIT` first | 6/6 pass, entry registered for `unix:uid:0` | With this PR none of the runs logs `no identity issued`, not even the transient retries every Jenkins cell shows today. The gate releases 4-5 s after the entry is created. - New `SpiffeWorkloadUidTest`: rootful, rootless, keep-id and unmapped `uid_map`s, using the lines as the kernel prints them (checked against real Docker and Podman containers). - `camel-spiffe` unit tests: 46/46. - Full reactor `mvn clean install -DskipTests -DskipITs`: 705/705 modules, no regenerated-file drift. - After rebasing onto `a16cccbde620`, ran the unit tests and both ITs again with `SpiffeWorkloadApiIT` first, on Docker and on Podman: all pass. --- _Claude Code on behalf of @oscerd_ 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
