goutamadwant opened a new pull request, #12478: URL: https://github.com/apache/seatunnel/pull/12478
### Purpose of this pull request Every S3-backed E2E test currently fails before it runs, on every PR and on `dev`, because the MinIO image can no longer be pulled from anywhere MinIO publishes it. #12287 and #12302 moved these tests from Docker Hub `minio/minio` to `quay.io/minio/minio:RELEASE.2024-06-13T22-53-53Z`. Since about 2026-09-24 12:30 UTC the quay.io image fails too: ``` $ docker pull quay.io/minio/minio:RELEASE.2024-06-13T22-53-53Z Error response from daemon: unknown: failed to resolve reference "quay.io/minio/minio:RELEASE.2024-06-13T22-53-53Z": unexpected status from HEAD request to https://quay.io/v2/minio/minio/manifests/RELEASE.2024-06-13T22-53-53Z: 401 UNAUTHORIZED $ docker pull minio/minio:RELEASE.2024-06-13T22-53-53Z Error response from daemon: pull access denied for minio/minio, repository does not exist or may require 'docker login' ``` On CI, Testcontainers retries the pull for 2 minutes and then the class fails in `startUp` with `ContainerFetchException`: ``` InternalServerErrorException: Status 500: {"message":"unauthorized: access to the requested resource is not authorized"} WARN tc.quay.io/minio/minio:RELEASE.2024-06-13T22-53-53Z - Retrying pull for image: quay.io/minio/minio:RELEASE.2024-06-13T22-53-53Z (119s remaining) ``` Examples: - `dev` build [35999263398](https://github.com/apache/seatunnel/actions/runs/35999263398) and nightly [36025021304](https://github.com/apache/seatunnel/actions/runs/36025021304): `IcebergSourceIT`, `S3FileConnectDryRunIT`, `S3FileWithFilterIT`, `PaimonWithS3IT`, `DatabendIT` and `DatabendCDCSinkIT` all fail in `startUp` with `ContainerFetch Can't get Docker image`. Jobs: [all-connectors-it-2 (8)](https://github.com/apache/seatunnel/actions/runs/36025021304/job/107723799532), [all-connectors-it-6 (8)](https://github.com/apache/seatunnel/actions/runs/36025021304/job/107723800378), [all-connectors-it-7 (8)](https://github.com/apache/seatunnel/actions/runs/36025021304/job/107723800319), [paimon-connector-it (8)](https://github.com/apache/seatunnel/actions/runs/36025021304/job/107723800395). - Fork PR runs: [all-connectors-it-7 (8)](https://github.com/goutamadwant/seatunnel/actions/runs/36093925266/job/107942628674), [all-connectors-it-2 (11)](https://github.com/goutamadwant/seatunnel/actions/runs/36093925266/job/107942628955), [all-connectors-it-6 (8)](https://github.com/goutamadwant/seatunnel/actions/runs/36093925266/job/107942628990), [paimon-connector-it (8)](https://github.com/goutamadwant/seatunnel/actions/runs/36093925266/job/107942629422). - The same signature shows up on at least 15 other open PRs since 2026-09-24. Affected test classes: the six above, plus `HudiSeatunnelS3MultiTableIT` and `HudiSparkS3MultiTableIT`, which use the same image. **Change** Use Bitnami's `bitnamilegacy/minio:2024.6.13` instead, pinned by digest: `bitnamilegacy/minio@sha256:aa1752895e6d2b420e394d55241d5b2c948960715db0a50bb648f430e447e645` (the multi-arch index, amd64 and arm64). It can be pulled anonymously. Its binary reports `DEVELOPMENT.2024-06-13T22-53-53Z` (commit `20960b6a2ddb`), which is the same source as the `RELEASE.2024-06-13T22-53-53Z` the tests already pin; it is a source build, hence the `DEVELOPMENT` prefix. The Java tests use the digest-only reference; the compose file uses `2024.6.13@sha256:...`. The image differs from the official one in two ways, so each container now starts the server with an explicit command: - Its entrypoint runs the container command as is (the official entrypoint prepends `minio`), so the command is `minio server ...`. The four `MinIOContainer` users keep `--console-address :9001`, as before. - It runs as a non-root user and has no `/data`, so the server uses the image's own data volume `/bitnami/minio/data`. Per file: - Databend (`DatabendIT`, `DatabendCDCSinkIT`), `S3FileConnectDryRunIT`: new image and command. - `S3FileWithFilterIT`: new image and command. It also waits for `/minio/health/ready` instead of the first log line, because this image prints a banner before the server starts listening. `S3FileConnectDryRunIT` already used the same wait. - Hudi (both), Iceberg S3, Paimon: these use Testcontainers' `MinIOContainer`, whose constructor sets `server --console-address :9001 /data`. The command is overridden after construction with `minio server --console-address :9001 /bitnami/minio/data`. `asCompatibleSubstituteFor("minio/minio")` stays and works with the digest reference. - `seatunnel-cli/benchmark/docker/docker-compose.yml` and its README had the same broken references (`quay.io/minio/minio` and `minio/mc`, which is also gone). The `minio-init` step now uses the same image, which includes `mc`. This file is not used by CI. Test logic, credentials, buckets and assertions are unchanged. The Paimon privilege tests use Paimon's file-based privilege manager on the bucket, not MinIO users or policies, so they need no MinIO admin features. Alternatives I looked at: - `cgr.dev/chainguard/minio`: the free tier only serves `latest`, which is a newer MinIO build and cannot be pinned by tag. - SeaweedFS or RustFS: different S3 implementations. `MinIOContainer` waits on MinIO's health endpoint, and `S3FileConnectDryRunIT` asserts MinIO's exact error codes (`403`, `NoSuchBucket`). `bitnamilegacy` is also a third-party namespace that could be removed later. A longer-term option is to publish a pinned copy under an Apache-controlled registry, as `publish-e2e-spark-images.yaml` does for the Spark images. I left that out of this PR. **Before vs after** (local, macOS arm64, Docker Desktop; engine images already local) | Test class | Engines run | Before (`dev`) | After, JDK 8 | After, JDK 11 | | --- | --- | --- | --- | --- | | S3FileConnectDryRunIT | none (no job) | error in `startUp`: `ContainerFetch Can't get Docker image` (401 from quay.io) after 432 s of pull retries | 6/6 pass | 6/6 pass | | S3FileWithFilterIT | Zeta | same error, after 443 s | 3/3 pass | 3/3 pass | | PaimonWithS3IT (includes the 2 privilege tests) | Zeta | same error on CI | 4/4 pass | 4/4 pass | | HudiSeatunnelS3MultiTableIT | Zeta | same error on CI | 1/1 pass | 1/1 pass | | HudiSparkS3MultiTableIT | Spark 3.3 | same error on CI | 1/1 pass | 1/1 pass | | IcebergSourceIT (s3) | Spark 3.3 | same error on CI | 1/1 pass | not run | | DatabendIT | Zeta, Flink 1.18, Flink 1.20, Spark 3.3 | same error on CI | 24/24 pass | not run | | DatabendCDCSinkIT | Zeta, Flink 1.18, Flink 1.20, Spark 3.3 | same error on CI | 4/4 pass | not run | Engines are the PR-mode set with `RUN_ALL_CONTAINER=false`. Hudi Spark and Iceberg S3 are already disabled on the other engines by `@DisabledOnContainer`. ### Does this PR introduce _any_ user-facing change? No. Only E2E test containers and the CLI benchmark's local Docker Compose file change. ### How was this patch tested? Local runs on macOS arm64 with Docker Desktop. JDK 8 is 1.8.0_172 and JDK 11 is Temurin 11.0.19. Env: `TEST_IN_PR=true RUN_ALL_CONTAINER=false RUN_ZETA_CONTAINER=true`. - Before: with the `dev` versions of the two S3File tests, `./mvnw -B verify -DskipUT=true -DskipIT=false -pl seatunnel-e2e/seatunnel-connector-v2-e2e/connector-file-s3-e2e -am -Dit.test='S3FileConnectDryRunIT,S3FileWithFilterIT'` (JDK 8). Both classes error in `startUp` with `ContainerFetch`, and the log shows `401 UNAUTHORIZED` from `quay.io`. - After: the same command for each module with its test classes, as in the table. JDK 8 covered all 8 classes. JDK 11 covered the S3File, Paimon and Hudi classes. - After pinning by digest and restoring `--console-address :9001`, reran on JDK 8: S3FileConnectDryRunIT 6/6, S3FileWithFilterIT 3/3, PaimonWithS3IT 4/4, both Hudi classes 1/1, IcebergSourceIT 1/1. This confirms `MinIOContainer` accepts the digest reference with `asCompatibleSubstituteFor("minio/minio")`. Since their 24/24 and 4/4 runs, the Databend classes only changed their image reference (to the same digest) and comment, and were not rerun. - Benchmark compose (with the digest-pinned image): `docker compose up -d minio minio-init`. `minio` became healthy through `mc ready local`, and `minio-init` exited 0 after `Bucket created successfully 'local/bench'`. - `./mvnw spotless:apply` on the changed modules. Not verified locally: - IcebergSourceIT and the Databend classes on JDK 11. The engine images run JDK 8 whatever the host JDK is, so the host JDK only affects the test client side. The PR's CI covers both JDKs. - Flink 1.13/1.15 and Spark 2.4. None of the Hudi, Iceberg S3 or Paimon S3 classes run on these. Databend runs on them in CI only. - linux/amd64 locally. The image is multi-arch, and CI runs amd64. During one local Databend run, a Flink 1.18 TaskManager container failed to start. That is a local engine-container problem unrelated to this change. A rerun passed 24/24. ### Check list * [ ] If any new Jar binary package adding in your PR, please add License Notice according [New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md) * [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs * [ ] If necessary, please update `incompatible-changes.md` to describe the incompatibility caused by this PR. * [ ] If you are contributing the connector code, please check that the following files are updated: 1. Update [plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it 2. Update the pom file of [seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml) 3. Add ci label in [label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml) 4. Add e2e testcase in [seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/) 5. Update connector [plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
