This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion.git
The following commit(s) were added to refs/heads/main by this push:
new bb21f51013 fix(ci): pull the MinIO test image from quay.io (#25216)
bb21f51013 is described below
commit bb21f51013a3f476c5e510de745c77e194fcb584
Author: Oleks V <[email protected]>
AuthorDate: Sat Sep 12 06:01:30 2026 +0000
fix(ci): pull the MinIO test image from quay.io (#25216)
MinIO withdrew the `minio/minio` repository from Docker Hub on
2026-09-11. The repository itself 404s, so the image that
`testcontainers-modules` pins no longer resolves and every
`datafusion-cli` storage integration test panics on startup.
Every `testcontainers-modules` version DataFusion has used (0.12 through
0.15) pins the identical tag, so downgrading does not help. quay.io
still serves that tag, so override the registry only and leave the tag
coming from the crate.
Also tighten `minio_image_matches_ci_prepull`: it matched a bare image
reference, which `quay.io/minio/minio:<tag>` satisfies as a substring,
and it now checks that the override still mirrors the crate's image
name.
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->
- Closes #25215 .
## Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
Please explain the problem you are trying to solve in terms of the
user-visible
behavior, rather than the implementation.
For example, "The code in `foo.rs` doesn't handle nulls" is a symptom of
the
implementation. "COUNT(DISTINCT) returns wrong results when the column
contains
nulls" is the user-visible problem.
-->
## What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here, but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->
## What is the testing strategy for this PR?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
Briefly describe how this PR is tested, and point to the specific tests
you added. For example: 'This new feature is covered by the
`sqllogictest` cases added in `foo.slt`'.
If this PR does not add tests, explain why. For example, if the change
is already covered by existing tests, please mention it.
You should also check the `codecov` bot reply on this PR to confirm the
changed code is exercised.
-->
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
---
.github/workflows/rust.yml | 9 +++++----
datafusion-cli/tests/cli_integration.rs | 34 ++++++++++++++++++++++++++-------
2 files changed, 32 insertions(+), 11 deletions(-)
diff --git a/.github/workflows/rust.yml b/.github/workflows/rust.yml
index e51215f991..77357da912 100644
--- a/.github/workflows/rust.yml
+++ b/.github/workflows/rust.yml
@@ -401,12 +401,13 @@ jobs:
# with transient Docker transport errors. The tests retry the pull
# themselves, so a failure here is only a warning.
#
- # MINIO_IMAGE must match the image used by the `minio` module of the
- # `testcontainers-modules` crate. The `minio_image_matches_ci_prepull`
- # test in `datafusion-cli/tests/cli_integration.rs` fails if it drifts.
+ # MINIO_IMAGE must match what the tests start: the tag comes from the
+ # `minio` module of `testcontainers-modules`, the registry from
+ # MINIO_IMAGE_NAME in `datafusion-cli/tests/cli_integration.rs`. The
+ # `minio_image_matches_ci_prepull` test fails if either drifts.
- name: Pre-pull MinIO image
env:
- MINIO_IMAGE: minio/minio:RELEASE.2025-02-28T09-55-16Z
+ MINIO_IMAGE: quay.io/minio/minio:RELEASE.2025-02-28T09-55-16Z
run: |
ci/scripts/retry timeout 120 docker pull "$MINIO_IMAGE" \
|| echo "::warning::Could not pre-pull $MINIO_IMAGE, the tests
will pull it themselves"
diff --git a/datafusion-cli/tests/cli_integration.rs
b/datafusion-cli/tests/cli_integration.rs
index a6591eaea8..a9209bc598 100644
--- a/datafusion-cli/tests/cli_integration.rs
+++ b/datafusion-cli/tests/cli_integration.rs
@@ -49,10 +49,17 @@ fn make_settings() -> Settings {
const MINIO_ROOT_USER: &str = "TEST-DataFusionLogin";
const MINIO_ROOT_PASSWORD: &str = "TEST-DataFusionPassword";
+/// Registry override for the image pinned by `testcontainers-modules`.
+///
+/// MinIO withdrew `minio/minio` from Docker Hub on 2026-09-11. quay.io still
+/// serves the same tag, so only the registry changes here. An unblock, not a
+/// fix: see <https://github.com/apache/datafusion/issues/25215>.
+const MINIO_IMAGE_NAME: &str = "quay.io/minio/minio";
+
/// How many times to try bringing up the MinIO container before failing.
///
-/// Both the Docker Hub image pull and the `mc` calls that provision the bucket
-/// fail intermittently on CI with transient errors such as
+/// Both the image pull and the `mc` calls that provision the bucket fail
+/// intermittently on CI with transient errors such as
/// `bytes remaining on stream`. Retrying is much cheaper than a flaky run.
const MINIO_SETUP_ATTEMPTS: u32 = 3;
@@ -67,8 +74,8 @@ const MINIO_SETUP_TIMEOUT: Duration = Duration::from_mins(3);
/// Docker failures.
///
/// Returns `None` when the test should be skipped, that is when
-/// `TEST_STORAGE_INTEGRATION` is unset or Docker Hub is rate limiting the
image
-/// pull. Panics if the container cannot be started for any other reason.
+/// `TEST_STORAGE_INTEGRATION` is unset or the registry is rate limiting the
+/// image pull. Panics if the container cannot be started for any other reason.
async fn start_minio_or_skip() -> Option<ContainerAsync<minio::MinIO>> {
if env::var("TEST_STORAGE_INTEGRATION").is_err() {
eprintln!("Skipping external storages integration tests");
@@ -85,7 +92,7 @@ async fn start_minio_or_skip() ->
Option<ContainerAsync<minio::MinIO>> {
}
}
-/// A Docker Hub pull rate limit does not clear up within a test run, so the
+/// A registry pull rate limit does not clear up within a test run, so the
/// affected tests are skipped rather than retried.
fn is_docker_pull_rate_limit(error: &str) -> bool {
error.contains("toomanyrequests")
@@ -158,6 +165,7 @@ async fn start_minio_container() ->
Result<ContainerAsync<minio::MinIO>, String>
.expect("Failed to get absolute path for test data");
minio::MinIO::default()
+ .with_name(MINIO_IMAGE_NAME)
.with_env_var("MINIO_ROOT_USER", MINIO_ROOT_USER)
.with_env_var("MINIO_ROOT_PASSWORD", MINIO_ROOT_PASSWORD)
.with_mount(Mount::bind_mount(
@@ -232,10 +240,22 @@ fn minio_image_matches_ci_prepull() {
};
let image = minio::MinIO::default();
- let image_ref = format!("{}:{}", image.name(), image.tag());
+
+ // The override only redirects the registry, so it would silently stop
+ // tracking upstream if the crate ever pinned a different image.
+ assert!(
+ MINIO_IMAGE_NAME.ends_with(&format!("/{}", image.name())),
+ "`testcontainers-modules` now uses `{}`, which MINIO_IMAGE_NAME \
+ (`{MINIO_IMAGE_NAME}`) no longer mirrors.",
+ image.name()
+ );
+
+ // Match the assignment: `minio/minio:<tag>` is a substring of the quay
+ // reference and would pass either way.
+ let image_ref = format!("{MINIO_IMAGE_NAME}:{}", image.tag());
assert!(
- contents.contains(&image_ref),
+ contents.contains(&format!("MINIO_IMAGE: {image_ref}")),
"{} does not pre-pull `{image_ref}`. Update MINIO_IMAGE in the \
`Pre-pull MinIO image` step to match the image used by the tests.",
workflow.display()
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
