Tracking the issue of the integ tests failures here: https://github.com/apache/iceberg/issues/18246 There is an open PR to get CI unblocked for now: https://github.com/apache/iceberg/pull/18247
On Thu, Sep 24, 2026 at 1:24 PM Steven Wu <[email protected]> wrote: > Where are we in this process? > > We have some Kafka connect integration test failures in the CI build that > seem related to the minio problem. e.g. > > https://github.com/apache/iceberg/actions/runs/36037765015/job/107788134738?pr=18213 > > https://github.com/apache/iceberg/actions/runs/36047551910/job/107794733276?pr=18248 > > Docker compose loads the MinIO service. > > https://github.com/apache/iceberg/blob/main/kafka-connect/kafka-connect-runtime/docker/docker-compose.yml#L23 > > > On Mon, Sep 14, 2026 at 11:12 AM Neelesh Salian <[email protected]> > wrote: > >> I know in another thread there were some CVE concerns raised about RustFS >> (mailing list thread titled: "[DISCUSS] Replacing minio with RustFS in >> quick start demo.") >> I see a blog referencing them is here: >> https://rustfs.com/blog/patch-release-rustfs-1-0-0-alpha-79-released/ >> Ok with either choice as long as there is long term support for RustFs >> and for security patches with prompt fixes when CVEs show up. >> >> On Mon, Sep 14, 2026 at 11:02 AM Szehon Ho <[email protected]> >> wrote: >> >>> It also sounds good to me, and also don't have any other strong >>> preference. >>> >>> Thanks >>> Szehon >>> >>> On Mon, Sep 14, 2026 at 9:03 AM Xuanwo <[email protected]> wrote: >>> >>>> RustFS LGTM. >>>> >>>> It's just for CI and should be easy to change in the future. I agree >>>> with Kevin that let's just move. >>>> >>>> On Mon, Sep 14, 2026, at 23:55, Kevin Liu wrote: >>>> >>>> Thanks for the PRs to fix the MinIO issue blocking CI. I think the >>>> fixes have now been merged across all the affected Iceberg repos. >>>> >>>> +1 on moving away from MinIO. Last time we discussed this, I think >>>> there was general agreement on moving to another object store, but we >>>> didn’t settle on which one. >>>> >>>> SeaweedFS seems fine, but PyIceberg has already moved to RustFS and >>>> Polaris is also using it. I don’t have a strong preference between the >>>> different options, so I think using RustFS across the Iceberg repos for >>>> consistency would be the simplest option, unless there’s something we >>>> specifically need from SeaweedFS. >>>> >>>> I think we should just go ahead and make the change. I can help review >>>> the PR. >>>> >>>> On Fri, Sep 11, 2026 at 6:21 PM Sreesh Maheshwar < >>>> [email protected]> wrote: >>>> >>>> Hi all, >>>> >>>> Wanted to revive this question - today, we saw some CI blockages [1] >>>> across Iceberg projects due to MinIO images being removed from Docker Hub, >>>> which adds another data point to moving away from MinIO. >>>> >>>> Worth noting that have been similar threads to this one in the past >>>> [2], [3] - also, Polaris now uses RustFS for its quickstart, with RustFS >>>> and Floci for integration tests (separating those concerns) [4], and >>>> PyIceberg has moved to RustFS [5]. >>>> >>>> Wondering what the community now thinks about moving away from MinIO >>>> for quickstarts and integration tests? >>>> >>>> [1] Iceberg Java: https://github.com/apache/iceberg/pull/18071 >>>> [2] https://lists.apache.org/thread/vnw9jonmfcsz6bwojhfch1nmywyl50h3 >>>> [3] https://lists.apache.org/thread/prmpbfy6v20spsmts19hs8gbkkxo16vv >>>> [4] https://lists.apache.org/thread/8o31ly7cd8ov70opjbtg630qlhrfl5yh >>>> [5] https://github.com/apache/iceberg-python/pull/3928 >>>> >>>> >>>> Thanks, >>>> >>>> Sreesh >>>> >>>> On Fri, Mar 20, 2026 at 5:38 AM Chris Lu <[email protected]> wrote: >>>> >>>> >>>> Hi everyone, >>>> >>>> I’ve been following the recent discussions around S3-compatible >>>> backends for local testing and CI, especially the challenges around S3 >>>> compatibility and credential-related testing. >>>> >>>> I’d like to propose an approach to help address: >>>> https://github.com/apache/iceberg/issues/14638 >>>> >>>> I’ve opened an initial PR that switches the test backend from MinIO to >>>> SeaweedFS: >>>> https://github.com/apache/iceberg/pull/15577 >>>> >>>> The goal is to improve test reliability and better support scenarios >>>> that are currently difficult to validate. >>>> >>>> SeaweedFS is an S3-compatible storage system with support for IAM-style >>>> access control and STS flows. One reason I explored this direction is that >>>> it may help with testing credential-related scenarios (e.g., temporary >>>> credentials / vended credentials), which have been discussed recently. >>>> >>>> There are also a couple of areas that could potentially expand test >>>> coverage over time: >>>> >>>> - Table-oriented bucket layouts could make it easier to exercise >>>> Iceberg-specific storage patterns that are currently hard to simulate in CI >>>> >>>> - Support for IAM and STS flows could allow better validation of >>>> credential-related behaviors used by some catalog integrations >>>> >>>> The current PR is intentionally minimal and focuses only on getting the >>>> existing tests running. If this direction is useful, I’m planning to follow >>>> up with additional integration tests (e.g., around table bucket behaviors). >>>> >>>> I want to emphasize that this is mainly to address #14638 and improve >>>> coverage. If replacing the backend is too disruptive, I’m also happy to >>>> explore alternatives such as running multiple backends or limiting usage to >>>> specific scenarios. >>>> >>>> I’d really appreciate feedback on: >>>> >>>> - Whether switching the backend is a reasonable approach for #14638 >>>> - Any concerns around CI stability or maintenance >>>> - Whether running multiple backends would be preferable >>>> - Any specific scenarios we should prioritize testing >>>> >>>> For context, I’m the author of SeaweedFS. SeaweedFS is open source >>>> under the Apache 2.0 license. >>>> >>>> Thanks for your time and guidance! >>>> >>>> Chris >>>> >>>> Xuanwo >>>> >>>> https://xuanwo.io/ >>>> >>>>
