Hi Yuri, I have finished reviewing the run, and unfortunately, I cannot approve the fs suite at this time.
The number of test failures remains high. While many failures appear infrastructure-related or existing issues, several new failures that seem to be related to the Python version change and possible new test case failures which require RCA. On Mon, Feb 16, 2026 at 4:07 PM Venky Shankar <[email protected]> wrote: > > Hi Yuri, > > On Thu, Feb 12, 2026 at 9:30 PM Yuri Weinstein via ceph-users > <[email protected]> wrote: > > > > Still seeking approvals for: > > > > (smoke - FAILED, was reviewed and approved by Laura and Patrick, see Slack > > thread for details) > > rados - FAILED, Afreen, Travis? ( Laura approved for core) > > > > rgw - FAILED, Adam E approved? > > > > (rbd, krbd - FAILED, Ilya approved) > > fs - FAILED, Venky approved? > > I wasn't able to finish reviewing the run and teuthology results are > unavailable right now due to the lab outage. > > > orch - FAILED, Adam K approved? > > > > upgrade/pacific-x (reef) - FAILED Laura, approved? > > upgrade/quincy-x (reef) - FAILED Laura, approved? > > upgrade/reef-p2p - FAILED Laura, approved? > > > > (powercycle - FAILED Brad, approved) > > ceph-volume - Approved Guillaume? > > > > On Wed, Feb 11, 2026 at 12:10 PM Laura Flores via Dev <[email protected]> wrote: > > > > > The rados suite is approved: > > > https://tracker.ceph.com/projects/rados/wiki/REEF#reef-v1828-httpstrackercephcomissues73906note-1 > > > > > > On Tue, Feb 10, 2026 at 6:33 PM Brad Hubbard via Dev <[email protected]> wrote: > > > > > >> Powercycle approved. > > >> > > >> On Tue, Feb 10, 2026 at 10:35 AM Brad Hubbard via Dev <[email protected]> > > >> wrote: > > >> > > >>> > > >>> > > >>> On Fri, Feb 6, 2026 at 12:36 PM Brad Hubbard <[email protected]> > > >>> wrote: > > >>> > > >>>> > > >>>> > > >>>> On Fri, Feb 6, 2026 at 8:33 AM Brad Hubbard via ceph-users < > > >>>> [email protected]> wrote: > > >>>> > > >>>>> On Fri, Feb 6, 2026 at 7:59 AM Yuri Weinstein via ceph-users < > > >>>>> [email protected]> wrote: > > >>>>> > > >>>>> > We are continuing on this release. > > >>>>> > > > >>>>> > > > >>>>> > powercycle - FAILED Brad, approved? > > >>>>> > > > >>>>> > > >>>>> Unfortunately, I don't see any logs anywhere where I have access to > > >>>>> them so > > >>>>> I have no way of checking this? > > >>>>> > > >>>> > > >>>> OK, I have the latest logs. The three failing jobs appear to have all > > >>>> timed out after running for about six hours. At this time I don't know > > >>>> why > > >>>> but I continue to investigate. > > >>>> > > >>> > > >>> One point of commonality of the three tests that failed is that they are > > >>> all running tests using ceph fuse. I'll reach out to the FS guys to see > > >>> if > > >>> they can help me get to the bottom of these failures. > > >>> > > >>> > > >>>> > > >>>> > > >>>>> > > >>>>> > > >>>>> > ceph-volume - Approved Guillaume? > > >>>>> > > > >>>>> > Patrick was reviewing some tests, please see his comments in the > > >>>>> > same > > >>>>> > tracker. > > >>>>> > TIA > > >>>>> > > > >>>>> > > > >>>>> > On Sat, Dec 13, 2025 at 1:19 AM Ilya Dryomov via ceph-users < > > >>>>> > [email protected]> wrote: > > >>>>> > > > >>>>> > > On Mon, Dec 8, 2025 at 11:31 PM Yuri Weinstein via Dev < > > >>>>> [email protected]> > > >>>>> > > wrote: > > >>>>> > > > > > >>>>> > > > This test branch was built loner then 2.5 weeks ago > > >>>>> > > > I have to rebuild it first and then will schedule krbd > > >>>>> > > > > >>>>> > > krbd approved based on > > >>>>> > > > > >>>>> > > > > >>>>> > > > > >>>>> > > > >>>>> https://pulpito.ceph.com/yuriw-2025-12-10_21:26:56-krbd-reef-release-testing-default-smithi/ > > >>>>> > > > > >>>>> > > > > >>>>> > > > >>>>> https://pulpito.ceph.com/yuriw-2025-12-12_18:20:32-krbd-reef-release-testing-default-smithi/ > > >>>>> > > > > >>>>> > > Thanks, > > >>>>> > > > > >>>>> > > Ilya > > >>>>> > > > > >>>>> > > > > > >>>>> > > > On Mon, Dec 8, 2025 at 1:39 PM Ilya Dryomov via ceph-users < > > >>>>> > > [email protected]> wrote: > > >>>>> > > >> > > >>>>> > > >> On Mon, Dec 8, 2025 at 8:50 PM Ilya Dryomov <[email protected] > > >>>>> > > > >>>>> > wrote: > > >>>>> > > >> > > > >>>>> > > >> > On Mon, Dec 8, 2025 at 5:00 PM Venky Shankar via ceph-users > > >>>>> > > >> > <[email protected]> wrote: > > >>>>> > > >> > > > > >>>>> > > >> > > Hi Yuri, > > >>>>> > > >> > > > > >>>>> > > >> > > On Fri, Nov 21, 2025 at 4:34 AM Yuri Weinstein < > > >>>>> > [email protected]> > > >>>>> > > wrote: > > >>>>> > > >> > > > > > >>>>> > > >> > > > Note: this is an attempt to test and publish post EOL > > >>>>> reef > > >>>>> > point > > >>>>> > > release > > >>>>> > > >> > > > > > >>>>> > > >> > > > Details of this release are summarized here: > > >>>>> > > >> > > > > > >>>>> > > >> > > > https://tracker.ceph.com/issues/73906#note-1 > > >>>>> > > >> > > > Release Notes - TBD > > >>>>> > > >> > > > LRC upgrade - N/A > > >>>>> > > >> > > > > > >>>>> > > >> > > > distro => jammy focal centos9 windows bookworm > > >>>>> > > >> > > > > > >>>>> > > >> > > > Seeking approvals/reviews for: > > >>>>> > > >> > > > > > >>>>> > > >> > > > smoke - Laura approved? > > >>>>> > > >> > > > > > >>>>> > > >> > > > rados - Radek, Laura approved? Travis? Nizamudeen? Adam > > >>>>> King > > >>>>> > > approved? > > >>>>> > > >> > > > (rados is also running on build 2 == build 1 + > > >>>>> > > >> > > > https://github.com/ceph/ceph/pull/66056 per Igor's > > >>>>> request) > > >>>>> > > >> > > > > > >>>>> > > >> > > > rgw - Adam E approved? > > >>>>> > > >> > > > > > >>>>> > > >> > > > fs - Venky approved? > > >>>>> > > >> > > > > >>>>> > > >> > > There are lot many failures in the fs suite resembling > > >>>>> > > >> > > > > >>>>> > > >> > > Error mounting /var/lib/ceph: mount: /var/lib/ceph: > > >>>>> wrong > > >>>>> > fs > > >>>>> > > >> > > type, bad option ... > > >>>>> > > >> > > > > >>>>> > > >> > > which is an issue with the testing kernel. Adding Alex who > > >>>>> was > > >>>>> > > looking > > >>>>> > > >> > > into this issue. > > >>>>> > > >> > > > > >>>>> > > >> > > As far as the run validation is concerned, it's risky to > > >>>>> ignore > > >>>>> > the > > >>>>> > > >> > > large number of failures. We have two options: > > >>>>> > > >> > > > > >>>>> > > >> > > - fix the testing kernel issue and rerun > > >>>>> > > >> > > - rerun with `for-linus` branch (with `-k for-linus` in the > > >>>>> > > >> > > teuthology-suite schedule command) > > >>>>> > > >> > > > > >>>>> > > >> > > The latter would be using an older kernel branch, but I > > >>>>> think it > > >>>>> > > >> > > should suffice for this case. > > >>>>> > > >> > > > >>>>> > > >> > Hi Venky, > > >>>>> > > >> > > > >>>>> > > >> > With your report of the fs suite being affected as well, I > > >>>>> carved > > >>>>> > out > > >>>>> > > >> > some time to take a look. > > >>>>> > > >> > > > >>>>> > > >> > I think it's happening because XFS maintainers disabled > > >>>>> support for > > >>>>> > V4 > > >>>>> > > >> > filesystems that was deprecated 5 years ago -- as of 6.18 > > >>>>> kernel > > >>>>> > it's > > >>>>> > > >> > no longer compiled in by default. The proper fix would be to > > >>>>> create > > >>>>> > > an > > >>>>> > > >> > image with/move to V5 on-disk format; I have opened [1] to > > >>>>> try to > > >>>>> > > >> > achieve that. We can also re-enable V4 on-disk format in our > > >>>>> kernel > > >>>>> > > >> > builds as a workaround. > > >>>>> > > >> > > > >>>>> > > >> > [1] https://github.com/ceph/ceph-cm-ansible/pull/799 > > >>>>> > > >> > > >>>>> > > >> This should be fixed now. Yuri, please schedule a rerun for > > >>>>> krbd. > > >>>>> > > >> > > >>>>> > > >> Thanks, > > >>>>> > > >> > > >>>>> > > >> Ilya > > >>>>> > > >> _______________________________________________ > > >>>>> > > >> ceph-users mailing list -- [email protected] > > >>>>> > > >> To unsubscribe send an email to [email protected] > > >>>>> > > > > > >>>>> > > > _______________________________________________ > > >>>>> > > > Dev mailing list -- [email protected] > > >>>>> > > > To unsubscribe send an email to [email protected] > > >>>>> > > _______________________________________________ > > >>>>> > > ceph-users mailing list -- [email protected] > > >>>>> > > To unsubscribe send an email to [email protected] > > >>>>> > > > > >>>>> > _______________________________________________ > > >>>>> > ceph-users mailing list -- [email protected] > > >>>>> > To unsubscribe send an email to [email protected] > > >>>>> > > > >>>>> > > >>>>> > > >>>>> -- > > >>>>> Cheers, > > >>>>> Brad > > >>>>> _______________________________________________ > > >>>>> ceph-users mailing list -- [email protected] > > >>>>> To unsubscribe send an email to [email protected] > > >>>>> > > >>>> > > >>>> > > >>>> -- > > >>>> Cheers, > > >>>> Brad > > >>>> > > >>> > > >>> > > >>> -- > > >>> Cheers, > > >>> Brad > > >>> _______________________________________________ > > >>> Dev mailing list -- [email protected] > > >>> To unsubscribe send an email to [email protected] > > >>> > > >> > > >> > > >> -- > > >> Cheers, > > >> Brad > > >> _______________________________________________ > > >> Dev mailing list -- [email protected] > > >> To unsubscribe send an email to [email protected] > > >> > > > > > > > > > -- > > > > > > Laura Flores > > > > > > She/Her/Hers > > > > > > Software Engineer, Ceph Storage <https://ceph.io> > > > > > > Chicago, IL > > > > > > [email protected] | [email protected] <[email protected]> > > > M: +17087388804 > > > > > > > > > _______________________________________________ > > > Dev mailing list -- [email protected] > > > To unsubscribe send an email to [email protected] > > > > > _______________________________________________ > > ceph-users mailing list -- [email protected] > > To unsubscribe send an email to [email protected] > > > > -- > Cheers, > Venky -- Cheers, Venky _______________________________________________ ceph-users mailing list -- [email protected] To unsubscribe send an email to [email protected]
