[Bug 1384062] Re: os-prober kills ceph OSD
Forgot to mention that the Ceph cluster has to be under write load in order to reproduce, i.e. running something like rados -p rbd bench 600 write -t 1 --show-time --run-length 60 There is no effect of running os-prober if the cluster is idle. Based with that information, though, I can also reproduce the issue by running fio on some partition and os-prober in parallel, getting: # fio --ioengine=libaio --filename=/dev/sdc4 --bs=64k --rw=randwrite --runtime=300 --size=1G --direct=1 --iodepth=8 --name=a a: (g=0): rw=randwrite, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=8 fio-2.2.13 Starting 1 process fio: io_u error on file /dev/sdc4: Operation not permitted: write offset=531300352, buflen=65536 fio: pid=17543, err=1/file:io_u.c:1596, func=io_u error, error=Operation not permitted So I think the error has nothing to do with Ceph in particular, but really os-prober should be made more conservative when trying to probe partitions. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to ceph in Ubuntu. https://bugs.launchpad.net/bugs/1384062 Title: os-prober kills ceph OSD To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1384062/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1384062] Re: os-prober kills ceph OSD
** Also affects: os-prober (Ubuntu) Importance: Undecided Status: New ** Changed in: os-prober (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to ceph in Ubuntu. https://bugs.launchpad.net/bugs/1384062 Title: os-prober kills ceph OSD To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1384062/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1384062] Re: os-prober kills ceph OSD
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: ceph (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to ceph in Ubuntu. https://bugs.launchpad.net/bugs/1384062 Title: os-prober kills ceph OSD To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1384062/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1384062] Re: os-prober kills ceph OSD
I can reproduce this on Wily with # apt-cache policy ceph ceph: Installed: 0.94.5-0ubuntu0.15.10.1 Candidate: 0.94.5-0ubuntu0.15.10.1 Version table: *** 0.94.5-0ubuntu0.15.10.1 0 500 http://eu.archive.ubuntu.com/ubuntu/ wily-updates/main amd64 Packages 100 /var/lib/dpkg/status 0.94.3-0ubuntu2 0 500 http://eu.archive.ubuntu.com/ubuntu/ wily/main amd64 Packages when I configure two OSDs both with their journal partition on a third disk. Running os-prober makes the second OSD fail with 0> 2016-02-16 15:15:17.604773 7fe2c7ec8700 -1 os/FileJournal.cc: In function 'void FileJournal::write_finish_thread_entry()' thread 7fe2c7ec8700 time 2016-02-16 15:15:17.603014 os/FileJournal.cc: 1426: FAILED assert(0 == "unexpected aio error") ceph version 0.94.5 (9764da52395923e0b32908d83a9f7304401fee43) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x80) [0x556db4ce44f0] 2: (FileJournal::write_finish_thread_entry()+0x775) [0x556db4bb5925] 3: (FileJournal::WriteFinisher::entry()+0xd) [0x556db4a990ed] 4: (()+0x76aa) [0x7fe2d32656aa] 5: (clone()+0x6d) [0x7fe2d173ceed] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. which seems to indicate that os-prober is doing unsafe things to the journal partitions. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to ceph in Ubuntu. https://bugs.launchpad.net/bugs/1384062 Title: os-prober kills ceph OSD To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1384062/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1384062] Re: os-prober kills ceph OSD
I did a quick test and was not able to reproduce - but my test environment is virtual so that may be making a difference. ** Also affects: ceph (Ubuntu) Importance: Undecided Status: New ** Changed in: ceph (Juju Charms Collection) Status: New => Invalid ** Changed in: ceph (Ubuntu) Importance: Undecided => Medium -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to ceph in Ubuntu. https://bugs.launchpad.net/bugs/1384062 Title: os-prober kills ceph OSD To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1384062/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
