[Bug 1371526] Re: ceph-disk-prepare command always fails; new partition table not avaliable until reboot
[Expired for linux (Ubuntu) because there has been no activity for 60 days.] ** Changed in: linux (Ubuntu) Status: Incomplete => Expired -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to ceph in Ubuntu. https://bugs.launchpad.net/bugs/1371526 Title: ceph-disk-prepare command always fails; new partition table not avaliable until reboot To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1371526/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1371526] Re: ceph-disk-prepare command always fails; new partition table not avaliable until reboot
** Changed in: ceph (Ubuntu) Status: New => Invalid -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to ceph in Ubuntu. https://bugs.launchpad.net/bugs/1371526 Title: ceph-disk-prepare command always fails; new partition table not avaliable until reboot To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1371526/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1371526] Re: ceph-disk-prepare command always fails; new partition table not avaliable until reboot
I believe this is affecting live migration of ubuntu 14.04 kvm guests from one kvm host to another. When i attempt this now I get read write errors on the guest, and the file system goes into read only mode. I have a test environment where I can verify this. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to ceph in Ubuntu. https://bugs.launchpad.net/bugs/1371526 Title: ceph-disk-prepare command always fails; new partition table not avaliable until reboot To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1371526/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1371526] Re: ceph-disk-prepare command always fails; new partition table not avaliable until reboot
I can't reproduce this problem any longer, so I'm assuming that something changed to fix this as I was seeing it 100% of the time. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to ceph in Ubuntu. https://bugs.launchpad.net/bugs/1371526 Title: ceph-disk-prepare command always fails; new partition table not avaliable until reboot To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1371526/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1371526] Re: ceph-disk-prepare command always fails; new partition table not avaliable until reboot
We would really need information about setting up a system that runs into issues. In particular, how is the cloud-init ephemeral disk created? I still cannot reproduce this (making an ext3 fs outside and put something into it, then mount it by label from the guest, unmount it, mount it again and write something, all works ok). So we need as much info about what is done to the ephemeral disk from the start to the point where umount fails. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to ceph in Ubuntu. https://bugs.launchpad.net/bugs/1371526 Title: ceph-disk-prepare command always fails; new partition table not avaliable until reboot To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1371526/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1371526] Re: ceph-disk-prepare command always fails; new partition table not avaliable until reboot
So to gather more pointers I tried a Trusty host and Utopic KVM guests. Either manually created (with virt-manager and not involving cloud-init then) and also using uvtool which is at least using a vdb for cloud-init data (in some way, though the image is a ro iso). Both ways the jbd2 process goes away after unmount. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to ceph in Ubuntu. https://bugs.launchpad.net/bugs/1371526 Title: ceph-disk-prepare command always fails; new partition table not avaliable until reboot To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1371526/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1371526] Re: ceph-disk-prepare command always fails; new partition table not avaliable until reboot
apport information ** Tags added: apport-collected ** Description changed: $ sudo ceph-disk-prepare --fs-type xfs --zap-disk /dev/vdb Caution: invalid backup GPT header, but valid main header; regenerating backup header from main header. Warning! Main and backup partition tables differ! Use the 'c' and 'e' options on the recovery & transformation menu to examine the two tables. Warning! One or more CRCs don't match. You should repair the disk! Caution: Found protective or hybrid MBR and corrupt GPT. Using GPT, but disk verification and recovery are STRONGLY recommended. Warning: The kernel is still using the old partition table. The new table will be used at the next reboot. GPT data structures destroyed! You may now partition the disk using fdisk or other utilities. Warning: The kernel is still using the old partition table. The new table will be used at the next reboot. The operation has completed successfully. Warning: The kernel is still using the old partition table. The new table will be used at the next reboot. The operation has completed successfully. Warning: The kernel is still using the old partition table. The new table will be used at the next reboot. The operation has completed successfully. mkfs.xfs: cannot open /dev/vdb1: Device or resource busy ceph-disk: Error: Command '['/sbin/mkfs', '-t', 'xfs', '-f', '-i', 'size=2048', '--', '/dev/vdb1']' returned non-zero exit status 1 I can reproduce this consistently across ceph nodes; also impacts on the way we use swift for storage as well. ProblemType: Bug DistroRelease: Ubuntu 14.10 Package: ceph 0.80.5-1 ProcVersionSignature: User Name 3.16.0-16.22-generic 3.16.2 Uname: Linux 3.16.0-16-generic x86_64 ApportVersion: 2.14.7-0ubuntu2 Architecture: amd64 Date: Fri Sep 19 09:39:18 2014 Ec2AMI: ami-0084 Ec2AMIManifest: FIXME Ec2AvailabilityZone: nova Ec2InstanceType: m1.small Ec2Kernel: aki-0002 Ec2Ramdisk: ari-0002 SourcePackage: ceph UpgradeStatus: No upgrade log present (probably fresh install) + --- + ApportVersion: 2.14.7-0ubuntu2 + Architecture: amd64 + DistroRelease: Ubuntu 14.10 + Ec2AMI: ami-0084 + Ec2AMIManifest: FIXME + Ec2AvailabilityZone: nova + Ec2InstanceType: m1.small + Ec2Kernel: aki-0002 + Ec2Ramdisk: ari-0002 + Package: linux + PackageArchitecture: amd64 + ProcVersionSignature: Ubuntu 3.16.0-16.22-generic 3.16.2 + Tags: utopic ec2-images + Uname: Linux 3.16.0-16-generic x86_64 + UpgradeStatus: No upgrade log present (probably fresh install) + UserGroups: + + _MarkForUpload: True ** Attachment added: "Dependencies.txt" https://bugs.launchpad.net/bugs/1371526/+attachment/4208774/+files/Dependencies.txt -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to ceph in Ubuntu. https://bugs.launchpad.net/bugs/1371526 Title: ceph-disk-prepare command always fails; new partition table not avaliable until reboot To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1371526/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1371526] Re: ceph-disk-prepare command always fails; new partition table not avaliable until reboot
This only appears to happen with the device on first boot; after a reboot mount/umount drops all jdb2 processes as I think it should. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to ceph in Ubuntu. https://bugs.launchpad.net/bugs/1371526 Title: ceph-disk-prepare command always fails; new partition table not avaliable until reboot To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1371526/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1371526] Re: ceph-disk-prepare command always fails; new partition table not avaliable until reboot
sudo lshw -class disk -class storage *-ide description: IDE interface product: 82371SB PIIX3 IDE [Natoma/Triton II] vendor: Intel Corporation physical id: 1.1 bus info: pci@:00:01.1 version: 00 width: 32 bits clock: 33MHz capabilities: ide bus_master configuration: driver=ata_piix latency=0 resources: irq:0 ioport:1f0(size=8) ioport:3f6 ioport:170(size=8) ioport:376 ioport:c0e0(size=16) *-scsi:0 description: SCSI storage controller product: Virtio block device vendor: Red Hat, Inc physical id: 4 bus info: pci@:00:04.0 version: 00 width: 32 bits clock: 33MHz capabilities: scsi msix bus_master cap_list configuration: driver=virtio-pci latency=0 resources: irq:11 ioport:c000(size=64) memory:febd2000-febd2fff *-scsi:1 description: SCSI storage controller product: Virtio block device vendor: Red Hat, Inc physical id: 5 bus info: pci@:00:05.0 version: 00 width: 32 bits clock: 33MHz capabilities: scsi msix bus_master cap_list configuration: driver=virtio-pci latency=0 resources: irq:10 ioport:c040(size=64) memory:febd3000-febd3fff *-scsi:2 description: SCSI storage controller product: Virtio block device vendor: Red Hat, Inc physical id: 7 bus info: pci@:00:07.0 version: 00 width: 32 bits clock: 33MHz capabilities: scsi msix bus_master cap_list configuration: driver=virtio-pci latency=0 resources: irq:10 ioport:1000(size=64) memory:8000-8fff -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to ceph in Ubuntu. https://bugs.launchpad.net/bugs/1371526 Title: ceph-disk-prepare command always fails; new partition table not avaliable until reboot To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1371526/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1371526] Re: ceph-disk-prepare command always fails; new partition table not avaliable until reboot
I poked at this with some additional volumes attached to the cloud instance: Offending device: ubuntu@juju-t-machine-22:~$ sudo umount /dev/vdb ubuntu@juju-t-machine-22:~$ sudo lsof | grep vdb jbd2/vdb- 1268 root cwd DIR 253,1 4096 2 / jbd2/vdb- 1268 root rtd DIR 253,1 4096 2 / jbd2/vdb- 1268 root txt unknown /proc/1268/exe Additional device: ubuntu@juju-t-machine-22:~$ sudo mount /dev/vdc /mnt2 ubuntu@juju-t-machine-22:~$ sudo lsof | grep vdc jbd2/vdc- 16058 root cwd DIR 253,1 4096 2 / jbd2/vdc- 16058 root rtd DIR 253,1 4096 2 / jbd2/vdc- 16058 root txt unknown /proc/16058/exe ubuntu@juju-t-machine-22:~$ sudo umount /dev/vdc ubuntu@juju-t-machine-22:~$ sudo lsof | grep vdc As you can see, the jbd2 process for vdb appears to hang around, which I think is what is keeping the partition table locked in kernel and hence stale. ** Also affects: linux (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to ceph in Ubuntu. https://bugs.launchpad.net/bugs/1371526 Title: ceph-disk-prepare command always fails; new partition table not avaliable until reboot To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1371526/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1371526] Re: ceph-disk-prepare command always fails; new partition table not avaliable until reboot
Its probably worth noting that this is in a cloud-instance which automatically formats and mounts /dev/vdb on first boot; however we do umount it prior to performing the ceph-disk-prepare command. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to ceph in Ubuntu. https://bugs.launchpad.net/bugs/1371526 Title: ceph-disk-prepare command always fails; new partition table not avaliable until reboot To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1371526/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1371526] Re: ceph-disk-prepare command always fails; new partition table not avaliable until reboot
This is what things should look like: sudo ceph-disk-prepare --fs-type xfs --zap-disk /dev/vdb Caution: invalid backup GPT header, but valid main header; regenerating backup header from main header. Caution! After loading partitions, the CRC doesn't check out! Warning! Main and backup partition tables differ! Use the 'c' and 'e' options on the recovery & transformation menu to examine the two tables. Warning! One or more CRCs don't match. You should repair the disk! Caution: Found protective or hybrid MBR and corrupt GPT. Using GPT, but disk verification and recovery are STRONGLY recommended. GPT data structures destroyed! You may now partition the disk using fdisk or other utilities. The operation has completed successfully. The operation has completed successfully. The operation has completed successfully. meta-data=/dev/vdb1 isize=2048 agcount=4, agsize=1245119 blks = sectsz=512 attr=2, projid32bit=1 = crc=0finobt=0 data = bsize=4096 blocks=4980475, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0 ftype=0 log =internal log bsize=4096 blocks=2560, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 The operation has completed successfully. (that was after a reboot). -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to ceph in Ubuntu. https://bugs.launchpad.net/bugs/1371526 Title: ceph-disk-prepare command always fails; new partition table not avaliable until reboot To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1371526/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
