[Yahoo-eng-team] [Bug 1773449] Re: VM rbd backed block devices inconsistent after unexpected host outage

2018-07-03 Thread Emilien Macchi
** Changed in: tripleo
   Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1773449

Title:
  VM rbd backed block devices inconsistent after unexpected host outage

Status in OpenStack ceph-mon charm:
  Fix Released
Status in charms.ceph:
  Fix Released
Status in Ubuntu Cloud Archive:
  Invalid
Status in OpenStack Compute (nova):
  Invalid
Status in tripleo:
  Fix Released
Status in ceph package in Ubuntu:
  Invalid
Status in nova package in Ubuntu:
  Invalid
Status in qemu package in Ubuntu:
  Invalid

Bug description:
  Reboot host that contains VMs with volumes and all VMs fail to boot.
  Happens with Queens on Bionic and Xenial

  [0.00] Initializing cgroup subsys cpuset

  [0.00] Initializing cgroup subsys cpu

  [0.00] Initializing cgroup subsys cpuacct

  [0.00] Linux version 4.4.0-124-generic
  (buildd@lcy01-amd64-028) (gcc version 5.4.0 20160609 (Ubuntu
  5.4.0-6ubuntu1~16.04.9) ) #148-Ubuntu SMP Wed May 2 13:00:18 UTC 2018
  (Ubuntu 4.4.0-124.148-generic 4.4.117)

  [0.00] Command line:
  BOOT_IMAGE=/boot/vmlinuz-4.4.0-124-generic
  root=UUID=bca2de6e-f774-4203-ae05-e8deeb05f64a ro console=tty1
  console=ttyS0

  [0.00] KERNEL supported cpus:

  [0.00]   Intel GenuineIntel

  [0.00]   AMD AuthenticAMD

  [0.00]   Centaur CentaurHauls

  [0.00] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256

  [0.00] x86/fpu: Supporting XSAVE feature 0x01: 'x87 floating
  point registers'

  [0.00] x86/fpu: Supporting XSAVE feature 0x02: 'SSE registers'

  [0.00] x86/fpu: Supporting XSAVE feature 0x04: 'AVX registers'

  [0.00] x86/fpu: Enabled xstate features 0x7, context size is
  832 bytes, using 'standard' format.

  [0.00] x86/fpu: Using 'eager' FPU context switches.

  [0.00] e820: BIOS-provided physical RAM map:

  [0.00] BIOS-e820: [mem 0x-0x0009fbff]
  usable

  [0.00] BIOS-e820: [mem 0x0009fc00-0x0009]
  reserved

  [0.00] BIOS-e820: [mem 0x000f-0x000f]
  reserved

  [0.00] BIOS-e820: [mem 0x0010-0x7ffdbfff]
  usable

  [0.00] BIOS-e820: [mem 0x7ffdc000-0x7fff]
  reserved

  [0.00] BIOS-e820: [mem 0xfeffc000-0xfeff]
  reserved

  [0.00] BIOS-e820: [mem 0xfffc-0x]
  reserved

  [0.00] NX (Execute Disable) protection: active

  [0.00] SMBIOS 2.8 present.

  [0.00] Hypervisor detected: KVM

  [0.00] e820: last_pfn = 0x7ffdc max_arch_pfn = 0x4

  [0.00] x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WC
  UC- WT

  [0.00] found SMP MP-table at [mem 0x000f6a20-0x000f6a2f]
  mapped at [880f6a20]

  [0.00] Scanning 1 areas for low memory corruption

  [0.00] Using GB pages for direct mapping

  [0.00] RAMDISK: [mem 0x361f4000-0x370f1fff]

  [0.00] ACPI: Early table checksum verification disabled

  [0.00] ACPI: RSDP 0x000F6780 14 (v00 BOCHS )

  [0.00] ACPI: RSDT 0x7FFE1649 2C (v01 BOCHS
  BXPCRSDT 0001 BXPC 0001)

  [0.00] ACPI: FACP 0x7FFE14CD 74 (v01 BOCHS
  BXPCFACP 0001 BXPC 0001)

  [0.00] ACPI: DSDT 0x7FFE0040 00148D (v01 BOCHS
  BXPCDSDT 0001 BXPC 0001)

  [0.00] ACPI: FACS 0x7FFE 40

  [0.00] ACPI: APIC 0x7FFE15C1 88 (v01 BOCHS
  BXPCAPIC 0001 BXPC 0001)

  [0.00] No NUMA configuration found

  [0.00] Faking a node at [mem
  0x-0x7ffdbfff]

  [0.00] NODE_DATA(0) allocated [mem 0x7ffd7000-0x7ffdbfff]

  [0.00] kvm-clock: Using msrs 4b564d01 and 4b564d00

  [0.00] kvm-clock: cpu 0, msr 0:7ffcf001, primary cpu clock

  [0.00] kvm-clock: using sched offset of 17590935813 cycles

  [0.00] clocksource: kvm-clock: mask: 0x
  max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns

  [0.00] Zone ranges:

  [0.00]   DMA  [mem 0x1000-0x00ff]

  [0.00]   DMA32[mem 0x0100-0x7ffdbfff]

  [0.00]   Normal   empty

  [0.00]   Device   empty

  [0.00] Movable zone start for each node

  [0.00] Early memory node ranges

  [0.00]   node   0: [mem 0x1000-0x0009efff]

  [0.00]   node   0: [mem 0x0010-0x7ffdbfff]

  [0.00] Initmem setup node 0 [mem
  0x1000-0x7ffdbfff]

  [0.00] ACPI: PM-Timer IO Port: 0x608

  [0.00] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])

  [

[Yahoo-eng-team] [Bug 1773449] Re: VM rbd backed block devices inconsistent after unexpected host outage

2018-06-21 Thread Giulio Fidente
Fixed in TripleO via change I9639d606bd538f6776c368a4f34aa6783ab91abb

** Also affects: tripleo
   Importance: Undecided
   Status: New

** Changed in: tripleo
   Status: New => Fix Committed

** Changed in: tripleo
   Importance: Undecided => High

** Changed in: tripleo
 Assignee: (unassigned) => Giulio Fidente (gfidente)

** Changed in: tripleo
Milestone: None => rocky-3

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1773449

Title:
  VM rbd backed block devices inconsistent after unexpected host outage

Status in OpenStack ceph-mon charm:
  Fix Released
Status in charms.ceph:
  Fix Released
Status in Ubuntu Cloud Archive:
  Invalid
Status in OpenStack Compute (nova):
  Invalid
Status in tripleo:
  Fix Committed
Status in ceph package in Ubuntu:
  Invalid
Status in nova package in Ubuntu:
  Invalid
Status in qemu package in Ubuntu:
  Invalid

Bug description:
  Reboot host that contains VMs with volumes and all VMs fail to boot.
  Happens with Queens on Bionic and Xenial

  [0.00] Initializing cgroup subsys cpuset

  [0.00] Initializing cgroup subsys cpu

  [0.00] Initializing cgroup subsys cpuacct

  [0.00] Linux version 4.4.0-124-generic
  (buildd@lcy01-amd64-028) (gcc version 5.4.0 20160609 (Ubuntu
  5.4.0-6ubuntu1~16.04.9) ) #148-Ubuntu SMP Wed May 2 13:00:18 UTC 2018
  (Ubuntu 4.4.0-124.148-generic 4.4.117)

  [0.00] Command line:
  BOOT_IMAGE=/boot/vmlinuz-4.4.0-124-generic
  root=UUID=bca2de6e-f774-4203-ae05-e8deeb05f64a ro console=tty1
  console=ttyS0

  [0.00] KERNEL supported cpus:

  [0.00]   Intel GenuineIntel

  [0.00]   AMD AuthenticAMD

  [0.00]   Centaur CentaurHauls

  [0.00] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256

  [0.00] x86/fpu: Supporting XSAVE feature 0x01: 'x87 floating
  point registers'

  [0.00] x86/fpu: Supporting XSAVE feature 0x02: 'SSE registers'

  [0.00] x86/fpu: Supporting XSAVE feature 0x04: 'AVX registers'

  [0.00] x86/fpu: Enabled xstate features 0x7, context size is
  832 bytes, using 'standard' format.

  [0.00] x86/fpu: Using 'eager' FPU context switches.

  [0.00] e820: BIOS-provided physical RAM map:

  [0.00] BIOS-e820: [mem 0x-0x0009fbff]
  usable

  [0.00] BIOS-e820: [mem 0x0009fc00-0x0009]
  reserved

  [0.00] BIOS-e820: [mem 0x000f-0x000f]
  reserved

  [0.00] BIOS-e820: [mem 0x0010-0x7ffdbfff]
  usable

  [0.00] BIOS-e820: [mem 0x7ffdc000-0x7fff]
  reserved

  [0.00] BIOS-e820: [mem 0xfeffc000-0xfeff]
  reserved

  [0.00] BIOS-e820: [mem 0xfffc-0x]
  reserved

  [0.00] NX (Execute Disable) protection: active

  [0.00] SMBIOS 2.8 present.

  [0.00] Hypervisor detected: KVM

  [0.00] e820: last_pfn = 0x7ffdc max_arch_pfn = 0x4

  [0.00] x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WC
  UC- WT

  [0.00] found SMP MP-table at [mem 0x000f6a20-0x000f6a2f]
  mapped at [880f6a20]

  [0.00] Scanning 1 areas for low memory corruption

  [0.00] Using GB pages for direct mapping

  [0.00] RAMDISK: [mem 0x361f4000-0x370f1fff]

  [0.00] ACPI: Early table checksum verification disabled

  [0.00] ACPI: RSDP 0x000F6780 14 (v00 BOCHS )

  [0.00] ACPI: RSDT 0x7FFE1649 2C (v01 BOCHS
  BXPCRSDT 0001 BXPC 0001)

  [0.00] ACPI: FACP 0x7FFE14CD 74 (v01 BOCHS
  BXPCFACP 0001 BXPC 0001)

  [0.00] ACPI: DSDT 0x7FFE0040 00148D (v01 BOCHS
  BXPCDSDT 0001 BXPC 0001)

  [0.00] ACPI: FACS 0x7FFE 40

  [0.00] ACPI: APIC 0x7FFE15C1 88 (v01 BOCHS
  BXPCAPIC 0001 BXPC 0001)

  [0.00] No NUMA configuration found

  [0.00] Faking a node at [mem
  0x-0x7ffdbfff]

  [0.00] NODE_DATA(0) allocated [mem 0x7ffd7000-0x7ffdbfff]

  [0.00] kvm-clock: Using msrs 4b564d01 and 4b564d00

  [0.00] kvm-clock: cpu 0, msr 0:7ffcf001, primary cpu clock

  [0.00] kvm-clock: using sched offset of 17590935813 cycles

  [0.00] clocksource: kvm-clock: mask: 0x
  max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns

  [0.00] Zone ranges:

  [0.00]   DMA  [mem 0x1000-0x00ff]

  [0.00]   DMA32[mem 0x0100-0x7ffdbfff]

  [0.00]   Normal   empty

  [0.00]   Device   empty

  [0.00] Movable zone start for each node

  [0.00] Early memory node ranges

  [0.00]   node   0: 

[Yahoo-eng-team] [Bug 1773449] Re: VM rbd backed block devices inconsistent after unexpected host outage

2018-06-12 Thread James Page
** Changed in: charm-ceph-mon
   Status: Fix Committed => Fix Released

** Changed in: charm-ceph-mon
Milestone: 18.08 => 18.05

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1773449

Title:
  VM rbd backed block devices inconsistent after unexpected host outage

Status in OpenStack ceph-mon charm:
  Fix Released
Status in charms.ceph:
  Fix Released
Status in Ubuntu Cloud Archive:
  Invalid
Status in OpenStack Compute (nova):
  Invalid
Status in ceph package in Ubuntu:
  Invalid
Status in nova package in Ubuntu:
  Invalid
Status in qemu package in Ubuntu:
  Invalid

Bug description:
  Reboot host that contains VMs with volumes and all VMs fail to boot.
  Happens with Queens on Bionic and Xenial

  [0.00] Initializing cgroup subsys cpuset

  [0.00] Initializing cgroup subsys cpu

  [0.00] Initializing cgroup subsys cpuacct

  [0.00] Linux version 4.4.0-124-generic
  (buildd@lcy01-amd64-028) (gcc version 5.4.0 20160609 (Ubuntu
  5.4.0-6ubuntu1~16.04.9) ) #148-Ubuntu SMP Wed May 2 13:00:18 UTC 2018
  (Ubuntu 4.4.0-124.148-generic 4.4.117)

  [0.00] Command line:
  BOOT_IMAGE=/boot/vmlinuz-4.4.0-124-generic
  root=UUID=bca2de6e-f774-4203-ae05-e8deeb05f64a ro console=tty1
  console=ttyS0

  [0.00] KERNEL supported cpus:

  [0.00]   Intel GenuineIntel

  [0.00]   AMD AuthenticAMD

  [0.00]   Centaur CentaurHauls

  [0.00] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256

  [0.00] x86/fpu: Supporting XSAVE feature 0x01: 'x87 floating
  point registers'

  [0.00] x86/fpu: Supporting XSAVE feature 0x02: 'SSE registers'

  [0.00] x86/fpu: Supporting XSAVE feature 0x04: 'AVX registers'

  [0.00] x86/fpu: Enabled xstate features 0x7, context size is
  832 bytes, using 'standard' format.

  [0.00] x86/fpu: Using 'eager' FPU context switches.

  [0.00] e820: BIOS-provided physical RAM map:

  [0.00] BIOS-e820: [mem 0x-0x0009fbff]
  usable

  [0.00] BIOS-e820: [mem 0x0009fc00-0x0009]
  reserved

  [0.00] BIOS-e820: [mem 0x000f-0x000f]
  reserved

  [0.00] BIOS-e820: [mem 0x0010-0x7ffdbfff]
  usable

  [0.00] BIOS-e820: [mem 0x7ffdc000-0x7fff]
  reserved

  [0.00] BIOS-e820: [mem 0xfeffc000-0xfeff]
  reserved

  [0.00] BIOS-e820: [mem 0xfffc-0x]
  reserved

  [0.00] NX (Execute Disable) protection: active

  [0.00] SMBIOS 2.8 present.

  [0.00] Hypervisor detected: KVM

  [0.00] e820: last_pfn = 0x7ffdc max_arch_pfn = 0x4

  [0.00] x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WC
  UC- WT

  [0.00] found SMP MP-table at [mem 0x000f6a20-0x000f6a2f]
  mapped at [880f6a20]

  [0.00] Scanning 1 areas for low memory corruption

  [0.00] Using GB pages for direct mapping

  [0.00] RAMDISK: [mem 0x361f4000-0x370f1fff]

  [0.00] ACPI: Early table checksum verification disabled

  [0.00] ACPI: RSDP 0x000F6780 14 (v00 BOCHS )

  [0.00] ACPI: RSDT 0x7FFE1649 2C (v01 BOCHS
  BXPCRSDT 0001 BXPC 0001)

  [0.00] ACPI: FACP 0x7FFE14CD 74 (v01 BOCHS
  BXPCFACP 0001 BXPC 0001)

  [0.00] ACPI: DSDT 0x7FFE0040 00148D (v01 BOCHS
  BXPCDSDT 0001 BXPC 0001)

  [0.00] ACPI: FACS 0x7FFE 40

  [0.00] ACPI: APIC 0x7FFE15C1 88 (v01 BOCHS
  BXPCAPIC 0001 BXPC 0001)

  [0.00] No NUMA configuration found

  [0.00] Faking a node at [mem
  0x-0x7ffdbfff]

  [0.00] NODE_DATA(0) allocated [mem 0x7ffd7000-0x7ffdbfff]

  [0.00] kvm-clock: Using msrs 4b564d01 and 4b564d00

  [0.00] kvm-clock: cpu 0, msr 0:7ffcf001, primary cpu clock

  [0.00] kvm-clock: using sched offset of 17590935813 cycles

  [0.00] clocksource: kvm-clock: mask: 0x
  max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns

  [0.00] Zone ranges:

  [0.00]   DMA  [mem 0x1000-0x00ff]

  [0.00]   DMA32[mem 0x0100-0x7ffdbfff]

  [0.00]   Normal   empty

  [0.00]   Device   empty

  [0.00] Movable zone start for each node

  [0.00] Early memory node ranges

  [0.00]   node   0: [mem 0x1000-0x0009efff]

  [0.00]   node   0: [mem 0x0010-0x7ffdbfff]

  [0.00] Initmem setup node 0 [mem
  0x1000-0x7ffdbfff]

  [0.00] ACPI: PM-Timer IO Port: 0x608

  [0.00] ACPI: LAPIC_NMI 

[Yahoo-eng-team] [Bug 1773449] Re: VM rbd backed block devices inconsistent after unexpected host outage

2018-06-11 Thread OpenStack Infra
Reviewed:  https://review.openstack.org/573659
Committed: 
https://git.openstack.org/cgit/openstack/charms.ceph/commit/?id=4d8f31d0ea0f47fd71939973cba86137d5daaef4
Submitter: Zuul
Branch:master

commit 4d8f31d0ea0f47fd71939973cba86137d5daaef4
Author: James Page 
Date:   Fri Jun 8 11:34:28 2018 +0100

Add 'osd blacklist' to default mon perms

Ensure that the default permissions for clients include the
'osd blacklist' command;  This ensures that in the event of
a client crashing (due to power outage or segfault), the
client and re-connect and write to any devices on reboot.

Change-Id: I0b43dece4e1c56fb838b0147bfb75fb9906e6657
Closes-Bug: 1773449


** Changed in: charms.ceph
   Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1773449

Title:
  VM rbd backed block devices inconsistent after unexpected host outage

Status in OpenStack ceph-mon charm:
  In Progress
Status in charms.ceph:
  Fix Released
Status in Ubuntu Cloud Archive:
  Invalid
Status in OpenStack Compute (nova):
  Invalid
Status in ceph package in Ubuntu:
  Invalid
Status in nova package in Ubuntu:
  Invalid
Status in qemu package in Ubuntu:
  Invalid

Bug description:
  Reboot host that contains VMs with volumes and all VMs fail to boot.
  Happens with Queens on Bionic and Xenial

  [0.00] Initializing cgroup subsys cpuset

  [0.00] Initializing cgroup subsys cpu

  [0.00] Initializing cgroup subsys cpuacct

  [0.00] Linux version 4.4.0-124-generic
  (buildd@lcy01-amd64-028) (gcc version 5.4.0 20160609 (Ubuntu
  5.4.0-6ubuntu1~16.04.9) ) #148-Ubuntu SMP Wed May 2 13:00:18 UTC 2018
  (Ubuntu 4.4.0-124.148-generic 4.4.117)

  [0.00] Command line:
  BOOT_IMAGE=/boot/vmlinuz-4.4.0-124-generic
  root=UUID=bca2de6e-f774-4203-ae05-e8deeb05f64a ro console=tty1
  console=ttyS0

  [0.00] KERNEL supported cpus:

  [0.00]   Intel GenuineIntel

  [0.00]   AMD AuthenticAMD

  [0.00]   Centaur CentaurHauls

  [0.00] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256

  [0.00] x86/fpu: Supporting XSAVE feature 0x01: 'x87 floating
  point registers'

  [0.00] x86/fpu: Supporting XSAVE feature 0x02: 'SSE registers'

  [0.00] x86/fpu: Supporting XSAVE feature 0x04: 'AVX registers'

  [0.00] x86/fpu: Enabled xstate features 0x7, context size is
  832 bytes, using 'standard' format.

  [0.00] x86/fpu: Using 'eager' FPU context switches.

  [0.00] e820: BIOS-provided physical RAM map:

  [0.00] BIOS-e820: [mem 0x-0x0009fbff]
  usable

  [0.00] BIOS-e820: [mem 0x0009fc00-0x0009]
  reserved

  [0.00] BIOS-e820: [mem 0x000f-0x000f]
  reserved

  [0.00] BIOS-e820: [mem 0x0010-0x7ffdbfff]
  usable

  [0.00] BIOS-e820: [mem 0x7ffdc000-0x7fff]
  reserved

  [0.00] BIOS-e820: [mem 0xfeffc000-0xfeff]
  reserved

  [0.00] BIOS-e820: [mem 0xfffc-0x]
  reserved

  [0.00] NX (Execute Disable) protection: active

  [0.00] SMBIOS 2.8 present.

  [0.00] Hypervisor detected: KVM

  [0.00] e820: last_pfn = 0x7ffdc max_arch_pfn = 0x4

  [0.00] x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WC
  UC- WT

  [0.00] found SMP MP-table at [mem 0x000f6a20-0x000f6a2f]
  mapped at [880f6a20]

  [0.00] Scanning 1 areas for low memory corruption

  [0.00] Using GB pages for direct mapping

  [0.00] RAMDISK: [mem 0x361f4000-0x370f1fff]

  [0.00] ACPI: Early table checksum verification disabled

  [0.00] ACPI: RSDP 0x000F6780 14 (v00 BOCHS )

  [0.00] ACPI: RSDT 0x7FFE1649 2C (v01 BOCHS
  BXPCRSDT 0001 BXPC 0001)

  [0.00] ACPI: FACP 0x7FFE14CD 74 (v01 BOCHS
  BXPCFACP 0001 BXPC 0001)

  [0.00] ACPI: DSDT 0x7FFE0040 00148D (v01 BOCHS
  BXPCDSDT 0001 BXPC 0001)

  [0.00] ACPI: FACS 0x7FFE 40

  [0.00] ACPI: APIC 0x7FFE15C1 88 (v01 BOCHS
  BXPCAPIC 0001 BXPC 0001)

  [0.00] No NUMA configuration found

  [0.00] Faking a node at [mem
  0x-0x7ffdbfff]

  [0.00] NODE_DATA(0) allocated [mem 0x7ffd7000-0x7ffdbfff]

  [0.00] kvm-clock: Using msrs 4b564d01 and 4b564d00

  [0.00] kvm-clock: cpu 0, msr 0:7ffcf001, primary cpu clock

  [0.00] kvm-clock: using sched offset of 17590935813 cycles

  [0.00] clocksource: kvm-clock: mask: 0x
  max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns

  [0.00] Zone ranges:

  [  

[Yahoo-eng-team] [Bug 1773449] Re: VM rbd backed block devices inconsistent after unexpected host outage

2018-06-08 Thread James Page
OK figured this one out - the cephx keys are missing a permission which
allows them to see blacklisted clients - as a result they can't deal
with a hard crash:

  mon 'allow command "osd blacklist"'

This is a charm issue after all.

As a workaround you can manually update the existing client keys for
nova-compute using:

  sudo ceph auth caps client.nova-compute mon 'allow r, allow command
"osd blacklist"' osd 'allow rwx'

from any mon unit.



** Changed in: nova
   Status: New => Invalid

** Changed in: ceph (Ubuntu)
   Status: New => Invalid

** Changed in: nova (Ubuntu)
   Status: New => Invalid

** Changed in: cloud-archive
   Status: New => Invalid

** Changed in: qemu (Ubuntu)
   Status: New => Invalid

** Also affects: charm-ceph-mon
   Importance: Undecided
   Status: New

** Also affects: charms.ceph
   Importance: Undecided
   Status: New

** Changed in: charms.ceph
   Status: New => Triaged

** Changed in: charm-ceph-mon
   Status: New => Triaged

** Changed in: charms.ceph
   Importance: Undecided => High

** Changed in: charm-ceph-mon
   Importance: Undecided => High

** Changed in: charm-ceph-mon
Milestone: None => 18.08

** Changed in: cloud-archive
 Assignee: Sean Feole (sfeole) => (unassigned)

** Changed in: nova (Ubuntu)
 Assignee: Sean Feole (sfeole) => (unassigned)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1773449

Title:
  VM rbd backed block devices inconsistent after unexpected host outage

Status in OpenStack ceph-mon charm:
  Triaged
Status in charms.ceph:
  Triaged
Status in Ubuntu Cloud Archive:
  Invalid
Status in OpenStack Compute (nova):
  Invalid
Status in ceph package in Ubuntu:
  Invalid
Status in nova package in Ubuntu:
  Invalid
Status in qemu package in Ubuntu:
  Invalid

Bug description:
  Reboot host that contains VMs with volumes and all VMs fail to boot.
  Happens with Queens on Bionic and Xenial

  [0.00] Initializing cgroup subsys cpuset

  [0.00] Initializing cgroup subsys cpu

  [0.00] Initializing cgroup subsys cpuacct

  [0.00] Linux version 4.4.0-124-generic
  (buildd@lcy01-amd64-028) (gcc version 5.4.0 20160609 (Ubuntu
  5.4.0-6ubuntu1~16.04.9) ) #148-Ubuntu SMP Wed May 2 13:00:18 UTC 2018
  (Ubuntu 4.4.0-124.148-generic 4.4.117)

  [0.00] Command line:
  BOOT_IMAGE=/boot/vmlinuz-4.4.0-124-generic
  root=UUID=bca2de6e-f774-4203-ae05-e8deeb05f64a ro console=tty1
  console=ttyS0

  [0.00] KERNEL supported cpus:

  [0.00]   Intel GenuineIntel

  [0.00]   AMD AuthenticAMD

  [0.00]   Centaur CentaurHauls

  [0.00] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256

  [0.00] x86/fpu: Supporting XSAVE feature 0x01: 'x87 floating
  point registers'

  [0.00] x86/fpu: Supporting XSAVE feature 0x02: 'SSE registers'

  [0.00] x86/fpu: Supporting XSAVE feature 0x04: 'AVX registers'

  [0.00] x86/fpu: Enabled xstate features 0x7, context size is
  832 bytes, using 'standard' format.

  [0.00] x86/fpu: Using 'eager' FPU context switches.

  [0.00] e820: BIOS-provided physical RAM map:

  [0.00] BIOS-e820: [mem 0x-0x0009fbff]
  usable

  [0.00] BIOS-e820: [mem 0x0009fc00-0x0009]
  reserved

  [0.00] BIOS-e820: [mem 0x000f-0x000f]
  reserved

  [0.00] BIOS-e820: [mem 0x0010-0x7ffdbfff]
  usable

  [0.00] BIOS-e820: [mem 0x7ffdc000-0x7fff]
  reserved

  [0.00] BIOS-e820: [mem 0xfeffc000-0xfeff]
  reserved

  [0.00] BIOS-e820: [mem 0xfffc-0x]
  reserved

  [0.00] NX (Execute Disable) protection: active

  [0.00] SMBIOS 2.8 present.

  [0.00] Hypervisor detected: KVM

  [0.00] e820: last_pfn = 0x7ffdc max_arch_pfn = 0x4

  [0.00] x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WC
  UC- WT

  [0.00] found SMP MP-table at [mem 0x000f6a20-0x000f6a2f]
  mapped at [880f6a20]

  [0.00] Scanning 1 areas for low memory corruption

  [0.00] Using GB pages for direct mapping

  [0.00] RAMDISK: [mem 0x361f4000-0x370f1fff]

  [0.00] ACPI: Early table checksum verification disabled

  [0.00] ACPI: RSDP 0x000F6780 14 (v00 BOCHS )

  [0.00] ACPI: RSDT 0x7FFE1649 2C (v01 BOCHS
  BXPCRSDT 0001 BXPC 0001)

  [0.00] ACPI: FACP 0x7FFE14CD 74 (v01 BOCHS
  BXPCFACP 0001 BXPC 0001)

  [0.00] ACPI: DSDT 0x7FFE0040 00148D (v01 BOCHS
  BXPCDSDT 0001 BXPC 0001)

  [0.00] ACPI: FACS 0x7FFE 40

  [0.00] ACPI: APIC 0x7FFE15C1 88 (v01 BOCHS
  BXPCAPIC