** Project changed: charm-nova-compute => nova -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1737879
Title: 2017-12-13 00:20:15.681+0000: shutting down, reason=crashed Status in OpenStack Compute (nova): New Bug description: 1)vm shutdown info: 43 2017-12-13 00:20:15.681+0000: shutting down, reason=crashed 2） nova start vm info: 2017-12-13 00:24:07.337+0000: starting up libvirt version: 3.2.0, package: 14.el7_4.3 (CentOS BuildSystem <http://bugs.centos.org>, 2017-09-07-11:27:44, c1bm.rdu2.centos.org), qemu versi on: 1.5.3 (qemu-kvm-1.5.3-141.el7_4.2), hostname: l23-41-5 45 LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name instance-0000005c -S -machine pc-i440fx-rhel7.0.0,accel=kvm,usb=off,dump-g uest-core=off -m 16384 -realtime mlock=off -smp 16,sockets=16,cores=16,threads=1 -uuid fc0cc515-5f7b-420c-bcda-d77b66610af2 -smbios 'type=1,manufacturer=Fedora Project,product=OpenStack Nova,version=13.1.2-1.el7,serial=f7f54ab4-0b4f-499c-8d1d-291e11f50bf2,uuid=fc0cc515-5f7b-420c-bcda-d77b66610af2,family=Virtual Machine' -no-user-config -nodefaults -chardev socket,id=cha rmonitor,path=/var/lib/libvirt/qemu/domain-37-instance-0000005c/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime,driftfix=slew -global kvm- pit.lost_tick_policy=delay -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive 'file=rbd:nova_instances/fc0cc515-5f7b-420c-bcda-d77b66610af2 _disk:id=cinder:key=AQBNLRRZswapABAAh6GCDOkWFqWj5uYAMVgIKA==:auth_supported=cephx\;none:mon_host=10.211.41.4\:6789\;10.211.41.6\:6789\;10.211.41.8\:6789,format=raw,if=none,id=drive-virtio-disk 0,cache=writeback,throttling.bps-read=52425500,throttling.bps-write=52425500' -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -dri ve 'file=rbd:cinder_volumes/volume-cd2fcd54-0661-4a4d-b13d-419b6585c676:id=cinder:key=AQBNLRRZswapABAAh6GCDOkWFqWj5uYAMVgIKA==:auth_supported=cephx\;none:mon_host=10.211.41.4\:6789\;10.211. 41.6\:6789\;10.211.41.8\:6789,format=raw,if=none,id=drive-virtio-disk1,serial=cd2fcd54-0661-4a4d-b13d-419b6585c676,cache=writeback' -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive= drive-virtio-disk1,id=virtio-disk1 -drive 'file=rbd:nova_instances/fc0cc515-5f7b-420c-bcda-d77b66610af2_disk.config:id=cinder:key=AQBNLRRZswapABAAh6GCDOkWFqWj5uYAMVgIKA==:auth_supported= cephx\;none:mon_host=10.211.41.4\:6789\;10.211.41.6\:6789\;10.211.41.8\:6789,format=raw,if=none,id=drive-ide0-1-1,readonly=on,cache=writeback,throttling.bps-read=52425500,throttling.bps-write= 52425500' -device ide-cd,bus=ide.1,unit=1,drive=drive-ide0-1-1,id=ide0-1-1 -netdev tap,fd=33,id=hostnet0,vhost=on,vhostfd=39 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:6 0:0b:06,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/data/nova/instances/fc0cc515-5f7b-420c-bcda-d77b66610af2/console.log -device isa-serial,chardev=charserial0,id=serial0 -char dev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0,bus=usb.0,port=1 -vnc 0.0.0.0:6 -k en-us -vga cirrus -device virtio-balloon-pci,id=b alloon0,bus=pci.0,addr=0x6 -msg timestamp=on 46 2017-12-13 00:24:07.337+0000: Domain id=37 is tainted: high-privileges 47 char device redirected to /dev/pts/7 (label charserial1) 3) Compute nodes system log: / var/log/messages Dec 13 08:20:15 l23-41-5 libvirtd: 2017-12-13 00:20:15.461+0000: 22613: error : qemuMonitorIO:697 : internal error: End of file from qemu monitor 4) openstack nova-compute log : /var/log/nova/nova-compute.log 2017-12-13 08:20:30.751 22579 INFO nova.compute.manager [-] [instance: fc0cc515-5f7b-420c-bcda-d77b66610af2] VM Stopped (Lifecycle Event) 2017-12-13 08:20:30.845 22579 INFO nova.compute.manager [req-1834e8c9-71b2-4a4c-8127-eee6888cf446 - - - - -] [instance: fc0cc515-5f7b-420c-bcda-d77b66610af2] During _sync_instance_power_state the DB power_state (1) does not match the vm_power_state from the hypervisor (4). Updating power_state in the DB to match the hypervisor. 2017-12-13 08:20:30.904 22579 WARNING nova.compute.manager [req-1834e8c9-71b2-4a4c-8127-eee6888cf446 - - - - -] [instance: fc0cc515-5f7b-420c-bcda-d77b66610af2] Instance shutdown by itself. Calling the stop API. Current vm_state: active, current task_state: None, original DB power_state: 1, current VM power_state: 4 2017-12-13 08:20:31.038 22579 INFO nova.extend.network [req-1834e8c9-71b2-4a4c-8127-eee6888cf446 - - - - -] Add william check_result: True 2017-12-13 08:20:31.039 22579 INFO nova.extend.network [req-1834e8c9-71b2-4a4c-8127-eee6888cf446 - - - - -] Add william,vlan1210 is exists,so is not create 2017-12-13 08:20:31.039 22579 INFO nova.compute.manager [req-1834e8c9-71b2-4a4c-8127-eee6888cf446 - - - - -] Add william, live migrate vm bridge_multiple_vlan True 2017-12-13 08:20:31.054 22579 INFO nova.compute.manager [req-1834e8c9-71b2-4a4c-8127-eee6888cf446 - - - - -] [instance: fc0cc515-5f7b-420c-bcda-d77b66610af2] Instance is already powered off in the hypervisor when stop is called. 2017-12-13 08:20:31.105 22579 INFO nova.virt.libvirt.driver [req-1834e8c9-71b2-4a4c-8127-eee6888cf446 - - - - -] [instance: fc0cc515-5f7b-420c-bcda-d77b66610af2] Instance already shutdown. 2017-12-13 08:20:31.114 22579 INFO nova.virt.libvirt.driver [-] [instance: fc0cc515-5f7b-420c-bcda-d77b66610af2] Instance destroyed successfully. 2017-12-13 08:24:05.008 22579 INFO nova.compute.resource_tracker [req-493ee54b-e894-471c-93ee-c5c8781f9d36 - - - - -] Compute_service record updated for l23-41-5.lq.cc:l23-41-5 2017-12-13 08:24:06.429 22579 INFO nova.extend.network [req-7360f51c-0e83-4eb2-b925-89949d620f5d 053d91950deb40d08b706988f4bf914a 8b79300ff2a546d896ef58a9c8004e68 - - -] Add william check_result: True 2017-12-13 08:24:06.430 22579 INFO nova.extend.network [req-7360f51c-0e83-4eb2-b925-89949d620f5d 053d91950deb40d08b706988f4bf914a 8b79300ff2a546d896ef58a9c8004e68 - - -] Add william,vlan1210 is exists,so is not create 2017-12-13 08:24:06.430 22579 INFO nova.compute.manager [req-7360f51c-0e83-4eb2-b925-89949d620f5d 053d91950deb40d08b706988f4bf914a 8b79300ff2a546d896ef58a9c8004e68 - - -] Add william, live migrate vm bridge_multiple_vlan True 2017-12-13 08:24:06.581 22579 INFO nova.virt.libvirt.driver [-] [instance: fc0cc515-5f7b-420c-bcda-d77b66610af2] Instance destroyed successfully. 2017-12-13 08:24:06.616 22579 WARNING nova.virt.osinfo [req-7360f51c-0e83-4eb2-b925-89949d620f5d 053d91950deb40d08b706988f4bf914a 8b79300ff2a546d896ef58a9c8004e68 - - -] Cannot find OS information - Reason: (No configuration information found for operating system Empty) 2017-12-13 08:24:07.260 22579 WARNING nova.virt.osinfo [req-7360f51c-0e83-4eb2-b925-89949d620f5d 053d91950deb40d08b706988f4bf914a 8b79300ff2a546d896ef58a9c8004e68 - - -] Cannot find OS information - Reason: (No configuration information found for operating system Empty) 2017-12-13 08:24:07.281 22579 INFO nova.virt.libvirt.config [req-7360f51c-0e83-4eb2-b925-89949d620f5d 053d91950deb40d08b706988f4bf914a 8b79300ff2a546d896ef58a9c8004e68 - - -] Add william , modify brdige type bridge = vlan1210 2017-12-13 08:24:07.282 22579 INFO nova.virt.libvirt.config [req-7360f51c-0e83-4eb2-b925-89949d620f5d 053d91950deb40d08b706988f4bf914a 8b79300ff2a546d896ef58a9c8004e68 - - -] wiliam bridge_multiple_vlan True 2017-12-13 08:24:07.282 22579 INFO nova.virt.libvirt.config [req-7360f51c-0e83-4eb2-b925-89949d620f5d 053d91950deb40d08b706988f4bf914a 8b79300ff2a546d896ef58a9c8004e68 - - -] Add William: Delete nwfilter rule <Element filterref at 0x7f84eb0> 2017-12-13 08:24:07.290 22579 INFO nova.virt.libvirt.firewall [req-7360f51c-0e83-4eb2-b925-89949d620f5d 053d91950deb40d08b706988f4bf914a 8b79300ff2a546d896ef58a9c8004e68 - - -] [instance: fc0cc515-5f7b-420c-bcda-d77b66610af2] Called setup_basic_filtering in nwfilter 2017-12-13 08:24:07.290 22579 INFO nova.virt.libvirt.firewall [req-7360f51c-0e83-4eb2-b925-89949d620f5d 053d91950deb40d08b706988f4bf914a 8b79300ff2a546d896ef58a9c8004e68 - - -] [instance: fc0cc515-5f7b-420c-bcda-d77b66610af2] Ensuring static filters 2017-12-13 08:24:08.083 22579 INFO nova.compute.manager [req-171953a6-8036-4ad1-a062-be5ca895b6aa - - - - -] [instance: fc0cc515-5f7b-420c-bcda-d77b66610af2] VM Resumed (Lifecycle Event) 2017-12-13 08:24:08.107 22579 INFO nova.virt.libvirt.driver [-] [instance: fc0cc515-5f7b-420c-bcda-d77b66610af2] Instance rebooted successfully. 2017-12-13 08:24:08.187 22579 INFO nova.compute.manager [req-171953a6-8036-4ad1-a062-be5ca895b6aa - - - - -] [instance: fc0cc515-5f7b-420c-bcda-d77b66610af2] During sync_power_state the instance has a pending task (powering-on). Skip. 2017-12-13 08:24:08.187 22579 INFO nova.compute.manager [req-171953a6-8036-4ad1-a062-be5ca895b6aa - - - - -] [instance: fc0cc515-5f7b-420c-bcda-d77b66610af2] VM Started (Lifecycle Event) 2017-12-13 08:25:06.395 22579 INFO nova.compute.resource_tracker [req-493ee54b-e894-471c-93ee-c5c8781f9d36 - - - - -] Auditing locally available compute resources for node l23-41-5 5) system info: 1) libvirt qemu version Virsh command line tool of libvirt 3.2.0 See web site at http://libvirt.org/ Compiled with support for: Hypervisors: QEMU/KVM LXC ESX Test Networking: Remote Network Bridging Interface netcf Nwfilter VirtualPort Storage: Dir Disk Filesystem SCSI Multipath iSCSI LVM RBD Gluster Miscellaneous: Daemon Nodedev SELinux Secrets Debug DTrace Readline Modular Note: instance-0000005c.log 8 hours time difference 2) openstack nova-compute openstack-nova-network-13.1.2-1.el7.noarch openstack-nova-compute-13.1.2-1.el7.noarch python-nova-13.1.2-1.el7.noarch openstack-nova-common-13.1.2-1.el7.noarch python2-novaclient-3.3.2-1.el7.noarch 3) ceph version ceph-10.2.2-0.el7.x86_64 ceph-common-10.2.2-0.el7.x86_64 ceph-base-10.2.2-0.el7.x86_64 ceph-osd-10.2.2-0.el7.x86_64 libcephfs1-10.2.2-0.el7.x86_64 ceph-mon-10.2.2-0.el7.x86_64 python-cephfs-10.2.2-0.el7.x86_64 ceph-selinux-10.2.2-0.el7.x86_64 ceph-mds-10.2.2-0.el7.x86_64 ceph-deploy-1.5.34-0.noarch Timing of VM downtime problems every day, don't know where the problem cause downtime phenomenon, hope the community people can give a solution Note: / var/log/libvirt/qemu/instance - 0000005 c. The log time is 8 hours To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1737879/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
