Re: [openstack-dev] [nova][libvirt] - 'nova reboot' causes console-log truncated
On 11/14/14 2:02 AM, Daniel P. Berrange wrote: On Thu, Nov 13, 2014 at 01:55:06PM -0800, Surojit Pathak wrote: Hi all, [Issue observed] If we issue 'nova reboot server', we get to have the console output of the latest bootup of the server only. The console output of the previous boot for the same server vanishes due to truncation[1]. If we do reboot from within the VM instance [ #sudo reboot ], or reboot the instance with 'virsh reboot instance' the behavior is not the same, where the console.log keeps increasing, with the new output being appended. This loss of history makes some debugging scenario difficult due to lack of information being available. Please point me to any solution/blueprint for this issue, if already planned. Otherwise, please comment on my analysis and proposals as solution, below - [Analysis] Nova's libvirt driver on compute node tries to do a graceful restart of the server instance, by attempting a soft_reboot first. If soft_reboot fails, it attempts a hard_reboot. As part of soft_reboot, it brings down the instance by calling shutdown(), and then calls createWithFlags() to bring this up. Because of this, qemu-kvm process for the instance gets terminated and new process is launched. In QEMU, the chardev file is opened with O_TRUNC, and thus we lose the previous content of the console.log file. On the other-hand, during 'virsh reboot instance', the same qemu-kvm process continues, and libvirt actually does a qemuDomainSetFakeReboot(). Thus the same file continues capturing the new console output as a continuation into the same file. Nova and libvirt have support for issuing a graceful reboot via the QEMU guest agent. So if you make sure that is installed, and tell Nova to use it, then Nova won't have to stop recreate the QEMU process and thus won't have the problem of overwriting the logs. Hi Daniel, Having GA to do graceful restart is nice option. But if it were to just preserve the same console file, even 'virsh reboot' achieves the purpose. As I explained in my original analysis, Nova seems to have not taken the path, as it does not want to have a false positive, where the GA does not respond or 'virDomain.reboot' fails later and the domain is not really restarted. [ CC-ed vish, author of nova http://tripsgrips.corp.gq1.yahoo.com:8080/source/xref/nova/nova//virt http://tripsgrips.corp.gq1.yahoo.com:8080/source/xref/nova/nova/virt//libvirt http://tripsgrips.corp.gq1.yahoo.com:8080/source/xref/nova/nova/virt/libvirt//driver.py http://tripsgrips.corp.gq1.yahoo.com:8080/source/xref/nova/nova/virt/libvirt/driver.py ] IMHO, QEMU should preserve the console-log file for a given domain, if it exists, by not opening with O_TRUNC option, instead opening with O_APPEND. I would like to draw a comparison of a real computer to which we might be connected over serial console, and the box gets powered down and up with external button press, and we do not lose the console history, if connected. And that's what is the experience console-log intends to provide. If you think, this is agreeable, please let me know, I will send the patch to qemu-devel@. -- Regards, SURO ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][libvirt] - 'nova reboot' causes console-log truncated
On Mon, Dec 22, 2014 at 04:36:02PM -0800, Surojit Pathak wrote: Hi Daniel, Having GA to do graceful restart is nice option. But if it were to just preserve the same console file, even 'virsh reboot' achieves the purpose. As I explained in my original analysis, Nova seems to have not taken the path, as it does not want to have a false positive, where the GA does not respond or 'virDomain.reboot' fails later and the domain is not really restarted. [ CC-ed vish, author of nova http://tripsgrips.corp.gq1.yahoo.com:8080/source/xref/nova/nova//virt http://tripsgrips.corp.gq1.yahoo.com:8080/source/xref/nova/nova/virt//libvirt http://tripsgrips.corp.gq1.yahoo.com:8080/source/xref/nova/nova/virt/libvirt//driver.py http://tripsgrips.corp.gq1.yahoo.com:8080/source/xref/nova/nova/virt/libvirt/driver.py ] IMHO, QEMU should preserve the console-log file for a given domain, if it exists, by not opening with O_TRUNC option, instead opening with O_APPEND. I would like to draw a comparison of a real computer to which we might be connected over serial console, and the box gets powered down and up with external button press, and we do not lose the console history, if connected. And that's what is the experience console-log intends to provide. If you think, this is agreeable, please let me know, I will send the patch to qemu-devel@. The issue is more complex than just removing the O_TRUNC from the open() flags. I havd a proposal that will (almost by accident) fix this in qemu by allowing console log files to be rotated. I'm also waorking on a similar feature in libvirt. I think the tl;dr: is that this /shoudl/ be fixed in kilo with a 'modern' libvirt. Yours Tony. pgp7TZH5n8wP4.pgp Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][libvirt] - 'nova reboot' causes console-log truncated
On 12/22/14 5:04 PM, Tony Breeds wrote: On Mon, Dec 22, 2014 at 04:36:02PM -0800, Surojit Pathak wrote: Hi Daniel, Having GA to do graceful restart is nice option. But if it were to just preserve the same console file, even 'virsh reboot' achieves the purpose. As I explained in my original analysis, Nova seems to have not taken the path, as it does not want to have a false positive, where the GA does not respond or 'virDomain.reboot' fails later and the domain is not really restarted. [ CC-ed vish, author of nova IMHO, QEMU should preserve the console-log file for a given domain, if it exists, by not opening with O_TRUNC option, instead opening with O_APPEND. I would like to draw a comparison of a real computer to which we might be connected over serial console, and the box gets powered down and up with external button press, and we do not lose the console history, if connected. And that's what is the experience console-log intends to provide. If you think, this is agreeable, please let me know, I will send the patch to qemu-devel@. The issue is more complex than just removing the O_TRUNC from the open() flags. I havd a proposal that will (almost by accident) fix this in qemu by allowing console log files to be rotated. I'm also waorking on a similar feature in libvirt. I think the tl;dr: is that this /shoudl/ be fixed in kilo with a 'modern' libvirt. Hi Tony, Can you please share some details of the effort, in terms of reference? Yours Tony. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Regards, SURO ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][libvirt] - 'nova reboot' causes console-log truncated
On Mon, Dec 22, 2014 at 07:16:27PM -0800, Surojit Pathak wrote: Hi Tony, Can you please share some details of the effort, in terms of reference? Well the initial discussions started with qemu at: http://lists.nongnu.org/archive/html/qemu-devel/2014-12/msg00765.html and then here: http://lists.openstack.org/pipermail/openstack-dev/2014-December/052356.html You'll note the the focus of the discussion is rotating the log files but I'm very much aware of the issue covered in theis thread and it will be covered in my fixes. Which is why I said 'almost' by accident ;P I have a partial implementation for the log rotation in qemu (you can issue a command from the monitor but I haven't looked at the HUP yet). I started looking at doing something in libvirt aswell but I haven't made much progress there due to conflicting priorities. Yours Tony. pgpNnUUPgRHYc.pgp Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][libvirt] - 'nova reboot' causes console-log truncated
On Thu, Nov 13, 2014 at 01:55:06PM -0800, Surojit Pathak wrote: Hi all, [Issue observed] If we issue 'nova reboot server', we get to have the console output of the latest bootup of the server only. The console output of the previous boot for the same server vanishes due to truncation[1]. If we do reboot from within the VM instance [ #sudo reboot ], or reboot the instance with 'virsh reboot instance' the behavior is not the same, where the console.log keeps increasing, with the new output being appended. This loss of history makes some debugging scenario difficult due to lack of information being available. Please point me to any solution/blueprint for this issue, if already planned. Otherwise, please comment on my analysis and proposals as solution, below - [Analysis] Nova's libvirt driver on compute node tries to do a graceful restart of the server instance, by attempting a soft_reboot first. If soft_reboot fails, it attempts a hard_reboot. As part of soft_reboot, it brings down the instance by calling shutdown(), and then calls createWithFlags() to bring this up. Because of this, qemu-kvm process for the instance gets terminated and new process is launched. In QEMU, the chardev file is opened with O_TRUNC, and thus we lose the previous content of the console.log file. On the other-hand, during 'virsh reboot instance', the same qemu-kvm process continues, and libvirt actually does a qemuDomainSetFakeReboot(). Thus the same file continues capturing the new console output as a continuation into the same file. Nova and libvirt have support for issuing a graceful reboot via the QEMU guest agent. So if you make sure that is installed, and tell Nova to use it, then Nova won't have to stop recreate the QEMU process and thus won't have the problem of overwriting the logs. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
