[Bug 1582990] Re: internal error: End of file from monitor
Hi s10, I know, but we never hit the deadlock they had fixed by the revert in Artful AFAIK. We followed the revert in the code for Bionic, but not yet intended to SRU the revert to Artful. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1582990 Title: internal error: End of file from monitor To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1582990/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1582990] Re: internal error: End of file from monitor
Upstream reverted this fix: https://github.com/libvirt/libvirt/commit/72adaf2f10509c3682f2c65ffad4176e00e5a2fb -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1582990 Title: internal error: End of file from monitor To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1582990/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1582990] Re: internal error: End of file from monitor
One has to be clear - any case where qemu stops executing in an unexpected way make libvirt go "huh ?" and this is that message. Please be careful that having the same "message" does not imply the same error other than "qemu went away" which can be for various reasons. There was some activity to improve the messages where possible, but for some libvirt just can't know. That "it will still happen anyway" made me forget about this. So thanks for the ping still @yanpeifei I checked and my fix went upstream in 3.4 which we have (newer) in Artful. I don't consider it SRU worth thou - so setting fix released to match reality. ** Changed in: libvirt (Ubuntu) Status: Triaged => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1582990 Title: internal error: End of file from monitor To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1582990/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1582990] Re: internal error: End of file from monitor
I met the same bug. Aug 10 08:01:41 controller libvirtd[12421]: internal error: End of file from monitor Aug 10 08:23:12 controller libvirtd[12421]: internal error: End of file from monitor Aug 10 08:24:06 controller libvirtd[12421]: internal error: End of file from monitor Aug 10 08:25:54 controller libvirtd[12421]: internal error: End of file from monitor Aug 10 08:28:03 controller libvirtd[12421]: internal error: End of file from monitor Aug 10 08:36:49 controller libvirtd[12421]: internal error: End of file from monitor Aug 10 08:37:00 controller libvirtd[12421]: internal error: End of file from monitor Aug 10 08:44:33 controller libvirtd[12421]: internal error: End of file from monitor Aug 10 08:46:12 controller libvirtd[12421]: internal error: End of file from monitor -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1582990 Title: internal error: End of file from monitor To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1582990/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1582990] Re: internal error: End of file from monitor
Great; then it feels like it's ready for upstream review. Thanks -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1582990 Title: internal error: End of file from monitor To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1582990/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1582990] Re: internal error: End of file from monitor
On Wed, Mar 8, 2017 at 2:01 AM, Seth Arnold <1582...@bugs.launchpad.net> wrote: > Is the bool getting the > correct initial value due to a 'memset 0' when the struct is > initialized? Or does this also need an explicit initializer? > Hi, as always in libvirt things are a bit more abstracted. After a trip through somewhat object oriented code in C (which I don't want to bother you) I can confirm that eventually is is a calloc that gets the structs memory (src/util/viralloc.c:560). Also an indicator - the other struct vars have no initiaizers either. So I think it does not need an explicit initializer. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1582990 Title: internal error: End of file from monitor To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1582990/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1582990] Re: internal error: End of file from monitor
Christian, many thanks for tracking this down. Is the bool getting the correct initial value due to a 'memset 0' when the struct is initialized? Or does this also need an explicit initializer? Thanks -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1582990 Title: internal error: End of file from monitor To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1582990/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1582990] Re: internal error: End of file from monitor
** Tags added: patch -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1582990 Title: internal error: End of file from monitor To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1582990/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1582990] Re: internal error: End of file from monitor
Do you think it makes sense suggesting that upstream? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1582990 Title: internal error: End of file from monitor To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1582990/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1582990] Re: internal error: End of file from monitor
Reproducable with current upstream (3.2 in development) Note that on newer versions with full virtlogd there also is a similar message from that: virtlogd[19517]: End of file while reading data: Input/output error There is some side effect that makes this slightly less straight forward. If the virReportError call is missing later on it reports "Error while processing monitor IO". I guess the report sets err and makes virGetLastError return something later on (what we still see is the generic catch all error message). It is like it is insisting on an error :-) If I don't get any from above I set the generic one down here. Yet just skipping those might have side effects I currently can't see. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1582990 Title: internal error: End of file from monitor To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1582990/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1582990] Re: internal error: End of file from monitor
The following patch (so far RFC level) would avoid the error for all qemu shutdown modes supported (acpi and agent). I checked that: - no error message on acpi shutdown - no error message on agent shutdown - still reports the error when the guest is killed from "other sources" (we want that). - still reports "other errors" on shutdown It is just not adding the EOF issue or the fallback "there has to be an error" in case qemuMonitorEmitShutdown was passed setting mon->willhangup = 1. ** Patch added: "do not report error on expected shutdown" https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1582990/+attachment/4832905/+files/0001-qemu-monitor-do-not-report-error-on-shutdown.patch -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1582990 Title: internal error: End of file from monitor To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1582990/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1582990] Re: internal error: End of file from monitor
On ther per guest qemu log one always can see:
2017-03-07 08:59:35.748+: shutting down
Then libvirt throws
Mär 07 09:59:35 lap libvirtd[3023]: internal error: End of file from monitor
It seems even on a clean shutdown I see before a kill signal like:
qemu-system-x86_64: terminating on signal 15 from pid 3023
So might that be "normal" if stopped like this.
There are several stop options for "virsh shutdown"
Shutdown methods:
"acpi", "agent", "initctl", "signal" and "paravirt"
And then one can also just shutdown from inside the guest.
Notation is "error log msg / kill signal", so Y/Y means both happened.
acpi: Y/Y
agent: Y/Y
shutdown -h 0 inside guest: Y/Y
initctl not-supported for qemu
signal: not-supported for qemu
paravirt: not-supported for qemu
So the "mode" doesn't matter - which at least tells us it is code a bit
outside the actual shutdown.
The signal comes from "/usr/sbin/libvirtd" itself.
And since it does so independent to the shutdown method it seems to be some
sort of generic "cleanup".
Lets find that in the code.
There are three kills in an strace of the shutdown action:
(some context to find these later)
0.20 write(22, "
0.32 fsync(22) = 0 <0.06>
0.18 close(22) = 0 <0.07>
0.20 rename("/var/run/libvirt/qemu/zesty-test-log.xml.new",
"/var/run/libvirt/qemu/zesty-test-log.xml") = 0 <0.24
0.39 unlink("/var/run/libvirt/qemu/zesty-test-log.xml.new") = -1
ENOENT (No such file or directory) <0.09>
XX 0.27 kill(11836, SIGTERM) = 0 <0.10>
0.31 poll([{fd=6, events=POLLIN}, {fd=8, events=POLLIN}, {fd=11,
events=POLLIN}, {fd=12, events=POLLIN}, {fd=13, even
0.000113 read(26, "{\"timestamp\": {\"seconds\": 148887"..., 1023) = 81
<0.12>
0.35 close(22) = 0 <0.44>
0.60
unlink("/var/lib/libvirt/qemu/channel/target/domain-zesty-test-log/org.qemu.guest_agent.0")
= -1 ENOENT (No such
XX 0.32 kill(11836, SIGTERM) = 0 <0.13>
0.29 nanosleep({0, 2}, NULL) = 0 <0.200086>
XX 0.200115 kill(11836, SIG_0)= -1 ESRCH (No such process) <0.07>
0.49 open("/dev/null", O_RDWR|O_CLOEXEC) = 22 <0.20>
0.45 pipe2([23, 24], O_CLOEXEC) = 0 <0.10>
0.27 pipe2([25, 26], O_CLOEXEC) = 0 <0.09>
0.21 rt_sigprocmask(SIG_SETMASK, ~[RTMIN RT_1], [ALRM], 8) = 0
<0.07>
Install libvirt0-dbg and break on those when called from the daemon.
Found the "kill" that does this at with a backtrace like:
#0 kill () at ../sysdeps/unix/syscall-template.S:84
#1 0x7f3511d3014a in virProcessKill (pid=, sig=) at ../../../src/util/virprocess.c:333
#2 0x7f34ea693928 in qemuProcessKill (vm=vm@entry=0x7f34dc004f60,
flags=flags@entry=2) at ../../../src/qemu/qemu_process.c:5213
#3 0x7f34ea693a0d in qemuProcessShutdownOrReboot (driver=,
vm=0x7f34dc004f60)
at ../../../src/qemu/qemu_process.c:612
#4 0x7f34ea693b95 in qemuProcessHandleShutdown (mon=,
vm=0x7f34dc004f60, opaque=0x7f34e03b3310)
at ../../../src/qemu/qemu_process.c:686
#5 0x7f34ea6a9682 in qemuMonitorEmitShutdown (mon=0x7f34c00170e0) at
../../../src/qemu/qemu_monitor.c:1232
#6 0x7f34ea6bda72 in qemuMonitorJSONIOProcessEvent (obj=0x55d05b5c8fa0,
mon=0x7f34c00170e0)
at ../../../src/qemu/qemu_monitor_json.c:173
#7 qemuMonitorJSONIOProcessLine (msg=0x0, line=,
mon=0x7f34c00170e0) at ../../../src/qemu/qemu_monitor_json.c:202
#8 qemuMonitorJSONIOProcess (mon=mon@entry=0x7f34c00170e0,
data=0x55d05b625130 "{\"timestamp\": {\"seconds\": 1488879468,
\"microseconds\": 318008}, \"event\": \"SHUTDOWN\"}\r\n", len=85,
msg=msg@entry=0x0) at ../../../src/qemu/qemu_monitor_json.c:244
#9 0x7f34ea6a874a in qemuMonitorIOProcess (mon=0x7f34c00170e0) at
../../../src/qemu/qemu_monitor.c:426
#10 qemuMonitorIO (watch=watch@entry=196, fd=, events=0,
events@entry=1, opaque=opaque@entry=0x7f34c00170e0)
at ../../../src/qemu/qemu_monitor.c:680
#11 0x7f3511d00393 in virEventPollDispatchHandles (fds=,
nfds=)
at ../../../src/util/vireventpoll.c:509
#12 virEventPollRunOnce () at ../../../src/util/vireventpoll.c:658
#13 0x7f3511cfec01 in virEventRunDefaultImpl () at
../../../src/util/virevent.c:314
#14 0x7f3511e4ab4d in virNetDaemonRun (dmn=0x55d05b5ddb10) at
../../../src/rpc/virnetdaemon.c:701
#15 0x55d059e9e407 in main (argc=, argv=) at
../../../daemon/libvirtd.c:1577
In qemuProcessHandleShutdown as a tail end cleanup (e.g. no matter if
priv-agent) was called or not it is killing that way.
I jumped the actual kill in gdb but that is the one place that makes the qemu
process go away.
If skipping it ends up stale (qemu there,
[Bug 1582990] Re: internal error: End of file from monitor
Not an excuse at all, but a statement of overall status - I checked a few other distros (Redhat, IBMKVM, Suse) as far as I could see they all have the same I'm analyzing the case at the moment (will post on that later) and I think I have found a place where once could tweak, but it seems to be an upstream bug/patch which then need to tickle down. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1582990 Title: internal error: End of file from monitor To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1582990/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1582990] Re: internal error: End of file from monitor
Hmm, this seems to be a non Fatal error. After explicitly looking for it I see this up to (including) zesty. Slight change in the Text: "End of file from qemu monitor" Which is as before +qemu This seems to be part of a normal non-error guest shutdown phase: dnsmasq-dhcp[5221]: DHCPRELEASE(virbr0) 192.168.122.235 52:54:00:b6:c3:65 kernel: virbr0: port 2(vnet0) entered disabled state kernel: device vnet0 left promiscuous mode kernel: virbr0: port 2(vnet0) entered disabled state libvirtd[18333]: internal error: End of file from qemu monitor virtlogd[18234]: End of file while reading data: Input/output error -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1582990 Title: internal error: End of file from monitor To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1582990/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1582990] Re: internal error: End of file from monitor
This was still happening as of mid-february 2017: Feb 24 20:34:45 hunt libvirtd[3871]: internal error: End of file from monitor $ dpkg -l libvirt-bin qemu-kvm | cat ... ii libvirt-bin1.3.1-1ubuntu10.8 amd64programs for the libvirt library ii qemu-kvm 1:2.5+dfsg-5ubuntu10.8 amd64QEMU Full virtualization Thanks -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1582990 Title: internal error: End of file from monitor To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1582990/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1582990] Re: internal error: End of file from monitor
** Changed in: libvirt (Ubuntu) Status: Confirmed => Triaged -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1582990 Title: internal error: End of file from monitor To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1582990/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1582990] Re: internal error: End of file from monitor
See: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=785011 Debian Bug report logs - #785011 libvirtd: log error message after successfully terminating machine ** Bug watch added: Debian Bug tracker #785011 http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=785011 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1582990 Title: internal error: End of file from monitor To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1582990/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1582990] Re: internal error: End of file from monitor
** Changed in: libvirt (Ubuntu) Importance: Undecided => Medium -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1582990 Title: internal error: End of file from monitor To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1582990/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1582990] Re: internal error: End of file from monitor
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: libvirt (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1582990 Title: internal error: End of file from monitor To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1582990/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
