Hey Bobby,
thanks Bobby.
Which version of QEMU do you use? You mean 4.12 and later? Which CentOS and
libvirt?
We used the actually master, SystemVM and qemu 3.10 and 3.12. it seems it
related to the cloudstack version 4.12 and above…
Example: qemu-kvm-ev-2.10.0-21.el7_5.7.1 / CentOS 7 / actually master /
SystemVM before 4.11.X
--VR Dumpxml (two channel, first in use from libvirt, I know its disconnected
but it seems here like the same not online)
<channel type='unix'>
<source mode='bind' path='/var/lib/libvirt/qemu/r-1741-VM.agent'/>
<target type='virtio' name='r-1712-VM.vport' state='disconnected'/>
<alias name='channel0'/>
<address type='virtio-serial' controller='0' bus='0' port='1'/>
</channel>
<channel type='unix'>
<source mode='bind'
path='/var/lib/libvirt/qemu/r-1741-VM.org.qemu.guest_agent.0'/>
<target type='virtio' name='org.qemu.guest_agent.0' state='connected'/>
<alias name='channel1'/>
<address type='virtio-serial' controller='0' bus='0' port='2'/>
</channel>
— Socket seems there...
[root@kvm08 /]# ll /var/lib/libvirt/qemu/r-1741-VM.org.qemu.guest_agent.0
srwxrwxr-x. 1 root root 0 Jul 23 11:49
/var/lib/libvirt/qemu/r-1741-VM.org.qemu.guest_agent.0
—related to the channel it seems to work
[root@kvm08 /]# virsh qemu-agent-command r-1741-VM '{"execute":"guest-ping"}'
{"return":{}}
Example: qemu-kvm-ev-2.10.0-21.el7_5.7.1 / CentOS 7 / actually master /
SystemVM 4.12.0 or later
—VR Dumpxml
<channel type='unix'>
<source mode='bind'
path='/var/lib/libvirt/qemu/r-1750-VM.org.qemu.guest_agent.0'/>
<target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/>
<alias name='channel0'/>
<address type='virtio-serial' controller='0' bus='0' port='1'/>
</channel>
— Socket seems there…
[root@kvm08 /]# ll /var/lib/libvirt/qemu/r-1750-VM.org.qemu.guest_agent.0
srwxrwxr-x. 1 root root 0 Jul 24 22:23
/var/lib/libvirt/qemu/r-1750-VM.org.qemu.guest_agent.0
—related to the channel it seems not to work
[root@kvm08 /]# virsh qemu-agent-command r-1750-VM '{"execute":"guest-ping"}'
error: Guest agent is not responding: QEMU guest agent is not connected
After „second" restart of the systemVM the agent gets connected, patch.sh is
running and cloud-early-config does configure the machine.
The second channel is always connected from the first boot.
The curious thing is that if we reboot the vpc provisioning is done correctly
and the first channel is up too.
It seems that libvirt use the first channel for its own. before it was a vport
and agent.0 … but it seems only the second channel was used.
—>https://wiki.libvirt.org/page/Qemu_guest_agent
Configure guest agent without libvirt interference
"In some cases, users might want to configure a guest agent in their domain XML
but don't want libvirt to connect to guest agent socket at all. Because libvirt
connects to a guest agent channel if and only if it is a virtio channel with
org.qemu.guest_agent.0 name, all one need to do is void at least one of these
conditions. However, the most feasible way is to change the target's name:"
It seems we need to configure a second one as org.qemu.guest_agent.1 or use two
of the channels!
1x org.qemu.guest_agent.1
or
org.qemu.guest_agent.0
And
org.qemu.guest_agent.1
Maybe I am wrong. Its only a theory that needs to be more investigation …
I think we have a problem with the channels or qemu agent inside the SystemVM.
Cheers,
Sven
__
Sven Vogel
Teamlead Platform
EWERK DIGITAL GmbH
Brühl 24, D-04109 Leipzig
P +49 341 42649 - 11
F +49 341 42649 - 18
[email protected]
www.ewerk.com
Geschäftsführer:
Dr. Erik Wende, Hendrik Schubert, Frank Richter
Registergericht: Leipzig HRB 17023
Zertifiziert nach:
ISO/IEC 27001:2013
DIN EN ISO 9001:2015
DIN ISO/IEC 20000-1:2011
ISAE 3402 Typ II Assessed
EWERK-Blog<https://blog.ewerk.com/> |
LinkedIn<https://www.linkedin.com/company/ewerk-group> |
Xing<https://www.xing.com/company/ewerk> |
Twitter<https://twitter.com/EWERK_Group> |
Facebook<https://de-de.facebook.com/EWERK.IT/>
Mit Handelsregistereintragung vom 09.07.2019 ist die EWERK RZ GmbH auf die
EWERK IT GmbH verschmolzen und firmiert nun gemeinsam unter dem Namen: EWERK
DIGITAL GmbH, für weitere Informationen klicken Sie
hier<https://www.ewerk.com/ewerkdigital>.
Auskünfte und Angebote per Mail sind freibleibend und unverbindlich.
Disclaimer Privacy:
Der Inhalt dieser E-Mail (einschließlich etwaiger beigefügter Dateien) ist
vertraulich und nur für den Empfänger bestimmt. Sollten Sie nicht der
bestimmungsgemäße Empfänger sein, ist Ihnen jegliche Offenlegung,
Vervielfältigung, Weitergabe oder Nutzung des Inhalts untersagt. Bitte
informieren Sie in diesem Fall unverzüglich den Absender und löschen Sie die
E-Mail (einschließlich etwaiger beigefügter Dateien) von Ihrem System. Vielen
Dank.
The contents of this e-mail (including any attachments) are confidential and
may be legally privileged. If you are not the intended recipient of this
e-mail, any disclosure, copying, distribution or use of its contents is
strictly prohibited, and you should please notify the sender immediately and
then delete it (including any attachments) from your system. Thank you.
Am 24.07.2019 um 09:47 schrieb Boris Stoyanov
<[email protected]<mailto:[email protected]>>:
Hi Sven,
I've just installed master and upgraded my hosts to latest qemu-ev, then was
able to deploy successfully the system vms, so looks like it's related to 4.12.
Bobby.
On 21.07.19, 18:23, "Sven Vogel" <[email protected]<mailto:[email protected]>>
wrote:
Hi Rohit,
Thx for input. We don’t checked the 11.X.X version. We tied the upcoming
master and 4.12. we check in the following week the XML from a actual Version
(Master). Maybe you are right and we see only one. I think it’s a good idea to
rid of the old adapter. I think what you mean is we should see only
„path='/var/lib/libvirt/qemu/r-1710-VM.org.qemu.guest_agent.0'/><target
type='virtio' name='org.qemu.guest_agent.0'/>“ that one. But why we don’t get
an connection to the router/systemVM. So what i see is that the Agent socket it
present. We will try the „virsh qemu-agent-command“ and see how if we get an
connection.
Do you think the problem is a Agent Problem into the SystemVM?
Cheers
Sven
__
Sven Vogel
Teamlead Platform
EWERK RZ GmbH
Brühl 24, D-04109 Leipzig
P +49 341 42649 - 11
F +49 341 42649 - 18
[email protected]<mailto:[email protected]>
www.ewerk.com<http://www.ewerk.com/>
Geschäftsführer:
Dr. Erik Wende, Hendrik Schubert, Frank Richter
Registergericht: Leipzig HRB 17023
Zertifiziert nach:
ISO/IEC 27001:2013
DIN EN ISO 9001:2015
DIN ISO/IEC 20000-1:2011
EWERK-Blog | LinkedIn | Xing | Twitter | Facebook
Auskünfte und Angebote per Mail sind freibleibend und unverbindlich.
Disclaimer Privacy:
Der Inhalt dieser E-Mail (einschließlich etwaiger beigefügter Dateien) ist
vertraulich und nur für den Empfänger bestimmt. Sollten Sie nicht der
bestimmungsgemäße Empfänger sein, ist Ihnen jegliche Offenlegung,
Vervielfältigung, Weitergabe oder Nutzung des Inhalts untersagt. Bitte
informieren Sie in diesem Fall unverzüglich den Absender und löschen Sie die
E-Mail (einschließlich etwaiger beigefügter Dateien) von Ihrem System. Vielen
Dank.
The contents of this e-mail (including any attachments) are confidential and
may be legally privileged. If you are not the intended recipient of this
e-mail, any disclosure, copying, distribution or use of its contents is
strictly prohibited, and you should please notify the sender immediately and
then delete it (including any attachments) from your system. Thank you.
Am 21.07.2019 um 16:50 schrieb Rohit Yadav
<[email protected]<mailto:[email protected]>>:
Hi Sven,
Are you able to reproduce the failure against latest KVM qemu-ev + 4.11.3
systemvmtemplate with ACS 4.11.3.0? The PR
https://github.com/apache/cloudstack/pull/3278 only fixes the issue towards
4.11.3.0 and upcoming master/4.13.0.0, qemu-ev will fail for 4.11.2.0 or
4.12.0.0. Also, with the PR/patch applied you should only see a single virtio
serial port device on the domain xml, we got rid of the old/special case which
would add a specific vport/serial-port device for systemvms.
Regards,
Rohit Yadav
Software Architect, ShapeBlue
https://www.shapeblue.com<https://www.shapeblue.com/>
________________________________
From: Sven Vogel <[email protected]<mailto:[email protected]>>
Sent: Saturday, July 20, 2019 3:45:08 PM
To: dev <[email protected]<mailto:[email protected]>>
Subject: Re: Latest Qemu KVM EV appears to be broken with ACS #3278
Hi Simon,
Thanks for your information. We looking forward it you on a more recent
mainline version. In the meantime we try find out more out more about this bug.
Please feel free if you have new information add them here.
In the meantime I wrote some comment below this PR #3278
Cheers
Sven
__
Sven Vogel
Teamlead Platform
EWERK DIGITAL GmbH
Brühl 24, D-04109 Leipzig
P +49 341 42649 - 11
F +49 341 42649 - 18
[email protected]<mailto:[email protected]>
www.ewerk.com<http://www.ewerk.com>
Geschäftsführer:
Dr. Erik Wende, Hendrik Schubert, Frank Richter
Registergericht: Leipzig HRB 17023
Zertifiziert nach:
ISO/IEC 27001:2013
DIN EN ISO 9001:2015
DIN ISO/IEC 20000-1:2011
ISAE 3402 Typ II Assessed
EWERK-Blog<https://blog.ewerk.com/> |
LinkedIn<https://www.linkedin.com/company/ewerk-group> |
Xing<https://www.xing.com/company/ewerk> |
Twitter<https://twitter.com/EWERK_Group> |
Facebook<https://de-de.facebook.com/EWERK.IT/>
Mit Handelsregistereintragung vom 10.07.2019 ist die EWERK RZ GmbH auf die
EWERK IT GmbH verschmolzen und firmiert zukünftig gemeinsam unter dem Namen:
EWERK DIGITAL GmbH, für weitere Informationen klicken Sie
hier<https://www.ewerk.com/ewerkdigital>.
Auskünfte und Angebote per Mail sind freibleibend und unverbindlich.
Disclaimer Privacy:
Der Inhalt dieser E-Mail (einschließlich etwaiger beigefügter Dateien) ist
vertraulich und nur für den Empfänger bestimmt. Sollten Sie nicht der
bestimmungsgemäße Empfänger sein, ist Ihnen jegliche Offenlegung,
Vervielfältigung, Weitergabe oder Nutzung des Inhalts untersagt. Bitte
informieren Sie in diesem Fall unverzüglich den Absender und löschen Sie die
E-Mail (einschließlich etwaiger beigefügter Dateien) von Ihrem System. Vielen
Dank.
The contents of this e-mail (including any attachments) are confidential and
may be legally privileged. If you are not the intended recipient of this
e-mail, any disclosure, copying, distribution or use of its contents is
strictly prohibited, and you should please notify the sender immediately and
then delete it (including any attachments) from your system. Thank you.
Am 19.07.2019 um 15:04 schrieb Simon Weller
<[email protected]<mailto:[email protected]>>:
Hi Sven,
We're still using 2.10 right now and we haven't tested the patches yet for
2.12. Having said that, we're currently trying to get our internal ACS release
up to a more recent mainline version, so I'd suspect we'll have to dive into
this fairly soon.
-Si
________________________________
From: Sven Vogel <[email protected]<mailto:[email protected]>>
Sent: Friday, July 19, 2019 6:48 AM
To: dev <[email protected]<mailto:[email protected]>>
Subject: Re: Latest Qemu KVM EV appears to be broken with ACS #3278
Hi Guys,
Sorry that I formal reopen this issue.
We tested the actual system with this patch #3278
We use CentOS 1805 or 1810 and OVS and QEMU 2.12. It seems the first agent
contact will not work so that the agent know the virtual machine is running.
With 2.10 and the old agent, SystemVM it works.
2019-07-19 00:17:30,571 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) (logid:8ad14091) Executing:
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patch.sh -n v-1728-VM -c
template=domP type=consoleproxy host=10.24.48.46 port=8250 name=v-1728-VM
premium=true zone=4 pod=4 guid=Proxy.1728 proxy_vm=1728 disable_rp_filter=true
eth2ip=185.232.219.237 eth2mask=255.255.255.248 gateway=185.232.219.233
eth0ip=169.254.0.193 eth0mask=255.255.0.0 eth1ip=10.24.48.124
eth1mask=255.255.255.0 mgmtcidr=10.24.48.0/24 localgw=10.24.48.1
internaldns1=10.24.48.33 internaldns2=10.24.48.34 dns1=217.69.224.73
dns2=85.232.28.146
2019-07-19 00:17:30,573 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) (logid:8ad14091) Executing while with timeout :
300000
2019-07-19 00:17:30,616 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-2:null) (logid:7577151e) Unable to logon to 169.254.2.189
2019-07-19 00:17:30,616 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-2:null) (logid:7577151e) Trying to connect to
169.254.2.189
2019-07-19 00:17:31,874 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-3:null) (logid:04b4d6ce) Could not connect to 169.254.1.21
2019-07-19 00:17:33,622 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-2:null) (logid:7577151e) Could not connect to
169.254.2.189
2019-07-19 00:17:36,874 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-3:null) (logid:04b4d6ce) Unable to logon to 169.254.1.21
2019-07-19 00:17:36,874 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-3:null) (logid:04b4d6ce) Trying to connect to 169.254.1.21
2019-07-19 00:17:38,622 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-2:null) (logid:7577151e) Trying to connect to
169.254.2.189
@simonweller you encountered the same problem?
@rohit maybe there is something what we forget or what’s wrong?
@wido do you have ideas?
thanks
Sven
__
Sven Vogel
Teamlead Platform
EWERK DIGITAL GmbH
Brühl 24, D-04109 Leipzig
P +49 341 42649 - 11
F +49 341 42649 - 18
[email protected]<mailto:[email protected]>
www.ewerk.com<http://www.ewerk.com/><http://www.ewerk.com<http://www.ewerk.com/>>
Geschäftsführer:
Dr. Erik Wende, Hendrik Schubert, Frank Richter
Registergericht: Leipzig HRB 17023
Zertifiziert nach:
ISO/IEC 27001:2013
DIN EN ISO 9001:2015
DIN ISO/IEC 20000-1:2011
ISAE 3402 Typ II Assessed
EWERK-Blog<https://blog.ewerk.com/> |
LinkedIn<https://www.linkedin.com/company/ewerk-group> |
Xing<https://www.xing.com/company/ewerk> |
Twitter<https://twitter.com/EWERK_Group> |
Facebook<https://de-de.facebook.com/EWERK.IT/>
Mit Handelsregistereintragung vom 10.07.2019 ist die EWERK RZ GmbH auf die
EWERK IT GmbH verschmolzen und firmiert zukünftig gemeinsam unter dem Namen:
EWERK DIGITAL GmbH, für weitere Informationen klicken Sie
hier<https://www.ewerk.com/ewerkdigital>.
Auskünfte und Angebote per Mail sind freibleibend und unverbindlich.
Disclaimer Privacy:
Der Inhalt dieser E-Mail (einschließlich etwaiger beigefügter Dateien) ist
vertraulich und nur für den Empfänger bestimmt. Sollten Sie nicht der
bestimmungsgemäße Empfänger sein, ist Ihnen jegliche Offenlegung,
Vervielfältigung, Weitergabe oder Nutzung des Inhalts untersagt. Bitte
informieren Sie in diesem Fall unverzüglich den Absender und löschen Sie die
E-Mail (einschließlich etwaiger beigefügter Dateien) von Ihrem System. Vielen
Dank.
The contents of this e-mail (including any attachments) are confidential and
may be legally privileged. If you are not the intended recipient of this
e-mail, any disclosure, copying, distribution or use of its contents is
strictly prohibited, and you should please notify the sender immediately and
then delete it (including any attachments) from your system. Thank you.
Am 22.04.2019 um 21:29 schrieb Simon Weller
<[email protected]<mailto:[email protected]><mailto:[email protected]>>:
In our case the SystemVMs were booting fine, but ACS wasn't able to inject the
payload via the socket.
[email protected]
www.shapeblue.com
Amadeus House, Floral Street, London WC2E 9DPUK
@shapeblue
[email protected]<mailto:[email protected]>
www.shapeblue.com<http://www.shapeblue.com/>
Amadeus House, Floral Street, London WC2E 9DPUK
@shapeblue
