Re: [ovirt-users] Networking issues with oVirt Guest VM.

[Gary Verhulp]

I encountered similar if not exact same issue.

Exploring KVM with oVirt. Testing on Nested Virtualization (KVM on VMWare)

I was able to solve this issue by setting on each node.

brctl setageing   0

I do not have access to the switching infrastructure to easily troubleshoot.
I'm not sure of the ramifications of setting that. I know it turns  the bridge 
from switch functionality to hub.

I need to look more into it.
This is not in production. I will have to see if this is isolated to the Nested 
Virt setup I  have or will it persist  on physical nodes.

G


This email and any attachments are intended for the sole use of the named 
recipient(s) and contain(s) confidential information that may be proprietary, 
privileged or copyrighted under applicable law. If you are not the intended 
recipient, do not read, copy, or forward this email message or any attachments. 
Delete this email message and any attachments immediately.

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] Networking issues with oVirt Guest VM.

[Clint Smith]

Hi Yevgeny,
Thanks for your help.

No luck yet!

Q: IIUC, ovirt-engine and its host are vSphere VMs. Then, a kind of
no-macspoof should be applied from the vSphere side.
A: I'm not exactly sure what setting this would be on the vmware side but
we have ensured the following settings are enabled; Promiscuous Mode, MAC
Address Changes, Forged Transmits.

Q: BTW, are both of them on the same vShepre host?
A: Yes, they are both on the same vsphere host.

Q: Is DHCP server another VM on that host?
A: Yes, the DHCP server is another VM on that vsphere host as well.

Q: Where/how did you "turn on Port Mirroring"?
A: I turned on Port Mirroring in the vNIC Profile called ovirtmgmt, it's
off for most of my tests, however, when I turn it on, everything works.

Q: I'd start the troubleshooting by using tcpdump utility in order to
pinpoint the component that blocks the traffic.
A: I have done this and I can see all DHCP messages including the DHCPACK
on the bridge named 'ovirtmgmt', but when I do tcpdump on the guest VM
network device 'eth0' (which is also 'vnet0' on the host)I never see the
DHCPACK. This same thing happens with ARP replies from the gateway.

Here is an ACK message that is returned from the DHCP server to the
'ovirtmgmt' bridge (I don't know if this is helpful, but thought I'd
include it, just in case).
16:35:22.958616 IP (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto
UDP (17), length 328)
10.255.233.125.67 > 10.255.233.185.68: [udp sum ok] BOOTP/DHCP, Reply,
length 300, xid 0x8d5c8e3d, Flags [none] (0x)
  Your-IP 10.255.233.185
  Client-Ethernet-Address 00:1a:4a:16:01:55
  Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message Option 53, length 1: ACK
Server-ID Option 54, length 4: 10.255.233.125
Lease-Time Option 51, length 4: 600
Subnet-Mask Option 1, length 4: 255.255.255.0
BR Option 28, length 4: 10.255.233.255
Domain-Name-Server Option 6, length 4: 10.255.233.102
Default-Gateway Option 3, length 4: 10.255.233.1


Q: Did you try assigning a static IP instead of DHCP and then check
connectivity? If that works, then the problem is on the DHCP sever side
probably.
A: Yes, with a static IP on the guest, everything works fine.  It's
possible, but in my naive understanding it seems unlikely, that the problem
is on the DHCP server side because the ACK makes it all the way back to the
bridge.  Again, I could be wrong.


Q: If you do not see any requests in the DHCP server log, then I guess,
"dhclient -B" wouldn't help.
A: I see the whole DHCP process in my DHCP Server log:

Jul  5 21:21:12 mydomain dhcpd: DHCPDISCOVER from 00:1a:4a:16:01:55 via eth0
Jul  5 21:21:12 mydomain dhcpd: DHCPOFFER on 10.255.233.185 to
00:1a:4a:16:01:55 via eth0
Jul  5 21:21:12 mydomain dhcpd: DHCPREQUEST for 10.255.233.185
(10.255.233.125) from 00:1a:4a:16:01:55 via eth0
Jul  5 21:21:12 mydomain dhcpd: DHCPACK on 10.255.233.185 to
00:1a:4a:16:01:55 via eth0

In addition to "dhclient -B" allowing the guest to successfully gain an IP
address from the DHCP server, I recently found that running the command
"brctl ovirtmgmt setageing 0" works also.  I read that this turns the
bridge into a hub, which is only a clue and not a solution, I think.  It
seems that Forwarding Database changes that occur when I run that command
are what allows ACK message to then pass to the guest.

Here is my Forwarding Database when the Ageing is set to 300:

01:00:5e:00:00:01 dev ens32 self permanent
33:33:00:00:02:02 dev ens32 self permanent
33:33:00:00:00:01 dev ens32 self permanent
33:33:00:00:00:01 dev bond0 self permanent
fe:4e:66:bf:cf:48 dev ;vdsmdummy; vlan 1 master ;vdsmdummy; permanent
00:50:56:8e:aa:2a dev ens32 master ovirtmgmt
e4:d3:f1:d1:99:c4 dev ens32 master ovirtmgmt
00:50:56:8e:d6:a2 dev ens32 master ovirtmgmt
00:50:56:8e:d2:cd dev ens32 master ovirtmgmt
00:50:56:8e:be:ca dev ens32 master ovirtmgmt permanent
00:50:56:8e:17:59 dev ens32 master ovirtmgmt
fe:1a:4a:16:01:55 dev vnet0 master ovirtmgmt permanent
fe:1a:4a:16:01:55 dev vnet0 vlan 1 master ovirtmgmt permanent
00:50:56:8e:be:ca dev ens32 vlan 1 master ovirtmgmt permanent
00:50:56:8e:3d:f1 dev ens32 master ovirtmgmt
e4:d3:f1:d1:99:8c dev ens32 master ovirtmgmt
e4:d3:f1:d1:99:8b dev ens32 master ovirtmgmt
76:f6:65:58:fe:f5 dev ovirtmgmt vlan 1 master ovirtmgmt permanent
33:33:00:00:00:01 dev vnet0 self permanent
01:00:5e:00:00:01 dev vnet0 self permanent
33:33:ff:16:01:55 dev vnet0 self permanent


Here is the Forwarding Database(fdb) after setting the Ageing to 0:

01:00:5e:00:00:01 dev ens32 self permanent
33:33:00:00:02:02 dev ens32 self permanent
33:33:00:00:00:01 dev ens32 self permanent
33:33:00:00:00:01 dev bond0 self permanent
fe:4e:66:bf:cf:48 dev ;vdsmdummy; vlan 1 master ;vdsmdummy; permanent
00:50:56:8e:be:ca dev ens32 master ovirtmgmt permanent
fe:1a:4a:16:01:55 dev vnet0 master ovirtmgmt permanent
fe:1a:4a:16:01:55 dev vnet0 

Re: [ovirt-users] Networking issues with oVirt Guest VM.

[Yevgeny Zaspitsky]

Adding mailing list back...

On Mon, Jul 4, 2016 at 3:38 PM, Yevgeny Zaspitsky 
wrote:

> Clint,
>
> Sorry, I missed that you already tried that.
>
> Here are my thoughts (some more shooting in the dark) after reading your
> description again:
>
>- You have quite complicate setup. IIUC, ovirt-engine and its host are
>vSphere VMs. Then, a kind of no-macspoof should be applied from the vSphere
>side. BTW, are both of them on the same vShepre host? Is DHCP server
>another VM on that host?
>- Where/how did you "turn on Port Mirroring"?
>- I'd start the troubleshooting by using tcpdump utility in order to
>pinpoint the component that blocks the traffic.
>- Did you try assigning a static IP instead of DHCP and then check
>connectivity? If that works, then the problem is on the DHCP sever side
>probably.
>- If you do not see any requests in the DHCP server log, then I guess,
>"dhclient -B" wouldn't help.
>- Please turn iptables/firewalld off.
>
>
> Regards,
> Yevgeny
>
> On Sun, Jul 3, 2016 at 9:06 PM, Yevgeny Zaspitsky 
> wrote:
>
>> Hello,
>>
>> IIUC using vdsm macspoof hook would help - reading [1] should help you
>> configuring that.
>>
>> [1] https://github.com/oVirt/vdsm/blob/master/vdsm_hooks/macspoof/README
>>
>> Hope that helps,
>> Yevgeny
>>
>> On Thu, Jun 30, 2016 at 6:11 AM, Clint Smith 
>> wrote:
>>
>>> Hello,
>>>
>>> I have been experimenting with oVirt for the last couple of weeks and I
>>> must say it has a lot of nice features.  I really like it, however, I am
>>> having a heck of a time getting the guest networking all set up correctly.
>>> I am hoping that someone can give me a little guidance in figuring this
>>> out.  I apologize in advance if some of my terminology is off,  I am new.
>>>
>>> Here is a brief intro to my setup:
>>> I created a Centos 7 VM within a vSphere/ESXI environment and then
>>> installed ovirt-engine on it.  I also created another Centos 7 VM and set
>>> it up as a host.  I have configured the Cluster and Host via the oVirt
>>> Administration Portal.  For simplicity, I am using the default ovirtmgmt
>>> network as my only logical network, however I have tried several different
>>> schemes with no luck.  I have a DHCP server and a DNS server that are
>>> siblings to the oVirt host and the engine.  Both the engine and the host
>>> have been upgraded to version 4.0.
>>>
>>> The problem:
>>> My thought was that I would have the guest VMs on the oVirt host use my
>>> existing DHCP server to get their IP addresses, at least at first.  The
>>> problem I am having is that the DHCPACK is not making it back across the
>>> ovirtmgmt bridge and on to the guest.  If I tell dhclient(from the guest)
>>> to force a Broadcast (by using the –B option) on the DHCP server, it will
>>> work.  This is not a solution, just a clue.  Another clue is that ARP
>>> replies from the gateway don’t make it back to the machine, preventing
>>> pings even when I force the IP.  Lastly, If I turn on Port Mirroring,
>>> everything works fine, but it’s my understanding that this is only for
>>> debugging purposes.
>>>
>>> What I have tried (in no particular order):
>>>
>>>- Reading the docs
>>>- Turning on VLAN tagging.
>>>- Installing the mac-spoofing hook, making the configuration changes
>>>to the engine, and then turning it on in the VM config.  I also verified
>>>that the ‘filterref’ tag was removed using virsh.
>>>- Setting up a second logical network on a different subnet, and
>>>connecting it to an additional network interface that I added to the 
>>> host.
>>>On the host, I setup dnsmasq as a DNS and DHCP server.  I got this 
>>> working
>>>up to the point of having the same issues that I was having using the
>>>existing DHCP and DNS servers on the ovirtmgmt network.
>>>- I have tried various changes to iptables as well as the original
>>>settings as well as verified that ebtables is not blocking any traffic.  
>>> I
>>>did configure iptables for logging and noticed it was dropping some 
>>> traffic
>>>related to DHCP, however it seemed like it was DISCOVER or REQUEST 
>>> traffic
>>>due to the IN, OUT, SRC, and DST variables in the log.  I have viewed the
>>>DHCP server logs multiple times and I can see that it is receiving the
>>>DISCOVER and the REQUEST from my guests MAC and sending the OFFER and ACK
>>>consistently.
>>>- Setting SELinux to Permissive
>>>- Setting ip_forward to 1
>>>- Turning STP ON on the bridge
>>>- Changing the bridge delay
>>>- Setting up a dhcrelay using dnsmasq (not sure I implemented this
>>>right though)
>>>
>>> I am really shooting in the dark when it comes to networking because I
>>> am learning a lot of this on the fly.   I feel like I must have a
>>> misconception about how networking should work with oVirt.  Is my entire
>>> approach naïve? Any 

[ovirt-users] Networking issues with oVirt Guest VM.

[Clint Smith]

Hello,

I have been experimenting with oVirt for the last couple of weeks and I must 
say it has a lot of nice features.  I really like it, however, I am having a 
heck of a time getting the guest networking all set up correctly. I am hoping 
that someone can give me a little guidance in figuring this out.  I apologize 
in advance if some of my terminology is off,  I am new.

Here is a brief intro to my setup:
I created a Centos 7 VM within a vSphere/ESXI environment and then installed 
ovirt-engine on it.  I also created another Centos 7 VM and set it up as a 
host.  I have configured the Cluster and Host via the oVirt Administration 
Portal.  For simplicity, I am using the default ovirtmgmt network as my only 
logical network, however I have tried several different schemes with no luck.  
I have a DHCP server and a DNS server that are siblings to the oVirt host and 
the engine.  Both the engine and the host have been upgraded to version 4.0.

The problem:
My thought was that I would have the guest VMs on the oVirt host use my 
existing DHCP server to get their IP addresses, at least at first.  The problem 
I am having is that the DHCPACK is not making it back across the ovirtmgmt 
bridge and on to the guest.  If I tell dhclient(from the guest) to force a 
Broadcast (by using the –B option) on the DHCP server, it will work.  This is 
not a solution, just a clue.  Another clue is that ARP replies from the gateway 
don’t make it back to the machine, preventing pings even when I force the IP.  
Lastly, If I turn on Port Mirroring, everything works fine, but it’s my 
understanding that this is only for debugging purposes.

What I have tried (in no particular order):
Reading the docs
Turning on VLAN tagging.
Installing the mac-spoofing hook, making the configuration changes to the 
engine, and then turning it on in the VM config.  I also verified that the 
‘filterref’ tag was removed using virsh.
Setting up a second logical network on a different subnet, and connecting it to 
an additional network interface that I added to the host.  On the host, I setup 
dnsmasq as a DNS and DHCP server.  I got this working up to the point of having 
the same issues that I was having using the existing DHCP and DNS servers on 
the ovirtmgmt network.
I have tried various changes to iptables as well as the original settings as 
well as verified that ebtables is not blocking any traffic.  I did configure 
iptables for logging and noticed it was dropping some traffic related to DHCP, 
however it seemed like it was DISCOVER or REQUEST traffic due to the IN, OUT, 
SRC, and DST variables in the log.  I have viewed the DHCP server logs multiple 
times and I can see that it is receiving the DISCOVER and the REQUEST from my 
guests MAC and sending the OFFER and ACK consistently.
Setting SELinux to Permissive
Setting ip_forward to 1
Turning STP ON on the bridge
Changing the bridge delay
Setting up a dhcrelay using dnsmasq (not sure I implemented this right though)
I am really shooting in the dark when it comes to networking because I am 
learning a lot of this on the fly.   I feel like I must have a misconception 
about how networking should work with oVirt.  Is my entire approach naïve? Any 
help/guidance that someone could offer would be much appreciated.

Thanks,
Clint




___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users