[ovirt-users] Re: Problems with OVN

[Samuli Heinonen]

Thanks Marcin! I set MTU to 1400 and connections seem to work. I haven't 
experienced any disconnects so far.


Is there any other way to set MTU rather than setting it per VM? Ie. 
setting it on oVirt/OVN side.


-samuli

Marcin Mirecki wrote:

Could you try the following:
on the vms, lower the mtu of the vnics connected to the ovn network?
And try again?


On Tue, May 8, 2018 at 11:40 AM, Samuli Heinonen<samp...@neutraali.net>
wrote:


Hi Marcin,

Here is ip addr output from virtual machines:

[root@testi2 ~]# ip addr
1: lo:<LOOPBACK,UP,LOWER_UP>  mtu 65536 qdisc noqueue state UNKNOWN qlen 1
 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
 inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
 inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0:<BROADCAST,MULTICAST,UP,LOWER_UP>  mtu 1500 qdisc pfifo_fast
state UP qlen 1000
 link/ether 00:1a:4a:16:01:05 brd ff:ff:ff:ff:ff:ff
 inet 10.0.1.25/24 brd 10.0.1.255 scope global dynamic eth0
valid_lft 86331sec preferred_lft 86331sec
 inet6 fe80::21a:4aff:fe16:105/64 scope link
valid_lft forever preferred_lft forever
3: eth2:<BROADCAST,MULTICAST,UP,LOWER_UP>  mtu 1500 qdisc pfifo_fast
state UP qlen 1000
 link/ether 00:1a:4a:16:01:03 brd ff:ff:ff:ff:ff:ff
 inet 10.0.200.10/24 brd 10.0.200.255 scope global dynamic eth2
valid_lft 86334sec preferred_lft 86334sec
 inet6 fe80::21a:4aff:fe16:103/64 scope link
valid_lft forever preferred_lft forever

eth0 connected to network ovirtmgmt
eth2 connected to OVN network vm-public

[root@testi6 ~]# ip addr
1: lo:<LOOPBACK,UP,LOWER_UP>  mtu 65536 qdisc noqueue state UNKNOWN qlen 1
 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
 inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
 inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0:<BROADCAST,MULTICAST,UP,LOWER_UP>  mtu 1500 qdisc pfifo_fast
state UP qlen 1000
 link/ether 00:1a:4a:16:01:0b brd ff:ff:ff:ff:ff:ff
 inet 10.0.1.27/24 brd 10.0.1.255 scope global dynamic eth0
valid_lft 86187sec preferred_lft 86187sec
 inet6 fe80::21a:4aff:fe16:10b/64 scope link
valid_lft forever preferred_lft forever
3: eth1:<BROADCAST,MULTICAST,UP,LOWER_UP>  mtu 1500 qdisc pfifo_fast
state UP qlen 1000
 link/ether 00:1a:4a:16:01:0c brd ff:ff:ff:ff:ff:ff
 inet 10.0.200.11/24 brd 10.0.200.255 scope global dynamic eth1
valid_lft 86301sec preferred_lft 86301sec
 inet6 fe80::21a:4aff:fe16:10c/64 scope link
valid_lft forever preferred_lft forever

eth0 connected to network ovirtmgmt
eth1 connected to OVN network vm-public

Best regards,
Samuli



Marcin Mirecki kirjoitti 08.05.2018 10:14:


Hi Samuli,

Your configuration looks correct.
Can you also send me the result of 'ip addr' on your vm's?

Thanks,
Marcin

On Mon, May 7, 2018 at 7:44 PM, Samuli Heinonen
<samp...@neutraali.net>  wrote:

Hi Marcin,

Thank you for your response.

I used engine-setup to do the configuration. Only exception is that
I had to run "vdsm-tool ovn-config engine-ip local-ip" (ie.
vdsm-tool ovn-config 10.0.1.101 10.0.1.21) on hypervisors.

Here is the output of requested commands:

[root@oe ~]# ovn-sbctl show
Chassis "049183d5-61b6-4b9c-bae3-c7b10d30f8cb"
hostname: "o2.hirundinidae.local"
Encap geneve
ip: "10.0.1.18"
options: {csum="true"}
Port_Binding "87c5e44a-7c8b-41b2-89a6-fa52f27643ed"
Chassis "972f1b7b-10de-4e4f-a5f9-f080890f087d"
hostname: "o3.hirundinidae.local"
Encap geneve
ip: "10.0.1.21"
options: {csum="true"}
Port_Binding "ccea5185-3efa-4d9c-9475-9e46009fea4f"
Port_Binding "e868219c-f16c-45c6-b7b1-72d044fee602"

[root@oe ~]# ovn-nbctl show
switch 7d264a6c-ea48-4a6d-9663-5244102dc9bb (vm-private)
port 4ec3ecf6-d04a-406c-8354-c5e195ffde05
addresses: ["00:1a:4a:16:01:06 dynamic"]
switch 40aedb7d-b1c3-400e-9ddb-16bee3bb312a (vm-public)
port 87c5e44a-7c8b-41b2-89a6-fa52f27643ed
addresses: ["00:1a:4a:16:01:03"]
port ccea5185-3efa-4d9c-9475-9e46009fea4f
addresses: ["00:1a:4a:16:01:0c"]
port e868219c-f16c-45c6-b7b1-72d044fee602
addresses: ["00:1a:4a:16:01:0a"]

[root@o2 ~]# ip addr
1: lo:<LOOPBACK,UP,LOWER_UP>  mtu 65536 qdisc noqueue state UNKNOWN
qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 [1] scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp0s31f6:<BROADCAST,MULTICAST,UP,LOWER_UP>  mtu 1500 qdisc
pfifo_fast master ovirtmgmt state UP qlen 1000
link/ether 78:f2:9e:90:bc:64 brd ff:ff:ff:ff:ff:ff
3: enp0s20f0u5c2:<BROADCAST,MULTICAST,UP,LOWER_UP>  mtu 1500 qdisc
pfifo_fast master public state UNKNOWN qlen 1000
link/ether 50:3e:aa:4c:9b:01 brd

[ovirt-users] Re: Problems with OVN

[Samuli Heinonen]

Hi Marcin,

Here is ip addr output from virtual machines:

[root@testi2 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 
1

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
   valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
   valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast 
state UP qlen 1000

link/ether 00:1a:4a:16:01:05 brd ff:ff:ff:ff:ff:ff
inet 10.0.1.25/24 brd 10.0.1.255 scope global dynamic eth0
   valid_lft 86331sec preferred_lft 86331sec
inet6 fe80::21a:4aff:fe16:105/64 scope link
   valid_lft forever preferred_lft forever
3: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast 
state UP qlen 1000

link/ether 00:1a:4a:16:01:03 brd ff:ff:ff:ff:ff:ff
inet 10.0.200.10/24 brd 10.0.200.255 scope global dynamic eth2
   valid_lft 86334sec preferred_lft 86334sec
inet6 fe80::21a:4aff:fe16:103/64 scope link
   valid_lft forever preferred_lft forever

eth0 connected to network ovirtmgmt
eth2 connected to OVN network vm-public

[root@testi6 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 
1

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
   valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
   valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast 
state UP qlen 1000

link/ether 00:1a:4a:16:01:0b brd ff:ff:ff:ff:ff:ff
inet 10.0.1.27/24 brd 10.0.1.255 scope global dynamic eth0
   valid_lft 86187sec preferred_lft 86187sec
inet6 fe80::21a:4aff:fe16:10b/64 scope link
   valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast 
state UP qlen 1000

link/ether 00:1a:4a:16:01:0c brd ff:ff:ff:ff:ff:ff
inet 10.0.200.11/24 brd 10.0.200.255 scope global dynamic eth1
   valid_lft 86301sec preferred_lft 86301sec
inet6 fe80::21a:4aff:fe16:10c/64 scope link
   valid_lft forever preferred_lft forever

eth0 connected to network ovirtmgmt
eth1 connected to OVN network vm-public

Best regards,
Samuli


Marcin Mirecki kirjoitti 08.05.2018 10:14:

Hi Samuli,

Your configuration looks correct.
Can you also send me the result of 'ip addr' on your vm's?

Thanks,
Marcin

On Mon, May 7, 2018 at 7:44 PM, Samuli Heinonen
<samp...@neutraali.net> wrote:


Hi Marcin,

Thank you for your response.

I used engine-setup to do the configuration. Only exception is that
I had to run "vdsm-tool ovn-config engine-ip local-ip" (ie.
vdsm-tool ovn-config 10.0.1.101 10.0.1.21) on hypervisors.

Here is the output of requested commands:

[root@oe ~]# ovn-sbctl show
Chassis "049183d5-61b6-4b9c-bae3-c7b10d30f8cb"
hostname: "o2.hirundinidae.local"
Encap geneve
ip: "10.0.1.18"
options: {csum="true"}
Port_Binding "87c5e44a-7c8b-41b2-89a6-fa52f27643ed"
Chassis "972f1b7b-10de-4e4f-a5f9-f080890f087d"
hostname: "o3.hirundinidae.local"
Encap geneve
ip: "10.0.1.21"
options: {csum="true"}
Port_Binding "ccea5185-3efa-4d9c-9475-9e46009fea4f"
Port_Binding "e868219c-f16c-45c6-b7b1-72d044fee602"

[root@oe ~]# ovn-nbctl show
switch 7d264a6c-ea48-4a6d-9663-5244102dc9bb (vm-private)
port 4ec3ecf6-d04a-406c-8354-c5e195ffde05
addresses: ["00:1a:4a:16:01:06 dynamic"]
switch 40aedb7d-b1c3-400e-9ddb-16bee3bb312a (vm-public)
port 87c5e44a-7c8b-41b2-89a6-fa52f27643ed
addresses: ["00:1a:4a:16:01:03"]
port ccea5185-3efa-4d9c-9475-9e46009fea4f
addresses: ["00:1a:4a:16:01:0c"]
port e868219c-f16c-45c6-b7b1-72d044fee602
addresses: ["00:1a:4a:16:01:0a"]

[root@o2 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 [1] scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp0s31f6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
pfifo_fast master ovirtmgmt state UP qlen 1000
link/ether 78:f2:9e:90:bc:64 brd ff:ff:ff:ff:ff:ff
3: enp0s20f0u5c2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
pfifo_fast master public state UNKNOWN qlen 1000
link/ether 50:3e:aa:4c:9b:01 brd ff:ff:ff:ff:ff:ff
4: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
qlen 1000
link/ether 82:49:e1:15:af:56 brd ff:ff:ff:ff:ff:ff
5: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen
1000
link/ether a2:bb:78:7e:35:4b brd ff:ff:ff:ff:ff:ff
21: public: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
state UP qlen 1000
link/ether 50:3e:aa:4c:9b:01 brd ff:ff:ff:ff:ff:ff
inet6 fe80::523e:aaff:fe4c:9b01/

Re: [ovirt-users] Problems with OVN

[Samuli Heinonen]

,UP,LOWER_UP> mtu 1500 qdisc 
pfifo_fast master public state UNKNOWN qlen 1000

link/ether 50:3e:aa:4c:9c:03 brd ff:ff:ff:ff:ff:ff
4: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 
1000

link/ether 7e:43:c1:b0:48:73 brd ff:ff:ff:ff:ff:ff
5: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 3a:fe:68:34:31:4c brd ff:ff:ff:ff:ff:ff
21: public: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue 
state UP qlen 1000

link/ether 50:3e:aa:4c:9c:03 brd ff:ff:ff:ff:ff:ff
inet6 fe80::523e:aaff:fe4c:9c03/64 scope link
   valid_lft forever preferred_lft forever
22: ovirtmgmt: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue 
state UP qlen 1000

link/ether 78:f2:9e:90:bc:50 brd ff:ff:ff:ff:ff:ff
inet 10.0.1.21/24 brd 10.0.1.255 scope global ovirtmgmt
   valid_lft forever preferred_lft forever
inet6 fe80::7af2:9eff:fe90:bc50/64 scope link
   valid_lft forever preferred_lft forever
24: ;vdsmdummy;: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
qlen 1000

link/ether 02:92:3f:89:f2:c7 brd ff:ff:ff:ff:ff:ff
25: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast 
master ovirtmgmt state UNKNOWN qlen 1000

link/ether fe:16:3e:0b:b1:2d brd ff:ff:ff:ff:ff:ff
inet6 fe80::fc16:3eff:fe0b:b12d/64 scope link
   valid_lft forever preferred_lft forever
27: vnet2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast 
master ovirtmgmt state UNKNOWN qlen 1000

link/ether fe:1a:4a:16:01:0b brd ff:ff:ff:ff:ff:ff
inet6 fe80::fc1a:4aff:fe16:10b/64 scope link
   valid_lft forever preferred_lft forever
29: vnet4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast 
master ovs-system state UNKNOWN qlen 1000

link/ether fe:1a:4a:16:01:0c brd ff:ff:ff:ff:ff:ff
inet6 fe80::fc1a:4aff:fe16:10c/64 scope link
   valid_lft forever preferred_lft forever
31: vnet6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast 
master ovirtmgmt state UNKNOWN qlen 1000

link/ether fe:1a:4a:16:01:07 brd ff:ff:ff:ff:ff:ff
inet6 fe80::fc1a:4aff:fe16:107/64 scope link
   valid_lft forever preferred_lft forever
32: vnet7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast 
master public state UNKNOWN qlen 1000

link/ether fe:1a:4a:16:01:09 brd ff:ff:ff:ff:ff:ff
inet6 fe80::fc1a:4aff:fe16:109/64 scope link
   valid_lft forever preferred_lft forever
33: vnet8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast 
master ovs-system state UNKNOWN qlen 1000

link/ether fe:1a:4a:16:01:0a brd ff:ff:ff:ff:ff:ff
inet6 fe80::fc1a:4aff:fe16:10a/64 scope link
   valid_lft forever preferred_lft forever
34: genev_sys_6081: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc 
noqueue master ovs-system state UNKNOWN qlen 1000

link/ether 46:88:1c:22:6f:c3 brd ff:ff:ff:ff:ff:ff
inet6 fe80::4488:1cff:fe22:6fc3/64 scope link
   valid_lft forever preferred_lft forever

[root@o3 ~]# ovs-vsctl show
8c2c19fc-d9e4-423d-afcb-f5ecff602ca7
Bridge br-int
fail_mode: secure
Port "vnet4"
Interface "vnet4"
Port "ovn-049183-0"
Interface "ovn-049183-0"
type: geneve
options: {csum="true", key=flow, remote_ip="10.0.1.18"}
Port "vnet8"
Interface "vnet8"
Port br-int
Interface br-int
type: internal
ovs_version: "2.9.0"

Best regards,
Samuli


Marcin Mirecki wrote:

Hi Samuli,

Let's first make sure the configuration is correct.
How did you configure the env? Did you use the automatic engine-setup
configuration?

Can you please send me the output of the following:

on engine:
   ovn-sbctl show
   ovn-nbctl show

on hosts:
   ip addr
   ovs-vsctl show

The 'vdsm-tool ovn-config' command configures the ovn controller to use the
first ip as the ovn central, and the local tunnel to use the second one.

Regards,
Marcin


On Sun, May 6, 2018 at 10:42 AM, Samuli Heinonen<samp...@neutraali.net>
wrote:


Hi all,

I'm building a home lab using oVirt+GlusterFS in hyperconverged(ish) setup.

My setup consists of 2x nodes with ASRock H110M-STX motherboard, Intel
Pentium G4560 3,5 GHz CPU and 16 GB RAM. Motherboard has integrated Intel
Gigabit I219V LAN. At the moment I'm using RaspberryPi as Gluster arbiter
node. Nodes are connected to basic "desktop switch" without any management
available.

Hardware is nowhere near perfect, but it get its job done and is enough
for playing around. However I'm having problems getting OVN to work
properly and I'm clueless where to look next.

oVirt is setup like this:
oVirt engine host oe / 10.0.1.101
oVirt hypervisor host o2 / 10.0.1.18
oVirt hypervisor host o3 / 10.0.1.21
OVN network 10.0.200.0/24

When I spin up 

[ovirt-users] Problems with OVN

[Samuli Heinonen]

Hi all,

I'm building a home lab using oVirt+GlusterFS in hyperconverged(ish) setup.

My setup consists of 2x nodes with ASRock H110M-STX motherboard, Intel 
Pentium G4560 3,5 GHz CPU and 16 GB RAM. Motherboard has integrated 
Intel Gigabit I219V LAN. At the moment I'm using RaspberryPi as Gluster 
arbiter node. Nodes are connected to basic "desktop switch" without any 
management available.


Hardware is nowhere near perfect, but it get its job done and is enough 
for playing around. However I'm having problems getting OVN to work 
properly and I'm clueless where to look next.


oVirt is setup like this:
oVirt engine host oe / 10.0.1.101
oVirt hypervisor host o2 / 10.0.1.18
oVirt hypervisor host o3 / 10.0.1.21
OVN network 10.0.200.0/24

When I spin up a VM in o2 and o3 with IP address in network 10.0.1.0/24 
everything works fine. VMs can interact between each other without any 
problems.


Problems show up when I try to use OVN based network between virtual 
machines. If virtual machines are on same hypervisor then everything 
seems to work ok. But if I have virtual machine on hypervisor o2 and 
another one on hypervisor o3 then TCP connections doesn't work very 
well. UDP seems to be ok and it's possible to ping hosts, do dns & ntp 
queries and so on.


Problem with TCP is that for example when taking SSH connection to 
another host at some point connection just hangs and most of the time 
it's not even possible to even log in before connectiong hangs. If I 
look into tcpdump at that point it looks like packets never reach 
destination. Also, if I have multiple connections, then all of them hang 
at the same time.


I have tried switching off tx checksum and other similar settings, but 
it didn't make any difference.


I'm suspecting that hardware is not good enough. Before investigating 
into new hardware I'd like to get some confirmation that everything is 
setup correctly.


When setting up oVirt/OVN I had to run following undocumented command to 
get it working at all: vdsm-tool ovn-config 10.0.1.101 10.0.1.21 (oVirt 
engine IP, hypervisor IP). Especially this makes me think that I have 
missed some crucial part in configuration.


On oVirt engine in /var/log/openvswitch/ovsdb-server-nb.log there are 
error messages:
2018-05-06T08:30:05.418Z|00913|stream_ssl|WARN|SSL_read: unexpected SSL 
connection close
2018-05-06T08:30:05.418Z|00914|jsonrpc|WARN|ssl:127.0.0.1:53152: receive 
error: Protocol error
2018-05-06T08:30:05.419Z|00915|reconnect|WARN|ssl:127.0.0.1:53152: 
connection dropped (Protocol error)


To be honest, I'm not sure what's causing those error messages or are 
they related. I found out some bug reports stating that they are not 
critical.


Any ideas what to do next or should I just get better hardware? :)

Best regards,
Samuli Heinonen
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt 3.6.6 and gluster 3.7.13

[Samuli Heinonen]

Hi,

> On 25 Jul 2016, at 12:34, David Gossage  wrote:
> 
> On Mon, Jul 25, 2016 at 1:01 AM, Krutika Dhananjay  
> wrote:
> Hi,
> 
> Thanks for the logs. So I have identified one issue from the logs for which 
> the fix is this: http://review.gluster.org/#/c/14669/. Because of a bug in 
> the code, ENOENT was getting converted to EPERM and being propagated up the 
> stack causing the reads to bail out early with 'Operation not permitted' 
> errors.
> I still need to find out two things:
> i) why there was a readv() sent on a non-existent (ENOENT) file (this is 
> important since some of the other users have not faced or reported this issue 
> on gluster-users with 3.7.13)
> ii) need to see if there's a way to work around this issue.
> 
> Do you mind sharing the steps needed to be executed to run into this issue? 
> This is so that we can apply our patches, test and ensure they fix the 
> problem.


Unfortunately I can’t test this right away nor give exact steps how to test 
this. This is just a theory but please correct me if you see some mistakes.

oVirt uses cache=none settings for VM’s by default which requires direct I/O. 
oVirt also uses dd with iflag=direct to check that storage has direct I/O 
enabled. Problems exist with GlusterFS with sharding enabled and bricks running 
on ZFS on Linux. Everything seems to be fine with GlusterFS 3.7.11 and problems 
exist at least with version .12 and .13. There has been some posts saying that 
GlusterFS 3.8.x is also affected.

Steps to reproduce:
1. Sharded file is created with GlusterFS 3.7.11. Everything works ok.
2. GlusterFS is upgraded to 3.7.12+
3. Sharded file cannot be read or written with direct I/O enabled. (Ie. oVirt 
uses to check storage connection with command "dd 
if=/rhev/data-center/0001-0001-0001-0001-02b6/mastersd/dom_md/inbox 
iflag=direct,fullblock count=1 bs=1024000”)

Please let me know if you need more information.

-samuli

> Well after upgrade of gluster all I did was start ovirt hosts up which 
> launched and started their ha-agent and broker processes.  I don't believe I 
> started getting any errors till it mounted GLUSTER1.  I had enabled sharding 
> but had no sharded disk images yet.  Not sure if the check for shards would 
> have caused that.  Unfortunately I can't just update this cluster and try and 
> see what caused it as it has sme VM's users expect to be available in few 
> hours.
> 
> I can see if I can get my test setup to recreate it.  I think I'll need to 
> de-activate data center so I can detach the storage thats on xfs and attach 
> the one thats over zfs with sharding enabled.  My test is 3 bricks on same 
> local machine, with 3 different volumes but I think im running into sanlock 
> issue or something as it won't mount more than one volume that was created 
> locally.
> 
> 
> -Krutika
> 
> On Fri, Jul 22, 2016 at 7:17 PM, David Gossage  
> wrote:
> Trimmed out the logs to just about when I was shutting down ovirt servers for 
> updates which was 14:30 UTC 2016-07-09
> 
> Pre-update settings were 
> 
> Volume Name: GLUSTER1
> Type: Replicate
> Volume ID: 167b8e57-28c3-447a-95cc-8410cbdf3f7f
> Status: Started
> Number of Bricks: 1 x 3 = 3
> Transport-type: tcp
> Bricks:
> Brick1: ccgl1.gl.local:/gluster1/BRICK1/1
> Brick2: ccgl2.gl.local:/gluster1/BRICK1/1
> Brick3: ccgl3.gl.local:/gluster1/BRICK1/1
> Options Reconfigured:
> performance.readdir-ahead: on
> storage.owner-uid: 36
> storage.owner-gid: 36
> performance.quick-read: off
> performance.read-ahead: off
> performance.io-cache: off
> performance.stat-prefetch: off
> cluster.eager-lock: enable
> network.remote-dio: enable
> cluster.quorum-type: auto
> cluster.server-quorum-type: server
> server.allow-insecure: on
> cluster.self-heal-window-size: 1024
> cluster.background-self-heal-count: 16
> performance.strict-write-ordering: off
> nfs.disable: on
> nfs.addr-namelookup: off
> nfs.enable-ino32: off
> 
> At the time of updates ccgl3 was offline from bad nic on server but had been 
> so for about a week with no issues in volume
> 
> Shortly after update I added these settings to enable sharding but did not as 
> of yet have any VM images sharded.
> features.shard-block-size: 64MB
> features.shard: on
> 
> 
> 
> 
> David Gossage
> Carousel Checks Inc. | System Administrator
> Office 708.613.2284
> 
> On Fri, Jul 22, 2016 at 5:00 AM, Krutika Dhananjay  
> wrote:
> Hi David,
> 
> Could you also share the brick logs from the affected volume? They're located 
> at /var/log/glusterfs/bricks/.log.
> 
> Also, could you share the volume configuration (output of `gluster volume 
> info `) for the affected volume(s) AND at the time you actually saw this 
> issue?
> 
> -Krutika
> 
> 
> 
> 
> On Thu, Jul 21, 2016 at 11:23 PM, David Gossage  
> wrote:
> On Thu, Jul 21, 2016 at 11:47 AM, Scott  wrote:
> Hi David,
> 

Re: [ovirt-users] Centos 7 no bootable device

[Samuli Heinonen]

Hi Johan,

I had similar problems with some VM’s when moving VM’s to oVirt 3.6 from older 
versions. Iirc it was unable to find the disk device at all. The weird pard is 
that when I booted VM with recovery cd everything was ok. I was able to boot 
VM’s when I enabled boot menu from VM options (Boot options - Enable boot 
menu). Can you try if that helps in your case?

Best regards,
Samuli Heinonen

 
> On 20 Jul 2016, at 15:12, Johan Kooijman <m...@johankooijman.com> wrote:
> 
> Hi all,
> 
> Situation as follows: mixed cluster with 3.5 and 3.6 nodes. Now in the 
> process of reinstalling the 3.5 nodes with 3.6 on CentOS 7.2. I can't live 
> migrate VM's while they're running on different versions. That's odd, but not 
> the weirdest issue.
> 
> The most interesting part is happening when I power down a VM, and then run 
> it on a 3.6 node. Only on CentOS 7 VM's, I'm getting a "no bootable device" 
> error. I have a mixed setup of ubuntu, CentOS 6 and CentOS 7. Ubuntu & CentOS 
> 6 are fine. 
> 
> Tried shooting grub in MBR again, to no effect. I start the VM then on a 3.5 
> node and all is fine.
> 
> -- 
> Met vriendelijke groeten / With kind regards,
> Johan Kooijman
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] QEMU GlusterFS support in oVirt

[Samuli Heinonen]

> On 12 Mar 2016, at 17:04, Nir Soffer <nsof...@redhat.com> wrote:
> 
> On Sat, Mar 12, 2016 at 1:55 PM, Samuli Heinonen <samp...@neutraali.net> 
> wrote:
>> Hello all,
>> 
>> It seems that oVirt 3.6 is still using FUSE to access GlusterFS storage 
>> domains instead of using QEMU driver (libgfapi). As far as I know libgfapi 
>> support should be available in Libvirt and QEMU packages provided in CentOS 
>> 7.
> 
> We started to work this during 3.6 development, but the work was
> suspended because
> libvirt and qemu do not support multiple gluster servers [1]. This
> means that if your single
> server is down, you will not be able to connect to glister.

Since this is only used to fetch volume information when connecting to Gluster 
volume I don’t think it should be treated as blocking issue. If we lose one 
storage server that’s a problem we have to fix as soon as possible anyway. Even 
then hypervisors are already connected to other storage servers and there is no 
need to fetch volume information again. Also there are other ways to work 
around this like having a separate hostname that is used to fetch volume 
information and which can then be pointed to server that’s up.

It would be great if libgfapi could be available even as selectable option so 
it could be tested in real world situation.

> Recently Niels suggested that we use DNS for this purpose - if the DNS
> return multiple
> servers, libgafpi should be able to failover to one of these servers,
> so connecting with
> single server address should good as multiple server support in libvirt or 
> qemu.
> 
> The changes needed to support this are not big as you can see in [2],
> [3]. However the work
> was not completed and I don't know if it will completed for 4.0.

Thank you for these links. I’ll see if this is something we can try out in our 
test environment

Best regards,
Samuli

> 
>> Is there any workarounds how to use libgfapi with oVirt before it’s 
>> officially available?
> 
> I don't know about any.
> 
> [1] https://bugzilla.redhat.com/1247521
> [2] https://gerrit.ovirt.org/44061
> [3] https://gerrit.ovirt.org/33768
> 
> Nir

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

[ovirt-users] QEMU GlusterFS support in oVirt

[Samuli Heinonen]

Hello all,

It seems that oVirt 3.6 is still using FUSE to access GlusterFS storage domains 
instead of using QEMU driver (libgfapi). As far as I know libgfapi support 
should be available in Libvirt and QEMU packages provided in CentOS 7. Is there 
any workarounds how to use libgfapi with oVirt before it’s officially available?

Best regards,
Samuli Heinonen
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [Users] Very bad write performance in VM (ovirt 3.3.3)

[Samuli Heinonen]

Hello,

What version of GlusterFS you are using?

ml ml mliebher...@googlemail.com kirjoitti 8.2.2014 kello 21.24:

 anyone?
 
 On Friday, February 7, 2014, ml ml mliebher...@googlemail.com wrote:
 Hello List,
 
 i set up a Cluster with 2 Nodes and Glusterfs.
 
 
 gluster volume info all
  
 Volume Name: Repl2
 Type: Replicate
 Volume ID: 8af9b282-8b60-4d71-a0fd-9116b8fdcca7
 Status: Started
 Number of Bricks: 1 x 2 = 2
 Transport-type: tcp
 Bricks:
 Brick1: node1.local:/data
 Brick2: node2.local:/data
 Options Reconfigured:
 auth.allow: *
 user.cifs: enable
 nfs.disable: off
 
 
 I turned node2 off. Just to make sure i have not network bottle neck and that 
 it will not replicate for my first benchmarks.
 
 
 My first test with bonnie on my local raw disk of node1 gave me 130MB/sec 
 write speed.
 Then i did the same test on my cluster dir /data: 130MB/sec
 Then i did the write test in a freshly installed Debian 7 vm: 10MB/sec 
 
 This is terrible and i wonder why?!
 
 My tests where made with:
 bonnie++ -u root -s double mem) -d dir
 
 Here are my bonnie results: http://oi62.tinypic.com/20aara0.jpg
 
 
 Since node2 is turned off, this cant be a network bottle neck.
 
 Any ideas? 
 
 Thanks,
 Mario
 ___
 Users mailing list
 Users@ovirt.org
 http://lists.ovirt.org/mailman/listinfo/users

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [Users] Keepalived on oVirt Hosts has engine networking issues

[Samuli Heinonen]

Andrew Lau and...@andrewklau.com kirjoitti 1.12.2013 kello 4.55:

 p.s. can anyone also confirm, does gluster support multi pathing by default? 
 If I'm using this keepalived method, am I bottle necking myself to one host?

If you are using native GlusterFS, either FUSE or libgfapi, then client only 
fetches volume information from server with floating IP. Client connects 
directly to servers defined in volume specification. If you are running Gluster 
daemon on every oVirt node, then you could also use localhost:/glusterVol to 
mount your GlusterFS volume. Of course this is not good way to go if you ever 
plan to add a node without Gluster daemon.

So, the answer is: you are not bottlenecking to one host if you are using 
native GlusterFS. With NFS it's connecting to one host only.

-samuli

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [Users] Problem adding an RHEL node/host

[Samuli Heinonen]

Hey!

Bob Doolittle b...@doolittle.us.com kirjoitti 8.11.2013 kello 21.21:

 
 The install fails, and when I look at the host-deploy log it appears to be 
 due to a failure for Yum to find a dependency.
 At Matt Curry's suggestion, I added the 'epel' repository (not mentioned in 
 QS Guide), which I think resolved some dependencies, but I still fail as 
 follows:
 
 2013-11-08 13:48:49 ERROR otopi.context context._executeMethod:146 Failed to 
 execute stage 'Package installation': [u'vdsm-4.12.1-4.el6.x86_64 requires 
 sanlock-python', u'vdsm-4.12.1-4.el6.x86_64 requires libvirt-lock-sanlock', 
 u'vdsm-4.12.1-4.el6.x86_64 requires sanlock = 2.3-4']
 
 
 As far as I can tell by googling around, sanlock belongs to the ovirt 
 project. So where should it be coming from?
 Is something wrong with ovirt-stable? I am using:

libvirt-lock-sanlock packages should be in RHEL Server Optional channel.

-samuili
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [Users] oVirt 3.3.1, RHEL 6.5 (beta) and Gluster SD

[Samuli Heinonen]

Dan Kenigsberg dan...@redhat.com kirjoitti 30.10.2013 kello 19.08:

 On Sat, Oct 19, 2013 at 09:38:41AM +0300, Samuli Heinonen wrote:
 Dear all,
 
 i'm testing QEMU-Gluster features with oVirt 3.3.1 beta and RHEL 6.5 beta.  
 For reason or another it seems to bypass Gluster support in QEMU and use VM 
 image through FUSE mount point instead.
 
 ie. Libvirt XML shown in vdsm.log looks like this:
 disk device=disk snapshot=no type=file
address bus=0x00 domain=0x function=0x0 
 slot=0x06 type=pci/
  source 
 file=/rhev/data-center/mnt/glusterSD/boar1.storage:dev-el6-sata1/402a2eee-40f6-4012-b188-63ec122ce399/images/f6ba1575-e6de-428c-93ef-8a27cf23de61/bc3de195-ccc4-4202-ade2-7e4d0cdb292e/
target bus=virtio dev=vda/
serialf6ba1575-e6de-428c-93ef-8a27cf23de61/serial
boot order=1/
driver cache=none error_policy=stop io=threads 
 name=qemu type=raw/
 /disk
 
 and also qemu-kvm process for this VM has option -drive 
 file=/rhev/data-center/mnt/glusterSD/boar1.storage:dev-el6-sata1/402a2eee-40f6-4012-b188-63ec122ce399/images/f6ba1575-e6de-428c-93ef-8a27cf23de61/bc3de195-ccc4-4202-ade2-7e4d0cdb292e,if=none,id=drive-virtio-disk0,format=raw,serial=f6ba1575-e6de-428c-93ef-8a27cf23de61,cache=none,werror=stop,rerror=stop,aio=threads
 
 oVirt 3.3 and Fedora 19 works as intended.  Any ideas what might be wrong?
 
 Steps to reproduce (hopefully?):
 1: Clean install of ovirt-engine 3.3.1 beta on CentOS 6.4
 2: Clean install of RHEL 6.5 beta with EPEL, Gluster-epel and oVirt repos 
 enabled (and all necessary Red Hat channels.)
 3: Install RHEL 6.5 host to oVirt and let it install necessary packages
 4: Switch datacenter storage type from NFS - GlusterFS
 5: Insert storage domain
 6: Start installing virtual machines
 
 Please let me know if more information is needed or you want me to open a 
 bug report for this.
 
 As we've learned during today's ovirt weekly meeting, this is probably a
 manifestation of
 
Bug 1022961 - Running a VM from a gluster domain uses mount instead
of gluster URI

Please see attachment if you still need vdsm.log during VM startup.



vdsm.log.gz
Description: GNU Zip compressed data
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

[Users] oVirt 3.3.1, RHEL 6.5 (beta) and Gluster SD

[Samuli Heinonen]

Dear all,

i'm testing QEMU-Gluster features with oVirt 3.3.1 beta and RHEL 6.5 beta.  For 
reason or another it seems to bypass Gluster support in QEMU and use VM image 
through FUSE mount point instead.

ie. Libvirt XML shown in vdsm.log looks like this:
disk device=disk snapshot=no type=file
address bus=0x00 domain=0x function=0x0 
slot=0x06 type=pci/
source 
file=/rhev/data-center/mnt/glusterSD/boar1.storage:dev-el6-sata1/402a2eee-40f6-4012-b188-63ec122ce399/images/f6ba1575-e6de-428c-93ef-8a27cf23de61/bc3de195-ccc4-4202-ade2-7e4d0cdb292e/
target bus=virtio dev=vda/
serialf6ba1575-e6de-428c-93ef-8a27cf23de61/serial
boot order=1/
driver cache=none error_policy=stop io=threads 
name=qemu type=raw/
 /disk

and also qemu-kvm process for this VM has option -drive 
file=/rhev/data-center/mnt/glusterSD/boar1.storage:dev-el6-sata1/402a2eee-40f6-4012-b188-63ec122ce399/images/f6ba1575-e6de-428c-93ef-8a27cf23de61/bc3de195-ccc4-4202-ade2-7e4d0cdb292e,if=none,id=drive-virtio-disk0,format=raw,serial=f6ba1575-e6de-428c-93ef-8a27cf23de61,cache=none,werror=stop,rerror=stop,aio=threads

oVirt 3.3 and Fedora 19 works as intended.  Any ideas what might be wrong?

Steps to reproduce (hopefully?):
1: Clean install of ovirt-engine 3.3.1 beta on CentOS 6.4
2: Clean install of RHEL 6.5 beta with EPEL, Gluster-epel and oVirt repos 
enabled (and all necessary Red Hat channels.)
3: Install RHEL 6.5 host to oVirt and let it install necessary packages
4: Switch datacenter storage type from NFS - GlusterFS
5: Insert storage domain
6: Start installing virtual machines

Please let me know if more information is needed or you want me to open a bug 
report for this.

-samuli

Cheers,
Samuli Heinonen




___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users