Hi,
I am currently upgrading my oVirt setup from 4.2.8 to 4.3.3.1.
The setup consists of:
Datacenter/Cluster Default: [fully upgraded to 4.3.3.1]
2 nodes (node04,node05)- NFS storage domain with self hosted engine
Datacenter Luise:
Cluster1: 3 nodes (node01,node02,node03) - Node NG with GlusterFS - Ceph
Cinder storage domain
[Node1 and Node3 are upgraded to 4.3.3.1, Node2 is on 4.2.8]
Cluster2: 1 node (node06) - only Ceph Cinder storage domain [fully upgraded
to 4.3.3.1]
Problems started when upgrading Luise/Cluster1 with GlusterFS:
(I always waited for GlusterFS to be fully synced before proceeding to the next
step)
- Upgrade node01 to 4.3.3 -> OK
- Upgrade node03 to 4.3.3.1 -> OK
- Upgrade node01 to 4.3.3.1 -> GlusterFS became unstable.
I now get the error message:
VDSM node03.infra.solutions.work command ConnectStoragePoolVDS failed: Cannot
find master domain: u'spUUID=f3218bf7-6158-4b2b-b272-51cdc3280376,
msdUUID=02a32017-cbe6-4407-b825-4e558b784157'
And on node03 there is a problem with Gluster:
node03#: ls -l
/rhev/data-center/mnt/glusterSD/node01.infra.solutions.work:_vmstore
ls: cannot access
/rhev/data-center/mnt/glusterSD/node01.infra.solutions.work:_vmstore: Transport
endpoint is not connected
The directory is available on node01 and node02.
The engine is reporting the brick on node03 as down. Node03 and Node06 are
shown as NonOperational, because they are not able to access the gluster
storage domain.
A “gluster peer status” on node1, node2, and node3 shows all peers connected.
“gluster volume heal vmstore info” shows for all nodes:
gluster volume heal vmstore info
Brick node01.infra.solutions.work:/gluster_bricks/vmstore/vmstore
Status: Transport endpoint is not connected
Number of entries: -
Brick node02.infra.solutions.work:/gluster_bricks/vmstore/vmstore
<gfid:0bcb7825-e649-4178-a899-c5cc04c95286>
<gfid:71ec8035-f5a5-4e61-bb34-5ad9db28c0eb>
<gfid:16d5961e-c3bb-4493-a51d-bf83074c4cc7>
/02a32017-cbe6-4407-b825-4e558b784157/dom_md/ids
/.shard/40948f85-2212-47f9-bd5e-102a8dd632b8.66
<gfid:5fe350e4-1eb5-4b6f-a3fb-42c98b7b2f8d>
/.shard/40948f85-2212-47f9-bd5e-102a8dd632b8.60
/02a32017-cbe6-4407-b825-4e558b784157/images/a3a10398-9698-4b73-84d9-9735448e3534/6161e310-4ad6-42d9-8117-5a89c5b2b4b6
<gfid:8eb9fd30-fdb9-442b-9c54-8ba256d7981b>
<gfid:c72001be-e7d3-4b34-bac5-9ab50b609eea>
/.shard/40948f85-2212-47f9-bd5e-102a8dd632b8.96
<gfid:447ec09b-336e-4d2b-8338-f31329ee7a55>
<gfid:9d7db516-d6fb-43d8-a069-dcbc1d72e62a>
/.shard/d66880de-3fa1-4362-8c43-574a173c5f7d.133
<gfid:2412d449-d3ed-40ef-b7eb-d81bdf7c5c05>
<gfid:0fae358b-2cdd-4064-b63c-7f31a35bc35a>
/.shard/40948f85-2212-47f9-bd5e-102a8dd632b8.38
/.shard/40948f85-2212-47f9-bd5e-102a8dd632b8.67
/__DIRECT_IO_TEST__
<gfid:a7945526-9ff3-40fe-b1e2-3921117ef738>
<gfid:e78e9c1f-ce6b-4871-b5bf-9bde34685b99>
/02a32017-cbe6-4407-b825-4e558b784157/images/493188b2-c137-4440-99ee-43a753842a7d/9aa2d139-e3bd-406b-8fe0-b189123eaa73
<gfid:3aed3fb6-044a-4371-9302-e0bd54cbd794>
/.shard/40948f85-2212-47f9-bd5e-102a8dd632b8.64
/.shard/d66880de-3fa1-4362-8c43-574a173c5f7d.132
<gfid:f7631be7-2ab5-4985-904d-69174c0e1267>
<gfid:43001625-1aad-4032-a76e-4cc2a51de2b3>
<gfid:6ae3fe7f-15c9-4103-960c-faba0ba59cb3>
/.shard/40948f85-2212-47f9-bd5e-102a8dd632b8.44
/.shard/40948f85-2212-47f9-bd5e-102a8dd632b8.9
/.shard/40948f85-2212-47f9-bd5e-102a8dd632b8.69
<gfid:f20cc6f7-9391-4260-9238-3e1d0cabbfa3>
/02a32017-cbe6-4407-b825-4e558b784157/images/12e647fb-20aa-4957-b659-05fa75a9215e/f7e4b2a3-ab84-4eb5-a4e7-7208ddad8156
<gfid:c540368a-4431-4405-9a59-e11a217d0ea6>
<gfid:4e698a74-39dc-40a3-ac9c-14456420ab66>
<gfid:afd48e71-ff23-42d7-aef4-b2e2167b75e8>
<gfid:194589b3-0760-4150-80ef-d87376813835>
<gfid:6e17ead1-88cc-4e3e-84fa-7495c4fc3a0e>
/.shard/40948f85-2212-47f9-bd5e-102a8dd632b8.35
/.shard/40948f85-2212-47f9-bd5e-102a8dd632b8.32
<gfid:982d071f-2081-4371-b007-bc48d8167e7c>
<gfid:e2285905-a8da-44f5-8c56-1f8a4d6326a8>
/.shard/40948f85-2212-47f9-bd5e-102a8dd632b8.39
<gfid:956c7d9c-2f96-42d1-bf6e-57bc9e534f84>
<gfid:16162bc7-201a-4842-a41a-af2cc4fb8a9e>
/.shard/40948f85-2212-47f9-bd5e-102a8dd632b8.34
/.shard/40948f85-2212-47f9-bd5e-102a8dd632b8.68
Status: Connected
Number of entries: 47
Brick node03.infra.solutions.work:/gluster_bricks/vmstore/vmstore
/02a32017-cbe6-4407-b825-4e558b784157/images/12e647fb-20aa-4957-b659-05fa75a9215e/f7e4b2a3-ab84-4eb5-a4e7-7208ddad8156
<gfid:c540368a-4431-4405-9a59-e11a217d0ea6>
<gfid:4e698a74-39dc-40a3-ac9c-14456420ab66>
<gfid:afd48e71-ff23-42d7-aef4-b2e2167b75e8>
<gfid:194589b3-0760-4150-80ef-d87376813835>
<gfid:6e17ead1-88cc-4e3e-84fa-7495c4fc3a0e>
<gfid:099284a6-9538-4f9a-928a-d9b704fe0735>
<gfid:75d3c8f7-d67a-49a4-9cd4-7fff202df40d>
<gfid:982d071f-2081-4371-b007-bc48d8167e7c>
<gfid:e2285905-a8da-44f5-8c56-1f8a4d6326a8>
<gfid:447ec09b-336e-4d2b-8338-f31329ee7a55>
<gfid:956c7d9c-2f96-42d1-bf6e-57bc9e534f84>
/.shard/d66880de-3fa1-4362-8c43-574a173c5f7d.133
<gfid:43001625-1aad-4032-a76e-4cc2a51de2b3>
<gfid:6ae3fe7f-15c9-4103-960c-faba0ba59cb3>
<gfid:1a0b2737-9172-4c51-aa77-e93e9671840c>
<gfid:eb471e13-6749-4f62-b1f5-15a44f8990c2>
<gfid:a7945526-9ff3-40fe-b1e2-3921117ef738>
<gfid:e78e9c1f-ce6b-4871-b5bf-9bde34685b99>
/02a32017-cbe6-4407-b825-4e558b784157/images/493188b2-c137-4440-99ee-43a753842a7d/9aa2d139-e3bd-406b-8fe0-b189123eaa73
<gfid:6b418e80-9f61-4d6e-ba77-8a1969d9a99b>
<gfid:914c72d2-e45e-48f2-b7ef-5846b13f7a91>
<gfid:2bd28bdb-1dc6-41d5-96be-c696f452e3f2>
<gfid:9d7db516-d6fb-43d8-a069-dcbc1d72e62a>
<gfid:f7631be7-2ab5-4985-904d-69174c0e1267>
<gfid:16162bc7-201a-4842-a41a-af2cc4fb8a9e>
/.shard/40948f85-2212-47f9-bd5e-102a8dd632b8.44
<gfid:afc6d611-528d-441b-b74e-c5fae6672088>
<gfid:707308e9-e8e5-487a-b0f1-a816720c4243>
<gfid:5f1a81a7-7c42-4226-9142-1b5b35c2b1e9>
<gfid:f20cc6f7-9391-4260-9238-3e1d0cabbfa3>
<gfid:0bcb7825-e649-4178-a899-c5cc04c95286>
<gfid:71ec8035-f5a5-4e61-bb34-5ad9db28c0eb>
<gfid:16d5961e-c3bb-4493-a51d-bf83074c4cc7>
/02a32017-cbe6-4407-b825-4e558b784157/dom_md/ids
<gfid:7661180b-1917-4a7b-9749-5dfb826c4449>
<gfid:5fe350e4-1eb5-4b6f-a3fb-42c98b7b2f8d>
<gfid:a6197593-7e09-4d3f-b538-9cd1ebadd6c9>
/02a32017-cbe6-4407-b825-4e558b784157/images/a3a10398-9698-4b73-84d9-9735448e3534/6161e310-4ad6-42d9-8117-5a89c5b2b4b6
<gfid:8eb9fd30-fdb9-442b-9c54-8ba256d7981b>
<gfid:c72001be-e7d3-4b34-bac5-9ab50b609eea>
<gfid:3aed3fb6-044a-4371-9302-e0bd54cbd794>
/.shard/d66880de-3fa1-4362-8c43-574a173c5f7d.132
<gfid:2412d449-d3ed-40ef-b7eb-d81bdf7c5c05>
<gfid:0fae358b-2cdd-4064-b63c-7f31a35bc35a>
<gfid:c0ca2784-a8af-44b3-9091-a1eaf4c8676f>
/__DIRECT_IO_TEST__
Status: Connected
Number of entries: 47
On Node03 there are several self healing processes, that seem to be doing
nothing.
Oh well.. What now?
Best regards,
- Andreas
_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/R5GS6AQXTEQRMUQNMEBDC72YG3A5JFF6/
