Re: [ovirt-users] Fwd: Re: ovirt - can't attach master domain II
used replica2 with volume option
Volume Name: 2KVM12-P2
Type: Replicate
Volume ID: 9745551f-4696-4a6c-820a-619e359a61fd
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: 16.0.0.164:/STORAGES/g1r5p2/GFS
Brick2: 16.0.0.163:/STORAGES/g1r5p2/GFS
Options Reconfigured:
storage.owner-uid: 36
storage.owner-gid: 36
performance.io-cache: off
performance.read-ahead: off
network.remote-dio: enable
cluster.eager-lock: enable
performance.stat-prefetch: off
performance.quick-read: off
cluster.quorum-count: 1
cluster.server-quorum-type: none
cluster.quorum-type: fixed
was runnig over year with no problems ( reboots, ..etc... )
On 24.2.2016 12:34, Ravishankar N wrote:
On 02/24/2016 04:48 PM, p...@email.cz wrote:
prereq: 2KVM12-P2 = master domain
-
YES - I'm using gluster.fuse NFS
localhost:/2KVM12-P2 on
/rhev/data-center/mnt/glusterSD/localhost:_2KVM12-P2 type
fuse.glusterfs
(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
-
Healing
==
# gluster volume heal 2KVM12-P2 info
Brick 16.0.0.164:/STORAGES/g1r5p2/GFS
Number of entries: 0
Brick 16.0.0.163:/STORAGES/g1r5p2/GFS
Number of entries: 0
# while true; do for vol in `gluster volume list`; do gluster volume
heal $vol info | sort | grep "Number of entries" | awk -F: '{tot+=$2}
END { printf("Heal entries for '"$vol"': %d\n", $tot);}'; done; sleep
120; echo -e "\n==\n"; done
Heal entries for 1KVM12-BCK: 1
Heal entries for 1KVM12-P1: 1
Heal entries for 1KVM12-P2: 0
Heal entries for 1KVM12-P3: 0
Heal entries for 1KVM12-P4: 0
Heal entries for 1KVM12-P5: 0
Heal entries for 2KVM12-P1: 1
Heal entries for 2KVM12-P2: 0
Heal entries for 2KVM12-P3: 0
Heal entries for 2KVM12-P5: 0
Heal entries for 2KVM12_P4: 1
# gluster volume heal 1KVM12-BCK info split-brain
Brick 16.0.0.161:/STORAGES/g2r5p1/GFS
/0fcad888-d573-47be-bef3-0bc0b7a99fb7/dom_md/ids
Number of entries in split-brain: 1
Brick 16.0.0.162:/STORAGES/g2r5p1/GFS
/0fcad888-d573-47be-bef3-0bc0b7a99fb7/dom_md/ids
Number of entries in split-brain: 1
# gluster volume heal 1KVM12-P1 info split-brain
Brick 16.0.0.161:/STORAGES/g1r5p1/GFS
/__DIRECT_IO_TEST__
Number of entries in split-brain: 1
Brick 16.0.0.162:/STORAGES/g1r5p1/GFS
/__DIRECT_IO_TEST__
Number of entries in split-brain: 1
etc..
YES - in split brain , but NOT master domain ( will solve later,
after master - if possible )
I'm not sure if it is related, but you could try to resolve the
split-brain first and see if it helps. Also, I see that you are using
replica-2. It is recommended to use replica-3 or arbiter volumes to
avoid split-brains.
-Ravi
---
vdsm.log
=
Thread-461::DEBUG::2016-02-24
11:12:45,328::fileSD::262::Storage.Misc.excCmd::(getReadDelay)
SUCCESS: = '0+1 records in\n0+1 records out\n333 bytes (333 B)
copied, 0.000724379 s, 460 kB/s\n'; = 0
Thread-461::INFO::2016-02-24
11:12:45,331::clusterlock::219::Storage.SANLock::(acquireHostId)
Acquiring host id for domain 88adbd49-62d6-45b1-9992-b04464a04112 (id: 3)
Thread-461::DEBUG::2016-02-24
11:12:45,331::clusterlock::237::Storage.SANLock::(acquireHostId) Host
id for domain 88adbd49-62d6-45b1-9992-b04464a04112 successfully
acquired (id: 3)
Thread-33186::DEBUG::2016-02-24
11:12:46,067::__init__::481::jsonrpc.JsonRpcServer::(_serveRequest)
Calling 'GlusterVolume.list' in bridge with {}
Thread-33186::DEBUG::2016-02-24
11:12:46,204::__init__::514::jsonrpc.JsonRpcServer::(_serveRequest)
Return 'GlusterVolume.list' in bridge with {'volumes': {'2KVM12-P5':
{'transportType': ['TCP'], 'uuid':
'4a6d775d-4a51-4f6c-9bfa-f7ef57f3ca1d', 'bricks':
['16.0.0.164:/STORAGES/g1r5p5/GFS',
'16.0.0.163:/STORAGES/g1r5p5/GFS'], 'volumeName': '2KVM12-P5',
'volumeType': 'REPLICATE', 'replicaCount': '2', 'brickCount': '2',
'distCount': '2', 'volumeStatus': 'ONLINE', 'stripeCount': '1',
'bricksInfo': [{'name': '16.0.0.164:/STORAGES/g1r5p5/GFS',
'hostUuid': '06854ac0-2ef1-4c12-bb8d-56cf9bf95ec9'}, {'name':
'16.0.0.163:/STORAGES/g1r5p5/GFS', 'hostUuid':
'6482ae32-25ac-41b5-b41d-b7ddf49bac2c'}], 'options':
{'cluster.server-quorum-type': 'none', 'cluster.eager-lock':
'enable', 'performance.stat-prefetch': 'off', 'cluster.quorum-type':
'fixed', 'performance.quick-read': 'off', 'network.remote-dio':
'enable', 'cluster.quorum-count': '1', 'performance.io-cache': 'off',
'storage.owner-uid': '36', 'performance.read-ahead': 'off',
'storage.owner-gid': '36'}}, '2KVM12_P4': {'transportType': ['TCP'],
'uuid': '18310aeb-639f-4b6d-9ef4-9ef560d6175c', 'bricks':
['16.0.0.163:/STORAGES/g1r5p4/GFS',
'16.0.0.164:/STORAGES/g1r5p4/GFS'], 'volumeName': '2KVM12_P4',
'volumeType': 'REPLICATE', 'replicaCount': '2', 'brickCount': '2',
'distCount': '2', 'volumeStatus': 'ONLINE', 'stripeCount':
Re: [ovirt-users] Fwd: Re: ovirt - can't attach master domain II
On 02/24/2016 04:48 PM, p...@email.cz wrote:
prereq: 2KVM12-P2 = master domain
-
YES - I'm using gluster.fuse NFS
localhost:/2KVM12-P2 on
/rhev/data-center/mnt/glusterSD/localhost:_2KVM12-P2 type
fuse.glusterfs
(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
-
Healing
==
# gluster volume heal 2KVM12-P2 info
Brick 16.0.0.164:/STORAGES/g1r5p2/GFS
Number of entries: 0
Brick 16.0.0.163:/STORAGES/g1r5p2/GFS
Number of entries: 0
# while true; do for vol in `gluster volume list`; do gluster volume
heal $vol info | sort | grep "Number of entries" | awk -F: '{tot+=$2}
END { printf("Heal entries for '"$vol"': %d\n", $tot);}'; done; sleep
120; echo -e "\n==\n"; done
Heal entries for 1KVM12-BCK: 1
Heal entries for 1KVM12-P1: 1
Heal entries for 1KVM12-P2: 0
Heal entries for 1KVM12-P3: 0
Heal entries for 1KVM12-P4: 0
Heal entries for 1KVM12-P5: 0
Heal entries for 2KVM12-P1: 1
Heal entries for 2KVM12-P2: 0
Heal entries for 2KVM12-P3: 0
Heal entries for 2KVM12-P5: 0
Heal entries for 2KVM12_P4: 1
# gluster volume heal 1KVM12-BCK info split-brain
Brick 16.0.0.161:/STORAGES/g2r5p1/GFS
/0fcad888-d573-47be-bef3-0bc0b7a99fb7/dom_md/ids
Number of entries in split-brain: 1
Brick 16.0.0.162:/STORAGES/g2r5p1/GFS
/0fcad888-d573-47be-bef3-0bc0b7a99fb7/dom_md/ids
Number of entries in split-brain: 1
# gluster volume heal 1KVM12-P1 info split-brain
Brick 16.0.0.161:/STORAGES/g1r5p1/GFS
/__DIRECT_IO_TEST__
Number of entries in split-brain: 1
Brick 16.0.0.162:/STORAGES/g1r5p1/GFS
/__DIRECT_IO_TEST__
Number of entries in split-brain: 1
etc..
YES - in split brain , but NOT master domain ( will solve later, after
master - if possible )
I'm not sure if it is related, but you could try to resolve the
split-brain first and see if it helps. Also, I see that you are using
replica-2. It is recommended to use replica-3 or arbiter volumes to
avoid split-brains.
-Ravi
---
vdsm.log
=
Thread-461::DEBUG::2016-02-24
11:12:45,328::fileSD::262::Storage.Misc.excCmd::(getReadDelay)
SUCCESS: = '0+1 records in\n0+1 records out\n333 bytes (333 B)
copied, 0.000724379 s, 460 kB/s\n'; = 0
Thread-461::INFO::2016-02-24
11:12:45,331::clusterlock::219::Storage.SANLock::(acquireHostId)
Acquiring host id for domain 88adbd49-62d6-45b1-9992-b04464a04112 (id: 3)
Thread-461::DEBUG::2016-02-24
11:12:45,331::clusterlock::237::Storage.SANLock::(acquireHostId) Host
id for domain 88adbd49-62d6-45b1-9992-b04464a04112 successfully
acquired (id: 3)
Thread-33186::DEBUG::2016-02-24
11:12:46,067::__init__::481::jsonrpc.JsonRpcServer::(_serveRequest)
Calling 'GlusterVolume.list' in bridge with {}
Thread-33186::DEBUG::2016-02-24
11:12:46,204::__init__::514::jsonrpc.JsonRpcServer::(_serveRequest)
Return 'GlusterVolume.list' in bridge with {'volumes': {'2KVM12-P5':
{'transportType': ['TCP'], 'uuid':
'4a6d775d-4a51-4f6c-9bfa-f7ef57f3ca1d', 'bricks':
['16.0.0.164:/STORAGES/g1r5p5/GFS',
'16.0.0.163:/STORAGES/g1r5p5/GFS'], 'volumeName': '2KVM12-P5',
'volumeType': 'REPLICATE', 'replicaCount': '2', 'brickCount': '2',
'distCount': '2', 'volumeStatus': 'ONLINE', 'stripeCount': '1',
'bricksInfo': [{'name': '16.0.0.164:/STORAGES/g1r5p5/GFS', 'hostUuid':
'06854ac0-2ef1-4c12-bb8d-56cf9bf95ec9'}, {'name':
'16.0.0.163:/STORAGES/g1r5p5/GFS', 'hostUuid':
'6482ae32-25ac-41b5-b41d-b7ddf49bac2c'}], 'options':
{'cluster.server-quorum-type': 'none', 'cluster.eager-lock': 'enable',
'performance.stat-prefetch': 'off', 'cluster.quorum-type': 'fixed',
'performance.quick-read': 'off', 'network.remote-dio': 'enable',
'cluster.quorum-count': '1', 'performance.io-cache': 'off',
'storage.owner-uid': '36', 'performance.read-ahead': 'off',
'storage.owner-gid': '36'}}, '2KVM12_P4': {'transportType': ['TCP'],
'uuid': '18310aeb-639f-4b6d-9ef4-9ef560d6175c', 'bricks':
['16.0.0.163:/STORAGES/g1r5p4/GFS',
'16.0.0.164:/STORAGES/g1r5p4/GFS'], 'volumeName': '2KVM12_P4',
'volumeType': 'REPLICATE', 'replicaCount': '2', 'brickCount': '2',
'distCount': '2', 'volumeStatus': 'ONLINE', 'stripeCount': '1',
'bricksInfo': [{'name': '16.0.0.163:/STORAGES/g1r5p4/GFS', 'hostUuid':
'6482ae32-25ac-41b5-b41d-b7ddf49bac2c'}, {'name':
'16.0.0.164:/STORAGES/g1r5p4/GFS', 'hostUuid':
'06854ac0-2ef1-4c12-bb8d-56cf9bf95ec9'}], 'options':
{'cluster.server-quorum-type': 'none', 'cluster.eager-lock': 'enable',
'performance.io-cache': 'off', 'performance.stat-prefetch': 'off',
'cluster.quorum-type': 'fixed', 'performance.quick-read': 'off',
'network.remote-dio': 'enable', 'cluster.quorum-count': '1',
'storage.owner-uid': '36', 'performance.read-ahead': 'off',
'storage.owner-gid': '36'}}, '2KVM12-P1': {'transportType': ['TCP'],
'uuid': 'cbf142f8-a40b-4cf4-ad29-2243c81d30c1', 'bricks':
[ovirt-users] Fwd: Re: ovirt - can't attach master domain II
prereq: 2KVM12-P2 = master domain
-
YES - I'm using gluster.fuse NFS
localhost:/2KVM12-P2 on
/rhev/data-center/mnt/glusterSD/localhost:_2KVM12-P2 type fuse.glusterfs
(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
-
Healing
==
# gluster volume heal 2KVM12-P2 info
Brick 16.0.0.164:/STORAGES/g1r5p2/GFS
Number of entries: 0
Brick 16.0.0.163:/STORAGES/g1r5p2/GFS
Number of entries: 0
# while true; do for vol in `gluster volume list`; do gluster volume
heal $vol info | sort | grep "Number of entries" | awk -F: '{tot+=$2}
END { printf("Heal entries for '"$vol"': %d\n", $tot);}'; done; sleep
120; echo -e "\n==\n"; done
Heal entries for 1KVM12-BCK: 1
Heal entries for 1KVM12-P1: 1
Heal entries for 1KVM12-P2: 0
Heal entries for 1KVM12-P3: 0
Heal entries for 1KVM12-P4: 0
Heal entries for 1KVM12-P5: 0
Heal entries for 2KVM12-P1: 1
Heal entries for 2KVM12-P2: 0
Heal entries for 2KVM12-P3: 0
Heal entries for 2KVM12-P5: 0
Heal entries for 2KVM12_P4: 1
# gluster volume heal 1KVM12-BCK info split-brain
Brick 16.0.0.161:/STORAGES/g2r5p1/GFS
/0fcad888-d573-47be-bef3-0bc0b7a99fb7/dom_md/ids
Number of entries in split-brain: 1
Brick 16.0.0.162:/STORAGES/g2r5p1/GFS
/0fcad888-d573-47be-bef3-0bc0b7a99fb7/dom_md/ids
Number of entries in split-brain: 1
# gluster volume heal 1KVM12-P1 info split-brain
Brick 16.0.0.161:/STORAGES/g1r5p1/GFS
/__DIRECT_IO_TEST__
Number of entries in split-brain: 1
Brick 16.0.0.162:/STORAGES/g1r5p1/GFS
/__DIRECT_IO_TEST__
Number of entries in split-brain: 1
etc..
YES - in split brain , but NOT master domain ( will solve later, after
master - if possible )
---
vdsm.log
=
Thread-461::DEBUG::2016-02-24
11:12:45,328::fileSD::262::Storage.Misc.excCmd::(getReadDelay) SUCCESS:
= '0+1 records in\n0+1 records out\n333 bytes (333 B) copied,
0.000724379 s, 460 kB/s\n'; = 0
Thread-461::INFO::2016-02-24
11:12:45,331::clusterlock::219::Storage.SANLock::(acquireHostId)
Acquiring host id for domain 88adbd49-62d6-45b1-9992-b04464a04112 (id: 3)
Thread-461::DEBUG::2016-02-24
11:12:45,331::clusterlock::237::Storage.SANLock::(acquireHostId) Host id
for domain 88adbd49-62d6-45b1-9992-b04464a04112 successfully acquired
(id: 3)
Thread-33186::DEBUG::2016-02-24
11:12:46,067::__init__::481::jsonrpc.JsonRpcServer::(_serveRequest)
Calling 'GlusterVolume.list' in bridge with {}
Thread-33186::DEBUG::2016-02-24
11:12:46,204::__init__::514::jsonrpc.JsonRpcServer::(_serveRequest)
Return 'GlusterVolume.list' in bridge with {'volumes': {'2KVM12-P5':
{'transportType': ['TCP'], 'uuid':
'4a6d775d-4a51-4f6c-9bfa-f7ef57f3ca1d', 'bricks':
['16.0.0.164:/STORAGES/g1r5p5/GFS', '16.0.0.163:/STORAGES/g1r5p5/GFS'],
'volumeName': '2KVM12-P5', 'volumeType': 'REPLICATE', 'replicaCount':
'2', 'brickCount': '2', 'distCount': '2', 'volumeStatus': 'ONLINE',
'stripeCount': '1', 'bricksInfo': [{'name':
'16.0.0.164:/STORAGES/g1r5p5/GFS', 'hostUuid':
'06854ac0-2ef1-4c12-bb8d-56cf9bf95ec9'}, {'name':
'16.0.0.163:/STORAGES/g1r5p5/GFS', 'hostUuid':
'6482ae32-25ac-41b5-b41d-b7ddf49bac2c'}], 'options':
{'cluster.server-quorum-type': 'none', 'cluster.eager-lock': 'enable',
'performance.stat-prefetch': 'off', 'cluster.quorum-type': 'fixed',
'performance.quick-read': 'off', 'network.remote-dio': 'enable',
'cluster.quorum-count': '1', 'performance.io-cache': 'off',
'storage.owner-uid': '36', 'performance.read-ahead': 'off',
'storage.owner-gid': '36'}}, '2KVM12_P4': {'transportType': ['TCP'],
'uuid': '18310aeb-639f-4b6d-9ef4-9ef560d6175c', 'bricks':
['16.0.0.163:/STORAGES/g1r5p4/GFS', '16.0.0.164:/STORAGES/g1r5p4/GFS'],
'volumeName': '2KVM12_P4', 'volumeType': 'REPLICATE', 'replicaCount':
'2', 'brickCount': '2', 'distCount': '2', 'volumeStatus': 'ONLINE',
'stripeCount': '1', 'bricksInfo': [{'name':
'16.0.0.163:/STORAGES/g1r5p4/GFS', 'hostUuid':
'6482ae32-25ac-41b5-b41d-b7ddf49bac2c'}, {'name':
'16.0.0.164:/STORAGES/g1r5p4/GFS', 'hostUuid':
'06854ac0-2ef1-4c12-bb8d-56cf9bf95ec9'}], 'options':
{'cluster.server-quorum-type': 'none', 'cluster.eager-lock': 'enable',
'performance.io-cache': 'off', 'performance.stat-prefetch': 'off',
'cluster.quorum-type': 'fixed', 'performance.quick-read': 'off',
'network.remote-dio': 'enable', 'cluster.quorum-count': '1',
'storage.owner-uid': '36', 'performance.read-ahead': 'off',
'storage.owner-gid': '36'}}, '2KVM12-P1': {'transportType': ['TCP'],
'uuid': 'cbf142f8-a40b-4cf4-ad29-2243c81d30c1', 'bricks':
['16.0.0.163:/STORAGES/g1r5p1/GFS', '16.0.0.164:/STORAGES/g1r5p1/GFS'],
'volumeName': '2KVM12-P1', 'volumeType': 'REPLICATE', 'replicaCount':
'2', 'brickCount': '2', 'distCount': '2', 'volumeStatus': 'ONLINE',
'stripeCount': '1', 'bricksInfo': [{'name':
'16.0.0.163:/STORAGES/g1r5p1/GFS',
