Hi Mauro, Yes, I can provide you step by step procedure to correct it. Is it fine If i provide you the steps tomorrow as it is quite late over here and I don't want to miss anything in hurry?
--- Ashish ----- Original Message ----- From: "Mauro Tridici" <[email protected]> To: "Ashish Pandey" <[email protected]> Cc: "gluster-users" <[email protected]> Sent: Wednesday, September 26, 2018 6:54:19 PM Subject: Re: [Gluster-users] Rebalance failed on Distributed Disperse volume based on 3.12.14 version Hi Ashish, in attachment you can find the rebalance log file and the last updated brick log file (the other files in /var/log/glusterfs/bricks directory seem to be too old). I just stopped the running rebalance (as you can see at the bottom of the rebalance log file). So, if exists a safe procedure to correct the problem I would like execute it. I don’t know if I can ask you it, but, if it is possible, could you please describe me step by step the right procedure to remove the newly added bricks without losing the data that have been already rebalanced? The following outputs show the result of “df -h” command executed on one of the first 3 nodes (s01, s02, s03) already existing and on one of the last 3 nodes (s04, s05, s06) added recently. [root@s06 bricks]# df -h File system Dim. Usati Dispon. Uso% Montato su /dev/mapper/cl_s06-root 100G 2,1G 98G 3% / devtmpfs 32G 0 32G 0% /dev tmpfs 32G 4,0K 32G 1% /dev/shm tmpfs 32G 26M 32G 1% /run tmpfs 32G 0 32G 0% /sys/fs/cgroup /dev/mapper/cl_s06-var 100G 2,0G 99G 2% /var /dev/mapper/cl_s06-gluster 100G 33M 100G 1% /gluster /dev/sda1 1014M 152M 863M 15% /boot /dev/mapper/gluster_vgd-gluster_lvd 9,0T 807G 8,3T 9% /gluster/mnt3 /dev/mapper/gluster_vgg-gluster_lvg 9,0T 807G 8,3T 9% /gluster/mnt6 /dev/mapper/gluster_vgc-gluster_lvc 9,0T 807G 8,3T 9% /gluster/mnt2 /dev/mapper/gluster_vge-gluster_lve 9,0T 807G 8,3T 9% /gluster/mnt4 /dev/mapper/gluster_vgj-gluster_lvj 9,0T 887G 8,2T 10% /gluster/mnt9 /dev/mapper/gluster_vgb-gluster_lvb 9,0T 807G 8,3T 9% /gluster/mnt1 /dev/mapper/gluster_vgh-gluster_lvh 9,0T 887G 8,2T 10% /gluster/mnt7 /dev/mapper/gluster_vgf-gluster_lvf 9,0T 807G 8,3T 9% /gluster/mnt5 /dev/mapper/gluster_vgi-gluster_lvi 9,0T 887G 8,2T 10% /gluster/mnt8 /dev/mapper/gluster_vgl-gluster_lvl 9,0T 887G 8,2T 10% /gluster/mnt11 /dev/mapper/gluster_vgk-gluster_lvk 9,0T 887G 8,2T 10% /gluster/mnt10 /dev/mapper/gluster_vgm-gluster_lvm 9,0T 887G 8,2T 10% /gluster/mnt12 tmpfs 6,3G 0 6,3G 0% /run/user/0 [root@s01 ~]# df -h File system Dim. Usati Dispon. Uso% Montato su /dev/mapper/cl_s01-root 100G 5,3G 95G 6% / devtmpfs 32G 0 32G 0% /dev tmpfs 32G 39M 32G 1% /dev/shm tmpfs 32G 26M 32G 1% /run tmpfs 32G 0 32G 0% /sys/fs/cgroup /dev/mapper/cl_s01-var 100G 11G 90G 11% /var /dev/md127 1015M 151M 865M 15% /boot /dev/mapper/cl_s01-gluster 100G 33M 100G 1% /gluster /dev/mapper/gluster_vgi-gluster_lvi 9,0T 5,5T 3,6T 61% /gluster/mnt7 /dev/mapper/gluster_vgm-gluster_lvm 9,0T 5,4T 3,6T 61% /gluster/mnt11 /dev/mapper/gluster_vgf-gluster_lvf 9,0T 5,7T 3,4T 63% /gluster/mnt4 /dev/mapper/gluster_vgl-gluster_lvl 9,0T 5,8T 3,3T 64% /gluster/mnt10 /dev/mapper/gluster_vgj-gluster_lvj 9,0T 5,5T 3,6T 61% /gluster/mnt8 /dev/mapper/gluster_vgn-gluster_lvn 9,0T 5,4T 3,6T 61% /gluster/mnt12 /dev/mapper/gluster_vgk-gluster_lvk 9,0T 5,8T 3,3T 64% /gluster/mnt9 /dev/mapper/gluster_vgh-gluster_lvh 9,0T 5,6T 3,5T 63% /gluster/mnt6 /dev/mapper/gluster_vgg-gluster_lvg 9,0T 5,6T 3,5T 63% /gluster/mnt5 /dev/mapper/gluster_vge-gluster_lve 9,0T 5,7T 3,4T 63% /gluster/mnt3 /dev/mapper/gluster_vgc-gluster_lvc 9,0T 5,6T 3,5T 62% /gluster/mnt1 /dev/mapper/gluster_vgd-gluster_lvd 9,0T 5,6T 3,5T 62% /gluster/mnt2 tmpfs 6,3G 0 6,3G 0% /run/user/0 s01-stg:tier2 420T 159T 262T 38% /tier2 As you can see, used space value of each brick of the last servers is about 800GB. Thank you, Mauro Il giorno 26 set 2018, alle ore 14:51, Ashish Pandey < [email protected] > ha scritto: Hi Mauro, rebalance and brick logs should be the first thing we should go through. There is a procedure to correct the configuration/setup but the situation you are in is difficult to follow that procedure. You should have added the bricks hosted on s04-stg , s05-stg and s06-stg the same way you had the previous configuration. That means 2 bricks on each node for one subvolume. The procedure will require a lot of replace bricks which will again need healing and all. In addition to that we have to wait for re-balance to complete. I would suggest that if whole data has not been rebalanced and if you can stop the rebalance and remove these newly added bricks properly then you should remove these newly added bricks. After that, add these bricks so that you have 2 bricks of each volume on 3 newly added nodes. Yes, it is like undoing whole effort but it is better to do it now then facing issues in future when it will be almost impossible to correct these things if you have lots of data. --- Ashish ----- Original Message ----- From: "Mauro Tridici" < [email protected] > To: "Ashish Pandey" < [email protected] > Cc: "gluster-users" < [email protected] > Sent: Wednesday, September 26, 2018 5:55:02 PM Subject: Re: [Gluster-users] Rebalance failed on Distributed Disperse volume based on 3.12.14 version Dear Ashish, thank you for you answer. I could provide you the entire log file related to glusterd, glusterfsd and rebalance. Please, could you indicate which one you need first? Yes, we added the last 36 bricks after creating vol. Is there a procedure to correct this error? Is it still possible to do it? Many thanks, Mauro <blockquote> Il giorno 26 set 2018, alle ore 14:13, Ashish Pandey < [email protected] > ha scritto: I think we don't have enough logs to debug this so I would suggest you to provide more logs/info. I have also observed that the configuration and setup of your volume is not very efficient. For example: Brick37: s04-stg:/gluster/mnt1/brick Brick38: s04-stg:/gluster/mnt2/brick Brick39: s04-stg:/gluster/mnt3/brick Brick40: s04-stg:/gluster/mnt4/brick Brick41: s04-stg:/gluster/mnt5/brick Brick42: s04-stg:/gluster/mnt6/brick Brick43: s04-stg:/gluster/mnt7/brick Brick44: s04-stg:/gluster/mnt8/brick Brick45: s04-stg:/gluster/mnt9/brick Brick46: s04-stg:/gluster/mnt10/brick Brick47: s04-stg:/gluster/mnt11/brick Brick48: s04-stg:/gluster/mnt12/brick These 12 bricks are on same node and the sub volume made up of these bricks will be of same subvolume, which is not good. Same is true for the bricks hosted on s05-stg and s06-stg I think you have added these bricks after creating vol. The probability of disruption in connection of these bricks will be higher in this case. --- Ashish ----- Original Message ----- From: "Mauro Tridici" < [email protected] > To: "gluster-users" < [email protected] > Sent: Wednesday, September 26, 2018 3:38:35 PM Subject: [Gluster-users] Rebalance failed on Distributed Disperse volume based on 3.12.14 version Dear All, Dear Nithya, after upgrading from 3.10.5 version to 3.12.14, I tried to start a rebalance process to distribute data across the bricks, but something goes wrong. Rebalance failed on different nodes and the time value needed to complete the procedure seems to be very high. [root@s01 ~]# gluster volume rebalance tier2 status Node Rebalanced-files size scanned failures skipped status run time in h:m:s --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- localhost 19 161.6GB 537 2 2 in progress 0:32:23 s02-stg 25 212.7GB 526 5 2 in progress 0:32:25 s03-stg 4 69.1GB 511 0 0 in progress 0:32:25 s04-stg 4 484Bytes 12283 0 3 in progress 0:32:25 s05-stg 23 484Bytes 11049 0 10 in progress 0:32:25 s06-stg 3 1.2GB 8032 11 3 failed 0:17:57 Estimated time left for rebalance to complete : 3601:05:41 volume rebalance: tier2: success When rebalance processes fail, I can see the following kind of errors in /var/log/glusterfs/tier2-rebalance.log Error type 1) [2018-09-26 08:50:19.872575] W [MSGID: 122053] [ec-common.c:269:ec_check_status] 0-tier2-disperse-10: Operation failed on 2 of 6 subvolumes.(up=111111, mask=100111, remaining= 000000, good=100111, bad=011000) [2018-09-26 08:50:19.901792] W [MSGID: 122053] [ec-common.c:269:ec_check_status] 0-tier2-disperse-11: Operation failed on 1 of 6 subvolumes.(up=111111, mask=111101, remaining= 000000, good=111101, bad=000010) Error type 2) [2018-09-26 08:53:31.566836] W [socket.c:600:__socket_rwv] 0-tier2-client-53: readv on 192.168.0.55:49153 failed (Connection reset by peer) Error type 3) [2018-09-26 08:57:37.852590] W [MSGID: 122035] [ec-common.c:571:ec_child_select] 0-tier2-disperse-9: Executing operation with some subvolumes unavailable (10) [2018-09-26 08:57:39.282306] W [MSGID: 122035] [ec-common.c:571:ec_child_select] 0-tier2-disperse-9: Executing operation with some subvolumes unavailable (10) [2018-09-26 09:02:04.928408] W [MSGID: 109023] [dht-rebalance.c:1013:__dht_check_free_space] 0-tier2-dht: data movement of file {blocks:0 name:(/OPA/archive/historical/dts/MRE A/Observations/Observations/MREA14/Cs-1/CMCC/raw/CS013.ext)} would result in dst node (tier2-disperse-5:2440190848) having lower disk space than the source node (tier2-dispers e-11:71373083776).Skipping file. Error type 4) W [rpc-clnt-ping.c:223:rpc_clnt_ping_cbk] 0-tier2-client-7: socket disconnected Error type 5) [2018-09-26 09:07:42.333720] W [glusterfsd.c:1375:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7e25) [0x7f0417e0ee25] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x55 90086004b5] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x55900860032b] ) 0-: received signum (15), shutting down Error type 6) [2018-09-25 08:09:18.340658] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-4: server 192.168.0.52:49153 has not responded in the last 42 seconds, disconnecting. It seems that there are some network or timeout problems, but the network usage/traffic values are not so high. Do you think that, in my volume configuration, I have to modify some volume options related to thread and/or network parameters? Could you, please, help me to understand the cause of the problems above? You can find below our volume info: (volume is implemented on 6 servers; each server configuration: 2 cpu 10-cores, 64GB RAM, 1 SSD dedicated to the OS, 12 x 10TB HD) [root@s04 ~]# gluster vol info Volume Name: tier2 Type: Distributed-Disperse Volume ID: a28d88c5-3295-4e35-98d4-210b3af9358c Status: Started Snapshot Count: 0 Number of Bricks: 12 x (4 + 2) = 72 Transport-type: tcp Bricks: Brick1: s01-stg:/gluster/mnt1/brick Brick2: s02-stg:/gluster/mnt1/brick Brick3: s03-stg:/gluster/mnt1/brick Brick4: s01-stg:/gluster/mnt2/brick Brick5: s02-stg:/gluster/mnt2/brick Brick6: s03-stg:/gluster/mnt2/brick Brick7: s01-stg:/gluster/mnt3/brick Brick8: s02-stg:/gluster/mnt3/brick Brick9: s03-stg:/gluster/mnt3/brick Brick10: s01-stg:/gluster/mnt4/brick Brick11: s02-stg:/gluster/mnt4/brick Brick12: s03-stg:/gluster/mnt4/brick Brick13: s01-stg:/gluster/mnt5/brick Brick14: s02-stg:/gluster/mnt5/brick Brick15: s03-stg:/gluster/mnt5/brick Brick16: s01-stg:/gluster/mnt6/brick Brick17: s02-stg:/gluster/mnt6/brick Brick18: s03-stg:/gluster/mnt6/brick Brick19: s01-stg:/gluster/mnt7/brick Brick20: s02-stg:/gluster/mnt7/brick Brick21: s03-stg:/gluster/mnt7/brick Brick22: s01-stg:/gluster/mnt8/brick Brick23: s02-stg:/gluster/mnt8/brick Brick24: s03-stg:/gluster/mnt8/brick Brick25: s01-stg:/gluster/mnt9/brick Brick26: s02-stg:/gluster/mnt9/brick Brick27: s03-stg:/gluster/mnt9/brick Brick28: s01-stg:/gluster/mnt10/brick Brick29: s02-stg:/gluster/mnt10/brick Brick30: s03-stg:/gluster/mnt10/brick Brick31: s01-stg:/gluster/mnt11/brick Brick32: s02-stg:/gluster/mnt11/brick Brick33: s03-stg:/gluster/mnt11/brick Brick34: s01-stg:/gluster/mnt12/brick Brick35: s02-stg:/gluster/mnt12/brick Brick36: s03-stg:/gluster/mnt12/brick Brick37: s04-stg:/gluster/mnt1/brick Brick38: s04-stg:/gluster/mnt2/brick Brick39: s04-stg:/gluster/mnt3/brick Brick40: s04-stg:/gluster/mnt4/brick Brick41: s04-stg:/gluster/mnt5/brick Brick42: s04-stg:/gluster/mnt6/brick Brick43: s04-stg:/gluster/mnt7/brick Brick44: s04-stg:/gluster/mnt8/brick Brick45: s04-stg:/gluster/mnt9/brick Brick46: s04-stg:/gluster/mnt10/brick Brick47: s04-stg:/gluster/mnt11/brick Brick48: s04-stg:/gluster/mnt12/brick Brick49: s05-stg:/gluster/mnt1/brick Brick50: s05-stg:/gluster/mnt2/brick Brick51: s05-stg:/gluster/mnt3/brick Brick52: s05-stg:/gluster/mnt4/brick Brick53: s05-stg:/gluster/mnt5/brick Brick54: s05-stg:/gluster/mnt6/brick Brick55: s05-stg:/gluster/mnt7/brick Brick56: s05-stg:/gluster/mnt8/brick Brick57: s05-stg:/gluster/mnt9/brick Brick58: s05-stg:/gluster/mnt10/brick Brick59: s05-stg:/gluster/mnt11/brick Brick60: s05-stg:/gluster/mnt12/brick Brick61: s06-stg:/gluster/mnt1/brick Brick62: s06-stg:/gluster/mnt2/brick Brick63: s06-stg:/gluster/mnt3/brick Brick64: s06-stg:/gluster/mnt4/brick Brick65: s06-stg:/gluster/mnt5/brick Brick66: s06-stg:/gluster/mnt6/brick Brick67: s06-stg:/gluster/mnt7/brick Brick68: s06-stg:/gluster/mnt8/brick Brick69: s06-stg:/gluster/mnt9/brick Brick70: s06-stg:/gluster/mnt10/brick Brick71: s06-stg:/gluster/mnt11/brick Brick72: s06-stg:/gluster/mnt12/brick Options Reconfigured: network.ping-timeout: 60 diagnostics.count-fop-hits: on diagnostics.latency-measurement: on cluster.server-quorum-type: server features.default-soft-limit: 90 features.quota-deem-statfs: on performance.io -thread-count: 16 disperse.cpu-extensions: auto performance.io -cache: off network.inode-lru-limit: 50000 performance.md-cache-timeout: 600 performance.cache-invalidation: on performance.stat-prefetch: on features.cache-invalidation-timeout: 600 features.cache-invalidation: on cluster.readdir-optimize: on performance.parallel-readdir: off performance.readdir-ahead: on cluster.lookup-optimize: on client.event-threads: 4 server.event-threads: 4 nfs.disable: on transport.address-family: inet cluster.quorum-type: auto cluster.min-free-disk: 10 performance.client-io-threads: on features.quota: on features.inode-quota: on features.bitrot: on features.scrub: Active cluster.brick-multiplex: on cluster.server-quorum-ratio: 51% If it can help, I paste here the output of “free -m” command executed on all the cluster nodes: The result is almost the same on every nodes. In your opinion, the available RAM is enough to support data movement? [root@s06 ~]# free -m total used free shared buff/cache available Mem: 64309 10409 464 15 53434 52998 Swap: 65535 103 65432 Thank you in advance. Sorry for my long message, but I’m trying to notify you all available information. Regards, Mauro _______________________________________________ Gluster-users mailing list [email protected] https://lists.gluster.org/mailman/listinfo/gluster-users _______________________________________________ Gluster-users mailing list [email protected] https://lists.gluster.org/mailman/listinfo/gluster-users </blockquote> _______________________________________________ Gluster-users mailing list [email protected] https://lists.gluster.org/mailman/listinfo/gluster-users
_______________________________________________ Gluster-users mailing list [email protected] https://lists.gluster.org/mailman/listinfo/gluster-users
