I am trying to add a fourth server to my distributed gluster setup, but the 'add-brick' command keeps failing. I've tried several times, each time cleaning the new server by stopping and uninstalling gluster, unmounting and mkfs the new filesystem, deleting all gluster files (/var/log/glusterfs, /var/lib/glusterfs), then re-installing. On fs1, I remove-brick the new server and detach the peer, and then start over. It keeps failing at the same point.
The servers have identical hardware and software. What should I look for? CentOS 7.2 Gluster 3.7.14-1 Server names are fs1, fs2, fs3 and the new server fs4. Fs1, fs2 and fs3 have been running for several months. [root@fs1]# gluster volume add-brick cees-data fs4:/data/brick volume add-brick: failed: Commit failed on fs4. Please check log file for details. [root@fs1]# gluster volume info Volume Name: cees-data Type: Distribute Volume ID: 27d2a59c-bdac-4f66-bcd8-e6124e53a4a2 Status: Started Number of Bricks: 4 Transport-type: tcp,rdma Bricks: Brick1: fs1:/data/brick Brick2: fs2:/data/brick Brick3: fs3:/data/brick Brick4: fs4:/data/brick Options Reconfigured: performance.readdir-ahead: on features.quota: on features.inode-quota: on features.quota-deem-statfs: on [root@fs1]# gluster volume status Status of volume: cees-data Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick fs1:/data/brick 49152 49153 Y 1878 Brick fs2:/data/brick 49152 0 Y 1707 Brick fs3:/data/brick 49152 0 Y 4696 NFS Server on fs4 2049 0 Y 12190 NFS Server on localhost 2049 0 Y 4838 Quota Daemon on localhost N/A N/A Y 4846 Quota Daemon on fs4 N/A N/A Y 12198 NFS Server on fs3 2049 0 Y 11084 Quota Daemon on fs3 N/A N/A Y 11092 NFS Server on fs2 2049 0 Y 10199 Quota Daemon on fs2 N/A N/A Y 10207 Task Status of Volume cees-data ------------------------------------------------------------------------------ There are no active volume tasks from the logs on the new server fs4: [2016-09-26 22:44:38.605539] I [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.7.14/xlator/mgmt/glusterd.so(glusterd_op_commit_hook+0x195) [0x7f257bed20e5] -->/usr/lib64/glusterfs/3.7.14/xlator/mgmt/glusterd.so(glusterd_hooks_run_hooks+0x4c5) [0x7f257bf66e95] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7f25873cecd5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/add-brick/pre/S28Quota-enable-root-xattr-heal.sh --volname=cees-data --version=1 --volume-op=add-brick --gd-workdir=/var/lib/glusterd [2016-09-26 22:44:39.254422] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/brick on port 49152 [2016-09-26 22:44:39.254510] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/brick.rdma on port 49153 [2016-09-26 22:44:39.254921] E [MSGID: 106005] [glusterd-utils.c:4771:glusterd_brick_start] 0-management: Unable to start brick fs4:/data/brick [2016-09-26 22:44:39.254949] E [MSGID: 106074] [glusterd-brick-ops.c:2372:glusterd_op_add_brick] 0-glusterd: Unable to add bricks [2016-09-26 22:44:39.254958] E [MSGID: 106123] [glusterd-mgmt.c:294:gd_mgmt_v3_commit_fn] 0-management: Add-brick commit failed. [2016-09-26 22:44:39.254965] E [MSGID: 106123] [glusterd-mgmt-handler.c:603:glusterd_handle_commit_fn] 0-management: commit failed on operation Add brick [2016-09-26 22:45:38.146318] I [MSGID: 106144] [glusterd-pmap.c:276:pmap_registry_remove] 0-pmap: removing brick /data/brick on port 49152
_______________________________________________ Gluster-users mailing list [email protected] http://www.gluster.org/mailman/listinfo/gluster-users
