Are there any workarounds to this? RDMA is configured on my servers. Dennis
On Thu, Sep 29, 2016 at 7:19 AM, Atin Mukherjee <[email protected]> wrote: > Dennis, > > Thanks for sharing the logs. > > It seems like a volume configured created with tcp,rdma transport fails to > start (atleast in my local set up). The issue here is although the brick > process comes up, but glusterd receives a non zero ret code from the runner > interface which spawns the brick process(es). > > Raghavendra Talur/Rafi, > > Is this an intended behaviour if rdma device is not configured? Please > chime in with your thoughts > > > On Wed, Sep 28, 2016 at 10:22 AM, Atin Mukherjee <[email protected]> > wrote: > >> Dennis, >> >> It seems like that add-brick has definitely failed and the entry is not >> committed into glusterd store. volume status and volume info commands are >> referring the in-memory data for fs4 (which exist) but post a restart they >> are no longer available. Could you run glusterd with debug log enabled >> (systemctl stop glusterd; glusterd -LDEBUG) and provide us cmd_history.log, >> glusterd log along with fs4 brick log files to further analyze the issue? >> Regarding the missing RDMA ports for fs2, fs3 brick can you cross check if >> glusterfs-rdma package is installed on both the nodes? >> >> On Wed, Sep 28, 2016 at 7:14 AM, Ravishankar N <[email protected]> >> wrote: >> >>> On 09/27/2016 10:29 PM, Dennis Michael wrote: >>> >>> >>> >>> [root@fs4 bricks]# gluster volume info >>> >>> Volume Name: cees-data >>> Type: Distribute >>> Volume ID: 27d2a59c-bdac-4f66-bcd8-e6124e53a4a2 >>> Status: Started >>> Number of Bricks: 4 >>> Transport-type: tcp,rdma >>> Bricks: >>> Brick1: fs1:/data/brick >>> Brick2: fs2:/data/brick >>> Brick3: fs3:/data/brick >>> Brick4: fs4:/data/brick >>> Options Reconfigured: >>> features.quota-deem-statfs: on >>> features.inode-quota: on >>> features.quota: on >>> performance.readdir-ahead: on >>> [root@fs4 bricks]# gluster volume status >>> Status of volume: cees-data >>> Gluster process TCP Port RDMA Port Online >>> Pid >>> ------------------------------------------------------------ >>> ------------------ >>> Brick fs1:/data/brick 49152 49153 Y >>> 1878 >>> Brick fs2:/data/brick 49152 0 Y >>> 1707 >>> Brick fs3:/data/brick 49152 0 Y >>> 4696 >>> Brick fs4:/data/brick N/A N/A N >>> N/A >>> NFS Server on localhost 2049 0 Y >>> 13808 >>> Quota Daemon on localhost N/A N/A Y >>> 13813 >>> NFS Server on fs1 2049 0 Y >>> 6722 >>> Quota Daemon on fs1 N/A N/A Y >>> 6730 >>> NFS Server on fs3 2049 0 Y >>> 12553 >>> Quota Daemon on fs3 N/A N/A Y >>> 12561 >>> NFS Server on fs2 2049 0 Y >>> 11702 >>> Quota Daemon on fs2 N/A N/A Y >>> 11710 >>> >>> Task Status of Volume cees-data >>> ------------------------------------------------------------ >>> ------------------ >>> There are no active volume tasks >>> >>> [root@fs4 bricks]# ps auxww | grep gluster >>> root 13791 0.0 0.0 701472 19768 ? Ssl 09:06 0:00 >>> /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO >>> root 13808 0.0 0.0 560236 41420 ? Ssl 09:07 0:00 >>> /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p >>> /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S >>> /var/run/gluster/01c61523374369658a62b75c582b5ac2.socket >>> root 13813 0.0 0.0 443164 17908 ? Ssl 09:07 0:00 >>> /usr/sbin/glusterfs -s localhost --volfile-id gluster/quotad -p >>> /var/lib/glusterd/quotad/run/quotad.pid -l >>> /var/log/glusterfs/quotad.log -S >>> /var/run/gluster/3753def90f5c34f656513dba6a544f7d.socket >>> --xlator-option *replicate*.data-self-heal=off --xlator-option >>> *replicate*.metadata-self-heal=off --xlator-option >>> *replicate*.entry-self-heal=off >>> root 13874 0.0 0.0 1200472 31700 ? Ssl 09:16 0:00 >>> /usr/sbin/glusterfsd -s fs4 --volfile-id cees-data.fs4.data-brick -p >>> /var/lib/glusterd/vols/cees-data/run/fs4-data-brick.pid -S >>> /var/run/gluster/5203ab38be21e1d37c04f6bdfee77d4a.socket --brick-name >>> /data/brick -l /var/log/glusterfs/bricks/data-brick.log --xlator-option >>> *-posix.glusterd-uuid=f04b231e-63f8-4374-91ae-17c0c623f165 --brick-port >>> 49152 49153 --xlator-option >>> cees-data-server.transport.rdma.listen-port=49153 >>> --xlator-option cees-data-server.listen-port=49152 >>> --volfile-server-transport=socket,rdma >>> root 13941 0.0 0.0 112648 976 pts/0 S+ 09:50 0:00 grep >>> --color=auto gluster >>> >>> [root@fs4 bricks]# systemctl restart glusterfsd glusterd >>> >>> [root@fs4 bricks]# ps auxww | grep gluster >>> root 13808 0.0 0.0 560236 41420 ? Ssl 09:07 0:00 >>> /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p >>> /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S >>> /var/run/gluster/01c61523374369658a62b75c582b5ac2.socket >>> root 13813 0.0 0.0 443164 17908 ? Ssl 09:07 0:00 >>> /usr/sbin/glusterfs -s localhost --volfile-id gluster/quotad -p >>> /var/lib/glusterd/quotad/run/quotad.pid -l >>> /var/log/glusterfs/quotad.log -S >>> /var/run/gluster/3753def90f5c34f656513dba6a544f7d.socket >>> --xlator-option *replicate*.data-self-heal=off --xlator-option >>> *replicate*.metadata-self-heal=off --xlator-option >>> *replicate*.entry-self-heal=off >>> root 13953 0.1 0.0 570740 14988 ? Ssl 09:51 0:00 >>> /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO >>> root 13965 0.0 0.0 112648 976 pts/0 S+ 09:51 0:00 grep >>> --color=auto gluster >>> >>> [root@fs4 bricks]# gluster volume info >>> >>> Volume Name: cees-data >>> Type: Distribute >>> Volume ID: 27d2a59c-bdac-4f66-bcd8-e6124e53a4a2 >>> Status: Started >>> Number of Bricks: 3 >>> Transport-type: tcp,rdma >>> Bricks: >>> Brick1: fs1:/data/brick >>> Brick2: fs2:/data/brick >>> Brick3: fs3:/data/brick >>> Options Reconfigured: >>> performance.readdir-ahead: on >>> features.quota: on >>> features.inode-quota: on >>> features.quota-deem-statfs: on >>> >>> >>> >>> I'm not sure what's going on here. Restarting glusterd seems to change >>> the output of gluster volume info? I also see you are using RDMA. Not sure >>> why the RDMA ports for fs2 and fs3 are not shown in the volume status >>> output. CC'ing some glusterd/rdma devs for pointers. >>> >>> -Ravi >>> >>> >>> >>> [root@fs4 bricks]# gluster volume status >>> Status of volume: cees-data >>> Gluster process TCP Port RDMA Port Online >>> Pid >>> ------------------------------------------------------------ >>> ------------------ >>> Brick fs1:/data/brick 49152 49153 Y >>> 1878 >>> Brick fs2:/data/brick 49152 0 Y >>> 1707 >>> Brick fs3:/data/brick 49152 0 Y >>> 4696 >>> NFS Server on localhost 2049 0 Y >>> 13968 >>> Quota Daemon on localhost N/A N/A Y >>> 13976 >>> NFS Server on fs2 2049 0 Y >>> 11702 >>> Quota Daemon on fs2 N/A N/A Y >>> 11710 >>> NFS Server on fs3 2049 0 Y >>> 12553 >>> Quota Daemon on fs3 N/A N/A Y >>> 12561 >>> NFS Server on fs1 2049 0 Y >>> 6722 >>> >>> Task Status of Volume cees-data >>> ------------------------------------------------------------ >>> ------------------ >>> There are no active volume tasks >>> >>> [root@fs4 bricks]# gluster peer status >>> Number of Peers: 3 >>> >>> Hostname: fs1 >>> Uuid: ddc0a23e-05e5-48f7-993e-a37e43b21605 >>> State: Peer in Cluster (Connected) >>> >>> Hostname: fs2 >>> Uuid: e37108f8-d2f1-4f28-adc8-0b3d3401df29 >>> State: Peer in Cluster (Connected) >>> >>> Hostname: fs3 >>> Uuid: 19a42201-c932-44db-b1a7-8b5b1af32a36 >>> State: Peer in Cluster (Connected) >>> >>> Dennis >>> >>> >>> On Tue, Sep 27, 2016 at 9:40 AM, Ravishankar N <[email protected]> >>> wrote: >>> >>>> On 09/27/2016 09:53 PM, Dennis Michael wrote: >>>> >>>> Yes, you are right. I mixed up the logs. I just ran the add-brick >>>> command again after cleaning up fs4 and re-installing gluster. This is the >>>> complete fs4 data-brick.log. >>>> >>>> [root@fs1 ~]# gluster volume add-brick cees-data fs4:/data/brick >>>> volume add-brick: failed: Commit failed on fs4. Please check log file >>>> for details. >>>> >>>> [root@fs4 bricks]# pwd >>>> /var/log/glusterfs/bricks >>>> [root@fs4 bricks]# cat data-brick.log >>>> [2016-09-27 16:16:28.095661] I [MSGID: 100030] [glusterfsd.c:2338:main] >>>> 0-/usr/sbin/glusterfsd: Started running /usr/sbin/glusterfsd version 3.7.14 >>>> (args: /usr/sbin/glusterfsd -s fs4 --volfile-id cees-data.fs4.data-brick -p >>>> /var/lib/glusterd/vols/cees-data/run/fs4-data-brick.pid -S >>>> /var/run/gluster/5203ab38be21e1d37c04f6bdfee77d4a.socket --brick-name >>>> /data/brick -l /var/log/glusterfs/bricks/data-brick.log >>>> --xlator-option *-posix.glusterd-uuid=f04b231e-63f8-4374-91ae-17c0c623f165 >>>> --brick-port 49152 --xlator-option >>>> cees-data-server.transport.rdma.listen-port=49153 >>>> --xlator-option cees-data-server.listen-port=49152 >>>> --volfile-server-transport=socket,rdma) >>>> [2016-09-27 16:16:28.101547] I [MSGID: 101190] >>>> [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started >>>> thread with index 1 >>>> [2016-09-27 16:16:28.104637] I [graph.c:269:gf_add_cmdline_options] >>>> 0-cees-data-server: adding option 'listen-port' for volume >>>> 'cees-data-server' with value '49152' >>>> [2016-09-27 16:16:28.104646] I [graph.c:269:gf_add_cmdline_options] >>>> 0-cees-data-server: adding option 'transport.rdma.listen-port' for volume >>>> 'cees-data-server' with value '49153' >>>> [2016-09-27 16:16:28.104662] I [graph.c:269:gf_add_cmdline_options] >>>> 0-cees-data-posix: adding option 'glusterd-uuid' for volume >>>> 'cees-data-posix' with value 'f04b231e-63f8-4374-91ae-17c0c623f165' >>>> [2016-09-27 16:16:28.104808] I [MSGID: 115034] >>>> [server.c:403:_check_for_auth_option] 0-/data/brick: skip format check >>>> for non-addr auth option auth.login./data/brick.allow >>>> [2016-09-27 16:16:28.104814] I [MSGID: 115034] >>>> [server.c:403:_check_for_auth_option] 0-/data/brick: skip format check >>>> for non-addr auth option auth.login.18ddaf4c-ad98-4155- >>>> 9372-717eae718b4c.password >>>> [2016-09-27 16:16:28.104883] I [MSGID: 101190] >>>> [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started >>>> thread with index 2 >>>> [2016-09-27 16:16:28.105479] I >>>> [rpcsvc.c:2196:rpcsvc_set_outstanding_rpc_limit] >>>> 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64 >>>> [2016-09-27 16:16:28.105532] W [MSGID: 101002] >>>> [options.c:957:xl_opt_validate] 0-cees-data-server: option >>>> 'listen-port' is deprecated, preferred is 'transport.socket.listen-port', >>>> continuing with correction >>>> [2016-09-27 16:16:28.109456] W [socket.c:3665:reconfigure] >>>> 0-cees-data-quota: NBIO on -1 failed (Bad file descriptor) >>>> [2016-09-27 16:16:28.489255] I [MSGID: 121050] >>>> [ctr-helper.c:259:extract_ctr_options] 0-gfdbdatastore: CTR Xlator is >>>> disabled. >>>> [2016-09-27 16:16:28.489272] W [MSGID: 101105] >>>> [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-cees-data-changetimerecorder: >>>> Failed to retrieve sql-db-pagesize from params.Assigning default value: >>>> 4096 >>>> [2016-09-27 16:16:28.489278] W [MSGID: 101105] >>>> [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-cees-data-changetimerecorder: >>>> Failed to retrieve sql-db-journalmode from params.Assigning default value: >>>> wal >>>> [2016-09-27 16:16:28.489284] W [MSGID: 101105] >>>> [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-cees-data-changetimerecorder: >>>> Failed to retrieve sql-db-sync from params.Assigning default value: off >>>> [2016-09-27 16:16:28.489288] W [MSGID: 101105] >>>> [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-cees-data-changetimerecorder: >>>> Failed to retrieve sql-db-autovacuum from params.Assigning default value: >>>> none >>>> [2016-09-27 16:16:28.490431] I [trash.c:2412:init] 0-cees-data-trash: >>>> no option specified for 'eliminate', using NULL >>>> [2016-09-27 16:16:28.672814] W [graph.c:357:_log_if_unknown_option] >>>> 0-cees-data-server: option 'rpc-auth.auth-glusterfs' is not recognized >>>> [2016-09-27 16:16:28.672854] W [graph.c:357:_log_if_unknown_option] >>>> 0-cees-data-server: option 'rpc-auth.auth-unix' is not recognized >>>> [2016-09-27 16:16:28.672872] W [graph.c:357:_log_if_unknown_option] >>>> 0-cees-data-server: option 'rpc-auth.auth-null' is not recognized >>>> [2016-09-27 16:16:28.672924] W [graph.c:357:_log_if_unknown_option] >>>> 0-cees-data-quota: option 'timeout' is not recognized >>>> [2016-09-27 16:16:28.672955] W [graph.c:357:_log_if_unknown_option] >>>> 0-cees-data-trash: option 'brick-path' is not recognized >>>> Final graph: >>>> +----------------------------------------------------------- >>>> -------------------+ >>>> 1: volume cees-data-posix >>>> 2: type storage/posix >>>> 3: option glusterd-uuid f04b231e-63f8-4374-91ae-17c0c623f165 >>>> 4: option directory /data/brick >>>> 5: option volume-id 27d2a59c-bdac-4f66-bcd8-e6124e53a4a2 >>>> 6: option update-link-count-parent on >>>> 7: end-volume >>>> 8: >>>> 9: volume cees-data-trash >>>> 10: type features/trash >>>> 11: option trash-dir .trashcan >>>> 12: option brick-path /data/brick >>>> 13: option trash-internal-op off >>>> 14: subvolumes cees-data-posix >>>> 15: end-volume >>>> 16: >>>> 17: volume cees-data-changetimerecorder >>>> 18: type features/changetimerecorder >>>> 19: option db-type sqlite3 >>>> 20: option hot-brick off >>>> 21: option db-name brick.db >>>> 22: option db-path /data/brick/.glusterfs/ >>>> 23: option record-exit off >>>> 24: option ctr_link_consistency off >>>> 25: option ctr_lookupheal_link_timeout 300 >>>> 26: option ctr_lookupheal_inode_timeout 300 >>>> 27: option record-entry on >>>> 28: option ctr-enabled off >>>> 29: option record-counters off >>>> 30: option ctr-record-metadata-heat off >>>> 31: option sql-db-cachesize 1000 >>>> 32: option sql-db-wal-autocheckpoint 1000 >>>> 33: subvolumes cees-data-trash >>>> 34: end-volume >>>> 35: >>>> 36: volume cees-data-changelog >>>> 37: type features/changelog >>>> 38: option changelog-brick /data/brick >>>> 39: option changelog-dir /data/brick/.glusterfs/changelogs >>>> 40: option changelog-barrier-timeout 120 >>>> 41: subvolumes cees-data-changetimerecorder >>>> 42: end-volume >>>> 43: >>>> 44: volume cees-data-bitrot-stub >>>> 45: type features/bitrot-stub >>>> 46: option export /data/brick >>>> 47: subvolumes cees-data-changelog >>>> 48: end-volume >>>> 49: >>>> 50: volume cees-data-access-control >>>> 51: type features/access-control >>>> 52: subvolumes cees-data-bitrot-stub >>>> 53: end-volume >>>> 54: >>>> 55: volume cees-data-locks >>>> 56: type features/locks >>>> 57: subvolumes cees-data-access-control >>>> 58: end-volume >>>> 59: >>>> 60: volume cees-data-upcall >>>> 61: type features/upcall >>>> 62: option cache-invalidation off >>>> 63: subvolumes cees-data-locks >>>> 64: end-volume >>>> 65: >>>> 66: volume cees-data-io-threads >>>> 67: type performance/io-threads >>>> 68: subvolumes cees-data-upcall >>>> 69: end-volume >>>> 70: >>>> 71: volume cees-data-marker >>>> 72: type features/marker >>>> 73: option volume-uuid 27d2a59c-bdac-4f66-bcd8-e6124e53a4a2 >>>> 74: option timestamp-file /var/lib/glusterd/vols/cees-da >>>> ta/marker.tstamp >>>> 75: option quota-version 1 >>>> 76: option xtime off >>>> 77: option gsync-force-xtime off >>>> 78: option quota on >>>> 79: option inode-quota on >>>> 80: subvolumes cees-data-io-threads >>>> 81: end-volume >>>> 82: >>>> 83: volume cees-data-barrier >>>> 84: type features/barrier >>>> 85: option barrier disable >>>> 86: option barrier-timeout 120 >>>> 87: subvolumes cees-data-marker >>>> 88: end-volume >>>> 89: >>>> 90: volume cees-data-index >>>> 91: type features/index >>>> 92: option index-base /data/brick/.glusterfs/indices >>>> 93: subvolumes cees-data-barrier >>>> 94: end-volume >>>> 95: >>>> 96: volume cees-data-quota >>>> 97: type features/quota >>>> 98: option transport.socket.connect-path >>>> /var/run/gluster/quotad.socket >>>> 99: option transport-type socket >>>> 100: option transport.address-family unix >>>> 101: option volume-uuid cees-data >>>> 102: option server-quota on >>>> 103: option timeout 0 >>>> 104: option deem-statfs on >>>> 105: subvolumes cees-data-index >>>> 106: end-volume >>>> 107: >>>> 108: volume cees-data-worm >>>> 109: type features/worm >>>> 110: option worm off >>>> 111: subvolumes cees-data-quota >>>> 112: end-volume >>>> 113: >>>> 114: volume cees-data-read-only >>>> 115: type features/read-only >>>> 116: option read-only off >>>> 117: subvolumes cees-data-worm >>>> 118: end-volume >>>> 119: >>>> 120: volume /data/brick >>>> 121: type debug/io-stats >>>> 122: option log-level INFO >>>> 123: option latency-measurement off >>>> 124: option count-fop-hits off >>>> 125: subvolumes cees-data-read-only >>>> 126: end-volume >>>> 127: >>>> 128: volume cees-data-server >>>> 129: type protocol/server >>>> 130: option transport.socket.listen-port 49152 >>>> 131: option rpc-auth.auth-glusterfs on >>>> 132: option rpc-auth.auth-unix on >>>> 133: option rpc-auth.auth-null on >>>> 134: option rpc-auth-allow-insecure on >>>> 135: option transport.rdma.listen-port 49153 >>>> 136: option transport-type tcp,rdma >>>> 137: option auth.login./data/brick.allow >>>> 18ddaf4c-ad98-4155-9372-717eae718b4c >>>> 138: option auth.login.18ddaf4c-ad98-4155-9372-717eae718b4c.password >>>> 9e913e92-7de0-47f9-94ed-d08cbb130d23 >>>> 139: option auth.addr./data/brick.allow * >>>> 140: subvolumes /data/brick >>>> 141: end-volume >>>> 142: >>>> +----------------------------------------------------------- >>>> -------------------+ >>>> [2016-09-27 16:16:30.079541] I [login.c:81:gf_auth] 0-auth/login: >>>> allowed user names: 18ddaf4c-ad98-4155-9372-717eae718b4c >>>> [2016-09-27 16:16:30.079567] I [MSGID: 115029] >>>> [server-handshake.c:690:server_setvolume] 0-cees-data-server: accepted >>>> client from fs3-12560-2016/09/27-16:16:30:47674-cees-data-client-3-0-0 >>>> (version: 3.7.14) >>>> [2016-09-27 16:16:30.081487] I [login.c:81:gf_auth] 0-auth/login: >>>> allowed user names: 18ddaf4c-ad98-4155-9372-717eae718b4c >>>> [2016-09-27 16:16:30.081505] I [MSGID: 115029] >>>> [server-handshake.c:690:server_setvolume] 0-cees-data-server: accepted >>>> client from fs2-11709-2016/09/27-16:16:30:50047-cees-data-client-3-0-0 >>>> (version: 3.7.14) >>>> [2016-09-27 16:16:30.111091] I [login.c:81:gf_auth] 0-auth/login: >>>> allowed user names: 18ddaf4c-ad98-4155-9372-717eae718b4c >>>> [2016-09-27 16:16:30.111113] I [MSGID: 115029] >>>> [server-handshake.c:690:server_setvolume] 0-cees-data-server: accepted >>>> client from fs2-11701-2016/09/27-16:16:29:24060-cees-data-client-3-0-0 >>>> (version: 3.7.14) >>>> [2016-09-27 16:16:30.112822] I [login.c:81:gf_auth] 0-auth/login: >>>> allowed user names: 18ddaf4c-ad98-4155-9372-717eae718b4c >>>> [2016-09-27 16:16:30.112836] I [MSGID: 115029] >>>> [server-handshake.c:690:server_setvolume] 0-cees-data-server: accepted >>>> client from fs3-12552-2016/09/27-16:16:29:23041-cees-data-client-3-0-0 >>>> (version: 3.7.14) >>>> [2016-09-27 16:16:31.950978] I [login.c:81:gf_auth] 0-auth/login: >>>> allowed user names: 18ddaf4c-ad98-4155-9372-717eae718b4c >>>> [2016-09-27 16:16:31.950998] I [MSGID: 115029] >>>> [server-handshake.c:690:server_setvolume] 0-cees-data-server: accepted >>>> client from fs1-6721-2016/09/27-16:16:26:939991-cees-data-client-3-0-0 >>>> (version: 3.7.14) >>>> [2016-09-27 16:16:31.981977] I [login.c:81:gf_auth] 0-auth/login: >>>> allowed user names: 18ddaf4c-ad98-4155-9372-717eae718b4c >>>> [2016-09-27 16:16:31.981994] I [MSGID: 115029] >>>> [server-handshake.c:690:server_setvolume] 0-cees-data-server: accepted >>>> client from fs1-6729-2016/09/27-16:16:27:971228-cees-data-client-3-0-0 >>>> (version: 3.7.14) >>>> >>>> >>>> Hmm, this shows the brick has started. >>>> Does gluster volume info on fs4 shows all 4 bricks? (I guess it does >>>> based on your first email). >>>> Does gluster volume status on fs4 (or ps aux|grep glusterfsd) show the >>>> brick as running? >>>> Does gluster peer status on all nodes list the other 3 nodes as >>>> connected? >>>> >>>> If yes, you could try `service glusterd restart` on fs4 and see if if >>>> brings up the brick? I'm just shooting in the dark here for possible clues. >>>> -Ravi >>>> >>>> On Tue, Sep 27, 2016 at 8:46 AM, Ravishankar N <[email protected]> >>>> wrote: >>>> >>>>> On 09/27/2016 09:06 PM, Dennis Michael wrote: >>>>> >>>>> Yes, the brick log /var/log/glusterfs/bricks/data-brick.log is >>>>> created on fs4, and the snippets showing the errors were from that log. >>>>> >>>>> Unless I'm missing something, the snippet below is from glusterd's log >>>>> and not the brick's as is evident from the function names. >>>>> -Ravi >>>>> >>>>> Dennis >>>>> >>>>> On Mon, Sep 26, 2016 at 5:58 PM, Ravishankar N <[email protected] >>>>> > wrote: >>>>> >>>>>> On 09/27/2016 05:25 AM, Dennis Michael wrote: >>>>>> >>>>>>> [2016-09-26 22:44:39.254921] E [MSGID: 106005] >>>>>>> [glusterd-utils.c:4771:glusterd_brick_start] 0-management: Unable >>>>>>> to start brick fs4:/data/brick >>>>>>> [2016-09-26 22:44:39.254949] E [MSGID: 106074] >>>>>>> [glusterd-brick-ops.c:2372:glusterd_op_add_brick] 0-glusterd: >>>>>>> Unable to add bricks >>>>>>> >>>>>> >>>>>> Is the brick log created on fs4? Does it contain warnings/errors? >>>>>> >>>>>> -Ravi >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>> >>> >> >> >> -- >> >> --Atin >> > > > > -- > > --Atin >
_______________________________________________ Gluster-users mailing list [email protected] http://www.gluster.org/mailman/listinfo/gluster-users
