On Fri, Feb 24, 2017 at 11:06 PM, Deepak Naidu <[email protected]> wrote:
> Thanks Rafi for workaround. > > *>> To find the root cause we need to get logs for the first failure of > volume start or volume stop .* > > Below, exact steps to re-produce the issue and attached log file contents > from /varlog/gluster folder(parsed) > > > > *STEPS to re-produce the issue* > > > > root@hostname:~# gluster volume create home-folder transport tcp,rdma > storageN1:/gluster/disk1/home-folder storageN2:/gluster/disk1/home-folder > > volume create: home-folder: success: please start the volume to access data > > root@hostname:~# > > root@hostname:~# gluster volume info home-folder > > Volume Name: home-folder > > Type: Distribute > > Volume ID: 09abd02a-b760-459f-afde-95b374eafc53 > > Status: Created > > Snapshot Count: 0 > > Number of Bricks: 2 > > Transport-type: tcp,rdma > > Bricks: > > Brick1: storageN1:/gluster/disk1/home-folder > > Brick2: storageN2:/gluster/disk1/home-folder > > Options Reconfigured: > > performance.readdir-ahead: on > > nfs.disable: on > > > > root@hostname:~# gluster volume status home-folder > > Volume home-folder is not started > > root@hostname:~# gluster volume start home-folder > > volume start: home-folder: *failed: Commit failed on localhost*. Please > check log file for details. > > root@hostname:~# gluster volume status home-folder > > Volume home-folder is not started > > > > root@hostname:~# gluster volume start home-folder force > > volume start: home-folder: success > > root@hostname:~# gluster volume status home-folder > > Status of volume: home-folder > > Gluster process TCP Port RDMA Port Online > Pid > > ------------------------------------------------------------ > ------------------ > > Brick storageN1:/gluster/disk1/home-folder N/A N/A Y > 17737 > > Brick storageN2:/gluster/disk1/home-folder N/A N/A N > N/A > > > > Task Status of Volume home-folder > > ------------------------------------------------------------ > ------------------ > > There are no active volume tasks > > root@hostname:~# gluster volume info home-folder > > Volume Name: home-folder > > Type: Distribute > > Volume ID: 09abd02a-b760-459f-afde-95b374eafc53 > > Status: Started > > Snapshot Count: 0 > > Number of Bricks: 2 > > Transport-type: tcp,rdma > > Bricks: > > Brick1: storageN1:/gluster/disk1/home-folder > > Brick2: storageN2:/gluster/disk1/home-folder > > Options Reconfigured: > > performance.readdir-ahead: on > > nfs.disable: on > > root@hostname:~# > > > > > > -- > > Deepak > > > > > > *From:* Mohammed Rafi K C [mailto:[email protected]] > *Sent:* Friday, February 24, 2017 1:08 AM > *To:* Deepak Naidu; [email protected] > *Subject:* Re: [Gluster-users] volume start: data0: failed: Commit failed > on localhost. > > > > It looks like it is ended up in split brain kind of situation. To find the > root cause we need to get logs for the first failure of volume start or > volume stop . > > Or to work around it, you can do a volume start force. > > > > Regards > > Rafi KC > > > > On 02/24/2017 01:36 PM, Deepak Naidu wrote: > > I keep on getting this error when my config.transport is set to both > tcp,rdma. The volume doesn’t start. I get the below error during volume > start. > > > > To get around this, I end up delete the volume, then configure either only > rdma or tcp. May be I am missing something, just trying to get the volume > up. > > > > root@hostname:~# gluster volume start data0 > > volume start: data0: failed: Commit failed on localhost. Please check log > file for details. > > root@hostname:~# > > > > root@ hostname:~# gluster volume status data0 > > Staging failed on storageN2. Error: Volume data0 is not started > > root@ hostname:~ > > > > ============= > > [2017-02-24 08:00:29.923516] I [MSGID: 106499] > [glusterd-handler.c:4349:__glusterd_handle_status_volume] > 0-management: Received status volume req for volume data0 > > [2017-02-24 08:00:29.926140] E [MSGID: 106153] > [glusterd-syncop.c:113:gd_collate_errors] > 0-glusterd: Staging failed on storageN2. Error: Volume data0 is not started > > [2017-02-24 08:00:33.770505] I [MSGID: 106499] > [glusterd-handler.c:4349:__glusterd_handle_status_volume] > 0-management: Received status volume req for volume data0 > > [2017-02-24 08:00:33.772824] E [MSGID: 106153] > [glusterd-syncop.c:113:gd_collate_errors] > 0-glusterd: Staging failed on storageN2. Error: Volume data0 is not started > > ============= > > [2017-02-24 08:01:36.305165] E [MSGID: 106537] [glusterd-volume-ops.c:1660: > glusterd_op_stage_start_volume] 0-management: Volume data0 already started > > [2017-02-24 08:01:36.305191] W [MSGID: 106122] > [glusterd-mgmt.c:198:gd_mgmt_v3_pre_validate_fn] 0-management: Volume > start prevalidation failed. > > [2017-02-24 08:01:36.305198] E [MSGID: 106122] > [glusterd-mgmt.c:884:glusterd_mgmt_v3_pre_validate] 0-management: Pre > Validation failed for operation Start on local node > > [2017-02-24 08:01:36.305205] E [MSGID: 106122] [glusterd-mgmt.c:2009: > glusterd_mgmt_v3_initiate_all_phases] 0-management: Pre Validation Failed > > What you are seeing is one of the side effect of BZ https://bugzilla.redhat.com/show_bug.cgi?id=1386578 & https://review.gluster.org/#/c/15687/ has been already posted for review. So in this case, although the volume status shows up that volume is not started, the brick process(es) actually do start. As a workaround please use volume start force one more time. @Rafi - We should try to get this patch in early and make it part of the next release. > > > > > -- > > Deepak > > > ------------------------------ > > This email message is for the sole use of the intended recipient(s) and > may contain confidential information. Any unauthorized review, use, > disclosure or distribution is prohibited. If you are not the intended > recipient, please contact the sender by reply email and destroy all copies > of the original message. > ------------------------------ > > > > > _______________________________________________ > > Gluster-users mailing list > > [email protected] > > http://lists.gluster.org/mailman/listinfo/gluster-users > > > > _______________________________________________ > Gluster-users mailing list > [email protected] > http://lists.gluster.org/mailman/listinfo/gluster-users > -- ~ Atin (atinm)
_______________________________________________ Gluster-users mailing list [email protected] http://lists.gluster.org/mailman/listinfo/gluster-users
