On 11/28/2013 12:52 PM, Patrick Haley wrote:
Hi Ravi,
Thanks for the reply. If I interpret the output of gluster volume status
correctly, glusterfsd was running
[root@mseas-data ~]# gluster volume status
Status of volume: gdata
Gluster process Port Online Pid
------------------------------------------------------------------------------
Brick gluster-0-0:/mseas-data-0-0 24009 Y 27006
Brick gluster-0-1:/mseas-data-0-1 24009 Y 7063
Brick gluster-data:/data 24009 Y 2897
NFS Server on localhost 38467 Y 2903
NFS Server on gluster-0-1 38467 Y 7069
NFS Server on gluster-0-0 38467 Y 27012
For completeness, I tried both "service glusterd restart" and
"gluster volume start gdata force". Neither solved the problem.
Note that after "gluster volume start gdata force" the gluster volume status
failed
[root@mseas-data ~]# gluster volume status
operation failed
Failed to get names of volumes
Doing another "service glusterd restart" let the "gluster volume status"
command work, but the clients still don't see the files on mseas-data.
Are your clients using fuse mounts or NFS mounts?
A second piece of data, on the other bricks, "gluster volume status"does not
show gluster-data:/data:
Hmm, could you check if all 3 bricks are connected ? `gluster peer
status` on each brick should show the others as connected.
[root@nas-0-0 ~]# gluster volume status
Status of volume: gdata
Gluster process Port Online Pid
------------------------------------------------------------------------------
Brick gluster-0-0:/mseas-data-0-0 24009 Y 27006
Brick gluster-0-1:/mseas-data-0-1 24009 Y 7063
NFS Server on localhost 38467 Y 27012
NFS Server on gluster-0-1 38467 Y 8051
Any thoughts on what I should look at next?
Also noticed the NFS server process on gluster-0-1 (on which I guess no
commands were run ) seems to have changed it's pid from 7069 to 8051.
FWIW, I am able to observe a similar bug
(https://bugzilla.redhat.com/show_bug.cgi?id=1035586) which needs to be
investigated.
Thanks,
Ravi
Thanks again.
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley Email:
[email protected]
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213
http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
________________________________________
From: Ravishankar N [[email protected]]
Sent: Wednesday, November 27, 2013 11:21 PM
To: Patrick Haley; [email protected]
Subject: Re: [Gluster-users] After reboot, one brick is not being seen by
clients
On 11/28/2013 03:12 AM, Pat Haley wrote:
Hi,
We are currently using gluster with 3 bricks. We just
rebooted one of the bricks (mseas-data, also identified
as gluster-data) which is actually the main server. After
rebooting this brick, our client machine (mseas) only sees
the files on the other 2 bricks. Note that if I mount
the gluster filespace (/gdata) on the brick I rebooted,
it sees the entire space.
The last time I had this problem, there was an error in
one of our /etc/hosts file. This does not seem to be the
case now.
What else can I look at to debug this problem?
Some information I have from the gluster server
[root@mseas-data ~]# gluster --version
glusterfs 3.3.1 built on Oct 11 2012 22:01:05
[root@mseas-data ~]# gluster volume info
Volume Name: gdata
Type: Distribute
Volume ID: eccc3a90-212d-4563-ae8d-10a77758738d
Status: Started
Number of Bricks: 3
Transport-type: tcp
Bricks:
Brick1: gluster-0-0:/mseas-data-0-0
Brick2: gluster-0-1:/mseas-data-0-1
Brick3: gluster-data:/data
[root@mseas-data ~]# ps -ef | grep gluster
root 2781 1 0 15:16 ? 00:00:00 /usr/sbin/glusterd -p
/var/run/glusterd.pid
root 2897 1 0 15:16 ? 00:00:00 /usr/sbin/glusterfsd
-s localhost --volfile-id gdata.gluster-data.data -p
/var/lib/glusterd/vols/gdata/run/gluster-data-data.pid -S
/tmp/e3eac7ce95e786a3d909b8fc65ed2059.socket --brick-name /data -l
/var/log/glusterfs/bricks/data.log --xlator-option
*-posix.glusterd-uuid=22f1102a-08e6-482d-ad23-d8e063cf32ed
--brick-port 24009 --xlator-option gdata-server.listen-port=24009
root 2903 1 0 15:16 ? 00:00:00 /usr/sbin/glusterfs -s
localhost --volfile-id gluster/nfs -p
/var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S
/tmp/d5c892de43c28a1ee7481b780245b789.socket
root 4258 1 0 15:52 ? 00:00:00 /usr/sbin/glusterfs
--volfile-id=/gdata --volfile-server=mseas-data /gdata
root 4475 4033 0 16:35 pts/0 00:00:00 grep gluster
[
From the ps output, the brick process (glusterfsd) doesn't seem to be
running on the gluster-data server. Run `gluster volume status` and
check if that is indeed the case. If yes, you could either restart
glusterd on the brick node (`service glusterd restart`) or restart the
entire volume (`gluster volume start gdata force`) which should bring
back the brick process online.
I'm not sure why glusterd did not start the brick process when you
rebooted the machine in the first place. You could perhaps check the
glusterd log for clues).
Hope this helps,
Ravi
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley Email: [email protected]
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
_______________________________________________
Gluster-users mailing list
[email protected]
http://supercolony.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
[email protected]
http://supercolony.gluster.org/mailman/listinfo/gluster-users
