On 08/13/2013 06:21 PM, Cool wrote:
I'm pretty sure I did "watch ... remove-brick ... status" till it
mentioned everything is completed before trigger commit, I should make
it clear in my previous mail.
Actually you can read my mail again - in step #5, files on /sdc1 got
migrated instead of /sdd1, even though my command was trying to
remove-brick /sdd1,
Ah, my bad. Got it now. This is strange..
this is the root cause (to me) that caused the problem, as data on
/sdc1 migrated to /sdb1 and /sdd1, then commit simply remove /sdd1
from gfs_v0. It seems vol definition information got some problem in
gluster.
If you are able to reproduce the issue, does 'gluster volume info' show
the correct bricks before and after start-status-commit operations of
removing sdd1? You could also see if there are any error messages in
/var/log/glusterfs/<volname>-rebalance.log
-Ravi
-C.B.
On 8/12/2013 9:51 PM, Ravishankar N wrote:
On 08/13/2013 03:43 AM, Cool wrote:
remove-brick in 3.4.0 seems removing wrong bricks, can someone help
to review the environment/steps to see if I did anything stupid?
setup - Ubuntu 12.04LTS on gfs11 and gfs12, with following packages
from ppa, both nodes have 3 xfs partitions sdb1, sdc1, sdd1:
ii glusterfs-client 3.4.0final-ubuntu1~precise1 clustered
file-system (client package)
ii glusterfs-common 3.4.0final-ubuntu1~precise1 GlusterFS common
libraries and translator modules
ii glusterfs-server 3.4.0final-ubuntu1~precise1 clustered
file-system (server package)
step to reproduce the problem:
1. create volume gfs_v0 in replica 2 with gfs11:/sdb1 and gfs12:/sdb1
2. add-brick gfs11:/sdc1 and gfs12:/sdc1
3. add-brick gfs11:/sdd1 and gfs12:/sdd1
4. rebalance to make files distributed to all three pair of disks
5. remove-brick gfs11:/sdd1 and gfs12:/sdd1 start, files on
***/sdc1*** are migrating out
6. remove-brick commit led to data loss in gfs_v0
If between step 5 and 6 I initiate a remove-brick targeting /sdc1,
then after commit I would not lose anything since all data will be
migrated back to /sdb1.
You should ensure that a 'remove-brick start ' has completed and
then commit it before initiating the second one. The correct way to
do this would be:
5. # gluster volume remove-brick gfs_v0 gfs11:/sdd1 gfs12:/sdd1 start
6. Check that the data migration has been completed using the status
command:
# gluster volume remove-brick gfs_v0 gfs11:/sdd1 gfs12:/sdd1
status
7. #gluster volume remove-brick gfs_v0 gfs11:/sdd1 gfs12:/sdd1 commit
8. # gluster volume remove-brick gfs_v0 gfs11:/sdc1 gfs12:/sdc1 start
9. # gluster volume remove-brick gfs_v0 gfs11:/sdc1 gfs12:/sdc1 status
10. # gluster volume remove-brick gfs_v0 gfs11:/sdc1 gfs12:/sdc1 commit
This would leave you with the original replica 2 volume that you had
begun with. Hope this helps.
Note:
The latest version of glusterfs has the check that prevents a second
remove-brick operation until the first one has been committed.
(You would receive a message thus : "volume remove-brick start:
failed: An earlier remove-brick task exists for volume <volname>.
Either commit it or stop it before starting a new task." )
-Ravi
-C.B.
_______________________________________________
Gluster-users mailing list
[email protected]
http://supercolony.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
[email protected]
http://supercolony.gluster.org/mailman/listinfo/gluster-users
