On 15/08/11 20:00, [email protected] wrote:
Message: 1
Date: Sun, 14 Aug 2011 23:24:46 +0300
From: "Deyan Chepishev - SuperHosting.BG"<[email protected]>
Subject: [Gluster-users] cluster.min-free-disk separate for each
brick
To: [email protected]
Message-ID:<[email protected]>
Content-Type: text/plain; charset=UTF-8; format=flowed
Hello,
I have a gluster set up with very different brick sizes.
brick1: 9T
brick2: 9T
brick3: 37T
with this configuration if I set the parameter cluster.min-free-disk to 10% it
applies to all bricks which is quite uncomfortable with these brick sizes,
because 10% for the small bricks are ~ 1T but for the big brick it is ~3.7T and
what happens at the end is that if all brick go to 90% usage and I continue
writing, the small ones eventually fill up to 100% while the big one has enough
free space.
My question is, is there a way to set cluster.min-free-disk per brick instead
setting it for the entire volume or any other way to work around this problem ?
Thank you in advance
Regards,
Deyan
Hello Deyan,
I have exactly the same problem and I have asked about it before - see
links below.
http://community.gluster.org/q/in-version-3-1-4-how-can-i-set-the-minimum-amount-of-free-disk-space-on-the-bricks/
http://gluster.org/pipermail/gluster-users/2011-May/007788.html
My understanding is that the patch referred to in Amar's reply in the
May thread prevents a "migrate-data" rebalance operation failing by
running out of space on smaller bricks, but that doesn't solve the
problem we are having. Being able to set min-free-disk for each brick
separately would be useful, as would being able to set this value as a
number of bytes rather than a percentage. However, even if these
features were present we would still have a problem when the amount of
free space becomes less than min-free-disk, because this just results in
a warning message in the logs and doesn't actually prevent more files
from being written. In other words, min-free-disk is a soft limit
rather than a hard limit. When a volume is more than 90% full there may
still be hundreds of gigabytes of free space spread over the large
bricks, but the small bricks may each only have a few gigabytes left of
even less. Users do "df" and see lots of free space in the volume so
they continue writing files. However, when GlusterFS chooses to write a
file to a small brick, the write fails with "device full" errors if the
file grows too large, which is often the case here with files typically
several gigabytes in size for some applications.
I would really like to know if there is a way to make min-free-disk a
hard limit. Ideally, GlusterFS would chose a brick on which to write a
file based on how much free space it has left rather than choosing a
brick at random (or however it is done now). That would solve the
problem of non-uniform brick sizes without the need for a hard
min-free-disk limit.
Amar's comment in the May thread about QA testing being done only on
volumes with uniform brick sizes prompted me to start standardising on a
uniform brick size for each volume in my cluster. My impression is that
implementing the features needed for users with non-uniform brick sizes
is not a priority for Gluster, and that users are all expected to use
uniform brick sizes. I really think this fact should be stated clearly
in the GlusterFS documentation, in the sections on creating volumes in
the Administration Guide for example. That would stop other users from
going down the path that I did initially, which has given me a real
headache because I am now having to move tens of terabytes of data off
bricks that are larger than the new standard size.
Regards
Dan.
--
Mr. D.A. Bretherton
Computer System Manager
Environmental Systems Science Centre
Harry Pitt Building
3 Earley Gate
University of Reading
Reading, RG6 6AL
UK
Tel. +44 118 378 5205
Fax: +44 118 378 6413
_______________________________________________
Gluster-users mailing list
[email protected]
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
