On 17/08/11 16:19, Dan Bretherton wrote:
Dan Bretherton wrote:
On 15/08/11 20:00, [email protected] wrote:
Message: 1
Date: Sun, 14 Aug 2011 23:24:46 +0300
From: "Deyan Chepishev - SuperHosting.BG"<[email protected]>
Subject: [Gluster-users] cluster.min-free-disk separate for each
brick
To: [email protected]
Message-ID:<[email protected]>
Content-Type: text/plain; charset=UTF-8; format=flowed
Hello,
I have a gluster set up with very different brick sizes.
brick1: 9T
brick2: 9T
brick3: 37T
with this configuration if I set the parameter
cluster.min-free-disk to 10% it
applies to all bricks which is quite uncomfortable with these brick
sizes,
because 10% for the small bricks are ~ 1T but for the big brick it
is ~3.7T and
what happens at the end is that if all brick go to 90% usage and I
continue
writing, the small ones eventually fill up to 100% while the big
one has enough
free space.
My question is, is there a way to set cluster.min-free-disk per
brick instead
setting it for the entire volume or any other way to work around
this problem ?
Thank you in advance
Regards,
Deyan
Hello Deyan,
I have exactly the same problem and I have asked about it before -
see links below.
http://community.gluster.org/q/in-version-3-1-4-how-can-i-set-the-minimum-amount-of-free-disk-space-on-the-bricks/
http://gluster.org/pipermail/gluster-users/2011-May/007788.html
My understanding is that the patch referred to in Amar's reply in
the May thread prevents a "migrate-data" rebalance operation failing
by running out of space on smaller bricks, but that doesn't solve
the problem we are having. Being able to set min-free-disk for each
brick separately would be useful, as would being able to set this
value as a number of bytes rather than a percentage. However, even
if these features were present we would still have a problem when
the amount of free space becomes less than min-free-disk, because
this just results in a warning message in the logs and doesn't
actually prevent more files from being written. In other words,
min-free-disk is a soft limit rather than a hard limit. When a
volume is more than 90% full there may still be hundreds of
gigabytes of free space spread over the large bricks, but the small
bricks may each only have a few gigabytes left of even less. Users
do "df" and see lots of free space in the volume so they continue
writing files. However, when GlusterFS chooses to write a file to a
small brick, the write fails with "device full" errors if the file
grows too large, which is often the case here with files typically
several gigabytes in size for some applications.
I would really like to know if there is a way to make min-free-disk
a hard limit. Ideally, GlusterFS would chose a brick on which to
write a file based on how much free space it has left rather than
choosing a brick at random (or however it is done now). That would
solve the problem of non-uniform brick sizes without the need for a
hard min-free-disk limit.
Amar's comment in the May thread about QA testing being done only on
volumes with uniform brick sizes prompted me to start standardising
on a uniform brick size for each volume in my cluster. My
impression is that implementing the features needed for users with
non-uniform brick sizes is not a priority for Gluster, and that
users are all expected to use uniform brick sizes. I really think
this fact should be stated clearly in the GlusterFS documentation,
in the sections on creating volumes in the Administration Guide for
example. That would stop other users from going down the path that
I did initially, which has given me a real headache because I am now
having to move tens of terabytes of data off bricks that are larger
than the new standard size.
Regards
Dan.
Hello,
This is really bad news, because I already migrated my data and I
just realized that I am screwed because Gluster just does not care
about the brick sizes.
It is impossible to move to uniform brick sizes.
Currently we use 2TB HDDs, but the disks are growing and soon we
will probably use 3TB hdds or whatever other larges sizes appear on
the market. So if we choose to use raid5 and some level of redundancy
(for example 6hdds in raid5, no matter what their size is) this
sooner or later will lead us to non uniform bricks which is a problem
and it is not correct to expect that we always can or want to provide
uniform size bricks.
With this way of thinking if we currently have 10T from 6x2T in hdd5,
at some point when there is a 10T on a single disk we will have to
use no raid just because gluster can not handle non uniform bricks.
Regards,
Deyan
I think Amar might have provided the answer in his posting to the
thread yesterday, which has just appeared in my autospam folder.
http://gluster.org/pipermail/gluster-users/2011-August/008579.html
With size option, you can have a hardbound on min-free-disk
This means that you can set a hard limit on min-free-disk, and set a
value in GB that is bigger than the biggest file that is ever likely
to be written. This looks likely to solve our problem and make
non-uniform brick sizes a practical proposition. I wish I had known
about this back in May when I embarked on my cluster restructuring
exercise; the issue was discussed in this thread in May as well:
http://gluster.org/pipermail/gluster-users/2011-May/007794.html
Once I have moved all the data off the large bricks and standardised
on a uniform brick size, it will be relatively easy to stick to this
because I use LVM. I create logical volumes for new bricks when a
volume needs extending. The only problem with this approach is what
happens when the amount of free space left on a server is less than
the size of the brick you want to create. The only option then would
be to use new servers, potentially wasting several TB of free space on
existing servers. The standard brick size for most of my volumes is
3TB, which allows me to use a mixture of small servers and large
servers in a volume and limits the amount of free space that would be
wasted if there wasn't quite enough free space on a server to create
another brick. Another consequence of having 3TB bricks is that a
single server typically has two more more bricks belonging to a the
same volume, although I do my best to distribute the volumes across
different servers in order to spread the load. I am not aware of any
problems associated with exporting multiple bricks from a single
server and it has not caused me any problems so far that I am aware of.
-Dan.
Hello Deyan,
Have you tried giving min-free-disk a value in gigabytes, and if so does
it prevent new files being written to your bricks when they are nearly
full? I recently tried it myself and found that min-free-disk had no
effect all. I deliberately filled my test/backup volume and most of the
bricks became 100 full. I set min-free-disk to "20GB", as reported in
"gluster volume ... info" below.
cluster.min-free-disk: 20GB
Unless I am doing something wrong it seems as though we can not "have a
hardbound on min-free-disk" after all, and uniform brick size is
therefore an essential requirement. It still doesn't say that in the
documentation, at least not in the volume creation sections.
-Dan.
--
Mr. D.A. Bretherton
Computer System Manager
Environmental Systems Science Centre
Harry Pitt Building
3 Earley Gate
University of Reading
Reading, RG6 6AL
UK
Tel. +44 118 378 5205
Fax: +44 118 378 6413
_______________________________________________
Gluster-users mailing list
[email protected]
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
