Re: [Gluster-devel] GlusterFS Example
Hello the thread Nathan Allen Stratton a écrit : Which version of glusterfs are you using? tla, pre ? (that issue has been fixed for a little while now, so if your using pre6 you shoulnd't have come across it) Yep, that was it, I was using 1.2, latest 1.3 fixes the problme. Indeed, AFR is not available in the 1.2 release. I'm really interested in your configuration and the issues you may have (or have not !) Please, keep us in touch ! Regards, Sébastien. ___ Gluster-devel mailing list Gluster-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] GlusterFS Example
On Mon, 30 Jul 2007, Matt Paine wrote: Hi Nathan, http://www.gluster.org/docs/index.php/GlusterFS_High_Availability_Storage_with_GlusterFS If anyone hasthe time for a few questions: I will do my best :) Thanks! 1) In the past, my server configuration had only the local volume as in http://www.gluster.org/docs/index.php/GlusterFS_Configuration_Example_for_Four_Bricks Your config looks like each server has its volume and all the others. This setup (i'm assuming you are talking about the high availability storage with glusterfs article) is specifically for high availability, I am referencing the first link: http://www.gluster.org/docs/index.php/GlusterFS_High_Availability_Storage_with_GlusterFS I understand that the 2nd link is talking about doing everything on one server, my question is why in the first link that looks like it is talking about 3 physical boxes and each config file lists all the volumes (mailspool-santa2-ds, santa3-ds) where in the 2nd link a server.vol file only has the local volume information. if a brick goes down then there needs to be redundancy built in to provide a fallback option. If it was a simple unify solution, and a brick goes down, then you will be missing files (hence, not highly available at all unless you can guarantee 100% uptime of all bricks under all circumstances - which as far as I know noone offers :) In that specific example, it has been chosen to provide every brick with a copy of the files. In this case two bricks could fail and the cluster would still be up and running :). Extending this to 10 bricks, then you could potentially have 9 bricks fail and it would still be working. Point being, GlusterFS is highly configurable, you can set it up any way you like, more redundancy (afr), or more space (unify) or both. Your choice. 2) Why do you use loopback IP vs the public IP in the first example? The first example introduces itself with this paragraph 8- In this article, I will show you how to configure GlusterFS to simulate a cluster with: * one single mount point * 4 storage nodes serving the files I said simulate since we will deploy such a cluster on only one computer, but you can as well set it up over physical different computers by changing the IP addresses. -8 Its only a simulation, run on one computer. If you wish to use the example over an actual cluster of computers, you will need to change the IP numbers accordingly. Understood, however in the fist link the example has the first brick as a 127 address and the other 2 bricks as 192.168.252 address. My guess is that it was a typo since when I looked at the page this morning the 172 address was corrected to be 192.168.242.2, consistent with the rest of the example. 3) Can I change option replicate *:3 to *:2 to just have two copies? I like having an extra copy, but in my application having 3 or 3 note or 4 for 4 node is a bit overkill. Yes. As I said its highly configurable. You can even choose to replicate only certain files. E.g. option replicate *.tmp:1,*.html:4,*.php:4,*.db:1,*:2 In this case any .tmp or .db files will not get replicated (will sit on the first afr brick only), all html and php files will be mirrored across the first four bricks of afr, and all other files will be mirrorred across the first two bricks. (note all on one line, note comma delimited, note also no spaces before or after the comma). Right, but is it possible to just tell the config you want to store 2 copies of everything without telling each system where the redundant copies should be stored? I.E. I am looking for something like RAID 6, where I have say 8 bricks with two copies of every file. If I just enter *:2 then the spare copies will just be on the first two bricks and since each 8 brick has the same amount of storage I will run out of space on the first two before the others vs the redundant copies distributed over the 8 bricks. 4) In my application the server is also the client, in my current config when one out of 3 servers is done I can no longer write, even tho I am telling my config to write to the local brick. Is there any way to fix this? If a note is down I understand how I would not have access to the files, but I don't want to take down the full cluster. I'm note sure sorry. Maybe one of the devs can clarify that. I know there was an issue with an earlier relase to do with AFR. When the first brick was down then afr was no longer available. Ah... That could be it. P.S. Our setup consists of 3 servers each with a 4 port video card. They continually capture video from the 4 inputs and save them in /share. Each box also acts as a video server accepting connections and reading any file in /share. I don't care much about redundancy, but I do care that /share has files from all servers, or all servers that are up
Re: [Gluster-devel] GlusterFS Example
On Tue, 31 Jul 2007, Matt Paine wrote: Absolutly, except its there is a bit more work to do. AFR works by starting at the first brick and any replicating goes to the second, third etc as described above. But these are normal gluster bricks, so there is nothing stopping you from creating an AFR for the first brick, and AFR for the second, and just a posix brick for the rest (if thats what you want of course). Let me try to explain. Wow, you should put this on the wiki, this is a very clear example. I can see how this can work, but does it need to be that hard to setup? How hard would it be to add functionality to AFR where you only need to specify the number of copies you want and the scheduler takes care of the rest? Server one has one brick: S1 Server two has one brick: S2 Server three has one brick: S3 Server four has one brick: S4 Server five has one brick: S5 Client unifies S1 and S2 (call him U1) Client unifies S3 and S4 (call him U2) Client then AFR's together U1 and U2 and S5 In this example, if I save a file and its *:3, it will appear * on U1 (either S1 or S2 depending on scheduler) * on U2 (either S3 or S4 depending on sched.) * on S5 If I save a file as only *:1, it will appear only on U1 (either S1 or S2 depending on scheduler). Ad nausium. Of coures there is nothing stopping you from unifying three bricks, or even unifying an afr to afr. i.e. (might need a mono spaced font to see correctly...) I use Pine (old text based mail reader) so it looks great! A5 +---+ | | U1 U2 +-+ +-+ | | | | A1A2 A3A4 +---+---+ +---+---+---+ +---+ +---+---+ | | | | | | | | | | | | S01 S02 S03 S04 S05 S06 S07 S08 S09 S10 S11 S12 Where Sxx = Server bricks Ax = AFR brick Ux = Unify brick So in this configuration (which you have already worked out i'm sure) that if you save something *:2, then it will appear in both U1 and U2, which means (depending on the spec from a[1-4], assume *:2) it will appear in either A1 or A2 (because of the unify), AND it will also appear in either A3 or A4. etc etc etc. I think i've laboured the point far enough :) Thanks, it was great! Which version of glusterfs are you using? tla, pre ? (that issue has been fixed for a little while now, so if your using pre6 you shoulnd't have come across it) Yep, that was it, I was using 1.2, latest 1.3 fixes the problme. Nathan Stratton CTO, Voila IP Communications nathan at robotics.net nathan at voilaip.com http://www.robotics.net http://www.voilaip.com ___ Gluster-devel mailing list Gluster-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] GlusterFS Example
Client unifies S1 and S2 (call him U1) Client unifies S3 and S4 (call him U2) Client then AFR's together U1 and U2 and S5 In this example, if I save a file and its *:3, it will appear * on U1 (either S1 or S2 depending on scheduler) * on U2 (either S3 or S4 depending on sched.) * on S5 Is that supported? I was trying to do something similiar earlier, but was told that AFR is under(is a subvolume of) unify since I was having too many problems. Can I create a couple unifies, and then have both U1 and U2 as subvolumes of AFR? If so, let me know, and I can test it. Can I do something like the following? BrickS1-1 BrickS1-2 Unify US1 subvolumes S1-1,S1-2 BrickS2-1 BrickS2-2 Unify US2 subvolumes S2-1,S2-2 AFR subvolumes US1, US2 replicate *:2 From: Matt Paine [EMAIL PROTECTED] To: gluster-devel@nongnu.org Subject: Re: [Gluster-devel] GlusterFS Example Date: Tue, 31 Jul 2007 09:15:32 +1000 option replicate *.tmp:1,*.html:4,*.php:4,*.db:1,*:2 In this case any .tmp or .db files will not get replicated (will sit on the first afr brick only), all html and php files will be mirrored across the first four bricks of afr, and all other files will be mirrorred across the first two bricks. (note all on one line, note comma delimited, note also no spaces before or after the comma). Right, but is it possible to just tell the config you want to store 2 copies of everything without telling each system where the redundant copies should be stored? I.E. I am looking for something like RAID 6, where I have say 8 bricks with two copies of every file. If I just enter *:2 then the spare copies will just be on the first two bricks and since each 8 brick has the same amount of storage I will run out of space on the first two before the others vs the redundant copies distributed over the 8 bricks. Absolutly, except its there is a bit more work to do. AFR works by starting at the first brick and any replicating goes to the second, third etc as described above. But these are normal gluster bricks, so there is nothing stopping you from creating an AFR for the first brick, and AFR for the second, and just a posix brick for the rest (if thats what you want of course). Let me try to explain. Server one has one brick: S1 Server two has one brick: S2 Server three has one brick: S3 Server four has one brick: S4 Server five has one brick: S5 Client unifies S1 and S2 (call him U1) Client unifies S3 and S4 (call him U2) Client then AFR's together U1 and U2 and S5 In this example, if I save a file and its *:3, it will appear * on U1 (either S1 or S2 depending on scheduler) * on U2 (either S3 or S4 depending on sched.) * on S5 If I save a file as only *:1, it will appear only on U1 (either S1 or S2 depending on scheduler). Ad nausium. Of coures there is nothing stopping you from unifying three bricks, or even unifying an afr to afr. i.e. (might need a mono spaced font to see correctly...) A5 +---+ | | U1 U2 +-+ +-+ | | | | A1A2 A3A4 +---+---+ +---+---+---+ +---+ +---+---+ | | | | | | | | | | | | S01 S02 S03 S04 S05 S06 S07 S08 S09 S10 S11 S12 Where Sxx = Server bricks Ax = AFR brick Ux = Unify brick So in this configuration (which you have already worked out i'm sure) that if you save something *:2, then it will appear in both U1 and U2, which means (depending on the spec from a[1-4], assume *:2) it will appear in either A1 or A2 (because of the unify), AND it will also appear in either A3 or A4. etc etc etc. I think i've laboured the point far enough :) 4) In my application the server is also the client, in my current config when one out of 3 servers is done I can no longer write, even tho I am telling my config to write to the local brick. Is there any way to fix this? If a note is down I understand how I would not have access to the files, but I don't want to take down the full cluster. I'm note sure sorry. Maybe one of the devs can clarify that. I know there was an issue with an earlier relase to do with AFR. When the first brick was down then afr was no longer available. Ah... That could be it. Which version of glusterfs are you using? tla, pre ? (that issue has been fixed for a little while now, so if your using pre6 you shoulnd't have come across it) Matt. ___ Gluster-devel mailing list Gluster-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/gluster-devel _ http://newlivehotmail.com ___ Gluster-devel mailing list Gluster-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] GlusterFS Example
Is that supported? I was trying to do something similiar earlier, but was told that AFR is under(is a subvolume of) unify since I was having too many problems. Can I create a couple unifies, and then have both U1 and U2 as subvolumes of AFR? If so, let me know, and I can test it. The beuaty about Gluster is you can try any configuration you like and see if it suites your purpose. Give it a go and see how it reacts. I cant see any issues with doing this, but an actual developer would be better at saying definitivly yes or no. Give it a go and let us know your experiences. Can I do something like the following? BrickS1-1 BrickS1-2 Unify US1 subvolumes S1-1,S1-2 BrickS2-1 BrickS2-2 Unify US2 subvolumes S2-1,S2-2 AFR subvolumes US1, US2 replicate *:2 Thats how I would do it. But give it a go and see if it works for yourself. The fantastic thing about the modularity and flexability of the xlators is that it can be used in combinations that even the developers didn't imagine. Matt. ___ Gluster-devel mailing list Gluster-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] GlusterFS Example
Hi Nathan Wow, you should put this on the wiki, this is a very clear example. I can see how this can work, but does it need to be that hard to setup? How hard would it be to add functionality to AFR where you only need to specify the number of copies you want and the scheduler takes care of the rest? Possibly - i'll let the devs work that out - for the moment this is how I would do it. Personally i think AFR works great and I woudln't want to add any more complexity to it (afr does not do scheduling). If the devs think its a good idea i'm sure they will go for it. At the moment it might be a pain, but we are working on a management interface which will make things extremely easy (point and click in a web browser, fantastic :) Matt ___ Gluster-devel mailing list Gluster-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] GlusterFS Example
Hi Nathan, http://www.gluster.org/docs/index.php/GlusterFS_High_Availability_Storage_with_GlusterFS If anyone hasthe time for a few questions: I will do my best :) 1) In the past, my server configuration had only the local volume as in http://www.gluster.org/docs/index.php/GlusterFS_Configuration_Example_for_Four_Bricks Your config looks like each server has its volume and all the others. This setup (i'm assuming you are talking about the high availability storage with glusterfs article) is specifically for high availability, if a brick goes down then there needs to be redundancy built in to provide a fallback option. If it was a simple unify solution, and a brick goes down, then you will be missing files (hence, not highly available at all unless you can guarantee 100% uptime of all bricks under all circumstances - which as far as I know noone offers :) In that specific example, it has been chosen to provide every brick with a copy of the files. In this case two bricks could fail and the cluster would still be up and running :). Extending this to 10 bricks, then you could potentially have 9 bricks fail and it would still be working. Point being, GlusterFS is highly configurable, you can set it up any way you like, more redundancy (afr), or more space (unify) or both. Your choice. 2) Why do you use loopback IP vs the public IP in the first example? The first example introduces itself with this paragraph 8- In this article, I will show you how to configure GlusterFS to simulate a cluster with: * one single mount point * 4 storage nodes serving the files I said simulate since we will deploy such a cluster on only one computer, but you can as well set it up over physical different computers by changing the IP addresses. -8 Its only a simulation, run on one computer. If you wish to use the example over an actual cluster of computers, you will need to change the IP numbers accordingly. 3) Can I change option replicate *:3 to *:2 to just have two copies? I like having an extra copy, but in my application having 3 or 3 note or 4 for 4 node is a bit overkill. Yes. As I said its highly configurable. You can even choose to replicate only certain files. E.g. option replicate *.tmp:1,*.html:4,*.php:4,*.db:1,*:2 In this case any .tmp or .db files will not get replicated (will sit on the first afr brick only), all html and php files will be mirrored across the first four bricks of afr, and all other files will be mirrorred across the first two bricks. (note all on one line, note comma delimited, note also no spaces before or after the comma). 4) In my application the server is also the client, in my current config when one out of 3 servers is done I can no longer write, even tho I am telling my config to write to the local brick. Is there any way to fix this? If a note is down I understand how I would not have access to the files, but I don't want to take down the full cluster. I'm note sure sorry. Maybe one of the devs can clarify that. I know there was an issue with an earlier relase to do with AFR. When the first brick was down then afr was no longer available. P.S. Our setup consists of 3 servers each with a 4 port video card. They continually capture video from the 4 inputs and save them in /share. Each box also acts as a video server accepting connections and reading any file in /share. I don't care much about redundancy, but I do care that /share has files from all servers, or all servers that are up anyway. My server.vol file, only change is the bind IP address. volume brick type storage/posix option directory /md0 end-volume volume server type protocol/server option transport-type tcp/server option listen-port 6996 option bind-address 192.168.0.60 subvolumes brick option auth.ip.brick.allow 192.168.0.* end-volume My client.vol file, the only change is option nufa.local-volume-name client0 changes to client1 or client2 for each box. The above works as long as all 3 servers are up, if one is down I can no longer read or write to /share. I found that I could continue to write to /md0 on each server and it still was replicated correctly, however if any server is down reads on /share would break. : ( Could you post your client spec file? That would make it clearer to me. I have a feeling you are using separate client files for each of the clients, perhaps it would be easier for you to use a single spec file for your entire cluster, but we could clarify that if you post your client file :) Any help would rock! I hope that rocked at least a little :p Matt. ___ Gluster-devel mailing list Gluster-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/gluster-devel
