Re: Cheap VM cluster (software, storage) ideas
2009/3/5 Ira Abramov lists-linux...@ira.abramov.org Also OCFS and GFS require a common disk, which at this budget would be a Linux machine running an iSCSI target at best (or OpenNAS). Everything else said aside, we use DRBD in a master-slave configuration (with ext3) between pairs of xen guests for a couple of years now and it works great. We are experimenting with write-write and GFS but so far haven't had all the time needed to crack this. This is on CentOS 5.2 xne guests. CentOS 5.3 should bring GFS2 into production quality which might make things better. Cheers, --Amos ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Cheap VM cluster (software, storage) ideas
Quoting Ghiora Drori, from the post of Thu, 05 Mar: Hi, s3 is not used for block storage!! local ec2 instances storage or EBS drives are used. Yeah, I see that now. The problem was I have never heard of EBS till you email (not that I could find much about them, but I understand they are somewhat like an external storage you see as any other disk on the bus, only you get the ability to share it among many VMs. Now THAT is interesting. cheap common storage is a real boon and I can use that for smarter hosting of clusters without shelling out for an iSCSI or FC machine... All that is left to decide is which clustered FS is the fastest and most reliable that I can use with this setup. As to cost if you are using 4 servers it really does not matter. If you are you are using a 100 and there number can drop to 50 or go up depending on what you are doing the savings are significant. true, but this is not the case at the moment, unless I give each virtual web host a separate unique VM, but that would be insane. Koan: if apache runs just one website, but inside a VM. Is it still a Virtual host? Ira. On Thu, Mar 5, 2009 at 10:25 AM, Ira Abramov lists-linux...@ira.abramov.org wrote: Quoting Ghiora Drori, from the post of Wed, 04 Mar: Hi, Why would s3 bother you ? Its there it does not cost money if you do not use it it cost fairly little when you do and it works ok. Well, I was told it's not accassible as a simple filesystem, which means it won't work as shared storage for my current set of apps (various PHP and other tools expecting to find plain files in the directories they put them in... The only thing you will need s3 for is to store images of systems you want to launch, you could use public images but the cost of keeping private onse in s3 is negligible and this gives much more flexiblity in creating custom servers. Well, the hosting guy has 5 very different servers hosted right now. a winXP for Marcom, a couple of production Debian LAMPs, a test LAMP and a spare machine for sensitive sites. I think we are talking about too many details to try and just dump them P2V on a cloud and hope for the best... You do have to learn how to handle it aka instances can die and then you loose their disk this has been rare lately, was more common a few month ago. The EBS drives however do not die with the machine so data are you trying to cheer me up? :) I have machines with 500-600 days of uptime here, I don't need to move a bunch of Israeli sites to a far away cloud that occasionally has a lightning storm as well. You can put mysql, or any other software in images or load them when the instance is up. You can run RedHat, Gentoo, Ubuntu or even Windows... You can scale up and down the number of your servers and EBS disks as needed. If your servers were built for clustering in the first place, maybe. This is not a case of Drag'n'drop, as you can understand :) Make an account and play with it!! A small machine/instance (32 bits) is 10 cents an hour. If you cannot afford that you are not really commercial :) A few minor calculations should give you the correct cost numbers. 10 cents an hour are $2.4 a day or about $75 a month. this is more or less what the guy is paying in Israel now, and he gets less latency (all Hebrew sites and wanted only by Israelis), more disk space, etc. I guess the only real plus of histing the apps there is the Fun in the SAN. As sysadmin using it in a real web company it is very good, has a learning curve like everything else. yeah, only he's expecting definite answers from me, and Now I have to talk him into doing a pilot because I don't have all the correct answers. But that's the way the Internet works, right? :-) Cheers, Ira. -- Santa's little helper Ira Abramov http://ira.abramov.org/email/ ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il -- Constant change is here to stay! ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il -- Another word for awesome Ira Abramov http://ira.abramov.org/email/ ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Cheap VM cluster (software, storage) ideas
Quoting Ghiora Drori, from the post of Wed, 04 Mar: Hi, Why would s3 bother you ? Its there it does not cost money if you do not use it it cost fairly little when you do and it works ok. Well, I was told it's not accassible as a simple filesystem, which means it won't work as shared storage for my current set of apps (various PHP and other tools expecting to find plain files in the directories they put them in... The only thing you will need s3 for is to store images of systems you want to launch, you could use public images but the cost of keeping private onse in s3 is negligible and this gives much more flexiblity in creating custom servers. Well, the hosting guy has 5 very different servers hosted right now. a winXP for Marcom, a couple of production Debian LAMPs, a test LAMP and a spare machine for sensitive sites. I think we are talking about too many details to try and just dump them P2V on a cloud and hope for the best... You do have to learn how to handle it aka instances can die and then you loose their disk this has been rare lately, was more common a few month ago. The EBS drives however do not die with the machine so data are you trying to cheer me up? :) I have machines with 500-600 days of uptime here, I don't need to move a bunch of Israeli sites to a far away cloud that occasionally has a lightning storm as well. You can put mysql, or any other software in images or load them when the instance is up. You can run RedHat, Gentoo, Ubuntu or even Windows... You can scale up and down the number of your servers and EBS disks as needed. If your servers were built for clustering in the first place, maybe. This is not a case of Drag'n'drop, as you can understand :) Make an account and play with it!! A small machine/instance (32 bits) is 10 cents an hour. If you cannot afford that you are not really commercial :) A few minor calculations should give you the correct cost numbers. 10 cents an hour are $2.4 a day or about $75 a month. this is more or less what the guy is paying in Israel now, and he gets less latency (all Hebrew sites and wanted only by Israelis), more disk space, etc. I guess the only real plus of histing the apps there is the Fun in the SAN. As sysadmin using it in a real web company it is very good, has a learning curve like everything else. yeah, only he's expecting definite answers from me, and Now I have to talk him into doing a pilot because I don't have all the correct answers. But that's the way the Internet works, right? :-) Cheers, Ira. -- Santa's little helper Ira Abramov http://ira.abramov.org/email/ ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Cheap VM cluster (software, storage) ideas
Hi, s3 is not used for block storage!! local ec2 instances storage or EBS drives are used. I have instances running for 485 days. These are from the time we started using Amazon which was in beta. The fact that during the beta they some issued is not a big surprise. As to cost if you are using 4 servers it really does not matter. If you are you are using a 100 and there number can drop to 50 or go up depending on what you are doing the savings are significant. Being able to setup more servers to test a large new installation and then discard them paying only for actual usage gives you a lot of flexibility. On Thu, Mar 5, 2009 at 10:25 AM, Ira Abramov lists-linux...@ira.abramov.org wrote: Quoting Ghiora Drori, from the post of Wed, 04 Mar: Hi, Why would s3 bother you ? Its there it does not cost money if you do not use it it cost fairly little when you do and it works ok. Well, I was told it's not accassible as a simple filesystem, which means it won't work as shared storage for my current set of apps (various PHP and other tools expecting to find plain files in the directories they put them in... The only thing you will need s3 for is to store images of systems you want to launch, you could use public images but the cost of keeping private onse in s3 is negligible and this gives much more flexiblity in creating custom servers. Well, the hosting guy has 5 very different servers hosted right now. a winXP for Marcom, a couple of production Debian LAMPs, a test LAMP and a spare machine for sensitive sites. I think we are talking about too many details to try and just dump them P2V on a cloud and hope for the best... You do have to learn how to handle it aka instances can die and then you loose their disk this has been rare lately, was more common a few month ago. The EBS drives however do not die with the machine so data are you trying to cheer me up? :) I have machines with 500-600 days of uptime here, I don't need to move a bunch of Israeli sites to a far away cloud that occasionally has a lightning storm as well. You can put mysql, or any other software in images or load them when the instance is up. You can run RedHat, Gentoo, Ubuntu or even Windows... You can scale up and down the number of your servers and EBS disks as needed. If your servers were built for clustering in the first place, maybe. This is not a case of Drag'n'drop, as you can understand :) Make an account and play with it!! A small machine/instance (32 bits) is 10 cents an hour. If you cannot afford that you are not really commercial :) A few minor calculations should give you the correct cost numbers. 10 cents an hour are $2.4 a day or about $75 a month. this is more or less what the guy is paying in Israel now, and he gets less latency (all Hebrew sites and wanted only by Israelis), more disk space, etc. I guess the only real plus of histing the apps there is the Fun in the SAN. As sysadmin using it in a real web company it is very good, has a learning curve like everything else. yeah, only he's expecting definite answers from me, and Now I have to talk him into doing a pilot because I don't have all the correct answers. But that's the way the Internet works, right? :-) Cheers, Ira. -- Santa's little helper Ira Abramov http://ira.abramov.org/email/ ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il -- Constant change is here to stay! ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Cheap VM cluster (software, storage) ideas
related to the Citrix-vs-VMware question, in the spirit of the times... I want to create a way to host a cheap HA solution for a web hosting outfit. they are running a few pretty busy asymetric servers and want to start improving that infrastructure. each machine holds several dozens of virtual hosts. At the moment each server has its own local storage and mysql. every part is a SPOF other than the minimal RAID and such things. I'm thinking: * Move to a central non-virtual MySQL for the backend. * have two servers go P2V and have those VMs hosted back on their original hardwares (sadly this means some painful downtime), and find a way to let them crash-migrate for HA (still trying to figure this out) * Second stage, add a second MySQL in a master-master setup. I'd love to have two servers with symetrical setup, but as you can guess, the virtual hots are dozens of different apps that are too expensive to go and rewrite for clusters at this point, with the issues of user-uploaded files having to be available to both Apaches, etc. Assuming we want the cheapest reliable solution, i.e. not a $6K-20K SAN and FC, I am looking for an easier solution (easier on the pocket at least). However NFS proved to be a disaster in such cases (high-load web services), OCFS has not been nice to me with any setup other than maybe Oracle clusters, and GFS also never ran smoothly in my tests. Also OCFS and GFS require a common disk, which at this budget would be a Linux machine running an iSCSI target at best (or OpenNAS). Am I missing something? Can this kind of reliability be achieved without shelling out big bucks? Of course, The other option is just separate the MySQL, have a third machine rsync the files of the two (non virtual) servers every few minutes and have the hosting farm's layer4 switch redirect to the fallback if something happens. Not very smart nor scalable, but does 70% of what we need till a bigger investment is required. your thoughts, as before, are welcome... Thanks, Ira. -- Can't catch me yet Ira Abramov http://ira.abramov.org/email/ ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Cheap VM cluster (software, storage) ideas
Hi, Amazon EC2 instances plus their EBS disks. Scalable, available, reliable (from my experience) and you can experiment for a few $'s a day. On Wed, Mar 4, 2009 at 11:06 AM, Ira Abramov lists-linux...@ira.abramov.org wrote: related to the Citrix-vs-VMware question, in the spirit of the times... I want to create a way to host a cheap HA solution for a web hosting outfit. they are running a few pretty busy asymetric servers and want to start improving that infrastructure. each machine holds several dozens of virtual hosts. At the moment each server has its own local storage and mysql. every part is a SPOF other than the minimal RAID and such things. I'm thinking: * Move to a central non-virtual MySQL for the backend. * have two servers go P2V and have those VMs hosted back on their original hardwares (sadly this means some painful downtime), and find a way to let them crash-migrate for HA (still trying to figure this out) * Second stage, add a second MySQL in a master-master setup. I'd love to have two servers with symetrical setup, but as you can guess, the virtual hots are dozens of different apps that are too expensive to go and rewrite for clusters at this point, with the issues of user-uploaded files having to be available to both Apaches, etc. Assuming we want the cheapest reliable solution, i.e. not a $6K-20K SAN and FC, I am looking for an easier solution (easier on the pocket at least). However NFS proved to be a disaster in such cases (high-load web services), OCFS has not been nice to me with any setup other than maybe Oracle clusters, and GFS also never ran smoothly in my tests. Also OCFS and GFS require a common disk, which at this budget would be a Linux machine running an iSCSI target at best (or OpenNAS). Am I missing something? Can this kind of reliability be achieved without shelling out big bucks? Of course, The other option is just separate the MySQL, have a third machine rsync the files of the two (non virtual) servers every few minutes and have the hosting farm's layer4 switch redirect to the fallback if something happens. Not very smart nor scalable, but does 70% of what we need till a bigger investment is required. your thoughts, as before, are welcome... Thanks, Ira. -- Can't catch me yet Ira Abramov http://ira.abramov.org/email/ ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il -- Constant change is here to stay! ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Cheap VM cluster (software, storage) ideas
Quoting Ghiora Drori, from the post of Wed, 04 Mar: Hi, Amazon EC2 instances plus their EBS disks. Scalable, available, reliable (from my experience) and you can experiment for a few $'s a day. I tried to calculate the hosting costs, but was lost when I could not find the full explanation on how to calculate some of the parameters Amazon asked. Also it seemed the move to the EC2 would force me to use S3 and that was a deal breaker. this EBS feature is something I missed from the stroll in their site, so I suppose it's not something they are pushing hard yet. is it production-ready? can I stick a mysql server and all and just keep it all running as-is? As for hosting cost calculations - is there a tool you can recommend I can run on existing servers, or just a checklist to go over, that will help me judge if my system would make the move smoothly and at what price? I see so many conflicting suggestions on Google and I don't know which to trust, and have 0 time for useless trial and error. Also, knowing this client, I think he's not too happy about shooting his precious core business machines over the net to be hosted out of reach. It may be a psychological thing, but I can sympathize :-) On Wed, Mar 4, 2009 at 11:06 AM, Ira Abramov lists-linux...@ira.abramov.org wrote: related to the Citrix-vs-VMware question, in the spirit of the times... I want to create a way to host a cheap HA solution for a web hosting outfit. they are running a few pretty busy asymetric servers and want to start improving that infrastructure. each machine holds several dozens of virtual hosts. At the moment each server has its own local storage and mysql. every part is a SPOF other than the minimal RAID and such things. I'm thinking: * Move to a central non-virtual MySQL for the backend. * have two servers go P2V and have those VMs hosted back on their original hardwares (sadly this means some painful downtime), and find a way to let them crash-migrate for HA (still trying to figure this out) * Second stage, add a second MySQL in a master-master setup. I'd love to have two servers with symetrical setup, but as you can guess, the virtual hots are dozens of different apps that are too expensive to go and rewrite for clusters at this point, with the issues of user-uploaded files having to be available to both Apaches, etc. Assuming we want the cheapest reliable solution, i.e. not a $6K-20K SAN and FC, I am looking for an easier solution (easier on the pocket at least). However NFS proved to be a disaster in such cases (high-load web services), OCFS has not been nice to me with any setup other than maybe Oracle clusters, and GFS also never ran smoothly in my tests. Also OCFS and GFS require a common disk, which at this budget would be a Linux machine running an iSCSI target at best (or OpenNAS). Am I missing something? Can this kind of reliability be achieved without shelling out big bucks? Of course, The other option is just separate the MySQL, have a third machine rsync the files of the two (non virtual) servers every few minutes and have the hosting farm's layer4 switch redirect to the fallback if something happens. Not very smart nor scalable, but does 70% of what we need till a bigger investment is required. your thoughts, as before, are welcome... Thanks, Ira. -- Can't catch me yet Ira Abramov http://ira.abramov.org/email/ ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il -- Constant change is here to stay! ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il -- An out of body experiance Ira Abramov http://ira.abramov.org/email/ ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: Cheap VM cluster (software, storage) ideas
Hi, Why would s3 bother you ? Its there it does not cost money if you do not use it it cost fairly little when you do and it works ok. The only thing you will need s3 for is to store images of systems you want to launch, you could use public images but the cost of keeping private onse in s3 is negligible and this gives much more flexiblity in creating custom servers. EBS has been in production for quite a while (I also used it before it was in production.) So far after a few month no glitches. You can do anything you want with it software wise. You do have to learn how to handle it aka instances can die and then you loose their disk this has been rare lately, was more common a few month ago. The EBS drives however do not die with the machine so data on them is much safer, they are kept in 3 copies at Amazon in different locations. You can split your servers to 3 different zones for more reliability. You can use s3 for backups and it is probably much safer then anything else you have. You can put mysql, or any other software in images or load them when the instance is up. You can run RedHat, Gentoo, Ubuntu or even Windows... You can scale up and down the number of your servers and EBS disks as needed. Make an account and play with it!! A small machine/instance (32 bits) is 10 cents an hour. If you cannot afford that you are not really commercial :) A few minor calculations should give you the correct cost numbers. As sysadmin using it in a real web company it is very good, has a learning curve like everything else. On Wed, Mar 4, 2009 at 4:44 PM, Ira Abramov lists-linux...@ira.abramov.orgwrote: Quoting Ghiora Drori, from the post of Wed, 04 Mar: Hi, Amazon EC2 instances plus their EBS disks. Scalable, available, reliable (from my experience) and you can experiment for a few $'s a day. I tried to calculate the hosting costs, but was lost when I could not find the full explanation on how to calculate some of the parameters Amazon asked. Also it seemed the move to the EC2 would force me to use S3 and that was a deal breaker. this EBS feature is something I missed from the stroll in their site, so I suppose it's not something they are pushing hard yet. is it production-ready? can I stick a mysql server and all and just keep it all running as-is? As for hosting cost calculations - is there a tool you can recommend I can run on existing servers, or just a checklist to go over, that will help me judge if my system would make the move smoothly and at what price? I see so many conflicting suggestions on Google and I don't know which to trust, and have 0 time for useless trial and error. Also, knowing this client, I think he's not too happy about shooting his precious core business machines over the net to be hosted out of reach. It may be a psychological thing, but I can sympathize :-) On Wed, Mar 4, 2009 at 11:06 AM, Ira Abramov lists-linux...@ira.abramov.org wrote: related to the Citrix-vs-VMware question, in the spirit of the times... I want to create a way to host a cheap HA solution for a web hosting outfit. they are running a few pretty busy asymetric servers and want to start improving that infrastructure. each machine holds several dozens of virtual hosts. At the moment each server has its own local storage and mysql. every part is a SPOF other than the minimal RAID and such things. I'm thinking: * Move to a central non-virtual MySQL for the backend. * have two servers go P2V and have those VMs hosted back on their original hardwares (sadly this means some painful downtime), and find a way to let them crash-migrate for HA (still trying to figure this out) * Second stage, add a second MySQL in a master-master setup. I'd love to have two servers with symetrical setup, but as you can guess, the virtual hots are dozens of different apps that are too expensive to go and rewrite for clusters at this point, with the issues of user-uploaded files having to be available to both Apaches, etc. Assuming we want the cheapest reliable solution, i.e. not a $6K-20K SAN and FC, I am looking for an easier solution (easier on the pocket at least). However NFS proved to be a disaster in such cases (high-load web services), OCFS has not been nice to me with any setup other than maybe Oracle clusters, and GFS also never ran smoothly in my tests. Also OCFS and GFS require a common disk, which at this budget would be a Linux machine running an iSCSI target at best (or OpenNAS). Am I missing something? Can this kind of reliability be achieved without shelling out big bucks? Of course, The other option is just separate the MySQL, have a third machine rsync the files of the two (non virtual) servers every few minutes and have the hosting farm's layer4 switch redirect to the fallback if something happens. Not very smart nor scalable, but does 70% of what we need till a
