Mike raises some very good points (and a few that I don't agree with as well :) ), about using the virtualized setup. One important thing to keep in mind is that being able to scale shouldn't cause you to "over-scale" at the beginning. Start small and work your way up. Since both messages are included here, I've got replies going to two different people in some cases. Sorry for any confusion.
> I've been using Xen for almost a year now and we've been really happy with > it. We have >50 virtual servers running on three physical servers. Unless you have a lot of CPU cores and/or a REALLY fast SAN, you're overloading your hardware. While you *can* put a virtually unlimited number of domU's on a dom0, it doesn't necessarily mean that you *should*. :) I suggest keeping it to no more than one domU per CPU core unless you have a good reason to do otherwise. > # One server consuming too much CPU or RAM will not adversely affect the > others > > We don't want an app that chews up RAM to affect other services (like db). > We allocate a certain amount of RAM to each slice and can change the > allocation when required. See the statement above. Depending on how many CPU cores you have, one server consuming too much CPU time has the potential to adversely affect other servers. If it's financially feasible, it is much preferable to have *at least* one dedicated CPU core to each domU. > Nightly backups chew up CPU but our backup slice doesn't affect the other > slices running on the same server as CPU is shared fairly*. This is definitely an example of a *good* setup. We have a few applications that run some extremely CPU-intensive rake tasks, and those are run on separate domU's so that the primary application won't get bogged down. Note that you could accomplish the same thing by just tying processes on one domU to a particular processor, but that takes additional work. As a side note, though, if your dom0 has only one or two cores, the above wouldn't help you. The dom0 would still need to divide the processing time between machines which would end up slowing things down. > # We can manage disk more easily > > Isolation of disk between virtual servers is a benefit as it reduces the > damage a full disk would cause (our monitoring alerts us well before that > happens though!). This is largely a moot point. On a standalone server, you would have at *least* a RAID 1 setup to handle disk failures. If a disk did fail, you'd get a new one in place and off you go. The "disk" on a domU *will* *not* *fail*. Period. You might get a failed disk on your dom0, but the dom0 should either be talking to a SAN or be running a RAID 10 array, either of which would give you very good protection. Keep in mind that your filesystem and your disks are two distinct entities. While the disk on a domU won't fail, you could get a corrupt filesystem. But FS corruption is usually caused by a combination of physical problems with the disk and uncontrolled system shutdowns. The former is completely eliminated in a domU, and the latter is made extremely unlikely since your domU won't just "lose power". > # Security > > We expose nginx to the world, but our app servers, database, etc are on > private IPs. This reduces our exposure to attack. Also, putting different > services on different (virtual) servers means that if one service have an > vulnerability it doesn't necessarily mean that other others will be > vulnerable. If an exploit is found for your mailserver you don't want that > to mean your database gets owned. Specifically in regards to a mail server, unless your application needs to *receive* email directly, you have no business exposing it to the outside world in the first place. Every domU you set up should have an MTA installed (to allow things like ActionMailer to work seamlessly using :sendmail, for example), *but* that MTA should *never* accept connections from the outside world. It should only send email *out*. If you want to receive email from customers, go with a dedicated email setup where things like security are handled for you. Google Apps for domains is an excellent solution for this. If your application actually needs to receive emails then roll up your sleeves and be ready to waste a lot of time. Configuring a secure mailserver with a proper filtering setup takes a *lot* of time. That's why it should be avoided whenever possible. But to address the overall statement, once it is time to scale out to multiple domUs, you should divide them based on their role, not necessarily the software that they're running. For example, it's perfectly valid to have nginx *and* a mongrel cluster running on one machine, since working together they provide a full site. (nginx for static content, and the mongrels for dynamic content) You could, optionally, have another standalone machine running just nginx as a load balancer between the webservers. > There are a few things that I've still not achieved: > > # Migrating a live server from one physical server to another > > This requires a shared filesystem. I couldn't get GFS working on Ubuntu. I'm > wary of NFS as I hear it can be unstable under high load. I'd love to have > it but I'm still waiting for the HOWTO to be written. I would never run the filesystem for a domU off of NFS or any other network-based filesystem. What we do is share the filesystem by providing shared block devices over iSCSI. You can export individual LVs as iSCSI targets and connect to them via an initiator on your remote machine. Using a dedicated SAN is obviously a preferable solution, but if that's not an option this gives you a viable workaround. > # Sharing a filesystem between slices > > Same as above. This would be needed to allow separate app servers to handle > fileuploads. Considering what Fernando was actually trying to do here, my suggestion would be to not separate things out too much. In the case described, if you really *need* to have everything on a separate domU, I'd do a setup similar to what I outlined above, and set up one domU as your load balancer running nginx, and then set up each of your application domUs with nginx to allow them to serve static content from the application while still passing requests through to Thin. (or, optionally, design your application so that the static content isn't even part of your application, but a separate component that just gets deployed to your web server). > In response to your drawbacks: > > - Xen is an absolute pain to setup > > Once you know what you're doing it's pretty easy but it takes some learning. > Possibly not worth the time if you only have one app and don't need it to > scale. I agree with Mike here. I do this stuff for a living, so setting up a new dom0 takes me about an hour including the full OS install. But if it's your first time doing this, and if it's likely to be your only time, it would be a good idea to look at a Rails-specific managed hosting setup. We have a number of customers that need more in the way of resources than our standard hosting products provide, so we do managed hosting for them. They buy the hardware and ship it to us, and we handle the rest. The Rails community in general has been in a position where a developer needs to be a sysadmin as well, and this just isn't right. We have enough smart people out there that developers should be able to focus on developing applications, and admins should keep those applications running. With that said, if you're going to be the admin supporting a team of developers, every minute spent with getting familiar with Xen is a minute well spent. > - Setting up each VM is tedious > > Check out xen-tools for creating the VMs. Check out deprec automating your > setup. I couldn't agree more! :) xen-tools is definitely the way to go. I've spent quite a bit of time tweaking our setup to get to this point, but it currently takes me about 2 minutes to get a new domU fully set up and running. About 30 seconds of that is me typing in a few parameters, and about 1:30 of that is the dom0 actually building the image. If you're going to be setting up a lot of domU's, this is definitely the way to go. > - Nginx can't talk to Thin over Unix sockets, and moreover you need to > setup NFS to share static files between both See the above recommendation on how to handle this by using multiple nginx instances. > - you must setup the DB to be remotely accessible, and cannot use Unix > socket for communication > > You can make it remotely accessible on a private subnet that nginx can > access. I can't see any problem with this. Again, if you don't yet *need* to have your application split out to multiple domUs, don't. Just let it run on the same machine and let it use sockets. However, once it comes time to move these responsibilities out to individual domUs, Mike's suggestion is perfectly valid. You should always have a private subnet for your VM's to communicate over. > - actually upgrading the server is not easy as moving the VMs on another > box, there will still be a good downtime. > > Yep, putting more RAM into the physical server requires me to schedule > downtime for all the slices. I was sold on the "live migration" but it's > never become a reality for me (yet). Still, it's no worse than without > virtualization. I've learnt to fill it with RAM when we buy it. :-) Without a SAN, you're going to have downtime. One solution for minimizing downtime is using an rsync program that can handle block devices (the stock rsync can't do this). Then what you do is a copy of the disk while the machine is running (This is obviously going to be an invalid copy). Then you take the machine offline, do a quick sync (usually < 5 minutes) and bring the domU back up on the new machine. If the machines you're moving between have an identical hardware/ software setup (CPU and Xen version must be absolutely identical) you can shave off a few seconds by pausing the domU and copying it over to the new server during the sync. (the image will be only as big as the amount of RAM on the domU) This way you avoid the shutdown/startup delay. But unless you have processes that absolutely *must not* be stopped, you're much better off doing the regular shutdown/startup sequence. > Xen is cool. It can be useful. It may be that your needs are not currently > great enough to justify the investment of time getting to know it though. > > I'd love to hear from anyone who's got live migration working with Xen on > Ubuntu or Debian. We do them on a regular basis to balance out the load on our dom0s. All of our dom0s and domUs are running a (mostly) standard Debian install. The only real trick is setting up shared storage. Ideally, you'll have a real SAN to work with, but failing that setting up your dom0s as iSCSI initiators and targets works quite well. You can offload the processor and RAM usage to another machine with no downtime. -- Alex Malinovich Director of Deployment Services PLANET ARGON, LLC design // development // hosting http://www.planetargon.com http://www.the-love-shack.net [blog] +1 503 445 2457 +1 877 55 ARGON [toll free] +1 815 642 4068 [fax] --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Deploying Rails" group. To post to this group, send email to rubyonrails-deployment@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/rubyonrails-deployment?hl=en -~----------~----~----~----~------~----~------~--~---
