We find the benefits gained by using shared /usr are probably not worth the additional effort. We started our Linux under VM environment by allocating full volumes to our Linux guest installs. This made for straight forward management and also provided a pretty simple backup environment but requires touching every guest to apply maintenence. After running a few months we thought a shared /usr would make this easier but found some problems with this in our environment. First we still find we are touching every guest to make sure the r/w files are in sync with the r/o /usr, second our methodology requires we cycle each r/o guest after we apply maintenence and third now that we allocate mount points to mini-disk rather than full volumes our backup strategy is extremely more difficult. With tight bugdets at this time we cannot afford automation tools to help so we will probably re-install all our r/o /usr guests back to full volume r/w environments. The savings in disk have not come anywhere near payback for the complexities created. Al Schilla State of Minnesota
-----Original Message----- From: Eric Sammons [mailto:[EMAIL PROTECTED] Sent: Sunday, January 25, 2004 9:49 AM To: [EMAIL PROTECTED] Subject: Re: Pros and Cons to sharing /usr Mark, I agree I have seen demos for both products and the more systems I have to create and maintain clearly the larger the benefit of such a product like DML or Levanta. However, the cloning scheme that I would implement would likely be one that involves no sharing but instead simply copying a gold image. While I don't save on DASD I do ensure that at a minimum I have a good foundation to build upon whether the system is to become a dB server or a web server or even an application server. All systems have to start from a common OS install with the same security settings and LDAP client configs etc. . . At this point DML or Levanta would certainly make maintaining the environment easy. My issue is with the concept of sharing filesystems, there I see more pain than profit. I would like to ask, as clearly you know these product solutions, which would you recommend? I have heard that Levanta is great but once in your environment it is near impossible to remove. I will add this, we already a fairly big user of BMC products. I am looking at these products for our environment to simplify the management and deployment of guests, but again I think I am going to work with the gold image model v. the shared model. On the note of your experience, I saw and noted you are from EDS. I will pass no judgement :-) (we have a lot of EDS folks in our environment working on different projects) I used to work for CSC and CTG. Honestly part of my issue here is I am 10+ years in on true distributed systems covering every Unix platform that exists (well except SGI) and this is the first time I have been exposed to Linux on the S/390 (well actually going on 6 mos. now) and I have found that the idea or concept is a great one; however, I still question its benefits in a production environment that is spread across three locations around the US. All applications require on-site and off-site contingency. And finally, with Sun hardware and in fact AMD and Intel hardware being more powerful (performance wise for certain tasks (this per IBM)) than the MF I have issues, certainly when we have applications that run in fully loaded V880s (8 processors + 32 GB of RAM). There is no way for such applications to be cost justified in Linux/390 (zVM) install. -- Enough of my off the point discussion. I am trying to keep an open mind, but on the issue of sharing binaries, I have seen too many times in Unix that a new enhancement to a binary requires a new configuration file. In binary sharing this will be difficult to maintain, simply from a knowing and tracking what configuration files changed and then have to be updated w/o loosing production operability. And as a simple example in Solaris, with every Maintenance release a file is updated in /etc that specifies the update level of the system. Please keep the feed back coming, I am trying to deploy a Linux environment, both 390 and x86 based, with more success than my predecessors deployed our Solaris environment. Thank you! Eric Sammons "Post, Mark K" <[EMAIL PROTECTED]> Sent by: Linux on 390 Port <[EMAIL PROTECTED]> 01/25/2004 12:27 AM Please respond to Linux on 390 Port To: [EMAIL PROTECTED] cc: Subject: Re: Pros and Cons to sharing /usr Until you take a look at Levanta and DML, you won't have any idea just what kind of flexibility and power they provide, and just how much manpower savings as well. It would only take saving one or two full time system administrators to justify either of the products. If you're looking at 100+ systems, that would be very easy to accomplish. And, contrary to what you're thinking, the more systems you have, the more savings products like this will realize for you. If you look at the domain name my emails come from, you might get some sort of clue as to what experience _I_ have in supporting multiple clients. Mark Post -----Original Message----- From: Linux on 390 Port [mailto:[EMAIL PROTECTED] Behalf Of Eric Sammons Sent: Saturday, January 24, 2004 5:09 PM To: [EMAIL PROTECTED] Subject: Re: Pros and Cons to sharing /usr As previously mentioned in a reply to another response, I still question the savings and I still don't see the benefit. Keeping configuration files and binaries in sync with each other seems like a full time job if cloning and /usr sharing were to be used. And contrary to your comment I would believe that the more systems you have the worst off you are. Today if I were to go through an upgrade process or a patch process I would simply apply my patches to each guest as I get scheduled time from my customers. And continue to proceed in that manner, coordinate with the customer a time to upgrade / patch their system. No all or nothing stuff. Perhaps I should explain my business, we service, well about 13 very unique customers. Each has their own application and environment requirements. Each has their own operating hours and SLAs. I am not an ISP, but I am a service provider. I simply provide the service of an environment where my customers pay me to run their applications in my environment. This environment is primarily web and websphere applications with of course back end databases. So with this said you can see that management of said environment will be a challenge. And each guest will be unique in some way, be it from the actual configuration or simply the SLA. In this scenario I do not believe that a shared filesystem will work when the risk of having a mismatched /usr and /etc is so great. And then should I run a shared /usr filesystem how do I ensure that /etc and /var are kept up to date. We must remember that /var contains the rpm database. WebSphere, IBM HTTPD, and IBM DB2 I believe all make use of the rpm database, what are the risks associated with having the package database so far out of whack. Installs may not work correctly because a prerequisite is falsely listed as failed, though there are ways around this, it seems like a huge risk. I appreciate the feedback please keep it coming. I have a planning meeting with IBM on Monday and I need to argue one way or the other. I am still not sold on the idea that cloning will work in my environment. And is DASD really that expensive? I know my time is. Also, is levanta not the product that once its in its almost impossible to get it out. I am not certain I like that idea. Eric Sammons "Post, Mark K" <[EMAIL PROTECTED]> Sent by: Linux on 390 Port <[EMAIL PROTECTED]> 01/24/2004 03:20 PM Please respond to Linux on 390 Port To: [EMAIL PROTECTED] cc: Subject: Re: Pros and Cons to sharing /usr Yes, these are all tough issues. The answer as to "why" is that, done correctly, it can significantly reduce the amount of people time to keep systems maintained, and reduce the (not insignificant) cost of DASD storage. Doing it yourself is hard, which is why Linuxcare has produced Levanta, and Aduva/BMC has their Deployment Manager for Linux. Neither one of them is cheap, in Intel terms, but for mainframe class software they're not bad. The IBM Linux Community Development System has just a couple/few people managing it, which at one point had 600 concurrent Linux/390 guests running. They were able to do this because of their extensive home-grown automation for system creation/deletion, sharing of /usr, etc. (Richard Lewis has given a couple presentations on this automation.) I think they're down to "only" 250 or so at the moment, but they didn't have to remove anyone from the support team because of the reduced workload. The system was just that scalable that the number of guests didn't affect how much work they needed to do. In the final analysis, if you're only going to be dealing with, say, less than 10 Linux/390 systems, it may not be worth your time to worry about the whole thing. If you get up into the 20-30-40 range, suddenly the savings get pretty big, pretty fast. Mark Post -----Original Message----- From: Linux on 390 Port [mailto:[EMAIL PROTECTED] Behalf Of Eric Sammons Sent: Saturday, January 24, 2004 9:28 AM To: [EMAIL PROTECTED] Subject: Pros and Cons to sharing /usr In today's Unix world we tend to put everything, almost, under /. With the elimination of the 2GB filesystem limit there hasn't been a need to split out every filesystem for quite some time. However, in looking at the cloning documentation there is a suggestion that /usr/src and /usr be shared as R/O. I am wondering what others have found to be the pros and cons to this? Also, is it possible to share other filesystems? I have heard my management state that the sharing of WebSphere binaries would be desired, I am personally not sure that that is possible. And to be honest I am not sure that the sharing of /usr is a good idea. Being experienced in Linux/x86 and other variants of Unix this is my reasoning for not liking the sharing /usr. There have been and will continue to be numerous times where an upgrade to the binaries (found in /usr) depend on a configuration file in /etc. /etc clearly is not a filesystem nor is it part /usr. This being the case my greatest fear is that /usr binaries would be upgraded while leaving legacy /etc configurations behind that may or more likely will not work. One example might be the going from inetd to xinetd. Other examples might be configuration parameters specific to one version of a product, OpenLDAP, may not work in a future version thus preventing the applicaiton from starting. And to make matters worst /var, having to be unique to each system has been known to see changes that are directly impacted by a newer version of a binary. Take Apache as an example when they moved the default DOCROOT from /home to /var. And because apache installs its configurations in /etc, there lies the biggest issue, the potential to change or touch /etc, /home, /var, and /usr. So my question is why would sharing /usr be a good idea? and how would you ensure that the configurations are up-to-date on all guests after an upgrade has been done to the master copy. And finally the entire sharing thing, unless planned very carefully, could result in an all or nothing situation. All guests get upgraded and it works or all guests get upgraded and nothing works. Of course with planning this can be addressed. Thoughts? Is anyone cloning? Yes, how are you addressing the /etc issues? Thanks! Eric Sammons
