Adam Tauno Williams wrote: > On Mon, 2010-03-22 at 10:05 -0400, Patrick Cable wrote: > >> On Mon, Mar 22, 2010 at 9:45 AM, Adam Tauno Williams >> <[email protected]> wrote: >> >>> 3.) DO NOT BASTARDIZE YOU SYSTEM STARTUP SCRIPTS. You will eventually >>> get hit by a bus or get a different job. If you hack your system's >>> standard scripts - like mounting /tmp without an fsck or recreating tmp >>> at book - you will break something, or something will break eventually. >>> And the guy who has to fix it will have to figure out whatever his >>> misguided predecessor did to ruin the system. NEVER BASTARDIZE YOUR >>> SYSTEMS STARTUP SCRIPTS. JUST DON'T. EVER. Fix the issue correctly: >>> by provisioning an application specific scratch area and/or using tmpfs. >>> >> What if you document your processes? Wouldn't that mitigate that... issue? >> > > I don't believe so; a sys-admin is entirely correct to make reasonable > assumptions about a system. There is just no good reason to do this - > when legitimate and customary mechanisms exist to accomplish the exact > same thing. Especially with startup scripts - where messing with them > will invariably mean that someday a server is rebooted - and does not > boot correctly, very likely resulting a service outage. > > If I'm going to modify the DNS config, the user authentication, etc... > of a server I fully expect to look at any documentation first - because > of course those things are customized. But to install routine updates, > reboot, etc.... these shouldn't require special knowledge - if they do > something is just wrong. If they do then someone, including the > original sys-admin, will eventually forget and screw up. Routine things > should always be routine; and special is not routine. Maybe that can't > be accomplished in every case; but it should always be a primary goal > of a professional systems administrator - to make your systems conform > to norms and thus be as easy as possible to understand. > > If you have a properly controlled environment, i.e., someone/some group is responsible for the deployment of OSes and related material, you certainly can maintain startup scripts that are different than those distributed with the OS. $WORK has just such a department, and they are responsible for maintaining a consistent 'yum' repository that is used for all system deployments. They still do not change things like startup scripts just because 'they can', such changes are only done when found to be necessary in our environment of tens of thousands of machines.
'Routine updates' are never 'routine' here, they are first tested in a lab, then released to development boxes only, then to our staging environment, then finally to production. All changes are reviewed at each step. If you have solidly defined policies and procedures in place for implementing change in your environment, the place doesn't fall apart because some individual leaves the company. And yes, this can be done in shops both small and large - that's what documenting your procedures is all about. Even 'routine' procedures need to be documented, no one should ever decide to 'yum update' against some remote repository just because it looks like a good idea at the time. All updates must be tested before being applied to production systems, or at the very least, a backout procedure must be defined for recovering from updates that fail (in environments where it just isn't possible to replicate some expensive hardware into a test environment, for example). I'm currently working with a product that has deficient startup scripts - the product is run as 'root' - needlessly - making maintenance by the non-sysadmin team responsible for it awkward (we have rather strict 'sudo' policies here), so these scripts will get changed and file permissions within the product will also get altered, and it will all be documented for the next person who does an install or upgrade (which probably won't be me). This is considered to be a better business decision than trying to work around the issue with a complex (and likely incomplete) set of sudo commands to operate and maintain the product, and any time that root privileges are not required, they shouldn't be used. Process and procedure are what keep systems running - not leaving them wholly vanilla, or using workarounds that are put in place because some issue of the underlying system design is deficient. I do agree that such changes should be kept to a minimum, but not less than that. - Richard _______________________________________________ Tech mailing list [email protected] http://lopsa.org/cgi-bin/mailman/listinfo/tech This list provided by the League of Professional System Administrators http://lopsa.org/
