On Sun, 7 Jan 2007, Joe Landman wrote:
BTW, the cluster's servers were not (and I would not advise that servers
ever be) running the old distro -- we use a castle keep security model
where servers have extremely limited access, are the most tightly
monitored, and are kept aggressively up to date on a fully supported
distro like Centos. The idea is to give humans more time to detect
intruders that have successfully compromised an account at the
workstation LAN level and squash them like the nasty little dung beetles
that they are.
Yup. Even better is never letting the users log in to admin machines.
Provide machines for them to log into, submit and run jobs from. Just
not the admin nodes.
That would be the "servers have extremely limited access" part -- as in
sysadmins only.
For what I call production cycle shops, those places which have to churn
out processing 24x7x365, you want as little "upgrading" as possible, and
it has to be tested/functional with everything. Ask your favorite CIO
if they would consider upgrading their most critical systems nightly.
It all boils down to a CBA (as everything does). Upgrading carries
risk, no matter who does it, and how carefully things are packaged. The
CBA equation should look something like this:
value_of_upgrade = positive_benefits_of_upgrade -
potential_risks_of_upgrade
I completely agree with this. As I pointed out earlier in the thread,
companies such as banks make "conservative" seem downright radical when
it comes to OS upgrades. They have to do a complete, thorough,
comprehensive security audit to change ANYTHING on their machines -- as
a requirement in federal law, IIRC. To get them to take you seriously,
you MUST be prepared to support the OS they install on (once it is
successfully audited) forever -- until the hardware itself falls apart
into itty-bitty bits.
On clusters that add new hardware, usually bleeding edge, every four to
six months as research groups hit grant year boundaries and buy their
next bolus of nodes, FC really does make sense as Centos probably won't
"work" on those nodes in some important way and you'll be stuck
backporting kernels or worse on top of your key libraries e.g. the GSL.
Just upgrade FC regularly across the cluster, probably on an "every
other release" schedule like the one we use.
Chances are that anything Red Hat Enterprise based just won't work. New
hardware is always hard.
Heh. Try to point this out to a purchasing agent on an RFP which
demands a) newest possible hardware and b) RHEL 4 support. You get to
pick one or the other, not both. Which one do you want? Hint: "b" is
far less valuable.
The other (not-so-funny) aspect of this is when we deliver new hardware
with an OS load that supports the newer hardware and someone wants to
pull it back to the "corporate standard". In doing so, they give up
stability, performance, and often file system support. Or in the case
of our JackRabbit unit, when we deliver 30TB of 5U system and we get the
"ext3 is almost as good as xfs" line. Uh.... er.... no. Those who
really insist upon this must only want 16TB units with no possibility to
ever grow beyond this (we have a design cooked up to show how to do a 1
PB in 4 racks as a single file system, or better, an HA 1 PB in 9 racks
as a single file system). 16TB is great for some folks, but it is a
fundamental ext3 limit. You need the untried-in-the-real-world ext4 to
break that limit. Or xfs and jfs.
Proving once again that Joe's company provides a valuable service,
because companies like this fill in an important gap between e.g. FC and
a customer's conservative needs. However, I'll bet Joe is still just as
vulnerable to the other problem -- customer wants to run commercial
package X (which "requires" RHEL) but ALSO wants to run it on bleeding
edge hardware. I'll bet you really earn your keep on those ones...
;-)
rgb
--
Robert G. Brown http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567 Fax: 919-660-2525 email:[EMAIL PROTECTED]
_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
