|
Ok, so right about the time Oracle 6 went
production (we were a beta site), I was the lead DBA for a very large
project that put retail store systems in around 1000 locations around the
country (USA). Each one of those locations had their own Oracle database
on AT&T 3B2 Intel boxes. I also supported development, testing, and QA
too. But the earliest versions of Oracle 6 had some database corruption
problems -- not too often, say only once a year on the average. However,
multiplying that times 1000 yielded about 2 or 3 stores weekly! I became
very familar with workarounds for db corruptions -- those undocumented rollback
init.ora parameters, how to find corrupt blocks and create dummy tables to hold
them, etc. We set up nightly exports to /dev/nul to find the corruption
sooner than later. Lots of scripts and plan A, B, C ... depending on where
the corruption was. Tape backups and mainframe downloads ... static data
separated from dynamic data in different tablespaces and
files.
During the most intense time period, hotels were
booked and meals catered and we didn't leave the building complex for
weeks.
Aren't all these stories fun!!? (Though I'm
happy to say I was never on another project that intense again!)
Marc Perkowitz.
----- Original Message -----
Sent: Friday, August 31, 2001 8:26
PM
Subject: Re: Duhvelopers and
DB-ehs?
OK. I've resisted the temptation so far,
but (puffing out chest)...
In one job in my distant past, at a turnkey
software/systems shop, four (initially two) of us used to administer
about half a dozen in house Oracle servers and remotely administer
about 2-3 dozen of our clients' database servers. They were
scattered all over the US and Canada and we all did database and Unix
systems administration "part time". About 80% of our time was spent in
custom software/systems design and development. Everybody got Oracle DBA
training, but nobody was a full-time DBA or systems administrator. (A
very novel approach that resulted in very few deeply imbedded application
performance issues!)
From late 1997 until mid-1999 I administered up to about 45
Oracle databases (each was 10-200 GB & OLTP) by myself -
including design/modeling, code reviews and tuning, physical server
configuration (sizing systems, creating raw devices, etc.), and
production database administration. I was the only Oracle DBA in the
entire company for my first 18 months there! We actually had about
12 distinct Oracle-based application systems, most with distinct dev,
functional test, performance test, and production databases. These were
spread over more than 40 Sun servers ranging from Ultra 2s to
E10Ks. Three of the production systems were on Sun PDB cluster and
Oracle parallel server. Critical? At least four were the
core components of the trading systems at one of the world's
largest online brokerages - serious 24 x forever with (usually) million+
dollar costs for any downtime.
Actually the number was about 24 databases
until Jan 1999. At that time, they gave me some help - the guy who
was the (only) DBA for the Informix system (which was phased out the
following Fall) - to retrain as an Oracle DBA. Two weeks
later I was asked to create 21 new databases on 17 new servers -
within nine days. I made it with a little under an hour to
spare. That brought the total up to about 45. By August, the
ex-Informix DBA was doing Oracle full time and they had hired some more help
(all with no significant Oracle experience!) - and, of course, added
another 20-30 databases! When I left a couple of months ago (no surprise
is it?) we had 42 production Oracle databases on 35 production servers, with a
total of over 160 databases on over 120 servers, administered by 6
DBAs. We also had over 100 other servers (Web servers, WebLogic servers,
Tuxedo servers, etc.) that had the Oracle client (including Pro*C)installed -
which we also maintained.
Of course I was working an average of about
95 hours/week much of that time... And sleeping less than half
that.
For none of these did I ever have any significant
3rd party tools (BMC Patrol, etc.). I did develop a *LOT* of homegrown
scripts and automation though. [I still prefer, with a very few select
exceptions, homegrown tools over commercial tools.] The other key was in
*ADAMANTLY INSISTING* on being able to set things up "right" at the
start. Having enough space on (at least most) test systems to load
(or clone) production-sized databases helped also.
This sounds kinda like "When I was a kid, we had to walk five miles to
school in neck-deep snow - uphill both ways!" doesn't it?
Unfortunately, its true. I have references and the complete lack of an
outside life to prove it! Since I left, I've been decompressing and
going through social rehabilitation ["Hi, my name is Don. I
haven't been on a bridge call in over two months." (Smattering of
applause...)]
-Don Granaman
[certifiable OraSaurus and shameless
braggart]
PS: Most of, and certainly the best
of, the DBA's I've ever worked with had significant prior
experience as developers (not "duhvelopers").
----- Original Message -----
Sent: Friday, August 31, 2001 12:18
PM
Subject: RE: Duhvelopers and
DB-ehs?
Yeah, well, in a previous life I was in charge of 27 databases on 9
servers ALL BY MYSELF. Didn't have anybody working with me, man. Zilch.
Nada.
And, of course, I'm not going to talk about the nature of the systems
'cause then it wouldn't sound so impressive....
--Walt Weaver
Bozeman, Montana
We do about the same number with 6 DBAs...
(not that that means sh*t without talking about the nature of the
systems, of
course)
|
