Ideally you'd run all your ordinary workloads but in a stress testing way, with
simulated demand, ordinary concurrent batch runs, and valid test data. And then
dial up the stress to +50% (for example) above your prior peak if you want to
see that.
Simulated demand implies you'd have a load generating tool of some kind. It
probably wouldn't involve hiring 1,500 temporary workers to sit at PCs to
simulate employee and customer usage. IBM and other vendors offer load
generating tools.
Valid test data implies you'd have a tool that generates test data — that
removes/masks personally identifiable and other sensitive information but
maintains testing validity of the data so that the test data will benchmark
similarly to real data. IBM and other vendors offer test data tools.
You may or may not currently be able to run stress tests in your own data
center with your own infrastructure — although with IBM zSystems stress testing
is typically much easier to accommodate than with other types of servers.
(Please talk with "your friendly IBM representative" if you'd like to explore
on-site stress testing capacity options.) If you cannot handle a stress test
on-site then one alternative is to contract with a benchmarking center. IBM is
in that business, too. The benchmarking center will typically offer suitable
infrastructure with security protocols, professional help (to make sure the
tests are valid, to execute the tests with your participation, and to compile
results that are understandable), and temporary use of testing tools (load
generators, etc.) It's rare that you would have only one test run. Usually
benchmark testing is iterative. You would have a plan, but you would also be
open to surprises and then run additional tests (change a parameter, for
example) based on your initial findings.
IBM and other storage vendors may also be able to provide examples of
realistic, representative benchmarks they've run in order to characterize their
offerings, and that could be more than satisfying for whatever your concerns
are. And in some cases they might even be able to offer credible delivery
promises of specific performance attributes particularly if they're minimum
relative ones. (Example: "With these assumptions we promise you'll see at least
double your existing I/O throughput Y. In the unlikely event you don't see at
least double we'll deliver some additional consideration X.")
It's important to be concise in forming and putting down the open questions
you're trying to answer. ("What are we trying to prove? Do we even need to?")
The more demands you have the more likely it is those demands will translate
into higher costs and/or fewer vendors willing to participate in any contest.
Demands should be realistic and necessary. In principle you can benchmark
practically anything (and spend years doing it, by which time there's a new
model anyway). Many tests simply aren't worth running because they wouldn't
clear up any genuine mysteries. For example when I'm shopping for a smartphone
and want certain battery life I'll check the vendor's representations and also
what professional reviewers (i.e. previously run benchmarks) have to say about
battery life. And then I'll fundamentally trust that information as long as the
sources are credible. I don't need to run my own battery life tests, especially
when the expected battery life is 2X my minimum required X. I have better ways
to spend my time. In the rare event the battery is defective (doesn't hold a
charge) the reputable vendor that sold me the smartphone will receive my
warranty claim and will honor it. Other things I might worry about. I might
stop at the vendor's shop to see what "Starlight Blue" looks like (a
basic/quick benchmark), for example. Or not if the phone's color isn't
important enough to worry about, because it'll require some of my time and
effort to visit the shop.
As an occasional reminder my views are my own.
— — — — —
Timothy Sipples
Senior Architect
Digital Assets, Industry Solutions, and Cybersecurity
IBM zSystems/LinuxONE, Asia-Pacific
[email protected]
----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [email protected] with the message: INFO IBM-MAIN
