On 2010-11-16 at 16:22 -0800, Hugh Brown wrote:
> * Have you included acceptance testing for large hardware purchases
> before? Why or why not?
Yes. Because you're spending a lot of money and you want to end up with
something you can actually use, instead of something which does exactly
what it claims to do, but it turns out that there were subtleties to
those claims you didn't spot, or there are arguments about what those
claims mean?
> * What have you specified in the tests? Do you have a template you
> can share? How specific did you have to be in your RFP about what
> constituted failure?
The acceptance test plan is typically part of a contract. Try to be
specific. I don't currently work on buying external products and I've
only done one large purchase, so am not the best for sage advice, but
perhaps have closer memories of being in your situation and nervous
about the whole affair.
My RFP almost wasn't good enough to let us return the kit as I hadn't
pegged down figures on I/O performance in certain operations; I had
specified that we would be able to maintain certain rates for stuff I'd
predicted, but I'd missed a case and it turned out that performance
there truly sucked in a way that would have bitten us hard. I also
hadn't specified what I meant by "SNMP support", as I was too used to
NetApp's excellent MIBs and thought these were the norm for professional
equipment.
Turns out, for some vendors, ability to monitor a storage appliance
means either keeping a browser window permanently open, or spending
$100k on their proprietary monitoring software, as "SNMP support" means
that they have the basic host MIB data (physical location, etc) and
nothing else. That's how bad checkbox compliance can be and how easy it
is to be screwed.
I got very lucky with phrasing I'd used, combined with the vendor's
inability to demonstrate performance in a measurable way, for a product
on an unrouted subnet (which argued against browsers; I didn't point out
that I could have set up a small proxy, and their staff either couldn't
spot that, or were willing to concede the point politely because of how
badly they clearly were performing and how badly the business
relationship would have gone if they'd pressed the point).
So, after you think about what you need, think about how that term could
be abused. To use my bad experience as a learning example, for "SNMP
support", instead consider something like this (from the top of my head,
could probably still be refined):
* SNMP support consisting of the union of:
* all available metrics which the product exposes for any monitoring
service, including proprietary enterprise-grade monitoring product
* metrics which let the customer monitor how the product performs,
sufficient for problem detection, fault diagnosis and capacity
trending
* any metrics which the vendor uses to demonstrate RFP compliance
* metrics to include, but not be limited to:
* <checklist of items you want>
Generally, you're writing part of the contract, which should be polite
but firm and clear, as this is what *avoids* fights later.
> * What resistance, if any, have you found from the vendors? What
> about management? (I heard stories at LISA about acceptance testing
> being whittled down in negotiations in return for concessions on price
> or other things.) If you had to negotiate, what did you give
> up/trade?
You don't trade away the tests which determine whether or not the
product is worth spending time and money on. That's removing your
main safeguard. If your manager is prepared to do that and can't be
educated, but you're good enough to be working on the RFP, then you're
good enough to get another job, even in this economy.
If your manager is willing to trade away the part of the contract which
verifies that your employer is getting what it needs, then they're
inept. Educate them, or leave, or get them pushed out of the way of
being actively dangerous to your mutual employer.
> * Have you had to reject anything, or send anything back to be fixed?
> What reaction did you get from the vendor?
Yes, rejected. The VAR had strong words with the storage vendor about
what they needed to do if they were serious about breaking into the
ISP market.
Ultimately, you're the customer, spending money. You don't have
consumer protection laws and the acceptance testing probably replaces
whatever "fit for purpose" consumer laws might have applied. If the
vendor doesn't want your money for the product, that's their choice to
do so. For any large purchase, you should be able to verify that the
product does what it claims and what you need it to do.
-Phil
_______________________________________________
Discuss mailing list
[email protected]
https://lists.lopsa.org/cgi-bin/mailman/listinfo/discuss
This list provided by the League of Professional System Administrators
http://lopsa.org/
