[ Wednesday, February 2, 2000 ] Erik Petersen wrote:
> It's to bad our press release has caused some misunderstanding of our
> position with respect to Linux software RAID. This was not intended
> nor desired. Let me say just a couple of things on this subject:
Ok, from your response and others I now understand what it is that's
getting donated. First and foremost, forcing people to register
for code you're "giving away" feels... wrong to me :)
Not to beat the horse, but I want to make sure it's understood how
the point got confused in the first place :) I can't say I've
been able to fully track the set of patches that may have come
out of land-5, I just want to get this view out in the open :)
If I'm off-base and Ingo tells me to STFU, I'll happily comply :)
For clarity, I'm going to quote from the press release
at http://wsp018ed.wspros.net/1.31.00.htm as linked from
http://www.land-5.com/
Just to be clear up-front, I'm very frightened that this entire release
was based on using words and phrasings to make it appear that s/w raid's
current state of excellence is due to LAND-5 coming in and doing
massive R&D effort.
1) This *seriously* undermines the already-unappreciated work Ingo has done
2) It appears to aim for duping readers/investors into believing LAND-5
has done the same amount of work and effort as companies that *do* in
fact spend real R&D effort on *actual* improvements in functionality
and performance (or even existance) ala SGI, ReiserFS, Trillian, etc,
Let me try and point out the possible offending parties in the release.
"This code includes significant functionality and performance upgrades
for Linux RAID software, plus a user-friendly browser interface."
- functionality and *especially* performance upgrades *clearly* implies
changes in actual implementation code. Putting pretty wrappers around
something doesn't make the code change at its core :)
- "plus a user-friendly browser interface" clearly implies that the
GUI part is simply an add-on front-end, implying deeper lower-level
code or patches.
- Any wrappers or guis should almost exclusively stick with the term
"usability", and this piece of literature appears to be no exception.
"Research and development activities undertaken by LAND-5 have resulted
in considerable improvement in Linux software RAID performance, making
it a viable alternative to hardware-controlled RAID for mission-critical
storage."
- The only thing I could see that could justify this are patches in the
past to mingo to help things out. I can't personally recall them, but
I'd love a reference about them, especially the ones that were
"considerable improvements" in performance :)
- This entire sentence implies that R&D done by LAND-5 is what did/will
make s/w raid viable for mission-critical and a viable alternative
to h/w based... the phrase used was *making it* which regardless of
after-the-fact apologies is definitive in stating that it was *not*
a viable alternative without this code from LAND-5. Moreover,
this gain in viability is *specifically* tied in this sentence to
*performance* improvements.
> 1. LAND-5, as far as I know, does not feel the current s/w RAID is
> "lacking". We in fact believe the opposite.
This sentence only co-exists successfully with the press release if
core performance and functionality improvements in the current s/w raid
(and not just some... so many as to make the diff between viable and
not... we don't count usability since the PR references only performance
improvements) are due to LAND-5.
I can't say with 100% surity that this *isn't* the case (although
I'm 99.5% sure :) so I'll let it go for now and let Ingo comment
if he wishes.
> LAND-5 is the only major enterprise storage company to actually build
> a product line based on it.
I *hope* that's not true... such good code should get more backers :)
> Otherwise we would not be investing substantial $$$ in promoting it.
The only thing I can see getting promoted is land-5 R&D. Maybe it's
just me, but I'd rather see something like "Linux s/w raid kicks
butt anyway you look at it. We know it and work with it extensively.
We've written a raid manager to make it easy and quick for you
to get that awesome power onto your machines"
Perhaps I'm being too pick, but I don't see any promotion of current
linux s/w raid at all, just performance improvements supposedly
coming out of LAND-5.
> [...] days explaining to the commercial press why we feel this it true
> and why we feel that Linux is enterprise ready when our competition
> (namely IBM, Sun and HP), are promoting proprietary Unicies as enterprise
> level products.
At least in the IBM and HP cases, you're referencing companies that
have done *real* changes to help out Linux, not just provide wrappings
(namely, the Trillian project, which I can d/l the code for off kernel.org
with *no* forced registration). Sun's another issue :)
Also keep in mind the IBM is/will be donating AIX source to help out
Linux... I don't know anything about it more than the current /. and
linuxtoday entries, either, but we're talkijng about *real* changes that
benefit *everyone* using the Linux kernel, not just some gui interface
if you happen to feel like registering and d/l'ing a package.
> No other supplier has done this or shown nearly the same confidence in
> s/w RAID as we have, including IBM which ships Netfinity servers with
> hardware RAID controllers.
This gets into the realm of just plain FUD, IMHO.
Do you think s/w raid is always the right answer to a RAID problem?
I don't, I know Ingo doesn't, and neither should anyone understanding
different RAID situations.
I am heavily involved in IBM's benchmarking activies, esp. those
database-related (TPC-C, TPC-D, TPC-H, etc). These situations (and others
with high-end database set-ups) need the main processors *desperately*
for the core of the work being done (esp. the SQL optimization engine,
SGA, etc). With 8-way machines pegged at 99-100% CPU util per processor,
s/w raid just isn't a viable option in this scenario.
Also, in many cases even if you have spare CPU your most scalable
situation will be a hybrid (my favorites :) where you mix h/w and s/w raid
together (My fav is s/w raid5 over multiple 64-bit PCI Mylex DAC1164P's
doing h/w raid0 :) for huge and fast filesystems (esp. once we get
to put these underneath a reiser or xfs)
> 2. We have been testing s/w RAID for more than 8 months in both SCSI
> and FC systems and have found some issues which we intend to fix
> post haste. The most significant is performance. We have yet to get
> s/w RAID to perform any faster than 25MB/s (according to Bonnie)
> in any configuration or RAID level. When my test server comes back
> from the LinuxWorld show I will post the numbers I'm talking about
> and perhaps you will offer some answers to this problem.
I can tell you now: s/w raid perf is heavily tied to fs options (4k block
size), fs choice (b-tree based :), chunk size (in my exp) and *insanely*
tied to the device driver for the scsi controller you're using. Accept no
substitutes and go get a AHA-3960D (or other similar high-end 160MB/sec
scsi card from adaptec) and use the aic7xxx driver, reading up on *all*
options that may help.
Also, Bonnie is a poor choice (IMHO) for fs testing and tiotest at least
is a large good step in the right direction. http://www.iki.fi/miku/tiotest/
During dinner after our LUG meeting last night we chatted about a
100MB/sec s/w raid fs (mainly drooled over it :) so rest assured
you can get performance out of s/w raid.
> There are some bugs which we have fixed and we will be posting a patch
> soon to fix them. These mostly relate to hot-swapping drives and failure
> handling.
Looking forward to it. Do you add in current_size fields to md arrays
(sector count perhaps) so you can check requests before passing them
down to the block device? Just curious, because that's the patch
I'm working on now.
> > 1) Is the MPL a problem given the kernel is (99.44%) GPL?
>
> The code released is a management tool designed to make setting up and
> managing software RAIDs easier. Any and all enhancements we have made
> or will make will be GPL.
Excellent news :)
> I closing I just want to state that LAND-5 is 100% Linux, 100% software
> RAID. We have a number of engineers, including myself, working on
> improving it and those improvements will be GPL'd.
Glad to hear it, and I look forward to these future patches :)
> To say that s/w RAID cannot be improved is a somewhat arrogant
> statement. Everything can be improved upon and until s/w RAID beats the
> pants off any h/w RAID solution I will keep working on it.
1) Noone said it cannot be improved. Please do *not* misquote.
In fact I have (along with Mika) done performance-enhancing
and tuning-addition patches.
2) s/w raid *does* beat the pants of h/w raid given the right scenario.
As painfully obvious as it is, it's owrth reiterating that if your PCI
and memory busses are already clogged, switching from h/w to s/w raid
only drags down performance. With the right amount of free resources
({CPU, memory bus, PCI bus} cycles), though, it can be a huge win,
which we all acknowledge.
> I can't speak for the marketing hype, but we've all been around long
> enough to understand it.
Hype has its place, esp. in the promotion of the code *already in place*
but marketing that (IMHO) appears to take corporate credit for the
work of others is (at a minimum) in bad taste.
I *do*, though, *very much* look forward to patches coming down the
pipe from your experiences, and working together to make software
RAID all it can be for Linux in the enterprise (ugh, marketing :)
> The preceeding statements are my own and not those of my employer.
Ditto :) Might as well put in a disclaimer as I'm sure this entire note
will be coming off far too inflamatory :) My overriding motivation
is to protect Ingo and make sure he gets credit where due. Why?
Because as ESR even knows, OSS is about ego-gratification at its base,
and anything that gives him or anyone else a sense that they're work
benefits the pockets and reputations of others instead will only serve
to drive them away from an area that needs more, not less, knowledgable
attention from the few people (sct, mingo, alan, etc) that *really*
understand and deal with the *core* issues of a fast, efficient, and
stable s/w raid implementation in the Linux kernel.
Or, to just blatantly rip off Doug Ledford:
Opinions expressed are my own, but they should be everybody's.
James
--
Miscellaneous Engineer --- IBM Netfinity Performance Development
