>From the various answers I received, not all CCed to the mailing
list, it seems that the Tekram DC390U2B / DC390U2W controler is the
most appreciated LVD controler (it really suprised me since I had not
eared much about it earlier) and that Adaptec controler are also
working well now.
Lastly, the clear chalenger seams to be AcceleRAID, so let's come
to a concreat scenario (which I hope is fairly general):
- I have a storage server that need more space every few
monthes (it currently contains 13 disks for a total of 500 GB).
- The backup is performed on CDs (see details at the end)
So now the big question is how to setup a resonably reliable, and
expendable terabyte range file system under Linux ?
My experiments led to the following option:
- a set of RAID5 subsystems (each using 8 50GB disks in a cabinet,
so providing 300 GB (one spare disk))
- a linear subsystem on top of the several RAID5 subsystems.
(. CODA would have been a more expensive, but even more secured
alternative because it provides strong replicating features,
so they also protect agains full computer failure, but reading
the mailing list archives, I discovered that they cannot scale yet
to the 1TB range.
. XFS is far away, and nothing proves yet that it will mixe
nicely with Linux)
Each RAID5 can be seen as a resonably reliable 300GB disk.
The linear subsystem enables to extend the array without
corrupting the datas (I tested it only on small experimental
partition). Since datas are already stipped on various disks in
each RAID5 subsystem, there is no (more precisely nearly no,
because the io will append all on the same contoler instead of
beeing split to several ones) performances penality through using
linear rather than RAID0 for the top raid system.
RAID5 for the top raid system would provide extra security, but
would not allow to extend, yet.
'ext2resize' utility is used to extend the filesytem accordingly.
This lead to the following, unclearly answered yet questions:
- is ext2resize resonably stable in the 1TB range ?
- how big can an ext2 partition be (I am not talking about theorical
values, but rather usable): I have a 200GB ext2 partition, it is
e2fscked in 30 minutes without any problem on a simple 300Mhz PC
with 256MB or ram.
Would a 1TB take roughly 5 time more, or could it also fail
due to the lack of memory ? I somewhere red that the time
required by e2fsck was not linear with the size of the
partition, but in my case it seamed to be: when I mixed
for 50GB partitions to a single 200GB one, the e2fsck time
has not changed significantly.
- there is a clear unanswered question about the alternative
for setting the various RAID5 subsystems. Tekram controlers
with Linux 0.90 software RAID or AcceleRAID 250/1100
RAID controlers. I tryed both (not with Tekram but Buslogic),
and it seems that the Linux RAID software speed penality is not
that high if the server is mainly a file server, since the
bottleneck are still disks, so the clear question is about
reliability, and my experiments did not lead to a clear winner
because they both basicly worked well, just as expected, so,
on this front, I only believe collecting statistics from users
experiments can give us a clear picture.
It seems that, over the last year, Linux has mainly focused on
improving access to datas (SMP, more efficient cache buffer,
new IP stack), and that storing and improviding datas security has
improved slower (for the moment, software RAID seams to be still
considered as a feature, not as a security or stability issue,
what it truely is, so it will surprisingly not be updated in the
official stable kernels).
Secondly, a 1TB file system under Linux would probably have been a
crazy idea two years ago, so HOWTOs are not of much help about this,
even if general safety receipes are still good to read from time to
time. On the same level, when I set up the RAID5 system using
a Mylex controler, they had no 50GB disk in their appoved list, so
they are also clearly not experimented in this range.
As a result, exchanging experiments is the only alternative solution
I can see for people like me that need to scale today.
So, any experimented comment in terabyte range under Linux will be
greatly appreciated. I will send feedback to anybody requesting
it.
Hubert Tonneau
---
Backing up on CDs:
This is not very common, since backups on Unix systems are
traditionaly performed on tapes, but I have four good reasons
not to use tapes:
- nowdays computers are linked to the internet, and most networks,
(mine for sure) are not safe againts attacks, so backing up on
a write once only media is a clear advantage, that was not so
important a few years ago.
- backing up on CDs means an infinit incremental backup (I have a
database specifying on which CD each file is, and on wich all
it's previous versions where) which is a great value since in
case of a corrupted file descovered long time after, all
previous versions are still available. This is also an important
security agains piracy, that also was not so important a few
years ago.
- controling the content of the backup is much easyer since CD
readers are very common, so randomly chosen files can be checked
on a not networked computer (again for security reason) in order
to verify that no virus is corrupting the content of the backup.
- 500Gb needs an expensive tape library compared to a CD engraver.
- in case of failure, any file can be restored in a few seconds.
But it has the following drawbacks:
- I cannot use the backup as a temporary storage area so I need
to extend without recreating the partition.
- in case of failure, restoring all files is a rather long
operation (25 hours or even more in order to restore 500 CDs
using 5 CD readers)
- a single file must not be more that 650MB.
- the long time reliability of write once only CDs seams to be an
open question. I red that a user had noticed a not more that
5 years life, and another had noticed that non readable CDs
where nearly always the result of physical damages. 5 years
ago, many writable CDs had no plastic on top of the sensible
metal suface, so it may be very different nowdays, but I have
not clear picture about this also.
So the final decision probably comes mainly the the following
question: in case of massive data loss, do you need to restore
all the files, or only some, before production can restart ?
