If you weren't happy with GFS try OCFS2. It's oracle's cluster
filesystem and it's SOO easy to set up. Sadly I don't have answers to
any of your other questions other than the fact that Lustre's
performance with small files is abysmal for me too. I'm very much
interested in any tunables.
-Aaron
On Jun 21, 2007, at 9:20 AM, Balagopal Pillai wrote:
Hi,
I am using Lustre 1.6.0.1 with one OST and 20 clients in
an HPC cluster.
The OST/MDT/MGS has a 16 channel 3ware 9650 using raid6. I
currently have another lustre installation
(version 1.4.5) and it has been working trouble free for over an
year. The OS is CentOS 4. There are 4 network
ports in the storage server in adaptive load balanced mode and
aggregate network throughout is great (with 4 x netperf/iperf from
clients)
in an ideal situation when clients pick up different mac addresses
of the different interfaces in their arp table.
I have a few questions about Lustre and hope someone can
help me.
* I had to re-export the lustre volume via nfs on the new 1.6.0.1
setup to other infrastructure boxes.
After the export, i get the following error messages in the OSS -
Jun 21 09:31:11 lustre-3ware kernel: Lustre: 4946:0:
(lustre_fsfilt.h:205:fsfilt_start_log()) scratch-OST0000: slow
journal start 33s
Jun 21 09:31:11 lustre-3ware kernel: Lustre: 4946:0:
(lustre_fsfilt.h:205:fsfilt_start_log()) Skipped 22 previous
similar messages
Jun 21 09:31:11 lustre-3ware kernel: Lustre: 4874:0:(filter.c:
1139:filter_parent_lock()) scratch-OST0000: slow parent lock 33s
Jun 21 09:31:11 lustre-3ware kernel: Lustre: 4874:0:(filter.c:
1139:filter_parent_lock()) Skipped 6 previous similar messages
Also is the NFS re-export option stable in version 1.6? I read some
posts before in the list reporting kernel panics on Lustre 1.4.
*I was evaluating GFS for the past few weeks with GNBD and the
performance was amazing (at least for my purpose with one storage
server). It was very fast, especially for small files.
But i had to dump it because of stability reasons. The problems
were these - has 6 daemons that need to come up in a particular
order. If some of
the kernel modules crash on heavy load on a node, the whole cluster
freezes. It had the issue of quorum, which is beneficial on a HA
setup, may be not for HPC.
In some cases, i have to keep just one server running that re-
exports the volume via nfs even if the hpc nodes are down. Like
during a power failure for example. Quorum is a
problem in that case. But it was mostly stability that made me not
go with GFS + GNBD.
*Now the problem - Lustre performance dips a lot when it comes to
small files. Please see the following fileop -f 5 test comparing
NFS and Lustre -
Lustre -
Fileop: File size is 1, Output is in Ops/sec. (A=Avg,
B=Best, W=Worst)
. mkdir rmdir create read write close stat access
chmod readdir link unlink delete Total_files
A 5 1654 691 132 14228 719 4874 1987 32737
1718 2506 1262 1340 1608 125
NFS -
Fileop: File size is 1, Output is in Ops/sec. (A=Avg, B=Best,
W=Worst)
. mkdir rmdir create read write close stat access
chmod readdir link unlink delete Total_files
A 5 177 594 459 380747 137392 2282 1219 444312
502 1274 306 513 464 125
Could you please recommend any tunables to get a bit more
performance out of Lustre with lots of small files? Lots of small
files was bad in GFS too, but
it was better than NFS though.
*Also the read performance of Lustre seems to be a little behind
NFS. I had /opt which has all the software for users moved to
Lustre in the new setup. But
software like Matlab, Splus etc takes almost a minute to come up.
The second time is very fast though, maybe due to caching. So i am
thinking of putting /opt
back to NFS. Is it possible to boost the read performance of Lustre
a bit?
*Is there a way to make disk quotas activate at startup
automatically on a Lustre client? The lfs quotaon <mount point>
works sometimes. But
sometimes it gives an a resource busy error message.
*One last question. In the older Lustre setup (version 1.4.5), i
have 5 scsi drives one each as an OST for a single volume. The
volume became full. But df still reported
that there is 27GB free. There doesn't seem to be an lfs df option
in that version of Lustre. So i couldn't see the individual
utilization of each of the 5 OST. Is this a striping
problem?
I know it's a lot of questions. Hope some of them are
solvable. Thanks very much.
Best Regards
Balagopal Pillai
_______________________________________________
Lustre-discuss mailing list
[email protected]
https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
Aaron Knister
Systems Administrator/Web Master
Center for Research on Environment and Water
(301) 595-7001
[EMAIL PROTECTED]
_______________________________________________
Lustre-discuss mailing list
[email protected]
https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
