[Lustre-discuss] MDS question
What is the disadvantage of creating a MDS partition with smaller inodes per block? Now, its 4k per inode what happens if we go to the least blocks which is 1024k? This would let us create more smaller files which will lead to more inodes used. But what is the downside? TIA ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
[Lustre-discuss] Bug 15912
I see it is fixed now as we were being bit by this. We would like to put the new filesystem into use on Monday thus we are trying to get this resolved. Question, because it is just a parsing problem in mkfs, can after the filesystem is created can the problem be corrected? If not, how can we work around this? Do I just need to build the mkfs out of CVS for 1.6.6 ? Brock Palen www.umich.edu/~brockp Center for Advanced Computing [EMAIL PROTECTED] (734)936-1985 ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] drbd async mode
I'm trying a Lustre + DRBD set-up, I would like to use DRBD with the following setup: on MDS: DRBD emulates an MDT on shared storage. From Lustre perspective one MDS is active, the other is passive on OSS: BRBD emulates two OST on shared storage for each OSS couple. From Lustre perspective each OSS is active for one OST and passive for the other. I'm planning to use DRBD in active/passive mode, so that the storage is mounted only on the active server, and luster is mounted on top. I'm not sure how to sync the mount/umount of the DRBD partition and the mount of MDT/OST upon failover. The target setup shall have 2 MDS (1 couple) and 8 OST (4 couples), dedicate LANs for heartbeat and drbd. Do you have any details of your setup you may share about this? TIA! On Aug 12, 4:50 am, Mag Gam [EMAIL PROTECTED] wrote: Ha yes..I seen this before Sebastein. :-) I wanted to see if other people have tried it and their thoughts were. thats all TIA On Mon, Aug 11, 2008 at 2:33 AM, Sébastien Buisson [EMAIL PROTECTED] wrote: Hello, I ran some tests with DRBD and Lustre a few months ago (including in async mode). You can find my modest findings here on the Lustre wiki : http://wiki.lustre.org/index.php?title=Drbd_And_Lustre Cheers, Sebastien. Mag Gam a écrit : Has anyone been using DRBD and Lustre in async mode? I was curious what type of speeds you were getting and also what network are you using? TIA ___ Lustre-discuss mailing list [EMAIL PROTECTED] http://lists.lustre.org/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list [EMAIL PROTECTED]://lists.lustre.org/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
[Lustre-discuss] Log file opens/reads/etc?
Is there is a tool that shows what files are being accessed? Sort of like inotify, but not inotify? I'd like to compile file access statistics to try and balance the most accessed files across the OST's better. Thanks for any tips, Dale ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
[Lustre-discuss] Lustre 1.6.5.1 + RHEL4.7 + Infiniband
Hi everyone, I'm trying to build lustre 1.6.5.1 against Red Hat's kernel 2.6.9-78.EL for RHEL4, but the configure script doesn't detect the headers needed for OFED gen2 support (Infiniband). Does anybody know a fix for that problem? Regards, Götz Waschk ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
[Lustre-discuss] lustre + debian
Hello, I want to use lustre with my debian (etch or lenny) I try to compile lustre as module with debian kernel and I've got error How to run lustre with debian ? ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] diskless booting Debian
Jeff/Troy, - Jeff Darcy [EMAIL PROTECTED] wrote: We've tried loading canned MDT/OST images into memory on some nodes and serving from there, and it does seem to work. There are two downsides, though. One is that the Linux loopback driver is a real performance bottleneck, ever since some bright person had the idea to make it less multi-threaded than it had been. Another is that booting tends to involve metadata-heavy access patterns which are not exactly Lustre's strength - a situation made worse when you have nearly a thousand clients doing it at the same time and your MDS is a relatively small node like the others. So far we've found that NBD serves us better in the boot/root filesystem role, though that means a read-only root which involves its own complexity. Your mileage will almost certainly vary. A good trick with Lustre to get around the metadata bottleneck is to use disk image files on Lustre (e.g. SquashFS) and mount them using the loopback driver on each compute node (or lctl attach_device ?). So instead of having to bother the MDS you need only seek through the file on an OST. By either striping the read-only image across all your OSTs or having a round-robin image per OST you can get pretty good scalability. I tried this with our 700 node compute cluster but to be honest the overall booting performance was not that different to a couple of NFS servers serving a read-only root so it was not really worth the extra complexity in the end. We do still use SquashFS on Lustre from time to time when we have a directory tree with 30,000 small files in it that needs to be read by every farm machine. It's rare but it does happen and traditionally NFS does much better with such workloads. Daire ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
[Lustre-discuss] Re-using OST and MDT names
Hello, As a part of my continuing to benchmark Lustre to ascertain was may be best-suited for our needs here, I have re-created at the LSI ELP card level some of my arrays from my earlier benchmark posts. The card is now sending /dev/sdf 998999Mb with 128kB stripesize and /dev/sdg 6992995 Mb with 128kB stripesize to my OSS. The sdf and sdg formatted fine with Lustre and mounted without issue on the OSS. Recycling the MGS MDT's seem to have been a problem. When I tried to mount the MDT on the MGS after mounting the new OST's the mounts performed without error, but the bonnie benchmark test as run before would hang every time. Sample of errors in MGS file /var/log/messages: Aug 13 12:39:30 mds1 kernel: Lustre: crew5-OST0001-osc: Connection to service crew5-OST000 1 via nid [EMAIL PROTECTED] was lost; in progress operations using this service will wait for recovery to complete. Aug 13 12:39:30 mds1 kernel: LustreError: 167-0: This client was evicted by crew5-OST0001; in progress operations using this service will fail. Aug 13 12:39:30 mds1 kernel: Lustre: crew5-OST0001-osc: Connection restored to service crew5-OST0001 using nid [EMAIL PROTECTED] Aug 13 12:39:30 mds1 kernel: Lustre: MDS crew5-MDT: crew5-OST0001_UUID now active, resetting orphans Aug 13 12:42:42 mds1 kernel: Lustre: 3406:0:(ldlm_lib.c:519:target_handle_reconnect()) cre w5-MDT: 50b043bb-0e8c-7a5b-b0fe-6bdb67d21e0b reconnecting Aug 13 12:42:42 mds1 kernel: Lustre: 3406:0:(ldlm_lib.c:519:target_handle_reconnect()) Skipped 24 previous similar messages Aug 13 12:42:42 mds1 kernel: Lustre: 3406:0:(ldlm_lib.c:747:target_handle_connect()) crew5-MDT: refuse reconnection from [EMAIL PROTECTED]@o2ib to 0x81006994d000; still busy with 2 active RPCs Aug 13 12:42:42 mds1 kernel: Lustre: 3406:0:(ldlm_lib.c:747:target_handle_connect()) Skipped 24 previous similar messages Aug 13 12:42:42 mds1 kernel: LustreError: 3406:0:(ldlm_lib.c:1442:target_send_reply_msg()) @@@ processing error (-16) [EMAIL PROTECTED] x600107/t0 o38-[EMAIL PROTECTED]:-1 lens 304/200 ref 0 fl Interpret:/0/0 rc -16/0Aug 13 12:42:42 mds1 kernel: LustreError: 3406:0:(ldlm_lib.c:1442:target_send_reply_msg()) Skipped 24 previous similar messages Aug 13 12:43:40 mds1 kernel: LustreError: 11-0: an error occurred while communicating with [EMAIL PROTECTED] The ost_connect operation failed with -19 Aug 13 12:43:40 mds1 kernel: LustreError: Skipped 7 previous similar messages Aug 13 12:47:50 mds1 kernel: Lustre: crew5-OST0001-osc: Connection to service crew5-OST0001 via nid [EMAIL PROTECTED] was lost; in progress operations using this service will wait f or recovery to complete. Sample of errors on OSS file /var/log/messages: Aug 13 12:39:30 oss4 kernel: Lustre: crew5-OST0001: received MDS connection from [EMAIL PROTECTED] Aug 13 12:43:57 oss4 kernel: Lustre: crew5-OST0001: haven't heard from client crew5-mdtlov_UUID (at [EMAIL PROTECTED]) in 267 seconds. I think it's dead, and I am evicting it. Aug 13 12:46:27 oss4 kernel: LustreError: 137-5: UUID 'crew5-OST_UUID' is not available for connect (no target) Aug 13 12:46:27 oss4 kernel: LustreError: Skipped 51 previous similar messages Aug 13 12:46:27 oss4 kernel: LustreError: 4151:0:(ldlm_lib.c:1442:target_send_reply_msg()) @@@ processing error (-19) [EMAIL PROTECTED] x600171/t0 o8-?@?:-1 lens 240/0 ref 0 fl Interpret:/0/0 rc -19/0 Aug 13 12:46:27 oss4 kernel: LustreError: 4151:0:(ldlm_lib.c:1442:target_send_reply_msg()) Skipped 52 previous similar messages Aug 13 12:47:50 oss4 kernel: Lustre: crew5-OST0001: received MDS connection from [EMAIL PROTECTED] In lctl all pings were successful. Additionally files on Lustre disks on our live system using the same MGS were all fine; no errors in logfile. I thought that maybe changing the disk kB size and reformatting the OST without reformatting the MDT was a problem. So I unmounted OST and MDT and reformatted the MDT on the MGS. All okay. The OST remount without error.The MDT on the MGS will not remount: [EMAIL PROTECTED] ~]# mount -t lustre /dev/sdg1 /srv/lustre/mds/crew5-MDT mount.lustre: mount /dev/sdg1 at /srv/lustre/mds/crew5-MDT failed: Address already in use The target service's index is already in use. (/dev/sdg1) Again, the live systems on the MGS are still fine. A web search for the error suggested I try tunefs.lustre --reformat --index=0 --writeconf=/dev/sdg1 but I was unable to get a syntax of that command that would run for me (I tried various index= and adding /dev/sdg1 at the end of the line but it failed each time and only reprinted the help without indicating what about what I typed was unparseable). My current thought is that a stripesize of 128kB from the LSI ELP card is not testable on my Lustre 1.6.4 system. This does not seem to be an accurate statement from
Re: [Lustre-discuss] lustre + debian
Hello, On Thursday 14 August 2008 18:57:52 Robert LeBlanc wrote: It's easy and then it's not. The Debian way is to download the Lustre kernel packages and the Lustre binary packages. You build a kernel using make-kpkg the --with-patches Lustre option, we found a bug when trying to use the --append_to_kernel option so unless that got fixed, don't use it, Thanks for this informations, I'll have a look on it. them up. You are now ready to use Lustre Debian style. If deploying this on a cluster, just install your ready made .debs for the kernel and modules. None of this was explained in the README in /usr/share/doc/Lustre so we had to figure it out ourselves. Hopefully the Debian folks have fixed that. Could you please have a look on our Readme.Debian. I've attached the current version of this README. Do not use a vanilla kernel from kernel.org, only use the kernel source from the Debian repository as the Lustre kernel packages are tested against them. Vanilla kernels should be supported.. We've only packaged the upstream patches and modified them a bit for the debian kernels. So vanilla kernels should work as usual. Greetings Patrick Winnertz ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
[Lustre-discuss] mounting an OST from another node attached to a fibre channel switch
Hi, We have set up a couple of test systems where the OSTs are mounted on the same node as the MDT. The OSTs are LUNS on a SATA Beast controller accessible from multiple systems attached to a fibre channel switch,. We would like to umount an OST from the MDT system and mount it on another system. We've tried doing this and even though there is network traffic between the new system and the mds system, the mds system seems to be ignoring the OST mount. Can the an OSTs OSS change? Thanks, Ron ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] lustre + debian
On 8/14/08 12:51 PM, Patrick Winnertz [EMAIL PROTECTED] wrote: Hello, On Thursday 14 August 2008 18:57:52 Robert LeBlanc wrote: It's easy and then it's not. The Debian way is to download the Lustre kernel packages and the Lustre binary packages. You build a kernel using make-kpkg the --with-patches Lustre option, we found a bug when trying to use the --append_to_kernel option so unless that got fixed, don't use it, Thanks for this informations, I'll have a look on it. It was that the build script was very strict on what the kernel was named to determine what patches to apply, this was an issue with the script from Lustre. It's been almost a year since I've built Lustre, so this may have changed. them up. You are now ready to use Lustre Debian style. If deploying this on a cluster, just install your ready made .debs for the kernel and modules. None of this was explained in the README in /usr/share/doc/Lustre so we had to figure it out ourselves. Hopefully the Debian folks have fixed that. Could you please have a look on our Readme.Debian. I've attached the current version of this README. The attachment did not make it, I'm downloading the current Lenny Lustre packages to view the file. Do not use a vanilla kernel from kernel.org, only use the kernel source from the Debian repository as the Lustre kernel packages are tested against them. Vanilla kernels should be supported.. We've only packaged the upstream patches and modified them a bit for the debian kernels. So vanilla kernels should work as usual. Yes, all the vanilla patches that Lustre have are in the Debian package, the problem is that Lustre is usually behind Lenny so grabbing the newest kernel from kernel.org fails no matter what distro. What I've found about Debian is that the package team has worked hard to patch Lustre for the current kernel supported by Debian, this is usually a version or two ahead of Lustre. That is why I said not to use a vanilla kernel, your chance of an error free build is greater. -- Robert LeBlanc Life Sciences Computer Support Brigham Young University [EMAIL PROTECTED] (801)422-1882 ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
Re: [Lustre-discuss] mounting an OST from another node attached to a fibre channel switch
Yes. There is an entry in the manual on this topic. You'll have to stop Lustre and update the MGS configuration, but it's a pretty quick operation. cheers, Klaus On 8/14/08 2:29 PM, Ron [EMAIL PROTECTED]did etch on stone tablets: Hi, We have set up a couple of test systems where the OSTs are mounted on the same node as the MDT. The OSTs are LUNS on a SATA Beast controller accessible from multiple systems attached to a fibre channel switch,. We would like to umount an OST from the MDT system and mount it on another system. We've tried doing this and even though there is network traffic between the new system and the mds system, the mds system seems to be ignoring the OST mount. Can the an OSTs OSS change? Thanks, Ron ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
[Lustre-discuss] Startup Sequence 1.6.5.1
Is there a recommended startup mounting sequence for 1.6.5.1? We are internally debating MGS lun - MDS luns - OSTs vs. MGS lun - OSTs - MDS luns Mike ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
