[Lustre-discuss] stripe don't work
Hello! I just migrate my lustre filesystem to version 2.2. I am trying to activate the stripe, but when I copy a file into the filesystem, it's allocated into only one OST. This is my procedure: From client: # lfs setstripe -s 1M /mnt/data/ # lfs df UUID 1K-blocks Used Available Use% Mounted on cetafs-MDT_UUID 1462920404 501360 1364873324 0% /mnt/data[MDT:0] cetafs-OST_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:0] cetafs-OST0001_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:1] cetafs-OST0002_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:2] cetafs-OST0003_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:3] cetafs-OST0004_UUID 9760101272 450304 9267055468 0% /mnt/data[OST:4] cetafs-OST0005_UUID 9760101272 450304 9271366744 0% /mnt/data[OST:5] cetafs-OST0006_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:6] cetafs-OST0007_UUID 9760101272 450304 9271366744 0% /mnt/data[OST:7] cetafs-OST0008_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:8] cetafs-OST0009_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:9] cetafs-OST000a_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:10] cetafs-OST000b_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:11] cetafs-OST000c_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:12] cetafs-OST000d_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:13] cetafs-OST000e_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:14] cetafs-OST000f_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:15] cetafs-OST0010_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:16] cetafs-OST0011_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:17] filesystem summary: 175681822896 8105472 166884732464 0% /mnt/data Then, I copy a 4'5GB file, but...: # lfs getstripe CentOS-6.2-x86_64-bin-DVD1.iso CentOS-6.2-x86_64-bin-DVD1.iso lmm_stripe_count: 1 lmm_stripe_size: 262144 lmm_stripe_offset: 4 obdidx objid objid group 4 2 0x2 0 # lfs df UUID 1K-blocks Used Available Use% Mounted on cetafs-MDT_UUID 1462920404 501360 1364873324 0% /mnt/data[MDT:0] cetafs-OST_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:0] cetafs-OST0001_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:1] cetafs-OST0002_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:2] cetafs-OST0003_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:3] cetafs-OST0004_UUID 9760101272 4769772 9267055468 0% /mnt/data[OST:4] cetafs-OST0005_UUID 9760101272 450304 9271366744 0% /mnt/data[OST:5] cetafs-OST0006_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:6] cetafs-OST0007_UUID 9760101272 450304 9271366744 0% /mnt/data[OST:7] cetafs-OST0008_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:8] cetafs-OST0009_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:9] cetafs-OST000a_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:10] cetafs-OST000b_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:11] cetafs-OST000c_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:12] cetafs-OST000d_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:13] cetafs-OST000e_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:14] cetafs-OST000f_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:15] cetafs-OST0010_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:16] cetafs-OST0011_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:17] filesystem summary: 175681822896 12424940 166880412996 0% /mnt/data The file is only allocated into OST with index 4 Can someone help me? Thanks -- Alfonso Pardo Daz IT Manager c/ Sola n 1; 10200 Trujillo, ESPAA Tel: +34 927 65 93 17 Fax: +34 927 32 32 37 Confidencialidad: Este mensaje y sus ficheros adjuntos se dirige exclusivamente a su destinatario y puede contener información privilegiada o confidencial. Si no es vd. el destinatario indicado, queda notificado de que la utilización, divulgación y/o copia sin autorización está prohibida en virtud de la legislación vigente. Si ha recibido este mensaje por error, le rogamos que nos lo comunique inmediatamente respondiendo al
Re: [Lustre-discuss] stripe don't work
Hi, The output indicates stripe count of 1. Use lfs setstripe to set the stripe count to something other than 1. I.e. -c -1 for all osts. On May 28, 2012 7:34 AM, Alfonso Pardo alfonso.pa...@ciemat.es wrote: Hello! I just migrate my lustre filesystem to version 2.2. I am trying to activate the stripe, but when I copy a file into the filesystem, it's allocated into only one OST. This is my procedure: *From client: **# **lfs setstripe -s 1M /mnt/data/ **# **lfs df UUID 1K-blocksUsed Available Use% Mounted on cetafs-MDT_UUID 1462920404 501360 1364873324 0% /mnt/data[MDT:0] cetafs-OST_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:0] cetafs-OST0001_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:1] cetafs-OST0002_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:2] cetafs-OST0003_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:3] cetafs-OST0004_UUID 9760101272 450304 9267055468 0% /mnt/data[OST:4] cetafs-OST0005_UUID 9760101272 450304 9271366744 0% /mnt/data[OST:5] cetafs-OST0006_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:6] cetafs-OST0007_UUID 9760101272 450304 9271366744 0% /mnt/data[OST:7] cetafs-OST0008_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:8] cetafs-OST0009_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:9] cetafs-OST000a_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:10] cetafs-OST000b_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:11] cetafs-OST000c_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:12] cetafs-OST000d_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:13] cetafs-OST000e_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:14] cetafs-OST000f_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:15] cetafs-OST0010_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:16] cetafs-OST0011_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:17] filesystem summary: 175681822896 8105472 166884732464 0% /mnt/data *Then, I copy a 4'5GB file, but...:* # lfs getstripe CentOS-6.2-x86_64-bin-DVD1.iso CentOS-6.2-x86_64-bin-DVD1.iso lmm_stripe_count: 1 lmm_stripe_size:262144 lmm_stripe_offset: 4 obdidx objidobjid group 4 2 0x2 0 **# **lfs df UUID 1K-blocksUsed Available Use% Mounted on cetafs-MDT_UUID 1462920404 501360 1364873324 0% /mnt/data[MDT:0] cetafs-OST_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:0] cetafs-OST0001_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:1] cetafs-OST0002_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:2] cetafs-OST0003_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:3] cetafs-OST0004_UUID 9760101272 4769772 9267055468 0% /mnt/data[OST:4] cetafs-OST0005_UUID 9760101272 450304 9271366744 0% /mnt/data[OST:5] cetafs-OST0006_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:6] cetafs-OST0007_UUID 9760101272 450304 9271366744 0% /mnt/data[OST:7] cetafs-OST0008_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:8] cetafs-OST0009_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:9] cetafs-OST000a_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:10] cetafs-OST000b_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:11] cetafs-OST000c_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:12] cetafs-OST000d_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:13] cetafs-OST000e_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:14] cetafs-OST000f_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:15] cetafs-OST0010_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:16] cetafs-OST0011_UUID 9760101272 450304 9271374936 0% /mnt/data[OST:17] filesystem summary: 17568182289612424940 166880412996 0% /mnt/data* The file is only allocated into OST with index 4 Can someone help me? Thanks -- *Alfonso Pardo Díaz* *IT Manager* *c/ Sola nº 1; 10200 Trujillo, ESPAÑA* *Tel: +34 927 65 93 17 Fax: +34 927 32 32 37* [image: CETA-Ciemat logo] http://www.ceta-ciemat.es/ Confidencialidad: Este mensaje y sus ficheros adjuntos se dirige exclusivamente a su destinatario y puede contener información privilegiada o confidencial. Si no es vd. el destinatario indicado, queda notificado de que la utilización, divulgación y/o copia sin autorización está prohibida en virtud de la legislación vigente. Si ha recibido este mensaje por error, le rogamos que nos lo comunique inmediatamente respondiendo al mensaje y proceda a su destrucción. Disclaimer:
Re: [Lustre-discuss] stripe don't work
lmm_stripe_count: 1 [...] The file is only allocated into OST with index 4 Yes, because the stripe *count* is set to '1'. Set it to '-1' to use all OSTs: $ lfs setstripe -c -1 -s 1M . ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
[Lustre-discuss] Tar backup of MDT runs extremely slow, tar pauses on pointers to very large files
Greetings, I am aiding in the recovery of a multi-Petabyte Lustre filesystem (1.8.7) that went down hard due to site wide power loss. Power loss caused the MDT RAID volume to be put in a critical state and I was able to get the md raid based MDT device mounted read only and the MDT mounted read only as type ldiskfs. I was able to successfully backup the extended attributes of the MDT. This process took about 10 minutes. The tar backup of the MDT is taking a very long time. So far it has backed up 1.6GB of the 5.0GB used in nine hours. In watching the tar process pointers to small or average size files are backed up quickly and at a consistent pace. When tar encounters a pointer/inode belonging to a very large file (100GB+) the tar process stalls on that file for a very long time, as if it were trying to archive the real filesize amount of data rather than the pointer/inode. During this process there are no errors reported by kernel, ldiskfs, md or tar. Nothing that would indiciate why things are so slow on pointers to large files. In watching the tar process the CPU utilization is at or near 100% so it is doing something. Running iostat at the same time shows that while tar is at or near 100% CPU there are no reads taking place on the MDT device and no writes to the device where the tarball is being written. It appears that the tar process goes to outer space when it encounters pointers to very large files. Is this expected behavior? The backup command used is the one from the MDT backup process in the 1.8 manual: 'tar zcvf tarfile --sparse .' df reports the ldiskfs MDT as 5GB used: /dev/md0 2636788616 5192372 2455778504 1% /mnt/mdt df -i reports the ldiskfs MDT as having 10,300,000 inodes used: /dev/md0 1758199808 10353389 17478464191% /mnt/mdt Any feedback is appreciated! --Jeff -- -- Jeff Johnson Partner Aeon Computing jeff dot johnson at aeoncomputing.com www.aeoncomputing.com t: 858-412-3810 x101 f: 858-412-3845 4905 Morena Boulevard, Suite 1313 - San Diego, CA 92117 ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
