Andreas Dilger wrote:
On May 13, 2008 19:41 +0200, Thomas Roth wrote:
I'm still in trouble with numbers: the available, used and necessary
space on my MDT:
According to "lfs df", I have now filled my file system with 115.3 TB.
All of these files are sized 5 MB. That should be roughly 24 million files.
For the MDT, "lfs df" reports 28.2 GB used.
Now I believed that creating a file on Lustre means using one inode on
the MDT. Since all of my Lustre partitions were formatted with the
default options (all of this is running Lustre v. 1.6.4.3, btw), an
inode should eat up 4kB on the MDT partition. Of course, 24 million
files times 4 kB gives you 91 GB rather than 28GB.
Obviously, there is something I missed completely. Perhaps somebody
could illuminate me here?
This issue could also be phrased as "How large should my MDT be to
accommodate n TB storage space?" The manual's answer boils down to "=
number of files * 4 kB " (*2 per recommendation). That's how I
calculated above - maybe my test system is broken? I can't check on the
content of these files, it's just 5MB test files created with the
'stress' utility.
Please provide output of "lfs df" and "lfs df -i" so we can see the
actual numbers.
Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.
Ok, I'm attaching these two files with the output. You'll notice some
OSTs with a lower fill level: these OSS had some hardware problems
during the production of my (now counted) 22.7 million files.
Greetings,
Thomas
# lfs df
UUID 1K-blocks Used Available Use% Mounted on
gsilust1-MDT0000_UUID 495497804 29575416 465922388 5% /lustre[MDT:0]
gsilust1-OST0000_UUID 2403044744 2208409576 194635168 91% /lustre[OST:0]
gsilust1-OST0001_UUID 2883653852 2648332660 235321192 91% /lustre[OST:1]
gsilust1-OST0002_UUID 2403044744 2131046760 271997984 88% /lustre[OST:2]
gsilust1-OST0003_UUID 2883653852 2557946316 325707536 88% /lustre[OST:3]
gsilust1-OST0004_UUID 2403044744 2206128516 196916228 91% /lustre[OST:4]
gsilust1-OST0005_UUID 2883653852 2647658256 235995596 91% /lustre[OST:5]
gsilust1-OST0006_UUID 2403044744 2197323104 205721640 91% /lustre[OST:6]
gsilust1-OST0007_UUID 2883653852 2636058844 247595008 91% /lustre[OST:7]
gsilust1-OST0008_UUID 2403044744 2207045972 195998772 91% /lustre[OST:8]
gsilust1-OST0009_UUID 2883653852 2648436060 235217792 91% /lustre[OST:9]
gsilust1-OST000a_UUID 2403044744 2205203944 197840800 91% /lustre[OST:10]
gsilust1-OST000b_UUID 2883653852 2648915248 234738604 91% /lustre[OST:11]
gsilust1-OST000c_UUID 2403044744 2206725516 196319228 91% /lustre[OST:12]
gsilust1-OST000d_UUID 2883653852 1449668056 1433985796 50% /lustre[OST:13]
gsilust1-OST000e_UUID 2403044744 2206148640 196896104 91% /lustre[OST:14]
gsilust1-OST000f_UUID 2883653852 2648466960 235186892 91% /lustre[OST:15]
gsilust1-OST0010_UUID 2403044744 2205981044 197063700 91% /lustre[OST:16]
gsilust1-OST0011_UUID 2883653852 2648466180 235187672 91% /lustre[OST:17]
gsilust1-OST0012_UUID 2403044744 2197113560 205931184 91% /lustre[OST:18]
gsilust1-OST0013_UUID 2883653852 2638146024 245507828 91% /lustre[OST:19]
gsilust1-OST0014_UUID 2403044744 2206067992 196976752 91% /lustre[OST:20]
gsilust1-OST0015_UUID 2883653852 2647881580 235772272 91% /lustre[OST:21]
gsilust1-OST0016_UUID 2403044744 2205237108 197807636 91% /lustre[OST:22]
gsilust1-OST0017_UUID 2883653852 2646574192 237079660 91% /lustre[OST:23]
gsilust1-OST0018_UUID 2403044744 2205484856 197559888 91% /lustre[OST:24]
gsilust1-OST0019_UUID 2883653852 2645678088 237975764 91% /lustre[OST:25]
gsilust1-OST001a_UUID 2403044744 2206248696 196796048 91% /lustre[OST:26]
gsilust1-OST001b_UUID 2883653852 2647041328 236612524 91% /lustre[OST:27]
gsilust1-OST001c_UUID 2403044744 2207710496 195334248 91% /lustre[OST:28]
gsilust1-OST001d_UUID 2883653852 2649304508 234349344 91% /lustre[OST:29]
gsilust1-OST001e_UUID 2403044744 2206033636 197011108 91% /lustre[OST:30]
gsilust1-OST001f_UUID 2883653852 2648367388 235286464 91% /lustre[OST:31]
gsilust1-OST0020_UUID 2403044744 2207774816 195269928 91% /lustre[OST:32]
gsilust1-OST0021_UUID 2883653852 2649212840 234441012 91% /lustre[OST:33]
gsilust1-OST0022_UUID 2403044744 2203817988 199226756 91% /lustre[OST:34]
gsilust1-OST0023_UUID 2883653852 2646795736 236858116 91% /lustre[OST:35]
gsilust1-OST0024_UUID 2403044744 1868659544 534385200 77% /lustre[OST:36]
gsilust1-OST0025_UUID 2883653852 2243676508 639977344 77% /lustre[OST:37]
gsilust1-OST0026_UUID 2403044744 2205775772 197268972 91% /lustre[OST:38]
gsilust1-OST0027_UUID 2883653852 2647277724 236376128 91% /lustre[OST:39]
gsilust1-OST0028_UUID 2403044744 2208664348 194380396 91% /lustre[OST:40]
gsilust1-OST0029_UUID 2883653852 2647154096 236499756 91% /lustre[OST:41]
gsilust1-OST002a_UUID 2403044744 2206053488 196991256 91% /lustre[OST:42]
gsilust1-OST002b_UUID 2883653852 2647567536 236086316 91% /lustre[OST:43]
gsilust1-OST002c_UUID 2403044744 2197169140 205875604 91% /lustre[OST:44]
gsilust1-OST002d_UUID 2883653852 2636726236 246927616 91% /lustre[OST:45]
gsilust1-OST002e_UUID 2403044744 2196736784 206307960 91% /lustre[OST:46]
gsilust1-OST002f_UUID 2883653852 2636158280 247495572 91% /lustre[OST:47]
gsilust1-OST0030_UUID 2403044744 2127152780 275891964 88% /lustre[OST:48]
gsilust1-OST0031_UUID 2883653852 2555100512 328553340 88% /lustre[OST:49]
gsilust1-OST0032_UUID 2403044744 2205456928 197587816 91% /lustre[OST:50]
gsilust1-OST0033_UUID 2883653852 2650130140 233523712 91% /lustre[OST:51]
filesystem summary: 137454163496 123851912300 13602251196 90% /lustre
# lfs df -i
UUID Inodes IUsed IFree IUse% Mounted on
gsilust1-MDT0000_UUID 141590528 22816486 118774042 16% /lustre[MDT:0]
gsilust1-OST0000_UUID 79583043 407308 79175735 0% /lustre[OST:0]
gsilust1-OST0001_UUID 95939053 488426 95450627 0% /lustre[OST:1]
gsilust1-OST0002_UUID 98908655 392216 98516439 0% /lustre[OST:2]
gsilust1-OST0003_UUID 118518048 470835 118047213 0% /lustre[OST:3]
gsilust1-OST0004_UUID 80152929 406929 79746000 0% /lustre[OST:4]
gsilust1-OST0005_UUID 96107538 488310 95619228 0% /lustre[OST:5]
gsilust1-OST0006_UUID 82352633 405280 81947353 0% /lustre[OST:6]
gsilust1-OST0007_UUID 99005270 486189 98519081 0% /lustre[OST:7]
gsilust1-OST0008_UUID 79923681 407045 79516636 0% /lustre[OST:8]
gsilust1-OST0009_UUID 95913223 488446 95424777 0% /lustre[OST:9]
gsilust1-OST000a_UUID 80383880 406737 79977143 0% /lustre[OST:10]
gsilust1-OST000b_UUID 95793561 488581 95304980 0% /lustre[OST:11]
gsilust1-OST000c_UUID 80003809 407059 79596750 0% /lustre[OST:12]
gsilust1-OST000d_UUID 183107584 254523 182853061 0% /lustre[OST:13]
gsilust1-OST000e_UUID 80147843 406874 79740969 0% /lustre[OST:14]
gsilust1-OST000f_UUID 95905520 488468 95417052 0% /lustre[OST:15]
gsilust1-OST0010_UUID 80189759 406891 79782868 0% /lustre[OST:16]
gsilust1-OST0011_UUID 95905738 488491 95417247 0% /lustre[OST:17]
gsilust1-OST0012_UUID 82404960 405221 81999739 0% /lustre[OST:18]
gsilust1-OST0013_UUID 98483893 486607 97997286 0% /lustre[OST:19]
gsilust1-OST0014_UUID 80168024 406893 79761131 0% /lustre[OST:20]
gsilust1-OST0015_UUID 96051801 488404 95563397 0% /lustre[OST:21]
gsilust1-OST0016_UUID 80375578 406726 79968852 0% /lustre[OST:22]
gsilust1-OST0017_UUID 96378384 488140 95890244 0% /lustre[OST:23]
gsilust1-OST0018_UUID 80313665 406750 79906915 0% /lustre[OST:24]
gsilust1-OST0019_UUID 96602213 487943 96114270 0% /lustre[OST:25]
gsilust1-OST001a_UUID 80122878 406923 79715955 0% /lustre[OST:26]
gsilust1-OST001b_UUID 96261703 488243 95773460 0% /lustre[OST:27]
gsilust1-OST001c_UUID 79757758 407253 79350505 0% /lustre[OST:28]
gsilust1-OST001d_UUID 95696293 488628 95207665 0% /lustre[OST:29]
gsilust1-OST001e_UUID 80176610 406890 79769720 0% /lustre[OST:30]
gsilust1-OST001f_UUID 95930453 488508 95441945 0% /lustre[OST:31]
gsilust1-OST0020_UUID 79741629 407204 79334425 0% /lustre[OST:32]
gsilust1-OST0021_UUID 95719241 488659 95230582 0% /lustre[OST:33]
gsilust1-OST0022_UUID 80730098 406466 80323632 0% /lustre[OST:34]
gsilust1-OST0023_UUID 96323031 488173 95834858 0% /lustre[OST:35]
gsilust1-OST0024_UUID 152592384 341011 152251373 0% /lustre[OST:36]
gsilust1-OST0025_UUID 183107584 409411 182698173 0% /lustre[OST:37]
gsilust1-OST0026_UUID 80241000 406814 79834186 0% /lustre[OST:38]
gsilust1-OST0027_UUID 96202598 488237 95714361 0% /lustre[OST:39]
gsilust1-OST0028_UUID 79519409 407367 79112042 0% /lustre[OST:40]
gsilust1-OST0029_UUID 96233492 488224 95745268 0% /lustre[OST:41]
gsilust1-OST002a_UUID 80171622 406865 79764757 0% /lustre[OST:42]
gsilust1-OST002b_UUID 96130241 488333 95641908 0% /lustre[OST:43]
gsilust1-OST002c_UUID 82391112 405268 81985844 0% /lustre[OST:44]
gsilust1-OST002d_UUID 98838567 486334 98352233 0% /lustre[OST:45]
gsilust1-OST002e_UUID 82499023 405090 82093933 0% /lustre[OST:46]
gsilust1-OST002f_UUID 98980406 486184 98494222 0% /lustre[OST:47]
gsilust1-OST0030_UUID 99881387 391453 99489934 0% /lustre[OST:48]
gsilust1-OST0031_UUID 119228929 470265 118758664 0% /lustre[OST:49]
gsilust1-OST0032_UUID 80320717 406820 79913897 0% /lustre[OST:50]
gsilust1-OST0033_UUID 95490008 488751 95001257 0% /lustre[OST:51]
filesystem summary: 141590528 22816486 118774042 16% /lustre
_______________________________________________
Lustre-discuss mailing list
[email protected]
http://lists.lustre.org/mailman/listinfo/lustre-discuss
