The journal pages are effectively pinned I think, so with 400MB journals on each OST, count on losing 400MB of memory at least.
Here is some anecdotal data from our environment: We seem to do OK with 4G nodes having 3 OST's each, including sometimes running with 6 during failover. On the other hand 2G nodes with 2 OST's each would run out of memory when running with 4 during failover. We had to reduce the journal size from 400MB to 200MB for failover to work reliably on the 2G nodes. In summary, with 400MB journals: - 0.66G per OST works - 0.50G per OST doesn't That's with servers running Lustre 1.6.6, RHEL 5.3, x86_64 arch, and socklnd. Jim On Wed, Aug 19, 2009 at 02:40:35PM +0200, Arne Wiebalck wrote: > > OK, I will keep that in mind. > > Thanks, > Arne > > Sébastien Buisson wrote: > >Hi, > > > >Just a small feedback from our own experience. > >I agree with Brian about the fact that there is no strong limit on the > >number of OSTs per OSS in the Lustre code. But one should really take > >into account the available memory on OSSes when defining the number of > >OSTs per OSS (and so the size of each OST). If you do not have 1GB or > >1.2 GB of memory per OST on your OSSes, you will run into serious > >trouble with "out of memory" messages. > > > >For instance, if you want 8 OSTs per OSS, your OSSes should have at > >least 10GB of RAM. > > > >Unfortunately we experienced those "out of memory" problems, so I advise > >you to read Lustre Operations Manual chapter 33.12 "OSS RAM Size for a > >Single OST". > > > >Cheers, > >Sebastien. > > > > > >Arne Wiebalck a écrit : > >>Brian J. Murrell wrote: > >>>On Wed, 2009-08-19 at 11:45 +0200, Arne Wiebalck wrote: > >>>>is there a limit on the number of OSTs per OSS? > >>>I have never run across one so there is likely no *practical* limit. I > >>>could very well be corrected on this though. > >>> > >>>Unless you are making lots and lots of small OSTs -- which is not > >>>usually beneficial anyway -- typically, you will run into resource > >>>limitations (memory, bus bandwidth, etc.) on an OSS before you hit a > >>>limit on the number of OSTs. > >>I just thought I remembered there was something like 8 OSTs per OSS, > >>but apparently I was wrong. > >> > >>>Just how many OSTs are you considering, and how big will they be? > >>My OSSs will have 10 OSTs with 1TB each. > >> > >>Thanks for the quick reply, > >> Arne > >> > >> > >>------------------------------------------------------------------------ > >> > >>_______________________________________________ > >>Lustre-discuss mailing list > >>[email protected] > >>http://lists.lustre.org/mailman/listinfo/lustre-discuss > > _______________________________________________ > Lustre-discuss mailing list > [email protected] > http://*lists.lustre.org/mailman/listinfo/lustre-discuss _______________________________________________ Lustre-discuss mailing list [email protected] http://lists.lustre.org/mailman/listinfo/lustre-discuss
