Re: [lustre-discuss] Restrict who can assign OST pools to directories
Marco, One other idea is to give an unfriendly pool name that users can't guess. Like "myfs.mkpilaxluia" instead of myfs.flash or myfs.ssd so that it becomes difficult(not impossible though) for users to use :). Users don't have access to MDS to get the entire lists of pools defined. Thanks, Raj On Mon, Nov 7, 2022 at 4:28 AM Andreas Dilger via lustre-discuss wrote: > > Unfortunately, this is not possible today, though I don't think it would be > too hard for someone to implement this by copying "enable_remote_dir_gid" and > similar checks on the MDS. > > In Lustre 2.14 and later, it is possible to set an OST pool quota that can > restrict users from creating too many files in a pool. This doesn't directly > prevent them from setting the pool on a directory (though I guess this > _could_ be checked), but they would get an EDQUOT error when trying to create > in that directory, and quickly tire of trying to use it. > > Cheers, Andreas > > On Nov 4, 2022, at 05:57, Passerini Marco wrote: > > Hi, > > Is there a way in Lustre to restrict who can assign OST pools to directories? > In specific, can we limit the following command so that it can be run by root > only? > > lfs setstripe --pool myfs.mypool test_dir > > I would need something similar to what can be done for remote directories: > lctl set_param mdt.*.enable_remote_dir_gid=1 > > Regards, > Marco Passerini > > > Cheers, Andreas > -- > Andreas Dilger > Lustre Principal Architect > Whamcloud > > > > > > > > ___ > lustre-discuss mailing list > lustre-discuss@lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] Disk usage for users per project
I do not think this exists yet. But, if every user has individual area(sub folder) inside the main project folder, can you create a ‘lustre project’ per project-user? -Raj On Tue, May 17, 2022 at 7:21 AM Kenneth Waegeman via lustre-discuss < lustre-discuss@lists.lustre.org> wrote: > Hi all, > > We have a lustre file system with project quota enabled. We are able to > list the project quota usage with lfs quota -p, and the total usage of a > user on the file system with lfs quota -u, but is there any way to find out > the per user usage within a project ? > > Thank you! > > Kind regards, > Kenneth > > ___ > lustre-discuss mailing list > lustre-discuss@lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org > ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] Essential tools for Lustre
Andreas, Is there any IO penalties in enabling project quota? Will I see the same throughput from the FS? Thanks -Raj On Fri, Apr 15, 2022 at 1:32 PM Andreas Dilger via lustre-discuss < lustre-discuss@lists.lustre.org> wrote: > Note that in newer Lustre releases, if you have project IDs enabled (you > don't need to *enforce* project quotas, just have quota accounting > enabled), that "df" (statfs()) will return the quota for the project ID on > that directory tree. It isn't _quite_ "fast du" for arbitrary directory > trees, but quite useful in any case. > > > On Apr 15, 2022, at 11:17, Steve L via lustre-discuss < > lustre-discuss@lists.lustre.org> wrote: > > Hi Thomas, > > Thanks for the tips. > > I have a client that they are looking into implementing their own "fast > du". I will look into Robinhood to see if there are any duplicated > functions. Not sure if they are looking into advanced functionality like > policy management, thou. > > Does Robinhood offer APIs, or command-line tools to provide "du" like > data? If not, how about direct database accesses? > > /steve > -- > *From:* thomas.leibov...@cea.fr > *Sent:* Friday, April 15, 2022 4:03 AM > *To:* 'Steve L' > *Cc:* lustre-discuss@lists.lustre.org > *Subject:* RE: Essential tools for Lustre > > Dear Steve, > > > >What are the essential tools to make life easier in the Lustre ecosystem? > > > We use and maintain to those 2 open-source tools: > - Shine, for Lustre administration: > https://github.com/cea-hpc/shine > - RobinHood policy engine for managing data in filesystems: > https://github.com/cea-hpc/robinhood/wiki > > > Best regards, > Thomas > > > *De :* lustre-discuss [mailto:lustre-discuss-boun...@lists.lustre.org > ] *De la part de* Steve L via > lustre-discuss > *Envoyé :* samedi 9 avril 2022 00:52 > *À :* lustre-discuss@lists.lustre.org > *Objet :* [lustre-discuss] Essential tools for Lustre > > > Hi All, > > > I am new to Lustre. I like to know if anyone out there is actively using > IML (Integrated Manager for Lustre) to administer and/or monitor Lustre? > > > If yes, I plan to run Lustre LTS (e.g., 2.12.8). Do you know if IML can > run against 2.12.8? IML official website seems to be very quiet for the > last year. Is the project still alive? > > > If no, do you run any other GUI or non-GUI tools to help > administer/monitor? What are the essential tools to make life easier in the > Lustre ecosystem? > > > Any pointers are deeply appreciated. > > > If this is not the right forum to ask such questions, please accept my > apology. I searched the archive, but could not discover any items related > to IML. > > > Thanks ahead for your assistance. > > > Steve > > > ___ > lustre-discuss mailing list > lustre-discuss@lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org > > > Cheers, Andreas > -- > Andreas Dilger > Lustre Principal Architect > Whamcloud > > > > > > > > ___ > lustre-discuss mailing list > lustre-discuss@lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org > ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] Lustre Client Lockup Under Buffered I/O (2.14/2.15)
Ellis, I would also check the peer_credit between server and the client. They should be same. On Wed, Jan 19, 2022 at 9:27 AM Patrick Farrell via lustre-discuss < lustre-discuss@lists.lustre.org> wrote: > Ellis, > > As you may have guessed, that function just set looks like a node which is > doing buffered I/O and thrashing for memory. No particular insight > available from the count of functions there. > > Would you consider opening a bug report in the Whamcloud JIRA? You should > have enough for a good report, here's a few things that would be helpful as > well: > > It sounds like you can hang the node on demand. If you could collect > stack traces with: > > echo t > /proc/sysrq-trigger > > after creating the hang, that would be useful. (It will print to dmesg.) > > You've also collected debug logs - Could you include, say, the last 100 > MiB of that log set? That should be reasonable to attach if compressed. > > Regards, > Patrick > > -- > *From:* lustre-discuss on > behalf of Ellis Wilson via lustre-discuss > > *Sent:* Wednesday, January 19, 2022 8:32 AM > *To:* Andreas Dilger > *Cc:* lustre-discuss@lists.lustre.org > *Subject:* Re: [lustre-discuss] Lustre Client Lockup Under Buffered I/O > (2.14/2.15) > > > Hi Andreas, > > > > Apologies in advance for the top-post. I’m required to use Outlook for > work, and it doesn’t handle in-line or bottom-posting well. > > > > Client-side defaults prior to any tuning of mine (this is a very minimal > 1-client, 1-MDS/MGS, 2-OSS cluster): > > > ~# lctl get_param llite.*.max_cached_mb > > llite.lustrefs-8d52a9c52800.max_cached_mb= > > users: 5 > > max_cached_mb: 7748 > > used_mb: 0 > > unused_mb: 7748 > > reclaim_count: 0 > > ~# lctl get_param osc.*.max_dirty_mb > > osc.lustrefs-OST-osc-8d52a9c52800.max_dirty_mb=1938 > > osc.lustrefs-OST0001-osc-8d52a9c52800.max_dirty_mb=1938 > > ~# lctl get_param osc.*.max_rpcs_in_flight > > osc.lustrefs-OST-osc-8d52a9c52800.max_rpcs_in_flight=8 > > osc.lustrefs-OST0001-osc-8d52a9c52800.max_rpcs_in_flight=8 > > ~# lctl get_param osc.*.max_pages_per_rpc > > osc.lustrefs-OST-osc-8d52a9c52800.max_pages_per_rpc=1024 > > osc.lustrefs-OST0001-osc-8d52a9c52800.max_pages_per_rpc=1024 > > > > Thus far I’ve reduced the following to what I felt were really > conservative values for a 16GB RAM machine: > > > > ~# lctl set_param llite.*.max_cached_mb=1024 > > llite.lustrefs-8d52a9c52800.max_cached_mb=1024 > > ~# lctl set_param osc.*.max_dirty_mb=512 > > osc.lustrefs-OST-osc-8d52a9c52800.max_dirty_mb=512 > > osc.lustrefs-OST0001-osc-8d52a9c52800.max_dirty_mb=512 > > ~# lctl set_param osc.*.max_pages_per_rpc=128 > > osc.lustrefs-OST-osc-8d52a9c52800.max_pages_per_rpc=128 > > osc.lustrefs-OST0001-osc-8d52a9c52800.max_pages_per_rpc=128 > > ~# lctl set_param osc.*.max_rpcs_in_flight=2 > > osc.lustrefs-OST-osc-8d52a9c52800.max_rpcs_in_flight=2 > > osc.lustrefs-OST0001-osc-8d52a9c52800.max_rpcs_in_flight=2 > > > > This slows down how fast I get to basically OOM from <10 seconds to more > like 25 seconds, but the trend is identical. > > > > As an example of what I’m seeing on the client, you can see below we start > with most free, and then iozone rapidly (within ~10 seconds) causes all > memory to be marked used, and that stabilizes at about 140MB free until at > some point it stalls for 20 or more seconds and then some has been synced > out: > > > ~# dstat --mem > > --memory-usage- > > used free buff cach > > 1029M 13.9G 2756k 215M > > 1028M 13.9G 2756k 215M > > 1028M 13.9G 2756k 215M > > 1088M 13.9G 2756k 215M > > 2550M 11.5G 2764k 1238M > > 3989M 10.1G 2764k 1236M > > 5404M 8881M 2764k 1239M > > 6831M 7453M 2772k 1240M > > 8254M 6033M 2772k 1237M > > 9672M 4613M 2772k 1239M > > 10.6G 3462M 2772k 1240M > > 12.1G 1902M 2772k 1240M > > 13.4G 582M 2772k 1240M > > 13.9G 139M 2488k 1161M > > 13.9G 139M 1528k 1174M > > 13.9G 140M 896k 1175M > > 13.9G 139M 676k 1176M > > 13.9G 142M 528k 1177M > > 13.9G 140M 484k 1188M > > 13.9G 139M 492k 1188M > > 13.9G 139M 488k 1188M > > 13.9G 141M 488k 1186M > > 13.9G 141M 480k 1187M > > 13.9G 139M 492k 1188M > > 13.9G 141M 600k 1188M > > 13.9G 139M 580k 1187M > > 13.9G 140M 536k 1186M > > 13.9G 141M 668k 1186M > > 13.9G 139M 580k 1188M > > 13.9G 140M 568k 1187M > > 12.7G 1299M 2064k 1197M missed 20 ticks <-- client is totally unresponsive > during this time > > 11.0G 2972M 5404k 1238M^C > > > > Additionally, I’ve messed with sysctl settings. Defaults: > > vm.dirty_background_bytes = 0 > > vm.dirty_background_ratio = 10 > > vm.dirty_bytes = 0 > > vm.dirty_expire_centisecs = 3000 > > vm.dirty_ratio = 20 > > vm.dirty_writeback_centisecs = 500 > > > > Revised to conservative values: > > vm.dirty_background_bytes = 1073741824 > > vm.dirty_background_ratio = 0 > > vm.dirty_bytes = 2147483648 > > vm.dirty_expire_centisecs = 200 > > vm.dirty_ratio = 0 > >
Re: [lustre-discuss] [EXTERNAL] good ways to identify clients causing problems?
One other way is to install xltop(https://github.com/jhammond/xltop) and use xltop client (ncurses based linux top like tool) to watch for top client with more requests per sec (xltop -k q h). You can also use it to track jobs but you might have to write your own nodes to job mapping script (xltop-clusterd). On Fri, May 28, 2021 at 4:21 PM Mohr, Rick via lustre-discuss wrote: > > Bill, > > One option I have used in the past is to look at the rpc request history. > For example, on an oss server, you can run: > > lctl get_param ost.OSS.ost_io.req_history > > and then extract the client nid for each request. Based on that, you can > calculate the number of requests coming into the server and look for any > clients that are significantly higher than the others. Maybe something like: > > lctl get_param ost.OSS.ost_io.req_history | cut -d: -f3 | sort | uniq -c | > sort -n > > I have used that approach in the past to identify misbehaving clients (the > number of requests from such clients was usually one or two orders of > magnitude higher than the others). If multiple clients are unusually high, > you may be able to correlate the nodes with currently running jobs to > identify a particular job (assuming you don't already have lustre job stats > enabled). > > -Rick > > > On 5/4/21, 2:41 PM, "lustre-discuss on behalf of Bill Anderson via > lustre-discuss" lustre-discuss@lists.lustre.org> wrote: > > >Hi All, > >Can you recommend good ways to identify Lustre client hosts that might > be causing stability or performance problems for the entire filesystem? > >For example, if a user is inadvertently doing something that's > creating an RPC storm, what are good ways to identify the client host that > has triggered the storm? > >Thank you! > >Bill > > ___ > lustre-discuss mailing list > lustre-discuss@lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] Multiple IB interfaces
Alastair, Few scenarios which you may consider: 1) define 2 lnets one per IB interface (say o2ib1 and o2ib2) and share out one OST through o2ib1 and other one through o2ib2. You can map HBA and disk locality so that they are attached to the same cpu. 2) Same as above but share the ost/s from both lnets But configure odd clients (clients with odd ips) to use o2ib1 and even clients to use o2ib2. This may not be exactly what you are looking for but can efficiently utilize both interfaces. -Raj On Tue, Mar 9, 2021 at 9:18 AM Alastair Basden via lustre-discuss < lustre-discuss@lists.lustre.org> wrote: > Hi, > > We are installing some new Lustre servers with 2 InfiniBand cards, 1 > attached to each CPU socket. Storage is nvme, again, some drives attached > to each socket. > > We want to ensure that data to/from each drive uses the appropriate IB > card, and doesn't need to travel through the inter-cpu link. Data being > written is fairly easy I think, we just set that OST to the appropriate IP > address. However, data being read may well go out the other NIC, if I > understand correctly. > > What setup do we need for this? > > I think probably not bonding, as that will presumably not tie > NIC interfaces to cpus. But I also see a note in the Lustre manual: > > """If the server has multiple interfaces on the same subnet, the Linux > kernel will send all traffic using the first configured interface. This is > a limitation of Linux, not Lustre. In this case, network interface bonding > should be used. For more information about network interface bonding, see > Chapter 7, Setting Up Network Interface Bonding.""" > > (plus, no idea if bonding is supported on InfiniBand). > > Thanks, > Alastair. > ___ > lustre-discuss mailing list > lustre-discuss@lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org > ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
