Re: [gpfsug-discuss] Backup All Cluster GSS GPFS Storage Server
I'm not familiar with GSS, but we have a script that executes the following
before backing up a GPFS filesystem so that we have human-readable
configuration information:
mmlsconfig
mmlsnsd
mmlscluster
mmlsnode
mmlsdisk ${FS_NAME} -L
mmlsfileset ${FS_NAME} -L
mmlspool ${FS_NAME} all -L
mmlslicense -L
mmlspolicy ${FS_NAME} -L
And then executes this for the benefit of GPFS:
mmbackupconfig
Of course there's quite a bit of overlap for clusters that have more than
one filesystem, and even more for filesystems that we backup at the
fileset level, but disk is cheap and the hope is it'll make a DR scenario
a little bit less harrowing.
On Sun, Oct 15, 2017 at 12:44:42PM +, atmane khiredine wrote:
> Dear All,
>
> Is there a way to save the GPS configuration?
>
> OR how backup all GSS
>
> no backup of data or metadata only configuration for disaster recovery
>
> for example:
> stanza
> vdisk
> pdisk
> RAID code
> recovery group
> array
>
> Thank you
>
> Atmane Khiredine
> HPC System Administrator | Office National de la Météorologie
> Tél : +213 21 50 73 93 # 303 | Fax : +213 21 50 79 40 | E-mail :
> a.khired...@meteo.dz
> ___
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
--
-- Skylar Thompson (skyl...@u.washington.edu)
-- Genome Sciences Department, System Administrator
-- Foege Building S046, (206)-685-7354
-- University of Washington School of Medicine
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] FIle system vs Database
On Thu, Nov 30, 2017 at 01:34:05PM -0500, Marc A Kaplan wrote: > It would be interesting to know how well Spectrum Scale large directory > and small file features work in these sort of DB-ish applications. > > You might want to optimize by creating a file system provisioned and tuned > for such application... > > Regardless of file system, `ls -1 | grep ...` in a huge directory is not > going to be a good idea. But stats and/or opens on a huge directory to > look for a particular file should work pretty well... I've wondered if it would be worthwhile having POSIX look-alike commands like ls and find that plug into the GPFS API rather than making VFS calls. That's of course a project for my Copious Free Time... -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -- University of Washington School of Medicine ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] Backing up GPFS config
I can't remember if I replied to that post or a different one, but these
are the commands we capture output for before running mmbackup:
mmlsconfig
mmlsnsd
mmlscluster
mmlscluster --cnfs
mmlscluster --ces
mmlsnode
mmlsdisk ${FS_NAME} -L
mmlspool ${FS_NAME} all -L
mmlslicense -L
mmlspolicy ${FS_NAME} -L
mmbackupconfig ${FS_NAME}
All the commands but mmbackupconfig produce human-readable output, while
mmbackupconfig produces machine-readable output suitable for recovering the
filesystem in a disaster.
On Tue, Nov 14, 2017 at 10:16:44AM +, Sobey, Richard A wrote:
> All,
>
> A few months ago someone posted to the list all the commands they run to back
> up their GPFS configuration. Including mmlsfileset -L, the output of
> mmlsconfig etc, so that in the event of a proper "crap your pants" moment you
> can not only restore your data, but also your whole configuration.
>
> I cannot seem to find this post... does the OP remember and could kindly
> forward it on to me, or the list again?
>
> Thanks
> Richard
> ___
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
--
-- Skylar Thompson (skyl...@u.washington.edu)
-- Genome Sciences Department, System Administrator
-- Foege Building S046, (206)-685-7354
-- University of Washington School of Medicine
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] FreeBSD client?
I doubt it, since IBM would need to tailor a kernel layer for FreeBSD (not the kind of thing you can run with the x86 Linux userspace emulation in FreeBSD), which would be a lot of work for not a lot of demand. On Tue, Oct 31, 2017 at 01:10:47PM +, Peter Smith wrote: > Hi > > Does such a thing exist? :-) -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -- University of Washington School of Medicine ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] Not recommended, but why not?
Our experience is that CES (at least NFS/ganesha) can easily consume all of the CPU resources on a system. If you're running it on the same hardware as your NSD services, then you risk delaying native GPFS I/O requests as well. We haven't found a great way to limit the amount of resources that NFS/ganesha can use, though maybe in the future it could be put in a cgroup since it's all user-space? On Fri, May 04, 2018 at 03:38:57PM +, Buterbaugh, Kevin L wrote: > Hi All, > > In doing some research, I have come across numerous places (IBM docs, > DeveloperWorks posts, etc.) where it is stated that it is not recommended to > run CES on NSD servers ??? but I???ve not found any detailed explanation of > why not. > > I understand that CES, especially if you enable SMB, can be a resource hog. > But if I size the servers appropriately ??? say, late model boxes with 2 x 8 > core CPU???s, 256 GB RAM, 10 GbE networking ??? is there any reason why I > still should not combine the two? > > To answer the question of why I would want to ??? simple, server licenses. > > Thanks??? > > Kevin > > ??? > Kevin Buterbaugh - Senior System Administrator > Vanderbilt University - Advanced Computing Center for Research and Education > kevin.buterba...@vanderbilt.edu<mailto:kevin.buterba...@vanderbilt.edu> - > (615)875-9633 > > > > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -- University of Washington School of Medicine ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] Snapshots for backups
On Wed, May 09, 2018 at 04:33:26PM -0400, valdis.kletni...@vt.edu wrote: > On Wed, 09 May 2018 15:01:55 -0400, "Marc A Kaplan" said: > > > I see there are also low-power / zero-power disk archive/arrays available. > > Any experience with those? > > The last time I looked at those (which was a few years ago) they were > competitive > with tape for power consumption, but not on cost per terabyte - it takes a > lot less > cable and hardware to hook up a dozen tape drives and a robot arm that can > reach 10,000 volumes than it does to wire up 10,000 disks of which only 500 > are > actually spinning at any given time... I also wonder what the lifespan of cold-storage hard drives are relative to tape. With BaFe universal for LTO now, our failure rate for tapes has gone way down (not that it was very high relative to HDDs anyways). FWIW, the operating+capital costs we recharge our grants for tape storage is ~50% of what we recharge them for bulk disk storage. -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -- University of Washington School of Medicine ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] Question concerning integration of CES with AD authentication system
I haven't needed to change the LDAP attributes that CES uses, but I do see --user-id-attrib in the mmuserauth documentation. Unfortunately, I don't see an equivalent one for gidNumber. On Thu, May 24, 2018 at 08:45:00AM +, Dorigo Alvise (PSI) wrote: > Dear members, > at PSI I'm trying to integrate the CES service with our AD authentication > system. > > My understanding, after talking to expert people here, is that I should use > the RFC2307 model for ID mapping (described here: https://goo.gl/XvqHDH). The > problem is that our ID schema is slightly different than that one described > in RFC2307. In the RFC the relevant user identification fields are named > "uidNumber" and "gidNumber". But in our AD database schema we have: > > # egrep 'uid_number|gid_number' /etc/sssd/sssd.conf > ldap_user_uid_number = msSFU30UidNumber > ldap_user_gid_number = msSFU30GidNumber > ldap_group_gid_number = msSFU30GidNumber > > My question is: is it possible to configure CES to look for the custom field > labels (those ones listed above) instead the default ones officially > described in rfc2307 ? > > many thanks. > Regards, > >Alvise Dorigo > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -- University of Washington School of Medicine ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] Question concerning integration of CES with AD authentication system
On Thu, May 24, 2018 at 03:46:32PM +0100, Jonathan Buzzard wrote: > On Thu, 2018-05-24 at 14:16 +0000, Skylar Thompson wrote: > > I haven't needed to change the LDAP attributes that CES uses, but I > > do see --user-id-attrib in the mmuserauth documentation. > > Unfortunately, I don't see an equivalent one for gidNumber. > > > > Is it not doing the "Samba thing" where your GID is the GID of your > primary Active Directory group? This is usually "Domain Users" but not > always. > > Basically Samba ignores the separate GID field in RFC2307bis, so one > imagines the options for changing the LDAP attributes are none > existent. > > I know back in the day this had me stumped for a while because unless > you assign a GID number to the users primary group then Winbind does > not return anything, aka a "getent passwd" on the user fails. At least for us, it seems to be using the gidNumber attribute of our users. On the back-end, of course, it is Samba, but I don't know that there are mm* commands available for all of the tunables one can set in smb.conf. -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -- University of Washington School of Medicine ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] IBM Spectrum Scale Support Webinar - Spectrum Scale Disk Lease, Expel & Recovery
er | Request for > Enhancement | Product SMC IBM > | dWA > > We meet our service commitment only when you are very satisfied > and EXTREMELY LIKELY to > recommend IBM. > > > > > > > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -- University of Washington School of Medicine ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] mmchconfig subnets
My understanding is that GPFS uses the network configuration on each node to determine netmask. The subnets option can be applied to specific nodes or groups of nodes with "mmchconfig subnets=... -N ", so what you're doing is specificy the preferred subnets for GPFS node communication, just for that list of nodes. For instance, we have four GPFS clusters, with three subnets: * eichler-cluster, eichler-cluster2 (10.130.0.0/16) * grc-cluster (10.200.0.0/16) * gs-cluster (10.110.0.0/16) And one data transfer system weasel that is a member of gs-cluster, but provides transfer services to all the clusters, and has an IP address on each subnet to avoid a bunch of network cross-talk. Its subnets setting looks like this: [weasel] subnets 10.130.0.0/eichler-cluster*.grid.gs.washington.edu 10.200.0.0/grc-cluster.grid.gs.washington.edu 10.110.0.0/gs-cluster.grid.gs.washington.edu Of course, there's some policy routing too to keep replies on the right interface as well, but that's the extent of the GPFS configuration. On Mon, Jun 25, 2018 at 04:20:35PM -0700, Eric Horst wrote: > Hi, I'm hoping somebody has insights into how the subnets option actually > works. I've read the docs a dozen times and I want to make sure I > understand before I take my production cluster down to make the changes. > > On the current cluster the daemon addresses are on a gpfs private network > and the admin addresses are on a public network. I'm changing so both > daemon and admin are public and the subnets option is used to utilize the > private network. This is to facilitate remote mounts to an independent > cluster. > > The confusing factor in my case, not covered in the docs, is that the gpfs > private network is subnetted and static routes are used to reach them. That > is, there are three private networks, one for each datacenter and the > cluster nodes daemon interfaces are spread between the three. > > 172.16.141.32/27 > 172.16.141.24/29 > 172.16.141.128/27 > > A router connects these three networks but are otherwise 100% private. > > For my mmchconfig subnets command should I use this? > > mmchconfig subnets="172.16.141.24 172.16.141.32 172.16.141.128" > > Where I get confused is that I'm trying to reason through how Spectrum > Scale is utilizing the subnets setting to decide if this will have the > desired result on my cluster. If I change the node addresses to their > public addresses, ie the private addresses are not explicitly configured in > Scale, then how are the private addresses discovered? Does each node use > the subnets option to identify that it has a private address and then > dynamically shares that with the cluster? > > Thanks in advance for your clarifying comments. > > -Eric > > -- > > Eric Horst > University of Washington > _______ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -- University of Washington School of Medicine ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] Meltdown, Spectre, and impacts on GPFS
While I'm not fully versed on the vulnerability or the proposed fixes, my understanding is that most of the performance impact from the fix is around having kernel memory completely separately from a process's user-space, which means every system call will have cache/TLB misses. This might mean clusters which are using RDMA won't have as big of a performance hit versus ones that aren't able to use RDMA, since they can do a lot more in user-space without involving the kernel. On 01/04/2018 09:57 AM, Buterbaugh, Kevin L wrote: > Happy New Year everyone, > > I’m sure that everyone is aware of Meltdown and Spectre by now … we, > like many other institutions, will be patching for it at the earliest > possible opportunity. > > Our understanding is that the most serious of the negative performance > impacts of these patches will be for things like I/O (disk / network) > … given that, we are curious if IBM has any plans for a GPFS update > that could help mitigate those impacts? Or is there simply nothing > that can be done? > > If there is a GPFS update planned for this we’d be interested in > knowing so that we could coordinate the kernel and GPFS upgrades on > our cluster. > > Thanks… > > Kevin > > P.S. The “Happy New Year” wasn’t intended as sarcasm … I hope it is a > good year for everyone despite how it’s starting out. :-O > > — > Kevin Buterbaugh - Senior System Administrator > Vanderbilt University - Advanced Computing Center for Research and > Education > kevin.buterba...@vanderbilt.edu > <mailto:kevin.buterba...@vanderbilt.edu> - (615)875-9633 > > > > > > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] More Drives For DDN 12KX
I would ask DDN this, but my guess is that even if the drives work, you would run into support headaches proving that whatever problem you're running into isn't the result of 3rd-party drives. Even with supported drives, we've run into drive firmware issues with almost all of our storage systems (not just DDN, but Isilon, Hitachi, EMC, etc.); for supported drives, it's a hassle to prove and then get updated, but it would be even worse without support on your side. On Tue, Aug 28, 2018 at 10:51:08PM -0400, Douglas Duckworth wrote: > Hi > > We have a 12KX which will be under support until 2020. Users are currently > happy with throughput but we need greater capacity as approaching 80%. > The enclosures are only half full. > > Does DDN require adding disks through them or can we get more 6TB SAS > through someone else? We would want support contract for the new disks. > If possible I think this would be a good stopgap solution until 2020 when > we can buy a new faster cluster. > > Thank you for your feedback. > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -- University of Washington School of Medicine ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] Preferred NSD
I agree. We have a small Gluster filesystem we use to perform failover of our job scheduler, but it predates our use of GPFS. We've run into a number of strange failures and "soft failures" (i.e. filesystem admin tools don't work but the filesystem is available), and the logging is much more cryptic and jumbled than mmfs.log. We'll soon be retiring it in favor of GPFS. On Wed, Mar 14, 2018 at 11:28:53AM -0400, Aaron Knister wrote: > I don't want to start a religious filesystem war, but I'd give pause to > GlusterFS based on a number of operational issues I've personally > experienced and seen others experience with it. > > I'm curious how glusterfs would resolve the issue here of multiple clients > failing simultaneously (unless you're talking about using disperse volumes)? > That does, actually, bring up an interesting question to IBM which is -- > when will mestor see the light of day? This is admittedly something other > filesystems can do that GPFS cannot. > > -Aaron > > On 3/14/18 6:57 AM, Michal Zacek wrote: > > Hi, > > > > I don't think the GPFS is good choice for your setup. Did you consider > > GlusterFS? It's used at Max Planck Institute at Dresden for HPC > > computing of Molecular Biology data. They have similar setup, tens > > (hundreds) of computers with shared local storage in glusterfs. But you > > will need 10Gb network. > > > > Michal > > > > > > Dne 12.3.2018 v 16:23 Lukas Hejtmanek napsal(a): > > > On Mon, Mar 12, 2018 at 11:18:40AM -0400, valdis.kletni...@vt.edu wrote: > > > > On Mon, 12 Mar 2018 15:51:05 +0100, Lukas Hejtmanek said: > > > > > I don't think like 5 or more data/metadata replicas are practical > > > > > here. On the > > > > > other hand, multiple node failures is something really expected. > > > > Umm.. do I want to ask *why*, out of only 60 nodes, multiple node > > > > failures are an expected event - to the point that you're thinking > > > > about needing 5 replicas to keep things running? > > > as of my experience with cluster management, we have multiple nodes down > > > on > > > regular basis. (HW failure, SW maintenance and so on.) > > > > > > I'm basically thinking that 2-3 replicas might not be enough while 5 or > > > more > > > are becoming too expensive (both disk space and required bandwidth being > > > scratch space - high i/o load expected). > > > > > > > > > > > > > ___ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > -- > Aaron Knister > NASA Center for Climate Simulation (Code 606.2) > Goddard Space Flight Center > (301) 286-2776 > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -- University of Washington School of Medicine ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] CES - suspend a node and don't start smb/nfs at mmstartup/boot
Hi Heiner, Try doing "mmces service stop -N " and/or "mmces service disable -N ". You'll definitely want the node suspended first, since I don't think the service commands do an address migration first. On Wed, Nov 14, 2018 at 04:20:12PM +, Billich Heinrich Rainer (PSI) wrote: > Hello, > > how can I prevent smb, ctdb, nfs (and object) to start when I reboot the node > or restart gpfs on a suspended ces node? Being able to do this would make > updates much easier > > With > > # mmces node suspend ???stop > > I can move all IPs to other CES nodes and stop all CES services, what also > releases the ces-shared-root-directory and allows to unmount the underlying > filesystem. > But after a reboot/restart only the IPs stay on the on the other nodes, the > CES services start up. Hm, sometimes I would very much prefer the services to > stay down as long as the nodes is suspended and to keep the node out of the > CES cluster as much as possible. > > I did not try rough things like just renaming smbd, this seems likely to > create unwanted issues. > > Thank you, > > Cheers, > > Heiner Billich > -- > Paul Scherrer Institut > Heiner Billich > System Engineer Scientific Computing > Science IT / High Performance Computing > WHGA/106 > Forschungsstrasse 111 > 5232 Villigen PSI > Switzerland > > Phone +41 56 310 36 02 > heiner.bill...@psi.ch<mailto:heiner.bill...@psi.ch> > https://www.psi.ch > > > > From: on behalf of Madhu Konidena > > Reply-To: gpfsug main discussion list > Date: Sunday 11 November 2018 at 22:06 > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] If you're attending KubeCon'18 > > I will be there at both. Please stop by our booth at SC18 for a quick chat. > > Madhu Konidena > [cid:ii_d4d3894a4c2f4773] > ma...@corehive.com > > > > On Nov 10, 2018, at 3:37 PM, Jon Bernard > mailto:jonbern...@gmail.com>> wrote: > Hi Vasily, > I will be at Kubecon with colleagues from Tower Research Capital (and at SC). > We have a few hundred nodes across several Kubernetes clusters, most of them > mounting Scale from the host. > Jon > On Fri, Oct 26, 2018, 5:58 PM Vasily Tarasov > mailto:vtara...@us.ibm.com> wrote: > Folks, Please let me know if anyone is attending KubeCon'18 in Seattle this > December (via private e-mail). We will be there and would like to meet in > person with people that already use or consider using Kubernetes/Swarm/Mesos > with Scale. The goal is to share experiences, problems, visions. P.S. If > you are not attending KubeCon, but are interested in the topic, shoot me an > e-mail anyway. Best, -- Vasily Tarasov, Research Staff Member, Storage > Systems Research, IBM Research - Almaden > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org<http://spectrumscale.org> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org<http://spectrumscale.org> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -- University of Washington School of Medicine ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] Hanging file-systems
Despite its name, kswapd isn't directly involved in paging to disk; it's the kernel process that's involved in finding committed memory that can be reclaimed for use (either immediately, or possibly by flushing dirty pages to disk). If kswapd is using a lot of CPU, it's a sign that the kernel is spending a lot of time to find free pages to allocate to processes. On Tue, Nov 27, 2018 at 05:53:58PM +, Simon Thompson wrote: > Thanks Sven ??? > > We found a node with kswapd running 100% (and swap was off)??? > > Killing that node made access to the FS spring into life. > > Simon > > From: on behalf of > "oeh...@gmail.com" > Reply-To: "gpfsug-discuss@spectrumscale.org" > > Date: Tuesday, 27 November 2018 at 16:14 > To: "gpfsug-discuss@spectrumscale.org" > Subject: Re: [gpfsug-discuss] Hanging file-systems > > 1. are you under memory pressure or even worse started swapping . > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -- University of Washington School of Medicine ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] Get list of filesets _without_ running mmlsfileset?
I suppose you could run the underlying tslsfileset, though that's probably not the answer you're looking for. Out of curiousity, what are you hoping to gain by not running mmlsfileset? Is the problem scaling due to the number of filesets that you have defined? On Tue, Jan 08, 2019 at 10:12:22PM +, Buterbaugh, Kevin L wrote: > Hi All, > > Happy New Year to all! Personally, I???ll gladly and gratefully settle for > 2019 not being a dumpster fire like 2018 was (those who attended my talk at > the user group meeting at SC18 know what I???m referring to), but I certainly > wish all of you the best! > > Is there a way to get a list of the filesets in a filesystem without running > mmlsfileset? I was kind of expecting to find them in one of the config files > somewhere under /var/mmfs but haven???t found them yet in the searching > I???ve done. > > The reason I???m asking is that we have a Python script that users can run > that needs to get a list of all the filesets in a filesystem. There are > obviously multiple issues with that, so the workaround we???re using for now > is to have a cron job which runs mmlsfileset once a day and dumps it out to a > text file, which the script then reads. That???s sub-optimal for any day on > which a fileset gets created or deleted, so I???m looking for a better way > ??? one which doesn???t require root privileges and preferably doesn???t > involve running a GPFS command at all. > > Thanks in advance. > > Kevin > > P.S. I am still working on metadata and iSCSI testing and will report back > on that when complete. > P.P.S. We ended up adding our new NSDs comprised of (not really) 12 TB disks > to the capacity pool and things are working fine. > > ??? > Kevin Buterbaugh - Senior System Administrator > Vanderbilt University - Advanced Computing Center for Research and Education > kevin.buterba...@vanderbilt.edu<mailto:kevin.buterba...@vanderbilt.edu> - > (615)875-9633 > > > > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -- University of Washington School of Medicine ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] Steps for gracefully handling bandwidth reduction during network maintenance
IIRC, maxMBpS isn't really a limit, but more of a hint for how GPFS should use its in-memory buffers for read prefetches and dirty writes. On Mon, Jun 17, 2019 at 09:31:38AM -0700, Alex Chekholko wrote: > Hi Chris, > > I think the next thing to double-check is when the maxMBpS change takes > effect. You may need to restart the nsds. Otherwise I think your plan is > sound. > > Regards, > Alex > > > On Mon, Jun 17, 2019 at 9:24 AM Christopher Black > wrote: > > > Our network team sometimes needs to take down sections of our network for > > maintenance. Our systems have dual paths thru pairs of switches, but often > > the maintenance will take down one of the two paths leaving all our nsd > > servers with half bandwidth. > > > > Some of our systems are transmitting at a higher rate than can be handled > > by half network (2x40Gb hosts with tx of 50Gb+). > > > > What can we do to gracefully handle network maintenance reducing bandwidth > > in half? > > > > Should we set maxMBpS for affected nodes to a lower value? (default on our > > ess appears to be maxMBpS = 3, would I reduce this to ~4000 for 32Gbps?) > > > > Any other ideas or comments? > > > > Our hope is that metadata operations are not affected much and users just > > see jobs and processes read or write at a slower rate. > > > > > > > > Best, > > > > Chris > > -- > > This message is for the recipient???s use only, and may contain > > confidential, privileged or protected information. Any unauthorized use or > > dissemination of this communication is prohibited. If you received this > > message in error, please immediately notify the sender and destroy all > > copies of this message. The recipient should check this email and any > > attachments for the presence of viruses, as we accept no liability for any > > damage caused by any virus transmitted by this email. > > ___ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -- University of Washington School of Medicine ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] gpfs client and turning off swap
We don't run diskless systems, but use cgroups where possible to limit the damage users can do to a node. We derate the node's total usable memory by 2GB (OS) + GPFS pagepool to avoid paging whenever possible. On Fri, Nov 08, 2019 at 11:31:20AM -0500, david_john...@brown.edu wrote: > We have most of our clients network booted and diskless ??? no swap possible. > Gpfs still works until someone runs the node out of memory > > -- ddj > Dave Johnson > > > On Nov 8, 2019, at 11:25 AM, Damir Krstic wrote: > > > > ??? > > I was wondering if it's safe to turn off swap on gpfs client machines? we > > have a case where checkpointing is swapping and we would like to prevent it > > from doing so by disabling swap. However, the gpfs manual admin. manual > > states to have swap enabled and sufficiently large, but it does not > > distinguish between clients and IO servers. > > > > Let me know when you get a chance. -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -- University of Washington School of Medicine ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] default owner and group for POSIX ACLs
On Wed, Oct 16, 2019 at 09:32:50PM +, Jonathan Buzzard wrote: > On 15/10/2019 16:41, Simon Thompson wrote: > > I thought Spectrum Protect didn't actually backup again on a file > > owner change. Sure mmbackup considers it, but I think Protect just > > updates the metadata. There are also some other options for dsmc that > > can stop other similar issues if you change ctime maybe. > > > > (Other backup tools are available) > > > > It certainly used too. I spent six months carefully chown'ing files one > user at a time so as not to overwhelm the backup, because the first > group I did meant no backup for about a week... > > I have not kept a close eye on it and have just worked on the assumption > for the last decade of "don't do that". If it is no longer the case I > apologize for spreading incorrect information. TSM can store some amount of metadata in its database without spilling over to a storage pool, so whether a metadata update is cheap or expensive depends not just on ACLs/extended attributes but also the directory entry name length. It can definitely make for some seemingly non-deterministic backup behavior. -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -- University of Washington School of Medicine ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] mmbackup questions
On Thu, Oct 17, 2019 at 10:26:45AM +, Jonathan Buzzard wrote: > I have been looking to give mmbackup another go (a very long history > with it being a pile of steaming dinosaur droppings last time I tried, > but that was seven years ago). > > Anyway having done a backup last night I am curious about something > that does not appear to be explained in the documentation. > > Basically the output has a line like the following > > Total number of objects inspected: 474630 > > What is this number? Is it the number of files that have changed since > the last backup or something else as it is not the number of files on > the file system by any stretch of the imagination. One would hope that > it inspected everything on the file system... I believe this is the number of paths that matched some include rule (or didn't match some exclude rule) for mmbackup. I would assume it would differ from the "total number of objects backed up" line if there were include/exclude rules that mmbackup couldn't process, leaving it to dsmc to decide whether to process. > Also it appears that the shadow database is held on the GPFS file system > that is being backed up. Is there any way to change the location of that? > I am only using one node for backup (because I am cheap and don't like > paying for more PVU's than I need to) and would like to hold it on the > node doing the backup where I can put it on SSD. Which does to things > firstly hopefully goes a lot faster, and secondly reduces the impact on > the file system of the backup. I haven't tried it, but there is a MMBACKUP_RECORD_ROOT environment variable noted in the mmbackup man path: Specifies an alternative directory name for storing all temporary and permanent records for the backup. The directory name specified must be an existing directory and it cannot contain special characters (for example, a colon, semicolon, blank, tab, or comma). Which seems like it might provide a mechanism to store the shadow database elsewhere. For us, though, we provide storage via a cost center, so we would want our customers to eat the full cost of their excessive file counts. > Anyway a significant speed up (assuming it worked) was achieved but I > note even the ancient Xeon E3113 (dual core 3GHz) was never taxed (load > average never went above one) and we didn't touch the swap despite only > have 24GB of RAM. Though the 10GbE networking did get busy during the > transfer of data to the TSM server bit of the backup but during the > "assembly stage" it was all a bit quiet, and the DSS-G server nodes where > not busy either. What options are there for tuning things because I feel > it should be able to go a lot faster. We have some TSM nodes (corresponding to GPFS filesets) that stress out our mmbackup cluster at the sort step of mmbackup. UNIX sort is not RAM-friendly, as it happens. -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -- University of Washington School of Medicine ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] GPFS 5 and supported rhel OS
On Sun, Feb 23, 2020 at 04:58:03PM -0500, Valdis Klētnieks wrote: > On Sun, 23 Feb 2020 12:20:48 +, Jonathan Buzzard said: > > > > That's not *quite* so bad. As long as you trust *all* your vendors to > > > notify > > > you when they release a patch for an issue you hadn't heard about. > > > Er, what do you think I am paid for? Specifically it is IMHO the job of > > any systems administrator to know when any critical patch becomes > > available for any software/hardware that they are using. > > You missed the point. > > Unless you spend your time constantly e-mailing *all* of your vendors > "Are there new patches I don't know about?", you're relying on them to > notify you when there's a known issue, and when a patch comes out. > > Redhat is good about notification. IBM is. > > But how about things like your Infiniband stack? OFED? The firmware in all > your devices? The BIOS/UEFI on the servers? If you're an Intel shop, how do > you > get notified about security issues in the Management Engine stuff (and there's > been plenty of them). Do *all* of those vendors have security lists? Are you > subscribed to *all* of them? Do *all* of them actually post to those lists? We put our notification sources (Nessus, US-CERT, etc.) into our response plan. Of course it's still a problem if we don't get notified, but part of the plan is to make it clear where we're willing to accept risk, and to limit our own liability. No process is going to be perfect, but we at least know and accept where those imperfections are. -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -- University of Washington School of Medicine ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] GPFS 5 and supported rhel OS
On Thu, Feb 20, 2020 at 11:23:52AM +, Jonathan Buzzard wrote: > On 20/02/2020 10:41, Simon Thompson wrote: > > Well, if you were buying some form of extended Life Support for > > Scale, then you might also be expecting to buy extended life for > > RedHat. RHEL6 has extended life support until June 2024. Sure its an > > add on subscription cost, but some people might be prepared to do > > that over OS upgrades. > > I would recommend anyone going down that to route to take a *very* close > look at what you get for the extended support. Not all of the OS is > supported, with large chunks being moved to unsupported even if you pay > for the extended support. > > Consequently extended support is not suitable for HPC usage in my view, > so start planning the upgrade now. It's not like you haven't had 10 > years notice. > > If your GPFS is just a storage thing serving out on protocol nodes, > upgrade one node at a time to RHEL7 and then repeat upgrading to GPFS 5. > It's a relatively easy invisible to the users upgrade. I agree, we're having increasing difficulty running CentOS 6, not because of the lack of support from IBM/RedHat, but because the software our customers want to run has started depending on OS features that simply don't exist in CentOS 6. In particular, modern gcc and glibc, and containers are all features that many of our customers are expecting that we provide. The newer kernel available in CentOS 7 (and now 8) supports large numbers of CPUs and large amounts of memory far better than the ancient CentOS 6 kernel as well. -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -- University of Washington School of Medicine ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] GPFS 5 and supported rhel OS
On Thu, Feb 20, 2020 at 12:14:40PM -0500, David Johnson wrote: > Instead of keeping whole legacy systems around, could they achieve the same > with a container built from the legacy software? That is our hope, at least once we can get off CentOS 6 and run containers. :) Though containers aren't quite a panacea; there's still the issue of insecure software being baked into the container, but at least we can limit what the container can access more easily than running outside a container. > > On Feb 20, 2020, at 11:59 AM, Skylar Thompson wrote: > > > > On Thu, Feb 20, 2020 at 04:29:40PM +, Ken Atkinson wrote: > >> Fred, > >> It may be that some HPC users "have to" > >> reverify the results of their computations as being exactly the same as a > >> previous software stack and that is not a minor task. Any change may > >> require this verification process. > >> Ken Atkjnson > > > > We have this problem too, but at the same time the same people require us > > to run supported software and remove software versions with known > > vulnerabilities. The compromise we've worked out for the researchers is to > > have them track which software versions they used for a particular run/data > > release. The researchers who care more will have a validation suite that > > will (hopefully) call out problems as we do required upgrades. > > > > At some point, it's simply unrealistic to keep legacy systems around, > > though we do have a lab that needs a Solaris/SPARC system just to run a > > 15-year-old component of a pipeline for which they don't have source code... > > > > -- > > -- Skylar Thompson (skyl...@u.washington.edu) > > -- Genome Sciences Department, System Administrator > > -- Foege Building S046, (206)-685-7354 > > -- University of Washington School of Medicine > > ___ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -- University of Washington School of Medicine ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] GPFS 5 and supported rhel OS
On Thu, Feb 20, 2020 at 04:29:40PM +, Ken Atkinson wrote: > Fred, > It may be that some HPC users "have to" > reverify the results of their computations as being exactly the same as a > previous software stack and that is not a minor task. Any change may > require this verification process. > Ken Atkjnson We have this problem too, but at the same time the same people require us to run supported software and remove software versions with known vulnerabilities. The compromise we've worked out for the researchers is to have them track which software versions they used for a particular run/data release. The researchers who care more will have a validation suite that will (hopefully) call out problems as we do required upgrades. At some point, it's simply unrealistic to keep legacy systems around, though we do have a lab that needs a Solaris/SPARC system just to run a 15-year-old component of a pipeline for which they don't have source code... -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -- University of Washington School of Medicine ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] How to install efix with yum ?
> start bitching about the RPM database being modified outside of > themselves and all sorts of useful information gets lost when you purge > the package installation history to make the error go away. > > > We work around this by repackaging the three affected RPMS to include > > the orphaned files from the original RPMs (and eliminating the related > > but problematic checks from the RPMs??? scripts) so that our efix RPMs > > have been ???un-efix-ified??? and will install as expected when using ???yum > > upgrade???. To my knowledge no one???s published a way to do this, so we > > all just have to figure this out and run rpmrebuild for ourselves. > > > > IBM should be hanging their heads in shame if the replacement RPM is > missing files. > > JAB. > > -- > Jonathan A. Buzzard Tel: +44141-5483420 > HPC System Administrator, ARCHIE-WeSt. > University of Strathclyde, John Anderson Building, Glasgow. G4 0NG > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -- University of Washington School of Medicine ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] How to install efix with yum ?
Thanks for the pointer! We're in the process of upgrading from 4.2.3-6 to 4.2.3-19 so I'll make a note that we should start setting that environment variable when we build gplbin. On Thu, Jan 16, 2020 at 05:59:14PM -0500, IBM Spectrum Scale wrote: > On Spectrum Scale 4.2.3.15 or later and 5.0.2.2 or later, you can install > gplbin without stopping GPFS by using the following step: > > Build gpfs.gplbin using mmbuildgpl --build-packge > Set environment variable MM_INSTALL_ONLY to 1 before install gpfs.gplbin > package with rpm -i gpfs.gplbin*.rpm > > Regards, The Spectrum Scale (GPFS) team > > -- > If you feel that your question can benefit other users of Spectrum Scale > (GPFS), then please post it to the public IBM developerWroks Forum at > https://www.ibm.com/developerworks/community/forums/html/forum?id=----0479 > . > > If your query concerns a potential software error in Spectrum Scale (GPFS) > and you have an IBM software maintenance contract please contact > 1-800-237-5511 in the United States or your local IBM Service Center in > other countries. > > The forum is informally monitored as time permits and should not be used > for priority messages to the Spectrum Scale (GPFS) team. > > gpfsug-discuss-boun...@spectrumscale.org wrote on 01/16/2020 10:32:27 AM: > > > From: Skylar Thompson > > To: gpfsug-discuss@spectrumscale.org > > Date: 01/16/2020 10:35 AM > > Subject: [EXTERNAL] Re: [gpfsug-discuss] How to install efix with yum ? > > Sent by: gpfsug-discuss-boun...@spectrumscale.org > > > > Another problem we've run into with automating GPFS installs/upgrades is > > that the gplbin (kernel module) packages have a post-install script that > > will unmount the filesystem *even if the package isn't for the running > > kernel*. We needed to write some custom reporting in our configuration > > management system to only install gplbin if GPFS was already stopped on > the > > node. > > > > On Wed, Jan 15, 2020 at 10:35:23PM +, Sanchez, Paul wrote: > > > This reminds me that there is one more thing which drives the > > convoluted process I described earlier??? > > > > > > Automation. Deployment solutions which use yum to build new hosts > > are often the place where one notices the problem. They would need > > to determine that they should install both the base-version and efix > > RPMS and in that order. IIRC, there were no RPM dependencies > > connecting the efix RPMs to their base-version equivalents, so > > there was nothing to signal YUM that installing the efix requires > > that the base-version be installed first. > > > > > > (Our particular case is worse than just this though, since we > > prohibit installing two versions/releases for the same (non-kernel) > > package name. But that???s not the case for everyone.) > > > > > > -Paul > > > > > > From: gpfsug-discuss-boun...@spectrumscale.org > boun...@spectrumscale.org> On Behalf Of IBM Spectrum Scale > > > Sent: Wednesday, January 15, 2020 16:00 > > > To: gpfsug main discussion list > > > Cc: gpfsug-discuss-boun...@spectrumscale.org > > > Subject: Re: [gpfsug-discuss] How to install efix with yum ? > > > > > > > > > This message was sent by an external party. > > > > > > > > > >> I don't see any yum options which match rpm's '--force' option. > > > Actually, you do not need to use --force option since efix RPMs > > have incremental efix number in rpm name. > > > > > > Efix package provides update RPMs to be installed on top of > > corresponding PTF GA version. When you install 5.0.4.1 efix9, if 5. > > 0.4.1 is already installed on your system, "yum update" should work. > > > > > > Regards, The Spectrum Scale (GPFS) team > > > > > > > > > -- > > > If you feel that your question can benefit other users of Spectrum > > Scale (GPFS), then please post it to the public IBM developerWroks Forum > at > > https://www.ibm.com/developerworks/community/forums/html/forum? > > id=----0479. > > > > > > If your query concerns a potential software error in Spectrum > > Scale (GPFS) and you have an IBM software maintenance contract > > please contact 1-800-237-5511 in the Un
Re: [gpfsug-discuss] mmbackup monitoring
We execute mmbackup via a regular TSM client schedule with an incremental action, with a virtualmountpoint set to an empty, local "canary" directory. mmbackup runs as a preschedule command, and the client -domain parameter is set only to backup the canary directory. dsmc will backup the canary directory as a filespace only if mmbackup succeeds (exits with 0). We can then monitor the canary and infer the status of the associated GPFS filespace or fileset. On Wed, Mar 25, 2020 at 10:01:04AM +, Jonathan Buzzard wrote: > > What is the best way of monitoring whether or not mmbackup has managed to > complete a backup successfully? > > Traditionally one use a TSM monitoring solution of your choice to make sure > nodes where backing up (I am assuming mmbackup is being used in conjunction > with TSM here). > > However mmbackup does not update the backup_end column in the filespaceview > table (at least in 4.2) which makes things rather more complicated. > > The best I can come up with is querying the events table to see if the > client schedule completed, but that gives a false sense of security as the > schedule completing does not mean the backup completed as far as I know. > > What solutions are you all using, or does mmbackup in 5.x update the > filespaceview table? -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -- University of Washington School of Medicine ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] mmbackup monitoring
IIRC, I think you need to set 2 in the bit field of the DEBUGmmbackup environment variable. I had a long-term task to see what I could get out of that, but this just reminded me of it and current events might actually let me have time to look into it now... On Wed, Mar 25, 2020 at 10:38:55AM -0400, Jaime Pinto wrote: > Additionally, mmbackup creates by default a .mmbackupCfg directory on the > root of the fileset where it dumps several files and directories with the > progress of the backup. For instance: expiredFiles/, prepFiles/, > updatedFiles/, dsminstr.log, ... > > You may then create a script to search these directories for logs/lists of > what has happened, and generate a more detailed report of what happened > during the backup. In our case I generate a daily report of how many files > and how much data have been sent to the TSM server and deleted for each user, > including their paths. You can do more tricks if you want. > > Jaime > > > On 3/25/2020 10:15:59, Skylar Thompson wrote: > > We execute mmbackup via a regular TSM client schedule with an incremental > > action, with a virtualmountpoint set to an empty, local "canary" directory. > > mmbackup runs as a preschedule command, and the client -domain parameter is > > set only to backup the canary directory. dsmc will backup the canary > > directory as a filespace only if mmbackup succeeds (exits with 0). We can > > then monitor the canary and infer the status of the associated GPFS > > filespace or fileset. > > > > On Wed, Mar 25, 2020 at 10:01:04AM +, Jonathan Buzzard wrote: > > > > > > What is the best way of monitoring whether or not mmbackup has managed to > > > complete a backup successfully? > > > > > > Traditionally one use a TSM monitoring solution of your choice to make > > > sure > > > nodes where backing up (I am assuming mmbackup is being used in > > > conjunction > > > with TSM here). > > > > > > However mmbackup does not update the backup_end column in the > > > filespaceview > > > table (at least in 4.2) which makes things rather more complicated. > > > > > > The best I can come up with is querying the events table to see if the > > > client schedule completed, but that gives a false sense of security as the > > > schedule completing does not mean the backup completed as far as I know. > > > > > > What solutions are you all using, or does mmbackup in 5.x update the > > > filespaceview table? > > > > . > . > . > TELL US ABOUT YOUR SUCCESS STORIES > http://www.scinethpc.ca/testimonials > > --- > Jaime Pinto - Storage Analyst > SciNet HPC Consortium - Compute/Calcul Canada > www.scinet.utoronto.ca - www.computecanada.ca > University of Toronto > 661 University Ave. (MaRS), Suite 1140 > Toronto, ON, M5G1M1 > P: 416-978-2755 > C: 416-505-1477 > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -- University of Washington School of Medicine ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] wait for mount during gpfs startup
We use callbacks successfully to ensure Linux auditd rules are only loaded after GPFS is mounted. It was easy to setup, and there's very fine-grained events that you can trigger on: https://www.ibm.com/support/knowledgecenter/STXKQY_5.0.4/com.ibm.spectrum.scale.v5r04.doc/bl1adm_mmaddcallback.htm On Tue, Apr 28, 2020 at 11:30:38AM +, Frederick Stock wrote: > Have you looked a the mmaddcallback command and specifically the file system > mount callbacks? -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -- University of Washington School of Medicine ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] wait for mount during gpfs startup
On Thu, Apr 30, 2020 at 12:50:27PM +0200, Ulrich Sibiller wrote: > Am 28.04.20 um 15:57 schrieb Skylar Thompson: > >> Have you looked a the mmaddcallback command and specifically the file > system mount callbacks? > > > We use callbacks successfully to ensure Linux auditd rules are only loaded > > after GPFS is mounted. It was easy to setup, and there's very fine-grained > > events that you can trigger on: > > Thanks. But how do set this up for a systemd service? Disable the dependent > service and start it > from the callback? Create some kind of state file in the callback and let the > dependent systemd > service check that flag file in a busy loop? Use inotify for the flag file? In the pre-systemd days, I would say just disable the service and let the callback handle it. I do see your point, though, that you lose the other systemd ordering benefits if you start the service from the callback. Assuming you're still able to start the service via systemctl, I would probably just leave it disabled and let the callback handle it. In the case of auditd rules, it's not actually a service (just a command that needs to be run) so we didn't run into this specific problem. -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -- University of Washington School of Medicine ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] Best of spectrum scale
On Wed, Sep 09, 2020 at 12:02:53PM +0100, Jonathan Buzzard wrote: > On 08/09/2020 18:37, IBM Spectrum Scale wrote: > > I think it is incorrect to assume that a command that continues > > after detecting the working directory has been removed is going to > > cause damage to the file system. > > No I am not assuming it will cause damage. I am making the fairly reasonable > assumption that any command which fails has an increased probability of > causing damage to the file system over one that completes successfully. I think there is another angle here, which is that this command's output has the possibility of triggering an "oh " (fill in your preferred colorful metaphor here) moment, followed up by a panicked Ctrl-C. That reaction has the possibility of causing its own problems (i.e. not sure if mmafmctl touches CCR, but aborting it midway could leave CCR inconsistent). I'm with Jonathan here: the command should fail with an informative message, and the admin can correct the problem (just cd somewhere else). -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department (UW Medicine), System Administrator -- Foege Building S046, (206)-685-7354 -- Pronouns: He/Him/His ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] Portability interface
We've used the same built RPMs (generally built on Intel) on Intel and AMD x86-64 CPUs, and definitely have a mix of ISAs from both vendors, and haven't run into any problems. On Tue, Sep 22, 2020 at 10:18:05AM +0100, Jonathan Buzzard wrote: > > I have a question about using RPM's for the portability interface on > different CPU's. > > According to /usr/lpp/mmfs/src/README > >The generated RPM can ONLY be deployed to the machine with >identical architecture, distribution level, Linux kernel version >and GPFS version. > > So does this mean that if I have a heterogeneous cluster with some machines > on Skylake and some on Sandy Bridge but all running on say RHEL 7.8 and all > using GPFS 5.0.5 I have to have different RPM's for the two CPU's? > > Or when it says "identical architecture" does it mean x86-64, ppc etc. and > not variations with the x86-64, ppc class? Assuming some minimum level is > met. > > Obviously the actual Linux kernel being stock RedHat would be the same on > every machine regardless of whether it's Skylake or Sandy Bridge, or even > for that matter an AMD processor. > > Consequently it seems strange that I would need different portability > interfaces. Would it help to generate the portability layer RPM's on a Sandy > Bridge machine and work no the presumption anything that runs on Sandy > Bridge will run on Skylake? > > > JAB. > > -- > Jonathan A. Buzzard Tel: +44141-5483420 > HPC System Administrator, ARCHIE-WeSt. > University of Strathclyde, John Anderson Building, Glasgow. G4 0NG > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department (UW Medicine), System Administrator -- Foege Building S046, (206)-685-7354 -- Pronouns: He/Him/His ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] dependent versus independent filesets
We wanted to be able to snapshot and backup filesets separately with mmbackup, so went with independent filesets. On Tue, Jul 07, 2020 at 08:37:46AM -0500, Damir Krstic wrote: > We are deploying our new ESS and are considering moving to independent > filesets. The snapshot per fileset feature appeals to us. > > Has anyone considered independent vs. dependent filesets and what was your > reasoning to go with one as opposed to the other? Or perhaps you opted to > have both on your filesystem, and if, what was the reasoning for it? > > Thank you. > Damir > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department (UW Medicine), System Administrator -- Foege Building S046, (206)-685-7354 -- Pronouns: He/Him/His ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] dependent versus independent filesets
Ah, yes, I forgot about the quota rationale; we use independent filesets for that as well. We have run into confusion with inodes as one has to be careful to allocate inodes /and/ adjust a quota to expand a fileset. IIRC GPFS generates ENOSPC if it actually runs out of inodes, and EDQUOT if it hits a quota. We've also run into the quiescing issue but have been able to workaround it for now by increasing the splay between the different schedules. On Tue, Jul 07, 2020 at 02:44:16PM +, Wahl, Edward wrote: > We also went with independent filesets for both backup (and quota) reasons > for several years now, and have stuck with this across to 5.x. However we > still maintain a minor number of dependent filesets for administrative use. > Being able to mmbackup on many filesets at once can increase your > parallelization _quite_ nicely! We create and delete the individual snaps > before and after each backup, as you may expect. Just be aware that if you > do massive numbers of fast snapshot deletes and creates you WILL reach a > point where you will run into issues due to quiescing compute clients, and > that certain types of workloads have issues with snapshotting in general. > > You have to more closely watch what you pre-allocate, and what you have left > in the common metadata/inode pool. Once allocated, even if not being used, > you cannot reduce the inode allocation without removing the fileset and > re-creating. (say a fileset user had 5 million inodes and now only needs > 500,000) > > Growth can also be an issue if you do NOT fully pre-allocate each space. > This can be scary if you are not used to over-subscription in general. But I > imagine that most sites have some decent % of oversubscription if they use > filesets and quotas. > > Ed > OSC > > -Original Message----- > From: gpfsug-discuss-boun...@spectrumscale.org > On Behalf Of Skylar Thompson > Sent: Tuesday, July 7, 2020 10:00 AM > To: gpfsug-discuss@spectrumscale.org > Subject: Re: [gpfsug-discuss] dependent versus independent filesets > > We wanted to be able to snapshot and backup filesets separately with > mmbackup, so went with independent filesets. > > On Tue, Jul 07, 2020 at 08:37:46AM -0500, Damir Krstic wrote: > > We are deploying our new ESS and are considering moving to independent > > filesets. The snapshot per fileset feature appeals to us. > > > > Has anyone considered independent vs. dependent filesets and what was > > your reasoning to go with one as opposed to the other? Or perhaps you > > opted to have both on your filesystem, and if, what was the reasoning for > > it? > > > > Thank you. > > Damir > > > ___ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > https://urldefense.com/v3/__http://gpfsug.org/mailman/listinfo/gpfsug- > > discuss__;!!KGKeukY!j-c9kslUrEaNslhTbLLfaY8TES7Xf4eUCxysOaXwroHhTMwiVY > > vcGNh4M_no$ > > > -- > -- Skylar Thompson (skyl...@u.washington.edu) > -- Genome Sciences Department (UW Medicine), System Administrator > -- Foege Building S046, (206)-685-7354 > -- Pronouns: He/Him/His > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.com/v3/__http://gpfsug.org/mailman/listinfo/gpfsug-discuss__;!!KGKeukY!j-c9kslUrEaNslhTbLLfaY8TES7Xf4eUCxysOaXwroHhTMwiVYvcGNh4M_no$ > > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department (UW Medicine), System Administrator -- Foege Building S046, (206)-685-7354 -- Pronouns: He/Him/His ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] Migrate/syncronize data from Isilon to Scale over NFS?
When we did a similar (though larger, at ~2.5PB) migration, we used rsync as well, but ran one rsync process per Isilon node, and made sure the NFS clients were hitting separate Isilon nodes for their reads. We also didn't have more than one rsync process running per client, as the Linux NFS client (at least in CentOS 6) was terrible when it came to concurrent access. Whatever method you end up using, I can guarantee you will be much happier once you are on GPFS. :) On Mon, Nov 16, 2020 at 08:44:14PM +0100, Andi Christiansen wrote: > Hi all, > > i have got a case where a customer wants 700TB migrated from isilon to Scale > and the only way for him is exporting the same directory on NFS from two > different nodes... > > as of now we are using multiple rsync processes on different parts of folders > within the main directory. this is really slow and will take forever.. right > now 14 rsync processes spread across 3 nodes fetching from 2.. > > does anyone know of a way to speed it up? right now we see from 1Gbit to > 3Gbit if we are lucky(total bandwidth) and there is a total of 30Gbit from > scale nodes and 20Gbits from isilon so we should be able to reach just under > 20Gbit... > > > if anyone have any ideas they are welcome! > > > Thanks in advance > Andi Christiansen > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department (UW Medicine), System Administrator -- Foege Building S046, (206)-685-7354 -- Pronouns: He/Him/His ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] Mounting filesystem on top of an existing filesystem
Agreed, not sure how the GPFS tools would react. An alternative to symlinks would be bind mounts, if for some reason a tool doesn't behave properly with a symlink in the path. On Thu, Nov 19, 2020 at 06:34:05PM +0100, Jan-Frode Myklebust wrote: > I would not mount a GPFS filesystem within a GPFS filesystem. Technically > it should work, but I???d expect it to cause surprises if ever the lower > filesystem experienced problems. Alone, a filesystem might recover > automatically by remounting. But if there???s another filesystem mounted > within, I expect it will be a problem.. > > Much better to use symlinks. > > > > -jf > > tor. 19. nov. 2020 kl. 18:01 skrev Caubet Serrabou Marc (PSI) < > marc.cau...@psi.ch>: > > > Hi Simon, > > > > > > that's a very good point, thanks a lot :) I have it remotely mounted on a > > client cluster, so I will consider priorities when mounting the filesystems > > with remote cluster mount. That's very useful. > > > > Also, as far as I saw, same approach can be also applied to local mounts > > (via mmchfs) during daemon startup with the same option --mount-priority. > > > > > > Thanks a lot for the hints, these are very useful. I'll test that. > > > > > > Cheers, > > > > Marc > > _ > > Paul Scherrer Institut > > High Performance Computing & Emerging Technologies > > Marc Caubet Serrabou > > Building/Room: OHSA/014 > > Forschungsstrasse, 111 > > 5232 Villigen PSI > > Switzerland > > > > Telephone: +41 56 310 46 67 > > E-Mail: marc.cau...@psi.ch > > -- > > *From:* gpfsug-discuss-boun...@spectrumscale.org < > > gpfsug-discuss-boun...@spectrumscale.org> on behalf of Simon Thompson < > > s.j.thomp...@bham.ac.uk> > > *Sent:* Thursday, November 19, 2020 5:42:07 PM > > *To:* gpfsug main discussion list > > *Subject:* Re: [gpfsug-discuss] Mounting filesystem on top of an existing > > filesystem > > > > > > If it is a remote cluster mount from your clients (hopefully!), you might > > want to look at priority to order mounting of the file-systems. I don???t > > know what would happen if the overmounted file-system went away, you would > > likely want to test. > > > > > > > > Simon > > > > > > > > *From: * on behalf of " > > marc.cau...@psi.ch" > > *Reply to: *"gpfsug-discuss@spectrumscale.org" < > > gpfsug-discuss@spectrumscale.org> > > *Date: *Thursday, 19 November 2020 at 15:39 > > *To: *"gpfsug-discuss@spectrumscale.org" > > > > *Subject: *[gpfsug-discuss] Mounting filesystem on top of an existing > > filesystem > > > > > > > > Hi, > > > > > > > > I have a filesystem holding many projects (i.e., mounted under /projects), > > each project is managed with filesets. > > > > I have a new big project which should be placed on a separate filesystem > > (blocksize, replication policy, etc. will be different, and subprojects of > > it will be managed with filesets). Ideally, this filesystem should be > > mounted in /projects/newproject. > > > > > > > > Technically, mounting a filesystem on top of an existing filesystem should > > be possible, but, is this discouraged for any reason? How GPFS would behave > > with that and is there a technical reason for avoiding this setup? > > > > Another alternative would be independent mount point + symlink, but I > > really would prefer to avoid symlinks. > > > > > > > > Thanks a lot, > > > > Marc > > > > _ > > Paul Scherrer Institut > > High Performance Computing & Emerging Technologies > > Marc Caubet Serrabou > > Building/Room: OHSA/014 > > > > Forschungsstrasse, 111 > > > > 5232 Villigen PSI > > Switzerland > > > > Telephone: +41 56 310 46 67 > > E-Mail: marc.cau...@psi.ch > > ___ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department (UW Medicine), System Administrator -- Foege Building S046, (206)-685-7354 -- Pronouns: He/Him/His ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] Migrate/syncronize data from Isilon to Scale over NFS?
On Tue, Nov 17, 2020 at 01:53:43PM +, Jonathan Buzzard wrote: > On 17/11/2020 11:51, Andi Christiansen wrote: > > Hi all, > > > > thanks for all the information, there was some interesting things > > amount it.. > > > > I kept on going with rsync and ended up making a file with all top > > level user directories and splitting them into chunks of 347 per > > rsync session(total 42000 ish folders). yesterday we had only 14 > > sessions with 3000 folders in each and that was too much work for one > > rsync session.. > > Unless you use something similar to my DB suggestion it is almost inevitable > that some of those rsync sessions are going to have issues and you will have > no way to track it or even know it has happened unless you do a single final > giant catchup/check rsync. > > I should add that a copy of the sqlite DB is cover your backside protection > when a user pops up claiming that you failed to transfer one of their > vitally important files six months down the line and the old system is > turned off and scrapped. That's not a bad idea, and I like it more than the method I setup where we captured the output of find from both sides of the transfer and preserved it for posterity, but obviously did require a hard-stop date on the source. Fortunately, we seem committed to GPFS so it might be we never have to do another bulk transfer outside of the filesystem... -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department (UW Medicine), System Administrator -- Foege Building S046, (206)-685-7354 -- Pronouns: He/Him/His ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] SC20 Planning - What questions would you ask a panel?
Here's one: How is IBM working to improve the integration between TSM and GPFS? We're in the biomedical space and have some overlapping regulatory requirements around retention, which translate to complicated INCLUDE/EXCLUDE rules that mmbackup has always had trouble processing. In particular, we need to be able to INCLUDE particular paths to set a management class, but then EXCLUDE particular paths, which results in mmbackup generating file lists for dsmc including those excluded paths, which dsmc can exclude but it logs every single one every time it runs. On Thu, Oct 29, 2020 at 02:43:02PM -0700, Kristy Kallback-Rose wrote: > Really? There???s nothing you want to ask about GPFS/Spectrum Scale? There > will be access to developers and management alike, so I have to imagine you > have something to ask??? Don???t be shy. > > Please help make this a lively discussion by submitting a question, or two. > > Best, > Kristy > > > On Oct 13, 2020, at 2:34 PM, Kristy Kallback-Rose wrote: > > > > Hi all, > > > > By now you know SC will be digital this year. We are working towards > > some SC events for the Spectrum Scale User Group, and using our usual slot > > of Sunday did not seem like a great idea. So, we???re planning a couple > > 90-minute sessions and would like to do a panel during one of them. We???ll > > hope to do live Q, like an in-person Ask Me Anything session, but it???s > > probably a good idea to have a bank of questions ready as well, Also, that > > way your question may get asked, even if you can???t make the live session > > ???we???ll record these sessions for later viewing. > > > > So, please send your questions for the panel and we???ll get a list > > built up. Better yet, attend the sessions live! Details to come, but for > > now, hold these time slots: > > > > November 16th - 8:00 AM Pacific/3:00 PM UTC > > > > November 18th - 8:00 AM Pacific/3:00 PM UTC > > > > Best, > > Kristy > > > > Kristy Kallback-Rose > > Senior HPC Storage Systems Analyst > > National Energy Research Scientific Computing Center > > Lawrence Berkeley National Laboratory > > > > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department (UW Medicine), System Administrator -- Foege Building S046, (206)-685-7354 -- Pronouns: He/Him/His ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] Spectrum Protect and disk pools
I think the collocation settings of the target pool for the migration come into play as well. If you have multiple filespaces associated with a node and collocation is set to FILESPACE, then you should be able to get one migration process per filespace rather than one per node/collocation group. On Mon, Jan 04, 2021 at 12:21:05PM +, Simon Thompson wrote: > Hi All, > > We use Spectrum Protect (TSM) to backup our Scale filesystems. We have the > backup setup to use multiple nodes with the PROXY node function turned on > (and to some extent also use multiple target servers). > > This all feels like it is nice and parallel, on the TSM servers, we have disk > pools for any ???small??? files to drop into (I think we set anything smaller > than 20GB) to prevent lots of small files stalling tape drive writes. > > Whilst digging into why we have slow backups at times, we found that the disk > pool empties with a single thread (one drive). And looking at the docs: > https://www.ibm.com/support/pages/concurrent-migration-processes-and-constraints > > This implies that we are limited to the number of client nodes stored in the > pool. i.e. because we have one node and PROXY nodes, we are essentially > limited to a single thread streaming out of the disk pool when full. > > Have we understood this correctly as if so, this appears to make the whole > purpose of PROXY nodes sort of pointless if you have lots of small files. Or > is there some other setting we should be looking at to increase the number of > threads when the disk pool is emptying? (The disk pool itself has Migration > Processes: 6) > > Thanks > > Simon > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department (UW Medicine), System Administrator -- Foege Building S046, (206)-685-7354 -- Pronouns: He/Him/His ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] GPFS systemd and gpfs.gplbin
Thanks, Jonathan, I've been thinking about how to manage this as well and like it more than version-locking the kernel. On Wed, Jun 09, 2021 at 09:28:07PM +0100, Jonathan Buzzard wrote: > > So you need to apply a kernel update and that means a new gpfs.gplbin :-( So > after going around the houses with several different approaches on this I > have finally settled on what IMHO is a most elegant method of ensuring the > right gpfs.gplbin version is installed for the kernel that is running and > thought I would share it. > > This is assuming you don't like the look of the compile it option IBM > introduced. You may well not want compilers installed on nodes for example, > or you just think compiling the module on hundreds of nodes is suboptimal. > > This exact version works for RHEL and it's derivatives. Modify for your > preferred distribution. It also assumes you have a repository setup with the > relevant gpfs.gplbin package. > > The basics are to use the "ExecStartPre" option of a unit file in systemd. > So because you don't want to be modifying the unit file provided by IBM > something like the following > > mkdir -p /etc/systemd/system/gpfs.service.d > echo -e "[Service]\nExecStartPre=-/usr/bin/yum --assumeyes install > gpfs.gplbin-%v" >/etc/systemd/system/gpfs.service.d/install-module.conf > systemctl daemon-reload > > How it works is that the %v is a special systemd variable which is the same > as "uname -r". So before it attempts to start gpfs, it attempts to install > the gpfs.gplbin RPM for the kernel you are running on. If already installed > this is harmless and if it's not installed it gets installed. > > How you set that up on your system is up to you, xCAT postscript, RPM > package, or a configuration management solution all work. I have gone for a > very minimal RPM I call gpfs.helper > > We then abuse the queuing system on the HPC cluster to schedule a "admin" > priority job that runs as soon as the node becomes free, which does a yum > update and then restarts the node. > > > JAB. > > -- > Jonathan A. Buzzard Tel: +44141-5483420 > HPC System Administrator, ARCHIE-WeSt. > University of Strathclyde, John Anderson Building, Glasgow. G4 0NG > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department (UW Medicine), System Administrator -- Foege Building S046, (206)-685-7354 -- Pronouns: He/Him/His ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] Ransom attacks
You can get clever/complicated (the interpretation could go either way) with ACLs and SELinux but, at the end of the day, nothing beats the air-gap of tape backups, IMHO. You might consider a belt approach that includes all of the above plus other controls (2FA, network security, etc.), and in my experience combining multiple solutions gives flexibility in that it can be easier to avoid the higher-cost aspects of one solution taken to an extreme by having one layer mitigate the shortcomings of another layer. On Thu, May 27, 2021 at 04:10:39PM +0100, Henrik Morsing wrote: > > Hi, > > It struck me that switching a Spectrum Protect solution from tapes to a GPFS > filesystem offers much less protection against ransom encryption should the > SP server be compromised. Same goes really for compromising an ESS node > itself, it is an awful lot of data that can be encrypted very quickly. > > Is there anything that can protect the GPFS filesystem against this kind of > attack? -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department (UW Medicine), System Administrator -- Foege Building S046, (206)-685-7354 -- Pronouns: He/Him/His ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] How to do multiple mounts via GPFS
Assuming this is on Linux, you ought to be able to use bind mounts for that, something like this in fstab or equivalent: /home /gpfs1/home bind defaults 0 0 On Tue, Feb 22, 2022 at 12:24:09PM -0500, Justin Cantrell wrote: > We're trying to mount multiple mounts at boot up via gpfs. > We can mount the main gpfs mount /gpfs1, but would like to mount things > like: > /home /gpfs1/home > /other /gpfs1/other > /stuff /gpfs1/stuff > > But adding that to fstab doesn't work, because from what I understand, > that's not how gpfs works with mounts. > What's the standard way to accomplish something like this? > We've used systemd timers/mounts to accomplish it, but that's not ideal. > Is there a way to do this natively with gpfs or does this have to be done > through symlinks or gpfs over nfs? -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department (UW Medicine), System Administrator -- Foege Building S046, (206)-685-7354 -- Pronouns: He/Him/His ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] How to do multiple mounts via GPFS
The problem might be that the service indicates success when mmstartup returns rather than when the mount is actually active (requires quorum checking, arbitration, etc.). A couple tricks I can think of would be using ConditionPathIsMountPoint from systemd.unit[1], or maybe adding a callback[2] that triggers on the mount condition for your filesystem that makes the bind mount rather than systemd. [1] https://www.freedesktop.org/software/systemd/man/systemd.unit.html#ConditionPathIsMountPoint= [2] https://www.ibm.com/docs/en/spectrum-scale/5.1.2?topic=reference-mmaddcallback-command These are both on our todo list for improving our own GPFS mounting as we have problems with our job scheduler not starting reliably on reboot, but for us we can have Puppet start it on the next run so it just means nodes might not return to service for 30 minutes or so. On Tue, Feb 22, 2022 at 03:05:58PM -0500, Justin Cantrell wrote: > This is how we're currently solving this problem, with systemd timer and > mount. None of the requires seem to work with gpfs since it starts so late. > I would like a better solution. > > Is it normal for gpfs to start so late?? I think it doesn't mount until > after the gpfs.service starts, and even then it's 20-30 seconds. > > > On 2/22/22 14:42, Skylar Thompson wrote: > > Like Tina, we're doing bind mounts in autofs. I forgot that there might be > > a race condition if you're doing it in fstab. If you're on system with > > systemd, > > another option might be to do this directly with systemd.mount rather than > > let the fstab generator make the systemd.mount units: > > > > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.freedesktop.org%2Fsoftware%2Fsystemd%2Fman%2Fsystemd.mount.htmldata=04%7C01%7Cjcantrell1%40gsu.edu%7C2a65cd0ddefd48cb81a308d9f63bb840%7C515ad73d8d5e4169895c9789dc742a70%7C0%7C0%7C637811559082622923%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000sdata=%2BWWD7cCNSMeJEYwELldYT3pLdXVX3AxJj7gqZQCqUv4%3Dreserved=0 > > > > You could then set RequiresMountFor=gpfs1.mount in the bind mount unit. > > > > On Tue, Feb 22, 2022 at 02:23:53PM -0500, Justin Cantrell wrote: > > > I tried a bind mount, but perhaps I'm doing it wrong. The system fails > > > to boot because gpfs doesn't start until too late in the boot process. > > > In fact, the system boots and the gpfs1 partition isn't available for a > > > good 20-30 seconds. > > > > > > /gfs1/home??? /home??? none bind > > > I've tried adding mount options of x-systemd-requires=gpfs1, noauto. > > > The noauto lets it boot, but the mount is never mounted properly. Doing > > > a manual mount -a mounts it. > > > > > > On 2/22/22 12:37, Skylar Thompson wrote: > > > > Assuming this is on Linux, you ought to be able to use bind mounts for > > > > that, something like this in fstab or equivalent: > > > > > > > > /home /gpfs1/home bind defaults 0 0 > > > > > > > > On Tue, Feb 22, 2022 at 12:24:09PM -0500, Justin Cantrell wrote: > > > > > We're trying to mount multiple mounts at boot up via gpfs. > > > > > We can mount the main gpfs mount /gpfs1, but would like to mount > > > > > things > > > > > like: > > > > > /home /gpfs1/home > > > > > /other /gpfs1/other > > > > > /stuff /gpfs1/stuff > > > > > > > > > > But adding that to fstab doesn't work, because from what I understand, > > > > > that's not how gpfs works with mounts. > > > > > What's the standard way to accomplish something like this? > > > > > We've used systemd timers/mounts to accomplish it, but that's not > > > > > ideal. > > > > > Is there a way to do this natively with gpfs or does this have to be > > > > > done > > > > > through symlinks or gpfs over nfs? > > > ___ > > > gpfsug-discuss mailing list > > > gpfsug-discuss at spectrumscale.org > > > https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discussdata=04%7C01%7Cjcantrell1%40gsu.edu%7C2a65cd0ddefd48cb81a308d9f63bb840%7C515ad73d8d5e4169895c9789dc742a70%7C0%7C0%7C637811559082622923%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000sdata=F4oXAT0zdY%2BS1mR784ZGghUt0G%2F6Ofu36MfJ9WnPsPM%3Dreserved=0 > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department (UW Medicine), System Administrator -- Foege Building S046, (206)-685-7354 -- Pronouns: He/Him/His ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] How to do multiple mounts via GPFS
Like Tina, we're doing bind mounts in autofs. I forgot that there might be a race condition if you're doing it in fstab. If you're on system with systemd, another option might be to do this directly with systemd.mount rather than let the fstab generator make the systemd.mount units: https://www.freedesktop.org/software/systemd/man/systemd.mount.html You could then set RequiresMountFor=gpfs1.mount in the bind mount unit. On Tue, Feb 22, 2022 at 02:23:53PM -0500, Justin Cantrell wrote: > I tried a bind mount, but perhaps I'm doing it wrong. The system fails > to boot because gpfs doesn't start until too late in the boot process. > In fact, the system boots and the gpfs1 partition isn't available for a > good 20-30 seconds. > > /gfs1/home??? /home??? none bind > I've tried adding mount options of x-systemd-requires=gpfs1, noauto. > The noauto lets it boot, but the mount is never mounted properly. Doing > a manual mount -a mounts it. > > On 2/22/22 12:37, Skylar Thompson wrote: > > Assuming this is on Linux, you ought to be able to use bind mounts for > > that, something like this in fstab or equivalent: > > > > /home /gpfs1/home bind defaults 0 0 > > > > On Tue, Feb 22, 2022 at 12:24:09PM -0500, Justin Cantrell wrote: > > > We're trying to mount multiple mounts at boot up via gpfs. > > > We can mount the main gpfs mount /gpfs1, but would like to mount things > > > like: > > > /home /gpfs1/home > > > /other /gpfs1/other > > > /stuff /gpfs1/stuff > > > > > > But adding that to fstab doesn't work, because from what I understand, > > > that's not how gpfs works with mounts. > > > What's the standard way to accomplish something like this? > > > We've used systemd timers/mounts to accomplish it, but that's not ideal. > > > Is there a way to do this natively with gpfs or does this have to be done > > > through symlinks or gpfs over nfs? > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department (UW Medicine), System Administrator -- Foege Building S046, (206)-685-7354 -- Pronouns: He/Him/His ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] mmbackup file selections
Hi Paul, Not to toot my own horn and while the DEBUGmmbackup=2 method definitely does work, you might want to vote for this RFE I put in a month ago to get a more robust "dry run" mode with mmbackup, since guessing how include/exclude rules get translated from SP/dsmc to SS/mmbackup can be challenging: https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe_ID=153520 Somewhat selfishly, I think implementing the RFE would benefit you as well. :) On Tue, Jan 18, 2022 at 04:56:17PM +, Paul Ward wrote: > Hi, > > I am trying to work out what files have been sent to backup using mmbackup. > I have increased the -L value from 3 up to 6 but only seem to see the files > that are in scope, not the ones that are selected. > > I can see the three file lists generated during a backup, but can't seem to > find a list of what files were backed up. > > It should be the diff of the shadow and shadow-old, but the wc -l of the diff > doesn't match the number of files in the backup summary. > Wrong assumption? > > Where should I be looking - surely it shouldn't be this hard to see what > files are selected? > > > Kindest regards, > Paul > > Paul Ward > TS Infrastructure Architect > Natural History Museum > T: 02079426450 > E: p.w...@nhm.ac.uk<mailto:p.w...@nhm.ac.uk> > [A picture containing drawing Description automatically generated] > > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department (UW Medicine), System Administrator -- Foege Building S046, (206)-685-7354 -- Pronouns: He/Him/His ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] mmbackup file selections
ntries. > > [Inactive hide details for "Paul Ward" ---01/18/2022 11:56:40 AM---Hi, I am > trying to work out what files have been sent to back]"Paul Ward" > ---01/18/2022 11:56:40 AM---Hi, I am trying to work out what files have been > sent to backup using mmbackup. > > From: "Paul Ward" mailto:p.w...@nhm.ac.uk>> > To: > "gpfsug-discuss@spectrumscale.org<mailto:gpfsug-discuss@spectrumscale.org>" > mailto:gpfsug-discuss@spectrumscale.org>> > Date: 01/18/2022 11:56 AM > Subject: [EXTERNAL] [gpfsug-discuss] mmbackup file selections > Sent by: > gpfsug-discuss-boun...@spectrumscale.org<mailto:gpfsug-discuss-boun...@spectrumscale.org> > > > > > > > > Hi, I am trying to work out what files have been sent to backup using > mmbackup. I have increased the -L value from 3 up to 6 but only seem to see > the files that are in scope, not the ones that are selected. I can see the > three file lists generated ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > ZjQcmQRYFpfptBannerEnd > Hi, > > I am trying to work out what files have been sent to backup using mmbackup. > I have increased the -L value from 3 up to 6 but only seem to see the files > that are in scope, not the ones that are selected. > > I can see the three file lists generated during a backup, but can?t seem to > find a list of what files were backed up. > > It should be the diff of the shadow and shadow-old, but the wc -l of the diff > doesn?t match the number of files in the backup summary. > Wrong assumption? > > Where should I be looking ? surely it shouldn?t be this hard to see what > files are selected? > > > Kindest regards, > Paul > > Paul Ward > TS Infrastructure Architect > Natural History Museum > T: 02079426450 > E: p.w...@nhm.ac.uk<mailto:p.w...@nhm.ac.uk> > [A picture containing drawingDescription automatically generated] > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss=04%7C01%7Cp.ward%40nhm.ac.uk%7C17acea2a1b4944dc8daa08d9dd3f1fea%7C73a29c014e78437fa0d4c8553e1960c1%7C1%7C0%7C637784085889014099%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000=jJykq3GwRIajX24bWsBuk6ydadeiumBiSLZeKnMdiZw%3D=0> > > > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss=04%7C01%7Cp.ward%40nhm.ac.uk%7C17acea2a1b4944dc8daa08d9dd3f1fea%7C73a29c014e78437fa0d4c8553e1960c1%7C1%7C0%7C637784085889014099%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000=jJykq3GwRIajX24bWsBuk6ydadeiumBiSLZeKnMdiZw%3D=0> > > > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss=04%7C01%7Cp.ward%40nhm.ac.uk%7C17acea2a1b4944dc8daa08d9dd3f1fea%7C73a29c014e78437fa0d4c8553e1960c1%7C1%7C0%7C637784085889014099%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000=jJykq3GwRIajX24bWsBuk6ydadeiumBiSLZeKnMdiZw%3D=0> > > > > ___ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department (UW Medicine), System Administrator -- Foege Building S046, (206)-685-7354 -- Pronouns: He/Him/His ___ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Re: [gpfsug-discuss] mmbackup file selections
pool name:data > fileset name: hpc-workspaces-fset > snapshot name: > creation time:Wed Oct 20 17:55:18 2021 > Misc attributes: ARCHIVE > Encrypted:no > > Check active and inactive backups (it was backed up yesterday) > 11:18:52 [root@scale-sk-pn-1 .mmbackupCfg]# dsmcqbi > "/gpfs/nhmfsa/bulk/share/data/mbl/share/workspaces/groups/urban-nature-project/audiowaveform/300_40/unp-grounds-01-1604557538.png" > IBM Spectrum Protect > Command Line Backup-Archive Client Interface > Client Version 8, Release 1, Level 10.0 > Client date/time: 01/26/2022 11:19:02 > (c) Copyright by IBM Corporation and other(s) 1990, 2020. All Rights Reserved. > > Node Name: SC-PN-SK-01 > Session established with server TSM-JERSEY: Windows > Server Version 8, Release 1, Level 10.100 > Server date/time: 01/26/2022 11:19:02 Last access: 01/26/2022 11:07:05 > > Accessing as node: SCALE >SizeBackup DateMgmt Class A/I > File >----- --- > >545 B 01/25/2022 06:41:17 DEFAULT A > /gpfs/nhmfsa/bulk/share/data/mbl/share/workspaces/groups/urban-nature-project/audiowaveform/300_40/unp-grounds-01-1604557538.png >545 B 12/28/2021 21:19:18 DEFAULT I > /gpfs/nhmfsa/bulk/share/data/mbl/share/workspaces/groups/urban-nature-project/audiowaveform/300_40/unp-grounds-01-1604557538.png >545 B 01/04/2022 06:17:35 DEFAULT I > /gpfs/nhmfsa/bulk/share/data/mbl/share/workspaces/groups/urban-nature-project/audiowaveform/300_40/unp-grounds-01-1604557538.png >545 B 01/04/2022 06:18:05 DEFAULT I > /gpfs/nhmfsa/bulk/share/data/mbl/share/workspaces/groups/urban-nature-project/audiowaveform/300_40/unp-grounds-01-1604557538.png > > > It will be backed up again shortly, why? > > And it was backed up again: > # dsmcqbi > /gpfs/nhmfsa/bulk/share/data/mbl/share/workspaces/groups/urban-nature-project/audiowaveform/300_40/unp-grounds-01-1604557538.png > IBM Spectrum Protect > Command Line Backup-Archive Client Interface > Client Version 8, Release 1, Level 10.0 > Client date/time: 01/26/2022 15:54:09 > (c) Copyright by IBM Corporation and other(s) 1990, 2020. All Rights Reserved. > > Node Name: SC-PN-SK-01 > Session established with server TSM-JERSEY: Windows > Server Version 8, Release 1, Level 10.100 > Server date/time: 01/26/2022 15:54:10 Last access: 01/26/2022 15:30:03 > > Accessing as node: SCALE >SizeBackup DateMgmt Class A/I > File >----- --- > >545 B 01/26/2022 12:23:02 DEFAULT A > /gpfs/nhmfsa/bulk/share/data/mbl/share/workspaces/groups/urban-nature-project/audiowaveform/300_40/unp-grounds-01-1604557538.png >545 B 12/28/2021 21:19:18 DEFAULT I > /gpfs/nhmfsa/bulk/share/data/mbl/share/workspaces/groups/urban-nature-project/audiowaveform/300_40/unp-grounds-01-1604557538.png >545 B 01/04/2022 06:17:35 DEFAULT I > /gpfs/nhmfsa/bulk/share/data/mbl/share/workspaces/groups/urban-nature-project/audiowaveform/300_40/unp-grounds-01-1604557538.png >545 B 01/04/2022 06:18:05 DEFAULT I > /gpfs/nhmfsa/bulk/share/data/mbl/share/workspaces/groups/urban-nature-project/audiowaveform/300_40/unp-grounds-01-1604557538.png >545 B 01/25/2022 06:41:17 DEFAULT I > /gpfs/nhmfsa/bulk/share/data/mbl/share/workspaces/groups/urban-nature-project/audiowaveform/300_40/unp-grounds-01-1604557538.png > > Kindest regards, > Paul > > Paul Ward > TS Infrastructure Architect > Natural History Museum > T: 02079426450 > E: p.w...@nhm.ac.uk > > > -Original Message- > From: gpfsug-discuss-boun...@spectrumscale.org > On Behalf Of Skylar Thompson > Sent: 24 January 2022 15:37 > To: gpfsug main discussion list > Cc: gpfsug-discuss-boun...@spectrumscale.org > Subject: Re: [gpfsug-discuss] mmbackup file selections > > Hi Paul, > > Did you look for dot files? At least for us on 5.0.5 there's a > .list.1. file while the backups are running: > > /gpfs/grc6/.mmbackupCfg/updatedFiles/: > -r 1 root nickers 6158526821 Jan 23 18:28 .list.1.gpfs-grc6 > /gpfs/grc6/.mmbackupCfg/expiredFiles/: > -r 1 root nickers 85862211 Jan 23 18:28 .list.1.gpfs-grc6 > > On Mon, Jan 24, 2022 at 02:31:54PM +, Paul Ward wrote: > > Those di
