Re: [gpfsug-discuss] Automating Snapshots : cron jobs or use the GUI ?

[Jaap Jan Ouwehand]

Hi,

I also used a custom script (database driven) via cron which creates many 
fileset snapshots during the day via the "default helper nodes".  Because of 
the iops, the oldest snapshots are deleted at night.

Perhaps it's a good idea to take one global filesystem snapshot and make it 
available to the filesets with mmsnapdir.

Kind regards,
Jaap Jan Ouwehand





"Hannappel, Juergen"  schreef op 2 februari 2022 
16:04:24 CET:
>Hi, 
>I use a python script via cron job, it checks how many snapshots exist and 
>removes those that 
>exceed a configurable limit, then creates a new one. 
>Deployed via puppet it's much less hassle than click around in a GUI/ 
>
>> From: "Kidger, Daniel" 
>> To: "gpfsug main discussion list" 
>> Sent: Wednesday, 2 February, 2022 11:07:25
>> Subject: [gpfsug-discuss] Automating Snapshots : cron jobs or use the GUI ?
>
>> Hi all,
>
>> Since the subject of snapshots has come up, I also have a question ...
>
>> Snapshots can be created from the command line with mmcrsnapshot, and hence 
>> can
>> be automated via con jobs etc.
>> Snapshots can also be created from the Scale GUI. The GUI also provides its 
>> own
>> automation for the creation, retention, and deletion of snapshots.
>
>> My question is: do most customers use the former or the latter for 
>> automation?
>
>> (I also note that /usr/lpp/mmfs/gui/cli/mksnaprule exists and appears to do
>> exactly the same as what the GUI does it terms of creating automated 
>> snapshots.
>> It is a relic of V7000 Unified but still works fine in Spectrum Scale 
>> 5.1.2.2.
>> How many customers also use the commands found in /usr/lpp/mmfs/gui/cli / ? )
>
>> Daniel
>
>> Daniel Kidger
>> HPC Storage Solutions Architect, EMEA
>> [ mailto:daniel.kid...@hpe.com | daniel.kid...@hpe.com ]
>
>> +44 (0)7818 522266
>
>> [ http://www.hpe.com/ | hpe.com ]
>
>> ___
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] [External] Automating Snapshots : cron jobs or use the GUI ?

[mark . bergman]

Big vote for cron jobs.

Our snapshot are created by a script, installed on each GPFS node. The script 
handles naming, removing old snapshots, checking that sufficient disk space 
exists before creating a snapshot,
etc.  We do snapshots every 15 minutes, keeping them with lower frequency over 
longer intervals. For example:

current hour:   keep 4 snapshots
hours -2 .. -8  keep 3 snapshots per hour
hours -8 .. -24 keep 2 snapshots per hour
days -1 .. -5   keep 1 snapshot per hour
days -5 .. -15  keep 4 snapshots per day
days -15 .. -30 keep 1 snapshot per day

the duration & frequency & minimum disk space can be adjusted per-filesystem.

The automation is done through a cronjob that runs on each GPFS (DSS-G) server 
to create the snapshot only if the node is currently the cluster master, as in:

*/15 * * * * root mmlsmgr -Y | grep -q "clusterManager.*:$(hostname 
--long):" && /path/to/snapshotter

This requires no locking and ensures that only a single instance of snapshots 
is created at each time interval.

We use the same trick to gather GPFS health stats, etc., ensuring that the data 
collection only runs on a single node (the cluster manager).


-- 
Mark Bergman   voice: 215-746-4061  
 
mark.berg...@pennmedicine.upenn.edu  fax: 215-614-0266
http://www.med.upenn.edu/cbica/
IT Technical Director, Center for Biomedical Image Computing and Analytics
Department of Radiology University of Pennsylvania


___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] Automating Snapshots : cron jobs or use the GUI ?

[Hannappel, Juergen]

Hi, 
I use a python script via cron job, it checks how many snapshots exist and 
removes those that 
exceed a configurable limit, then creates a new one. 
Deployed via puppet it's much less hassle than click around in a GUI/ 

> From: "Kidger, Daniel" 
> To: "gpfsug main discussion list" 
> Sent: Wednesday, 2 February, 2022 11:07:25
> Subject: [gpfsug-discuss] Automating Snapshots : cron jobs or use the GUI ?

> Hi all,

> Since the subject of snapshots has come up, I also have a question ...

> Snapshots can be created from the command line with mmcrsnapshot, and hence 
> can
> be automated via con jobs etc.
> Snapshots can also be created from the Scale GUI. The GUI also provides its 
> own
> automation for the creation, retention, and deletion of snapshots.

> My question is: do most customers use the former or the latter for automation?

> (I also note that /usr/lpp/mmfs/gui/cli/mksnaprule exists and appears to do
> exactly the same as what the GUI does it terms of creating automated 
> snapshots.
> It is a relic of V7000 Unified but still works fine in Spectrum Scale 5.1.2.2.
> How many customers also use the commands found in /usr/lpp/mmfs/gui/cli / ? )

> Daniel

> Daniel Kidger
> HPC Storage Solutions Architect, EMEA
> [ mailto:daniel.kid...@hpe.com | daniel.kid...@hpe.com ]

> +44 (0)7818 522266

> [ http://www.hpe.com/ | hpe.com ]

> ___
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] snapshots causing filesystem quiesce

[Jordi Caubet Serrabou]

Maybe some colleagues at IBM devel can correct me, but pagepool size should not 
make much difference. Afaik, it is mostly read cache data. Another think could 
be if using HAWC function, I am not sure in such case.

Anyhow, looking at your node name, your system seems a DSS from Lenovo so you 
NSD servers are running GPFS Native RAID and the reason why the pagepool is 
large there, not for the NSD server role itself, it is for the GNR role that 
caches disk tracks. Lowering will impact performance.

--
Jordi Caubet Serrabou
IBM Software Defined Infrastructure (SDI) and Flash Technical Sales Specialist
Technical Computing and HPC IT Specialist and Architect

> On 2 Feb 2022, at 14:03, Talamo Ivano Giuseppe (PSI)  
> wrote:
> 
> 
> That's true, although I would not expect the memory to be flushed for just 
> snapshots deletion. But it could well be a problem at snapshot creation time.
> 
> Anyway for changing the pagepool we should contact the vendor, since this is 
> configured by their installation scripts, so we better have them to agree.
> 
> 
> 
> Cheers,
> 
> Ivano
> 
> 
> 
> __
> Paul Scherrer Institut
> Ivano Talamo
> WHGA/038
> Forschungsstrasse 111
> 5232 Villigen PSI
> Schweiz
> 
> Telefon: +41 56 310 47 11
> E-Mail: ivano.tal...@psi.ch
> 
> 
> 
> From: gpfsug-discuss-boun...@spectrumscale.org 
>  on behalf of Alec 
> 
> Sent: Wednesday, February 2, 2022 1:41 PM
> To: gpfsug main discussion list
> Subject: Re: [gpfsug-discuss] snapshots causing filesystem quiesce
>  
> Might it be a case of being over built?  In the old days you could really 
> mess up an Oracle DW by giving it too much RAM... It would spend all day 
> reading in and out data to the ram that it didn't really need, because it had 
> the SGA available to load the whole table.
> 
> Perhaps the pagepool is so large that the time it takes to clear that much 
> RAM is the actual time out?
> 
> My environment has only a million files but has quite a bit more storage and 
> has only an 8gb pagepool.  Seems you are saying you have 618gb of RAM for 
> pagepool...  Even at 8GB/second that would take 77 seconds to flush it out..
> 
> Perhaps drop the pagepool in half and see if your timeout adjusts accordingly?
> 
> Alec
> 
> 
>> On Wed, Feb 2, 2022, 4:09 AM Olaf Weiser  wrote:
>> keep in mind... creating many snapshots... means ;-) .. you'll have to 
>> delete many snapshots..
>> at a certain level, which depends on #files, #directories, ~workload, 
>> #nodes, #networks etc we ve seen cases, where generating just full 
>> snapshots (whole file system)  is the better approach instead of maintaining 
>> snapshots for each file set individually ..
>>  
>> sure. this has other side effects , like space consumption etc...
>> so as always.. it depends..
>>  
>>  
>>  
>> - Ursprüngliche Nachricht -
>> Von: "Jan-Frode Myklebust" 
>> Gesendet von: gpfsug-discuss-boun...@spectrumscale.org
>> An: "gpfsug main discussion list" 
>> CC:
>> Betreff: [EXTERNAL] Re: [gpfsug-discuss] snapshots causing filesystem quiesce
>> Datum: Mi, 2. Feb 2022 12:54
>>  
>> Also, if snapshotting multiple filesets, it's important to group these into 
>> a single mmcrsnapshot command. Then you get a single quiesce, instead of one 
>> per fileset.
>>  
>> i.e. do:
>>  
>> snapname=$(date --utc +@GMT-%Y.%m.%d-%H.%M.%S)
>> mmcrsnapshot gpfs0 fileset1:$snapname,filset2:snapname,fileset3:snapname
>>  
>> instead of:
>>  
>> mmcrsnapshot gpfs0 fileset1:$snapname
>> mmcrsnapshot gpfs0 fileset2:$snapname
>> mmcrsnapshot gpfs0 fileset3:$snapname   
>>  
>>  
>>   -jf
>>  
>>  
>> On Wed, Feb 2, 2022 at 12:07 PM Jordi Caubet Serrabou 
>>  wrote:
>> Ivano,
>>  
>> if it happens frequently, I would recommend to open a support case.
>>  
>> The creation or deletion of a snapshot requires a quiesce of the nodes to 
>> obtain a consistent point-in-time image of the file system and/or update 
>> some internal structures afaik. Quiesce is required for nodes at the storage 
>> cluster but also remote clusters. Quiesce means stop activities (incl. I/O) 
>> for a short period of time to get such consistent image. Also waiting to 
>> flush any data in-flight to disk that does not allow a consistent 
>> point-in-time image.
>>  
>> Nodes receive a quiesce request and acknowledge when ready. When all nodes 
>> acknowledge, snapshot operation can proceed and immediately I/O can resume. 
>> It usually takes few seconds at most and the operation performed is short 
>> but time I/O is stopped depends of how long it takes to quiesce the nodes. 
>> If some node take longer to agree stop the activities, such node will be 
>> delay the completion of the quiesce and keep I/O paused on the rest.
>> There could many things while some nodes delay quiesce ack.
>>  
>> The larger the cluster, the more difficult it gets. The more network 
>> congestion or I/O load, the more difficult it gets. I recommend to open a 
>> ticket for support 

Re: [gpfsug-discuss] snapshots causing filesystem quiesce

[Talamo Ivano Giuseppe (PSI)]

That's true, although I would not expect the memory to be flushed for just 
snapshots deletion. But it could well be a problem at snapshot creation time.

Anyway for changing the pagepool we should contact the vendor, since this is 
configured by their installation scripts, so we better have them to agree.


Cheers,

Ivano


__
Paul Scherrer Institut
Ivano Talamo
WHGA/038
Forschungsstrasse 111
5232 Villigen PSI
Schweiz

Telefon: +41 56 310 47 11
E-Mail: ivano.tal...@psi.ch




From: gpfsug-discuss-boun...@spectrumscale.org 
 on behalf of Alec 
Sent: Wednesday, February 2, 2022 1:41 PM
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] snapshots causing filesystem quiesce

Might it be a case of being over built?  In the old days you could really mess 
up an Oracle DW by giving it too much RAM... It would spend all day reading in 
and out data to the ram that it didn't really need, because it had the SGA 
available to load the whole table.

Perhaps the pagepool is so large that the time it takes to clear that much RAM 
is the actual time out?

My environment has only a million files but has quite a bit more storage and 
has only an 8gb pagepool.  Seems you are saying you have 618gb of RAM for 
pagepool...  Even at 8GB/second that would take 77 seconds to flush it out..

Perhaps drop the pagepool in half and see if your timeout adjusts accordingly?

Alec


On Wed, Feb 2, 2022, 4:09 AM Olaf Weiser 
mailto:olaf.wei...@de.ibm.com>> wrote:
keep in mind... creating many snapshots... means ;-) .. you'll have to delete 
many snapshots..
at a certain level, which depends on #files, #directories, ~workload, #nodes, 
#networks etc we ve seen cases, where generating just full snapshots (whole 
file system)  is the better approach instead of maintaining snapshots for each 
file set individually ..

sure. this has other side effects , like space consumption etc...
so as always.. it depends..



- Ursprüngliche Nachricht -
Von: "Jan-Frode Myklebust" mailto:janfr...@tanso.net>>
Gesendet von: 
gpfsug-discuss-boun...@spectrumscale.org
An: "gpfsug main discussion list" 
mailto:gpfsug-discuss@spectrumscale.org>>
CC:
Betreff: [EXTERNAL] Re: [gpfsug-discuss] snapshots causing filesystem quiesce
Datum: Mi, 2. Feb 2022 12:54

Also, if snapshotting multiple filesets, it's important to group these into a 
single mmcrsnapshot command. Then you get a single quiesce, instead of one per 
fileset.

i.e. do:

snapname=$(date --utc +@GMT-%Y.%m.%d-%H.%M.%S)
mmcrsnapshot gpfs0 fileset1:$snapname,filset2:snapname,fileset3:snapname

instead of:

mmcrsnapshot gpfs0 fileset1:$snapname
mmcrsnapshot gpfs0 fileset2:$snapname
mmcrsnapshot gpfs0 fileset3:$snapname


  -jf


On Wed, Feb 2, 2022 at 12:07 PM Jordi Caubet Serrabou 
mailto:jordi.cau...@es.ibm.com>> wrote:
Ivano,

if it happens frequently, I would recommend to open a support case.

The creation or deletion of a snapshot requires a quiesce of the nodes to 
obtain a consistent point-in-time image of the file system and/or update some 
internal structures afaik. Quiesce is required for nodes at the storage cluster 
but also remote clusters. Quiesce means stop activities (incl. I/O) for a short 
period of time to get such consistent image. Also waiting to flush any data 
in-flight to disk that does not allow a consistent point-in-time image.

Nodes receive a quiesce request and acknowledge when ready. When all nodes 
acknowledge, snapshot operation can proceed and immediately I/O can resume. It 
usually takes few seconds at most and the operation performed is short but time 
I/O is stopped depends of how long it takes to quiesce the nodes. If some node 
take longer to agree stop the activities, such node will be delay the 
completion of the quiesce and keep I/O paused on the rest.
There could many things while some nodes delay quiesce ack.

The larger the cluster, the more difficult it gets. The more network congestion 
or I/O load, the more difficult it gets. I recommend to open a ticket for 
support to try to identify the root cause of which nodes not acknowledge the 
quiesce  and maybe find the root cause. If I recall some previous thread, 
default timeout was 60 seconds which match your log message. After such 
timeout, snapshot is considered failed to complete.

Support might help you understand the root cause and provide some 
recommendations if it happens frequently.

Best Regards,
--
Jordi Caubet Serrabou
IBM Storage Client Technical Specialist (IBM Spain)

- Original message -
From: "Talamo Ivano Giuseppe (PSI)" 
mailto:ivano.tal...@psi.ch>>
Sent by: 
gpfsug-discuss-boun...@spectrumscale.org
To: "gpfsug main discussion list" 
mailto:gpfsug-discuss@spectrumscale.org>>
Cc:
Subject: [EXTERNAL] Re: [gpfsug-discuss] snapshots causing filesystem quiesce
Date: Wed, Feb 2, 

Re: [gpfsug-discuss] snapshots causing filesystem quiesce

[Talamo Ivano Giuseppe (PSI)]

Ok that sounds a good candidate for an improvement. Thanks.

We didn't want to do a full filesystem snapshot for the space consumption 
indeed. But we may consider it, keeping an eye on the space.


Cheers,

Ivano


__
Paul Scherrer Institut
Ivano Talamo
WHGA/038
Forschungsstrasse 111
5232 Villigen PSI
Schweiz

Telefon: +41 56 310 47 11
E-Mail: ivano.tal...@psi.ch




From: gpfsug-discuss-boun...@spectrumscale.org 
 on behalf of Olaf Weiser 

Sent: Wednesday, February 2, 2022 1:09 PM
To: gpfsug-discuss@spectrumscale.org
Cc: gpfsug-discuss@spectrumscale.org
Subject: Re: [gpfsug-discuss] snapshots causing filesystem quiesce

keep in mind... creating many snapshots... means ;-) .. you'll have to delete 
many snapshots..
at a certain level, which depends on #files, #directories, ~workload, #nodes, 
#networks etc we ve seen cases, where generating just full snapshots (whole 
file system)  is the better approach instead of maintaining snapshots for each 
file set individually ..

sure. this has other side effects , like space consumption etc...
so as always.. it depends..



- Ursprüngliche Nachricht -
Von: "Jan-Frode Myklebust" 
Gesendet von: gpfsug-discuss-boun...@spectrumscale.org
An: "gpfsug main discussion list" 
CC:
Betreff: [EXTERNAL] Re: [gpfsug-discuss] snapshots causing filesystem quiesce
Datum: Mi, 2. Feb 2022 12:54

Also, if snapshotting multiple filesets, it's important to group these into a 
single mmcrsnapshot command. Then you get a single quiesce, instead of one per 
fileset.

i.e. do:

snapname=$(date --utc +@GMT-%Y.%m.%d-%H.%M.%S)
mmcrsnapshot gpfs0 fileset1:$snapname,filset2:snapname,fileset3:snapname

instead of:

mmcrsnapshot gpfs0 fileset1:$snapname
mmcrsnapshot gpfs0 fileset2:$snapname
mmcrsnapshot gpfs0 fileset3:$snapname


  -jf


On Wed, Feb 2, 2022 at 12:07 PM Jordi Caubet Serrabou 
mailto:jordi.cau...@es.ibm.com>> wrote:
Ivano,

if it happens frequently, I would recommend to open a support case.

The creation or deletion of a snapshot requires a quiesce of the nodes to 
obtain a consistent point-in-time image of the file system and/or update some 
internal structures afaik. Quiesce is required for nodes at the storage cluster 
but also remote clusters. Quiesce means stop activities (incl. I/O) for a short 
period of time to get such consistent image. Also waiting to flush any data 
in-flight to disk that does not allow a consistent point-in-time image.

Nodes receive a quiesce request and acknowledge when ready. When all nodes 
acknowledge, snapshot operation can proceed and immediately I/O can resume. It 
usually takes few seconds at most and the operation performed is short but time 
I/O is stopped depends of how long it takes to quiesce the nodes. If some node 
take longer to agree stop the activities, such node will be delay the 
completion of the quiesce and keep I/O paused on the rest.
There could many things while some nodes delay quiesce ack.

The larger the cluster, the more difficult it gets. The more network congestion 
or I/O load, the more difficult it gets. I recommend to open a ticket for 
support to try to identify the root cause of which nodes not acknowledge the 
quiesce  and maybe find the root cause. If I recall some previous thread, 
default timeout was 60 seconds which match your log message. After such 
timeout, snapshot is considered failed to complete.

Support might help you understand the root cause and provide some 
recommendations if it happens frequently.

Best Regards,
--
Jordi Caubet Serrabou
IBM Storage Client Technical Specialist (IBM Spain)

- Original message -
From: "Talamo Ivano Giuseppe (PSI)" 
mailto:ivano.tal...@psi.ch>>
Sent by: 
gpfsug-discuss-boun...@spectrumscale.org
To: "gpfsug main discussion list" 
mailto:gpfsug-discuss@spectrumscale.org>>
Cc:
Subject: [EXTERNAL] Re: [gpfsug-discuss] snapshots causing filesystem quiesce
Date: Wed, Feb 2, 2022 11:45 AM


Hello Andrew,



Thanks for your questions.



We're not experiencing any other issue/slowness during normal activity.

The storage is a Lenovo DSS appliance with a dedicated SSD enclosure/pool for 
metadata only.



The two NSD servers have 750GB of RAM and 618 are configured as pagepool.



The issue we see is happening on both the two filesystems we have:



- perf filesystem:

 - 1.8 PB size (71% in use)

 - 570 milions of inodes (24% in use)



- tiered filesystem:

 - 400 TB size (34% in use)

 - 230 Milions of files (60% in use)



Cheers,

Ivano







__
Paul Scherrer Institut
Ivano Talamo
WHGA/038
Forschungsstrasse 111
5232 Villigen PSI
Schweiz

Telefon: +41 56 310 47 11
E-Mail: ivano.tal...@psi.ch






From: 
gpfsug-discuss-boun...@spectrumscale.org
 

Re: [gpfsug-discuss] snapshots causing filesystem quiesce

[Talamo Ivano Giuseppe (PSI)]

Sure, that makes a lot of sense and we were already doing in that way.


Cheers,

Ivano


__
Paul Scherrer Institut
Ivano Talamo
WHGA/038
Forschungsstrasse 111
5232 Villigen PSI
Schweiz

Telefon: +41 56 310 47 11
E-Mail: ivano.tal...@psi.ch




From: gpfsug-discuss-boun...@spectrumscale.org 
 on behalf of Jan-Frode Myklebust 

Sent: Wednesday, February 2, 2022 12:53 PM
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] snapshots causing filesystem quiesce

Also, if snapshotting multiple filesets, it's important to group these into a 
single mmcrsnapshot command. Then you get a single quiesce, instead of one per 
fileset.

i.e. do:

snapname=$(date --utc +@GMT-%Y.%m.%d-%H.%M.%S)
mmcrsnapshot gpfs0 fileset1:$snapname,filset2:snapname,fileset3:snapname

instead of:

mmcrsnapshot gpfs0 fileset1:$snapname
mmcrsnapshot gpfs0 fileset2:$snapname
mmcrsnapshot gpfs0 fileset3:$snapname


  -jf


On Wed, Feb 2, 2022 at 12:07 PM Jordi Caubet Serrabou 
mailto:jordi.cau...@es.ibm.com>> wrote:
Ivano,

if it happens frequently, I would recommend to open a support case.

The creation or deletion of a snapshot requires a quiesce of the nodes to 
obtain a consistent point-in-time image of the file system and/or update some 
internal structures afaik. Quiesce is required for nodes at the storage cluster 
but also remote clusters. Quiesce means stop activities (incl. I/O) for a short 
period of time to get such consistent image. Also waiting to flush any data 
in-flight to disk that does not allow a consistent point-in-time image.

Nodes receive a quiesce request and acknowledge when ready. When all nodes 
acknowledge, snapshot operation can proceed and immediately I/O can resume. It 
usually takes few seconds at most and the operation performed is short but time 
I/O is stopped depends of how long it takes to quiesce the nodes. If some node 
take longer to agree stop the activities, such node will be delay the 
completion of the quiesce and keep I/O paused on the rest.
There could many things while some nodes delay quiesce ack.

The larger the cluster, the more difficult it gets. The more network congestion 
or I/O load, the more difficult it gets. I recommend to open a ticket for 
support to try to identify the root cause of which nodes not acknowledge the 
quiesce  and maybe find the root cause. If I recall some previous thread, 
default timeout was 60 seconds which match your log message. After such 
timeout, snapshot is considered failed to complete.

Support might help you understand the root cause and provide some 
recommendations if it happens frequently.

Best Regards,
--
Jordi Caubet Serrabou
IBM Storage Client Technical Specialist (IBM Spain)

- Original message -
From: "Talamo Ivano Giuseppe (PSI)" 
mailto:ivano.tal...@psi.ch>>
Sent by: 
gpfsug-discuss-boun...@spectrumscale.org
To: "gpfsug main discussion list" 
mailto:gpfsug-discuss@spectrumscale.org>>
Cc:
Subject: [EXTERNAL] Re: [gpfsug-discuss] snapshots causing filesystem quiesce
Date: Wed, Feb 2, 2022 11:45 AM


Hello Andrew,



Thanks for your questions.



We're not experiencing any other issue/slowness during normal activity.

The storage is a Lenovo DSS appliance with a dedicated SSD enclosure/pool for 
metadata only.



The two NSD servers have 750GB of RAM and 618 are configured as pagepool.



The issue we see is happening on both the two filesystems we have:



- perf filesystem:

 - 1.8 PB size (71% in use)

 - 570 milions of inodes (24% in use)



- tiered filesystem:

 - 400 TB size (34% in use)

 - 230 Milions of files (60% in use)



Cheers,

Ivano







__
Paul Scherrer Institut
Ivano Talamo
WHGA/038
Forschungsstrasse 111
5232 Villigen PSI
Schweiz

Telefon: +41 56 310 47 11
E-Mail: ivano.tal...@psi.ch






From: 
gpfsug-discuss-boun...@spectrumscale.org
 
mailto:gpfsug-discuss-boun...@spectrumscale.org>>
 on behalf of Andrew Beattie mailto:abeat...@au1.ibm.com>>
Sent: Wednesday, February 2, 2022 10:33 AM
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] snapshots causing filesystem quiesce

Ivano,

How big is the filesystem in terms of number of files?
How big is the filesystem in terms of capacity?
Is the Metadata on Flash or Spinning disk?
Do you see issues when users do an LS of the filesystem or only when you are 
doing snapshots.

How much memory do the NSD servers have?
How much is allocated to the OS / Spectrum
 Scale  Pagepool

Regards

Andrew Beattie
Technical Specialist - Storage for Big Data & AI
IBM Technology Group
IBM Australia & New Zealand
P. +61 421 337 927
E. abeat...@au1.ibm.com



On 2 Feb 2022, at 19:14, Talamo Ivano Giuseppe (PSI) 
mailto:ivano.tal...@psi.ch>> wrote:





Dear 

Re: [gpfsug-discuss] snapshots causing filesystem quiesce

[Talamo Ivano Giuseppe (PSI)]

Hi Jordi,


thanks for the explanation, I can now see better why something like that would 
happen. Indeed the cluster has a lot of clients, coming via different clusters 
and even some NFS/SMB via protocol nodes.

So I think opening a case makes a lot of sense to track it down. Not sure how 
we can make the debug transparent to the users, but we'll see.


Cheers,

Ivano


__
Paul Scherrer Institut
Ivano Talamo
WHGA/038
Forschungsstrasse 111
5232 Villigen PSI
Schweiz

Telefon: +41 56 310 47 11
E-Mail: ivano.tal...@psi.ch




From: gpfsug-discuss-boun...@spectrumscale.org 
 on behalf of Jordi Caubet Serrabou 

Sent: Wednesday, February 2, 2022 12:07 PM
To: gpfsug-discuss@spectrumscale.org
Cc: gpfsug-discuss@spectrumscale.org
Subject: Re: [gpfsug-discuss] snapshots causing filesystem quiesce

Ivano,

if it happens frequently, I would recommend to open a support case.

The creation or deletion of a snapshot requires a quiesce of the nodes to 
obtain a consistent point-in-time image of the file system and/or update some 
internal structures afaik. Quiesce is required for nodes at the storage cluster 
but also remote clusters. Quiesce means stop activities (incl. I/O) for a short 
period of time to get such consistent image. Also waiting to flush any data 
in-flight to disk that does not allow a consistent point-in-time image.

Nodes receive a quiesce request and acknowledge when ready. When all nodes 
acknowledge, snapshot operation can proceed and immediately I/O can resume. It 
usually takes few seconds at most and the operation performed is short but time 
I/O is stopped depends of how long it takes to quiesce the nodes. If some node 
take longer to agree stop the activities, such node will be delay the 
completion of the quiesce and keep I/O paused on the rest.
There could many things while some nodes delay quiesce ack.

The larger the cluster, the more difficult it gets. The more network congestion 
or I/O load, the more difficult it gets. I recommend to open a ticket for 
support to try to identify the root cause of which nodes not acknowledge the 
quiesce  and maybe find the root cause. If I recall some previous thread, 
default timeout was 60 seconds which match your log message. After such 
timeout, snapshot is considered failed to complete.

Support might help you understand the root cause and provide some 
recommendations if it happens frequently.

Best Regards,
--
Jordi Caubet Serrabou
IBM Storage Client Technical Specialist (IBM Spain)

- Original message -
From: "Talamo Ivano Giuseppe (PSI)" 
Sent by: gpfsug-discuss-boun...@spectrumscale.org
To: "gpfsug main discussion list" 
Cc:
Subject: [EXTERNAL] Re: [gpfsug-discuss] snapshots causing filesystem quiesce
Date: Wed, Feb 2, 2022 11:45 AM


Hello Andrew,



Thanks for your questions.



We're not experiencing any other issue/slowness during normal activity.

The storage is a Lenovo DSS appliance with a dedicated SSD enclosure/pool for 
metadata only.



The two NSD servers have 750GB of RAM and 618 are configured as pagepool.



The issue we see is happening on both the two filesystems we have:



- perf filesystem:

 - 1.8 PB size (71% in use)

 - 570 milions of inodes (24% in use)



- tiered filesystem:

 - 400 TB size (34% in use)

 - 230 Milions of files (60% in use)



Cheers,

Ivano







__
Paul Scherrer Institut
Ivano Talamo
WHGA/038
Forschungsstrasse 111
5232 Villigen PSI
Schweiz

Telefon: +41 56 310 47 11
E-Mail: ivano.tal...@psi.ch






From: gpfsug-discuss-boun...@spectrumscale.org 
 on behalf of Andrew Beattie 

Sent: Wednesday, February 2, 2022 10:33 AM
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] snapshots causing filesystem quiesce

Ivano,

How big is the filesystem in terms of number of files?
How big is the filesystem in terms of capacity?
Is the Metadata on Flash or Spinning disk?
Do you see issues when users do an LS of the filesystem or only when you are 
doing snapshots.

How much memory do the NSD servers have?
How much is allocated to the OS / Spectrum
 Scale  Pagepool

Regards

Andrew Beattie
Technical Specialist - Storage for Big Data & AI
IBM Technology Group
IBM Australia & New Zealand
P. +61 421 337 927
E. abeat...@au1.ibm.com



On 2 Feb 2022, at 19:14, Talamo Ivano Giuseppe (PSI)  
wrote:





Dear all,

Since a while we are experiencing an issue when dealing with snapshots.
Basically what happens is that when deleting a fileset snapshot (and maybe also 
when creating new ones) the filesystem becomes inaccessible on the clients for 
the duration of the operation (can take a few minutes).

The clients and the storage are on two different clusters, using remote cluster 
mount for the access.

On the log files many lines like the following appear (on both clusters):
Snapshot whole quiesce of SG perf from xbldssio1 on this node lasted 60166 msec

By 

Re: [gpfsug-discuss] snapshots causing filesystem quiesce

[Alec]

Might it be a case of being over built?  In the old days you could really
mess up an Oracle DW by giving it too much RAM... It would spend all day
reading in and out data to the ram that it didn't really need, because it
had the SGA available to load the whole table.

Perhaps the pagepool is so large that the time it takes to clear that much
RAM is the actual time out?

My environment has only a million files but has quite a bit more storage
and has only an 8gb pagepool.  Seems you are saying you have 618gb of RAM
for pagepool...  Even at 8GB/second that would take 77 seconds to flush it
out..

Perhaps drop the pagepool in half and see if your timeout adjusts
accordingly?

Alec


On Wed, Feb 2, 2022, 4:09 AM Olaf Weiser  wrote:

> keep in mind... creating many snapshots... means ;-) .. you'll have to
> delete many snapshots..
> at a certain level, which depends on #files, #directories, ~workload,
> #nodes, #networks etc we ve seen cases, where generating just full
> snapshots (whole file system)  is the better approach instead of
> maintaining snapshots for each file set individually ..
>
> sure. this has other side effects , like space consumption etc...
> so as always.. it depends..
>
>
>
>
> - Ursprüngliche Nachricht -
> Von: "Jan-Frode Myklebust" 
> Gesendet von: gpfsug-discuss-boun...@spectrumscale.org
> An: "gpfsug main discussion list" 
> CC:
> Betreff: [EXTERNAL] Re: [gpfsug-discuss] snapshots causing filesystem
> quiesce
> Datum: Mi, 2. Feb 2022 12:54
>
> Also, if snapshotting multiple filesets, it's important to group these
> into a single mmcrsnapshot command. Then you get a single quiesce,
> instead of one per fileset.
>
> i.e. do:
>
> snapname=$(date --utc +@GMT-%Y.%m.%d-%H.%M.%S)
> mmcrsnapshot gpfs0
> fileset1:$snapname,filset2:snapname,fileset3:snapname
>
> instead of:
>
> mmcrsnapshot gpfs0 fileset1:$snapname
> mmcrsnapshot gpfs0 fileset2:$snapname
> mmcrsnapshot gpfs0 fileset3:$snapname
>
>
>   -jf
>
>
> On Wed, Feb 2, 2022 at 12:07 PM Jordi Caubet Serrabou <
> jordi.cau...@es.ibm.com> wrote:
>
> Ivano,
>
> if it happens frequently, I would recommend to open a support case.
>
> The creation or deletion of a snapshot requires a quiesce of the nodes to
> obtain a consistent point-in-time image of the file system and/or update
> some internal structures afaik. Quiesce is required for nodes at the
> storage cluster but also remote clusters. Quiesce means stop activities
> (incl. I/O) for a short period of time to get such consistent image. Also
> waiting to flush any data in-flight to disk that does not allow a
> consistent point-in-time image.
>
> Nodes receive a quiesce request and acknowledge when ready. When all nodes
> acknowledge, snapshot operation can proceed and immediately I/O can resume.
> It usually takes few seconds at most and the operation performed is short
> but time I/O is stopped depends of how long it takes to quiesce the nodes.
> If some node take longer to agree stop the activities, such node will
> be delay the completion of the quiesce and keep I/O paused on the rest.
> There could many things while some nodes delay quiesce ack.
>
> The larger the cluster, the more difficult it gets. The more network
> congestion or I/O load, the more difficult it gets. I recommend to open a
> ticket for support to try to identify the root cause of which nodes not
> acknowledge the quiesce  and maybe find the root cause. If I recall some
> previous thread, default timeout was 60 seconds which match your log
> message. After such timeout, snapshot is considered failed to complete.
>
> Support might help you understand the root cause and provide some
> recommendations if it happens frequently.
>
> Best Regards,
> --
> Jordi Caubet Serrabou
> IBM Storage Client Technical Specialist (IBM Spain)
>
>
> - Original message -
> From: "Talamo Ivano Giuseppe (PSI)" 
> Sent by: gpfsug-discuss-boun...@spectrumscale.org
> To: "gpfsug main discussion list" 
> Cc:
> Subject: [EXTERNAL] Re: [gpfsug-discuss] snapshots causing filesystem
> quiesce
> Date: Wed, Feb 2, 2022 11:45 AM
>
>
> Hello Andrew,
>
>
>
> Thanks for your questions.
>
>
>
> We're not experiencing any other issue/slowness during normal activity.
>
> The storage is a Lenovo DSS appliance with a dedicated SSD enclosure/pool
> for metadata only.
>
>
>
> The two NSD servers have 750GB of RAM and 618 are configured as pagepool.
>
>
>
> The issue we see is happening on both the two filesystems we have:
>
>
>
> - perf filesystem:
>
>  - 1.8 PB size (71% in use)
>
>  - 570 milions of inodes (24% in use)
>
>
>
> - tiered filesystem:
>
>  - 400 TB size (34% in use)
>
>  - 230 Milions of files (60% in use)
>
>
>
> Cheers,
>
> Ivano
>
>
>
>
>
>
> __
> Paul Scherrer Institut
> Ivano Talamo
> WHGA/038
> Forschungsstrasse 111
> 5232 Villigen PSI
> Schweiz
>
> Telefon: +41 56 310 47 11
> E-Mail: ivano.tal...@psi.ch
>
>
>
>
> --
> *From:* 

Re: [gpfsug-discuss] Automating Snapshots : cron jobs or use the GUI ?

[Kidger, Daniel]

Simon,

Thanks - that is a good insight.

The HA 'feature' of the snapshot automation is perhaps a key feature as Linux 
still lacks a decent 'cluster cron'
Also, If "HA" do we know where the state is centrally kept?

On the point of snapshots being left undeleted, do you ever use 
/usr/lpp/mmfs/gui/cli/lssnapops to see what the queue of outstanding actions is 
like?
(There is also a notification tool:  lssnapnotify in that directory that is 
supposed to alert on failed snapshot actions, although personally I have never 
used it)


Daniel Kidger

HPC Storage Solutions Architect, EMEA
daniel.kid...@hpe.com

+44 (0)7818 522266

hpe.com


[cid:fce0ce85-6ae4-44ce-aa94-d7d099e68acb]


From: gpfsug-discuss-boun...@spectrumscale.org 
 on behalf of Simon Thompson2 

Sent: 02 February 2022 10:52
To: gpfsug main discussion list 
Subject: Re: [gpfsug-discuss] [External] Automating Snapshots : cron jobs or 
use the GUI ?


I always used the GUI for automating snapshots that were tagged with the YYMMDD 
format so that they were accessible via the previous versions tab from CES 
access.



This requires no locking if you have multiple GUI servers running, so in theory 
the snapshots creation is “HA”. BUT if you shutdown the GUI servers (say you 
are waiting for a log4j patch …) then you have no snapshot automation.



Due to the way we structured independent filesets, this could be 50 or so to 
automate and we wanted to set a say 4 day retention policy. So clicking in the 
GUI was pretty simple to do this for.



What we did found is it a snapshot failed to delete for some reason (quiesce 
etc), then the GUI never tried again to clean it up so we have monitoring to 
look for unexpected snapshots that needed cleaning up.



Simon







Simon Thompson
He/Him/His
Senior Storage Performance
WW HPC Customer Solutions
Lenovo UK

[Phone]+44 7788 320635
[Email]sthomps...@lenovo.com



Lenovo.com
Twitter | Instagram | 
Facebook | 
Linkedin | 
YouTube | 
Privacy

[cid:image003.png@01D81822.F63BAB90]





From: gpfsug-discuss-boun...@spectrumscale.org 
 On Behalf Of Kidger, Daniel
Sent: 02 February 2022 10:07
To: gpfsug-discuss@spectrumscale.org
Subject: [External] [gpfsug-discuss] Automating Snapshots : cron jobs or use 
the GUI ?



Hi all,



Since the subject of snapshots has come up, I also have a question ...



Snapshots can be created from the command line with mmcrsnapshot, and hence can 
be automated via con jobs etc.

Snapshots can also be created from the Scale GUI. The GUI also provides its own 
automation for the creation, retention, and deletion of snapshots.



My question is: do most customers use the former or the latter for automation?





(I also note that /usr/lpp/mmfs/gui/cli/mksnaprule exists and appears to do 
exactly the same as what the GUI does it terms of creating automated snapshots. 
It is a relic of V7000 Unified but still works fine in Spectrum Scale 5.1.2.2. 
How many customers also use the commands found in /usr/lpp/mmfs/gui/cli/  ? )



Daniel



Daniel Kidger
HPC Storage Solutions Architect, EMEA
daniel.kid...@hpe.com

+44 (0)7818 522266

hpe.com


[cid:image004.png@01D81822.F63BAB90]


Outlook-axuecxph
Description: Outlook-axuecxph
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] snapshots causing filesystem quiesce

[Olaf Weiser]

keep in mind... creating many snapshots... means ;-) .. you'll have to delete many snapshots..
at a certain level, which depends on #files, #directories, ~workload, #nodes, #networks etc we ve seen cases, where generating just full snapshots (whole file system)  is the better approach instead of maintaining snapshots for each file set individually ..
 
sure. this has other side effects , like space consumption etc...
so as always.. it depends..
 
 
 
- Ursprüngliche Nachricht -Von: "Jan-Frode Myklebust" Gesendet von: gpfsug-discuss-boun...@spectrumscale.orgAn: "gpfsug main discussion list" CC:Betreff: [EXTERNAL] Re: [gpfsug-discuss] snapshots causing filesystem quiesceDatum: Mi, 2. Feb 2022 12:54 
Also, if snapshotting multiple filesets, it's important to group these into a single mmcrsnapshot command. Then you get a single quiesce, instead of one per fileset.
 
i.e. do:
 
    snapname=$(date --utc +@GMT-%Y.%m.%d-%H.%M.%S)
    mmcrsnapshot gpfs0 fileset1:$snapname,filset2:snapname,fileset3:snapname
 
instead of:
 
    mmcrsnapshot gpfs0 fileset1:$snapname
    mmcrsnapshot gpfs0 fileset2:$snapname
    mmcrsnapshot gpfs0 fileset3:$snapname   
 
 
  -jf
  

On Wed, Feb 2, 2022 at 12:07 PM Jordi Caubet Serrabou  wrote:
Ivano,
 
if it happens frequently, I would recommend to open a support case.
 
The creation or deletion of a snapshot requires a quiesce of the nodes to obtain a consistent point-in-time image of the file system and/or update some internal structures afaik. Quiesce is required for nodes at the storage cluster but also remote clusters. Quiesce means stop activities (incl. I/O) for a short period of time to get such consistent image. Also waiting to flush any data in-flight to disk that does not allow a consistent point-in-time image.
 
Nodes receive a quiesce request and acknowledge when ready. When all nodes acknowledge, snapshot operation can proceed and immediately I/O can resume. It usually takes few seconds at most and the operation performed is short but time I/O is stopped depends of how long it takes to quiesce the nodes. If some node take longer to agree stop the activities, such node will be delay the completion of the quiesce and keep I/O paused on the rest.
There could many things while some nodes delay quiesce ack.
 
The larger the cluster, the more difficult it gets. The more network congestion or I/O load, the more difficult it gets. I recommend to open a ticket for support to try to identify the root cause of which nodes not acknowledge the quiesce  and maybe find the root cause. If I recall some previous thread, default timeout was 60 seconds which match your log message. After such timeout, snapshot is considered failed to complete.
 
Support might help you understand the root cause and provide some recommendations if it happens frequently.
 
Best Regards,--Jordi Caubet SerrabouIBM Storage Client Technical Specialist (IBM Spain)
 
- Original message -From: "Talamo Ivano Giuseppe (PSI)" Sent by: gpfsug-discuss-boun...@spectrumscale.orgTo: "gpfsug main discussion list" Cc:Subject: [EXTERNAL] Re: [gpfsug-discuss] snapshots causing filesystem quiesceDate: Wed, Feb 2, 2022 11:45 AM 
Hello Andrew,
 
Thanks for your questions.
 
We're not experiencing any other issue/slowness during normal activity.
The storage is a Lenovo DSS appliance with a dedicated SSD enclosure/pool for metadata only.
 
The two NSD servers have 750GB of RAM and 618 are configured as pagepool.
 
The issue we see is happening on both the two filesystems we have:
 
- perf filesystem:
 - 1.8 PB size (71% in use)
 - 570 milions of inodes (24% in use)
 
- tiered filesystem:
 - 400 TB size (34% in use)
 - 230 Milions of files (60% in use)
 
Cheers,
Ivano
 
 
 
__
Paul Scherrer Institut
Ivano Talamo
WHGA/038
Forschungsstrasse 111
5232 Villigen PSI
Schweiz
 
Telefon: +41 56 310 47 11
E-Mail: ivano.tal...@psi.ch 

  

From: gpfsug-discuss-boun...@spectrumscale.org  on behalf of Andrew Beattie Sent: Wednesday, February 2, 2022 10:33 AMTo: gpfsug main discussion listSubject: Re: [gpfsug-discuss] snapshots causing filesystem quiesce
 
Ivano,
 
How big is the filesystem in terms of number of files?
How big is the filesystem in terms of capacity? 
Is the Metadata on Flash or Spinning disk? 
Do you see issues when users do an LS of the filesystem or only when you are doing snapshots.
 
How much memory do the NSD servers have?
How much is allocated to the OS / Spectrum
 Scale  Pagepool 
Regards
 
Andrew Beattie
Technical Specialist - Storage for Big Data & AI
IBM Technology Group
IBM Australia & New Zealand
P. +61 421 337 927
E. abeat...@au1.ibm.com
 
 
 
On 2 Feb 2022, at 19:14, Talamo Ivano Giuseppe (PSI)  wrote: 

 
Dear all,
 
Since a while we are experiencing an issue when dealing with snapshots.

Re: [gpfsug-discuss] snapshots causing filesystem quiesce

[Jan-Frode Myklebust]

Also, if snapshotting multiple filesets, it's important to group these into
a single mmcrsnapshot command. Then you get a single quiesce, instead of
one per fileset.

i.e. do:

snapname=$(date --utc +@GMT-%Y.%m.%d-%H.%M.%S)
mmcrsnapshot gpfs0 fileset1:$snapname,filset2:snapname,fileset3:snapname

instead of:

mmcrsnapshot gpfs0 fileset1:$snapname
mmcrsnapshot gpfs0 fileset2:$snapname
mmcrsnapshot gpfs0 fileset3:$snapname


  -jf


On Wed, Feb 2, 2022 at 12:07 PM Jordi Caubet Serrabou <
jordi.cau...@es.ibm.com> wrote:

> Ivano,
>
> if it happens frequently, I would recommend to open a support case.
>
> The creation or deletion of a snapshot requires a quiesce of the nodes to
> obtain a consistent point-in-time image of the file system and/or update
> some internal structures afaik. Quiesce is required for nodes at the
> storage cluster but also remote clusters. Quiesce means stop activities
> (incl. I/O) for a short period of time to get such consistent image. Also
> waiting to flush any data in-flight to disk that does not allow a
> consistent point-in-time image.
>
> Nodes receive a quiesce request and acknowledge when ready. When all nodes
> acknowledge, snapshot operation can proceed and immediately I/O can resume.
> It usually takes few seconds at most and the operation performed is short
> but time I/O is stopped depends of how long it takes to quiesce the nodes.
> If some node take longer to agree stop the activities, such node will
> be delay the completion of the quiesce and keep I/O paused on the rest.
> There could many things while some nodes delay quiesce ack.
>
> The larger the cluster, the more difficult it gets. The more network
> congestion or I/O load, the more difficult it gets. I recommend to open a
> ticket for support to try to identify the root cause of which nodes not
> acknowledge the quiesce  and maybe find the root cause. If I recall some
> previous thread, default timeout was 60 seconds which match your log
> message. After such timeout, snapshot is considered failed to complete.
>
> Support might help you understand the root cause and provide some
> recommendations if it happens frequently.
>
> Best Regards,
> --
> Jordi Caubet Serrabou
> IBM Storage Client Technical Specialist (IBM Spain)
>
>
> - Original message -
> From: "Talamo Ivano Giuseppe (PSI)" 
> Sent by: gpfsug-discuss-boun...@spectrumscale.org
> To: "gpfsug main discussion list" 
> Cc:
> Subject: [EXTERNAL] Re: [gpfsug-discuss] snapshots causing filesystem
> quiesce
> Date: Wed, Feb 2, 2022 11:45 AM
>
>
> Hello Andrew,
>
>
>
> Thanks for your questions.
>
>
>
> We're not experiencing any other issue/slowness during normal activity.
>
> The storage is a Lenovo DSS appliance with a dedicated SSD enclosure/pool
> for metadata only.
>
>
>
> The two NSD servers have 750GB of RAM and 618 are configured as pagepool.
>
>
>
> The issue we see is happening on both the two filesystems we have:
>
>
>
> - perf filesystem:
>
>  - 1.8 PB size (71% in use)
>
>  - 570 milions of inodes (24% in use)
>
>
>
> - tiered filesystem:
>
>  - 400 TB size (34% in use)
>
>  - 230 Milions of files (60% in use)
>
>
>
> Cheers,
>
> Ivano
>
>
>
>
>
>
> __
> Paul Scherrer Institut
> Ivano Talamo
> WHGA/038
> Forschungsstrasse 111
> 5232 Villigen PSI
> Schweiz
>
> Telefon: +41 56 310 47 11
> E-Mail: ivano.tal...@psi.ch
>
>
>
>
> --
> *From:* gpfsug-discuss-boun...@spectrumscale.org <
> gpfsug-discuss-boun...@spectrumscale.org> on behalf of Andrew Beattie <
> abeat...@au1.ibm.com>
> *Sent:* Wednesday, February 2, 2022 10:33 AM
> *To:* gpfsug main discussion list
> *Subject:* Re: [gpfsug-discuss] snapshots causing filesystem quiesce
>
> Ivano,
>
> How big is the filesystem in terms of number of files?
> How big is the filesystem in terms of capacity?
> Is the Metadata on Flash or Spinning disk?
> Do you see issues when users do an LS of the filesystem or only when you
> are doing snapshots.
>
> How much memory do the NSD servers have?
> How much is allocated to the OS / Spectrum
>  Scale  Pagepool
>
> Regards
>
> Andrew Beattie
> Technical Specialist - Storage for Big Data & AI
> IBM Technology Group
> IBM Australia & New Zealand
> P. +61 421 337 927
> E. abeat...@au1.ibm.com
>
>
>
>
> On 2 Feb 2022, at 19:14, Talamo Ivano Giuseppe (PSI) 
> wrote:
>
>
> 
>
>
> Dear all,
>
> Since a while we are experiencing an issue when dealing with snapshots.
> Basically what happens is that when deleting a fileset snapshot (and maybe
> also when creating new ones) the filesystem becomes inaccessible on the
> clients for the duration of the operation (can take a few minutes).
>
> The clients and the storage are on two different clusters, using remote
> cluster mount for the access.
>
> On the log files many lines like the following appear (on both clusters):
> Snapshot whole quiesce of SG perf from xbldssio1 on this node lasted 60166
> msec
>
> By looking around I see 

Re: [gpfsug-discuss] snapshots causing filesystem quiesce

[Jordi Caubet Serrabou]

Ivano,
 
if it happens frequently, I would recommend to open a support case.
 
The creation or deletion of a snapshot requires a quiesce of the nodes to obtain a consistent point-in-time image of the file system and/or update some internal structures afaik. Quiesce is required for nodes at the storage cluster but also remote clusters. Quiesce means stop activities (incl. I/O) for a short period of time to get such consistent image. Also waiting to flush any data in-flight to disk that does not allow a consistent point-in-time image.
 
Nodes receive a quiesce request and acknowledge when ready. When all nodes acknowledge, snapshot operation can proceed and immediately I/O can resume. It usually takes few seconds at most and the operation performed is short but time I/O is stopped depends of how long it takes to quiesce the nodes. If some node take longer to agree stop the activities, such node will be delay the completion of the quiesce and keep I/O paused on the rest.
There could many things while some nodes delay quiesce ack.
 
The larger the cluster, the more difficult it gets. The more network congestion or I/O load, the more difficult it gets. I recommend to open a ticket for support to try to identify the root cause of which nodes not acknowledge the quiesce  and maybe find the root cause. If I recall some previous thread, default timeout was 60 seconds which match your log message. After such timeout, snapshot is considered failed to complete.
 
Support might help you understand the root cause and provide some recommendations if it happens frequently.
 
Best Regards,--Jordi Caubet SerrabouIBM Storage Client Technical Specialist (IBM Spain)
 
- Original message -From: "Talamo Ivano Giuseppe (PSI)" Sent by: gpfsug-discuss-boun...@spectrumscale.orgTo: "gpfsug main discussion list" Cc:Subject: [EXTERNAL] Re: [gpfsug-discuss] snapshots causing filesystem quiesceDate: Wed, Feb 2, 2022 11:45 AM 
Hello Andrew,
 
Thanks for your questions.
 
We're not experiencing any other issue/slowness during normal activity.
The storage is a Lenovo DSS appliance with a dedicated SSD enclosure/pool for metadata only.
 
The two NSD servers have 750GB of RAM and 618 are configured as pagepool.
 
The issue we see is happening on both the two filesystems we have:
 
- perf filesystem:
 - 1.8 PB size (71% in use)
 - 570 milions of inodes (24% in use)
 
- tiered filesystem:
 - 400 TB size (34% in use)
 - 230 Milions of files (60% in use)
 
Cheers,
Ivano
 
 
 
__
Paul Scherrer Institut
Ivano Talamo
WHGA/038
Forschungsstrasse 111
5232 Villigen PSI
Schweiz
 
Telefon: +41 56 310 47 11
E-Mail: ivano.tal...@psi.ch 

  

From: gpfsug-discuss-boun...@spectrumscale.org  on behalf of Andrew Beattie Sent: Wednesday, February 2, 2022 10:33 AMTo: gpfsug main discussion listSubject: Re: [gpfsug-discuss] snapshots causing filesystem quiesce
 
Ivano,
 
How big is the filesystem in terms of number of files?
How big is the filesystem in terms of capacity? 
Is the Metadata on Flash or Spinning disk? 
Do you see issues when users do an LS of the filesystem or only when you are doing snapshots.
 
How much memory do the NSD servers have?
How much is allocated to the OS / Spectrum
 Scale  Pagepool 
Regards
 
Andrew Beattie
Technical Specialist - Storage for Big Data & AI
IBM Technology Group
IBM Australia & New Zealand
P. +61 421 337 927
E. abeat...@au1.ibm.com
 
 
 
On 2 Feb 2022, at 19:14, Talamo Ivano Giuseppe (PSI)  wrote: 

 
Dear all,
 
Since a while we are experiencing an issue when dealing with snapshots.
Basically what happens is that when deleting a fileset snapshot (and maybe also when creating new ones) the filesystem becomes inaccessible on the clients for the duration of the operation (can take a few minutes).
 
The clients and the storage are on two different clusters, using remote cluster mount for the access.
 
On the log files many lines like the following appear (on both clusters):
Snapshot whole quiesce of SG perf from xbldssio1 on this node lasted 60166 msec
 
By looking around I see we're not the first one. I am wondering if that's considered an unavoidable part of the snapshotting and if there's any tunable that can improve the situation. Since when this occurs all the clients are stuck and users are very quick to complain.
 
If it can help, the clients are running GPFS 5.1.2-1 while the storage cluster is on 5.1.1-0.
 
Thanks,
Ivano
 
  
___gpfsug-discuss mailing listgpfsug-discuss at spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss 
 


Salvo indicado de otro modo más arriba / Unless stated otherwise above:

International Business Machines, S.A.

Santa Hortensia, 26-28, 28002 Madrid

Registro Mercantil de Madrid; Folio 1; Tomo 1525; Hoja M-28146

CIF A28-010791


___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org

Re: [gpfsug-discuss] [External] Automating Snapshots : cron jobs or use the GUI ?

[Simon Thompson2]

I always used the GUI for automating snapshots that were tagged with the YYMMDD 
format so that they were accessible via the previous versions tab from CES 
access.

This requires no locking if you have multiple GUI servers running, so in theory 
the snapshots creation is "HA". BUT if you shutdown the GUI servers (say you 
are waiting for a log4j patch ...) then you have no snapshot automation.

Due to the way we structured independent filesets, this could be 50 or so to 
automate and we wanted to set a say 4 day retention policy. So clicking in the 
GUI was pretty simple to do this for.

What we did found is it a snapshot failed to delete for some reason (quiesce 
etc), then the GUI never tried again to clean it up so we have monitoring to 
look for unexpected snapshots that needed cleaning up.

Simon



Simon Thompson
He/Him/His
Senior Storage Performance
WW HPC Customer Solutions
Lenovo UK
[Phone]+44 7788 320635
[Email]sthomps...@lenovo.com

Lenovo.com
Twitter | Instagram | 
Facebook | 
Linkedin | 
YouTube | 
Privacy
[cid:image003.png@01D81822.F63BAB90]


From: gpfsug-discuss-boun...@spectrumscale.org 
 On Behalf Of Kidger, Daniel
Sent: 02 February 2022 10:07
To: gpfsug-discuss@spectrumscale.org
Subject: [External] [gpfsug-discuss] Automating Snapshots : cron jobs or use 
the GUI ?

Hi all,

Since the subject of snapshots has come up, I also have a question ...

Snapshots can be created from the command line with mmcrsnapshot, and hence can 
be automated via con jobs etc.
Snapshots can also be created from the Scale GUI. The GUI also provides its own 
automation for the creation, retention, and deletion of snapshots.

My question is: do most customers use the former or the latter for automation?


(I also note that /usr/lpp/mmfs/gui/cli/mksnaprule exists and appears to do 
exactly the same as what the GUI does it terms of creating automated snapshots. 
It is a relic of V7000 Unified but still works fine in Spectrum Scale 5.1.2.2. 
How many customers also use the commands found in /usr/lpp/mmfs/gui/cli/  ? )

Daniel


Daniel Kidger
HPC Storage Solutions Architect, EMEA
daniel.kid...@hpe.com

+44 (0)7818 522266

hpe.com


[cid:image004.png@01D81822.F63BAB90]
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] snapshots causing filesystem quiesce

[Talamo Ivano Giuseppe (PSI)]

Hello Andrew,


Thanks for your questions.


We're not experiencing any other issue/slowness during normal activity.

The storage is a Lenovo DSS appliance with a dedicated SSD enclosure/pool for 
metadata only.


The two NSD servers have 750GB of RAM and 618 are configured as pagepool.


The issue we see is happening on both the two filesystems we have:


- perf filesystem:

 - 1.8 PB size (71% in use)

 - 570 milions of inodes (24% in use)


- tiered filesystem:

 - 400 TB size (34% in use)

 - 230 Milions of files (60% in use)


Cheers,

Ivano


__
Paul Scherrer Institut
Ivano Talamo
WHGA/038
Forschungsstrasse 111
5232 Villigen PSI
Schweiz

Telefon: +41 56 310 47 11
E-Mail: ivano.tal...@psi.ch




From: gpfsug-discuss-boun...@spectrumscale.org 
 on behalf of Andrew Beattie 

Sent: Wednesday, February 2, 2022 10:33 AM
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] snapshots causing filesystem quiesce

Ivano,

How big is the filesystem in terms of number of files?
How big is the filesystem in terms of capacity?
Is the Metadata on Flash or Spinning disk?
Do you see issues when users do an LS of the filesystem or only when you are 
doing snapshots.

How much memory do the NSD servers have?
How much is allocated to the OS / Spectrum
 Scale  Pagepool

Regards

Andrew Beattie
Technical Specialist - Storage for Big Data & AI
IBM Technology Group
IBM Australia & New Zealand
P. +61 421 337 927
E. abeat...@au1.ibm.com



On 2 Feb 2022, at 19:14, Talamo Ivano Giuseppe (PSI)  
wrote:



Dear all,

Since a while we are experiencing an issue when dealing with snapshots.
Basically what happens is that when deleting a fileset snapshot (and maybe also 
when creating new ones) the filesystem becomes inaccessible on the clients for 
the duration of the operation (can take a few minutes).

The clients and the storage are on two different clusters, using remote cluster 
mount for the access.

On the log files many lines like the following appear (on both clusters):
Snapshot whole quiesce of SG perf from xbldssio1 on this node lasted 60166 msec

By looking around I see we're not the first one. I am wondering if that's 
considered an unavoidable part of the snapshotting and if there's any tunable 
that can improve the situation. Since when this occurs all the clients are 
stuck and users are very quick to complain.

If it can help, the clients are running GPFS 5.1.2-1 while the storage cluster 
is on 5.1.1-0.

Thanks,
Ivano


___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

[gpfsug-discuss] Automating Snapshots : cron jobs or use the GUI ?

[Kidger, Daniel]

Hi all,

Since the subject of snapshots has come up, I also have a question ...

Snapshots can be created from the command line with mmcrsnapshot, and hence can 
be automated via con jobs etc.
Snapshots can also be created from the Scale GUI. The GUI also provides its own 
automation for the creation, retention, and deletion of snapshots.

My question is: do most customers use the former or the latter for automation?


(I also note that /usr/lpp/mmfs/gui/cli/mksnaprule exists and appears to do 
exactly the same as what the GUI does it terms of creating automated snapshots. 
It is a relic of V7000 Unified but still works fine in Spectrum Scale 5.1.2.2. 
How many customers also use the commands found in /usr/lpp/mmfs/gui/cli/  ? )

Daniel


Daniel Kidger
HPC Storage Solutions Architect, EMEA
daniel.kid...@hpe.com

+44 (0)7818 522266

hpe.com


[cid:548be828-dcc2-4a88-ac2e-ff5106b3f802]


Outlook-iity4nk4
Description: Outlook-iity4nk4
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] snapshots causing filesystem quiesce

[Andrew Beattie]

Ivano,

How big is the filesystem in terms of number of files?
How big is the filesystem in terms of capacity? 
Is the Metadata on Flash or Spinning disk? 
Do you see issues when users do an LS of the filesystem or only when you are 
doing snapshots.

How much memory do the NSD servers have?
How much is allocated to the OS / Spectrum
 Scale  Pagepool

Regards

Andrew Beattie
Technical Specialist - Storage for Big Data & AI
IBM Technology Group
IBM Australia & New Zealand
P. +61 421 337 927
E. abeat...@au1.ibm.com



> On 2 Feb 2022, at 19:14, Talamo Ivano Giuseppe (PSI)  
> wrote:
> 
> 
> This Message Is From an External Sender
> This message came from outside your organization.
> Dear all,
> 
> Since a while we are experiencing an issue when dealing with snapshots.
> Basically what happens is that when deleting a fileset snapshot (and maybe 
> also when creating new ones) the filesystem becomes inaccessible on the 
> clients for the duration of the operation (can take a few minutes).
> 
> The clients and the storage are on two different clusters, using remote 
> cluster mount for the access.
> 
> On the log files many lines like the following appear (on both clusters):
> Snapshot whole quiesce of SG perf from xbldssio1 on this node lasted 60166 
> msec
> 
> By looking around I see we're not the first one. I am wondering if that's 
> considered an unavoidable part of the snapshotting and if there's any tunable 
> that can improve the situation. Since when this occurs all the clients are 
> stuck and users are very quick to complain.
> 
> If it can help, the clients are running GPFS 5.1.2-1 while the storage 
> cluster is on 5.1.1-0.
> 
> Thanks,
> Ivano

___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

[gpfsug-discuss] snapshots causing filesystem quiesce

[Talamo Ivano Giuseppe (PSI)]

Dear all,

Since a while we are experiencing an issue when dealing with snapshots.
Basically what happens is that when deleting a fileset snapshot (and maybe also 
when creating new ones) the filesystem becomes inaccessible on the clients for 
the duration of the operation (can take a few minutes).

The clients and the storage are on two different clusters, using remote cluster 
mount for the access.

On the log files many lines like the following appear (on both clusters):
Snapshot whole quiesce of SG perf from xbldssio1 on this node lasted 60166 msec

By looking around I see we're not the first one. I am wondering if that's 
considered an unavoidable part of the snapshotting and if there's any tunable 
that can improve the situation. Since when this occurs all the clients are 
stuck and users are very quick to complain.

If it can help, the clients are running GPFS 5.1.2-1 while the storage cluster 
is on 5.1.1-0.

Thanks,
Ivano
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss