Re: [gpfsug-discuss] Quota and hardlimit enforcement

2017-07-31 Thread Buterbaugh, Kevin L 
Jaime,

That’s heavily workload dependent.  We run a traditional HPC cluster and have a 
7 day grace on home and 14 days on scratch.  By setting the soft and hard 
limits appropriately we’ve slammed the door on many a runaway user / group / 
fileset.  YMMV…

Kevin

On Jul 31, 2017, at 3:03 PM, Jaime Pinto 
> wrote:

If your grace period is set to 7 days I can assure you that in an HPC 
environment it's the equivalent of not having quotas effectively. You should 
set it to 2 hours or 4 hours.

In an environment such as ours a runway process can easily generate 500TB of 
data or 1 billion inodes in few hours, and choke the file system to all 
users/jobs.

Jaime


—
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
kevin.buterba...@vanderbilt.edu - 
(615)875-9633



___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] Quota and hardlimit enforcement

2017-07-31 Thread Jaime Pinto 
In addition, the in_doubt column is a function of the data turn-over  
and the internal gpfs accounting synchronization period (beyond root  
control). The higher the in_doubt values the less accurate the real  
amount of space/inodes a user/group/fileset has in the filesystem.


What I noticed in practice is the the in_doubt values only get worst  
overtime, and work against the quotas, making them hit the limits  
sooner. Therefore, you may wish to run a 'mmcheckquota' crontab job  
once or twice a day, to reset the in_doubt column to zero mover often.  
GPFS has a very high lag to do this on its own in the most recent  
versions, and seldom really catches up on a very active filesystem.


If your grace period is set to 7 days I can assure you that in an HPC  
environment it's the equivalent of not having quotas effectively. You  
should set it to 2 hours or 4 hours.


In an environment such as ours a runway process can easily generate  
500TB of data or 1 billion inodes in few hours, and choke the file  
system to all users/jobs.


Jaime



Quoting "Buterbaugh, Kevin L" <kevin.buterba...@vanderbilt.edu>:


Hi Renar,

I?m sure this is the case, but I don?t see anywhere in this thread   
where this is explicitly stated ? you?re not doing your tests as   
root, are you?  root, of course, is not bound by any quotas.


Kevin

On Jul 31, 2017, at 2:04 PM, Grunenberg, Renar   
<renar.grunenb...@huk-coburg.de<mailto:renar.grunenb...@huk-coburg.de>>   
wrote:



Hallo J. Eric, hallo Jaime,
Ok after we hit the softlimit we see that the graceperiod are go to   
7 days. I think that?s the default. But was does it mean.

After we reach the ?hard?-limit. we see additionaly the gbytes  in_doubt.
My interpretation now we can write many gb to the nospace-left event  
 in the filesystem.
But our intention is to restricted some application to write only to  
 the hardlimit in the fileset. Any hints to accomplish this?




Renar Grunenberg
Abteilung Informatik ? Betrieb

HUK-COBURG
Bahnhofsplatz
96444 Coburg
Telefon:09561 96-44110
Telefax:09561 96-44104
E-Mail: renar.grunenb...@huk-coburg.de<mailto:renar.grunenb...@huk-coburg.de>
Internet:   www.huk.de<http://www.huk.de/>


HUK-COBURG Haftpflicht-Unterstützungs-Kasse kraftfahrender Beamter  
Deutschlands a. G. in Coburg

Reg.-Gericht Coburg HRB 100; St.-Nr. 9212/101/00021
Sitz der Gesellschaft: Bahnhofsplatz, 96444 Coburg
Vorsitzender des Aufsichtsrats: Prof. Dr. Heinrich R. Schradin.
Vorstand: Klaus-Jürgen Heitmann (Sprecher), Stefan Gronbach, Dr.  
Hans Olav Herøy, Dr. Jörg Rheinländer (stv.), Sarah Rössler, Daniel  
Thomas (stv.).


Diese Nachricht enthält vertrauliche und/oder rechtlich geschützte  
Informationen.
Wenn Sie nicht der richtige Adressat sind oder diese Nachricht  
irrtümlich erhalten haben,

informieren Sie bitte sofort den Absender und vernichten Sie diese Nachricht.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser   
Nachricht ist nicht gestattet.


This information may contain confidential and/or privileged information.
If you are not the intended recipient (or have received this   
information in error) please notify the

sender immediately and destroy this information.
Any unauthorized copying, disclosure or distribution of the material  
 in this information is strictly forbidden.





Von:   
gpfsug-discuss-boun...@spectrumscale.org<mailto:gpfsug-discuss-boun...@spectrumscale.org> [mailto:gpfsug-discuss-boun...@spectrumscale.org] Im Auftrag von J. Eric   
Wonderley

Gesendet: Montag, 31. Juli 2017 19:55
An: gpfsug main discussion list   
<gpfsug-discuss@spectrumscale.org<mailto:gpfsug-discuss@spectrumscale.org>>

Betreff: Re: [gpfsug-discuss] Quota and hardlimit enforcement

Hi Renar:
What does 'mmlsquota -j fileset filesystem' report?
I did not think you would get a grace period of none unless the   
hardlimit=softlimit.


On Mon, Jul 31, 2017 at 1:44 PM, Grunenberg, Renar   
<renar.grunenb...@huk-coburg.de<mailto:renar.grunenb...@huk-coburg.de>>   
wrote:

Hallo All,
we are on Version 4.2.3.2 and see some missunderstandig in the   
enforcement of hardlimit definitions on a flieset quota. What we see  
 is we put some 200 GB files on following quota definitions:  quota   
150 GB Limit 250 GB Grace none.
After the creating of one 200 GB we hit the softquota limit, thats   
ok. But After the the second file was created!! we expect an io   
error but it don?t happen. We define all well know Parameters   
(-Q,..) on the filesystem . Is this a bug or a Feature? mmcheckquota  
 are already running at first.

Regards Renar.


Renar Grunenberg
Abteilung Informatik ? Betrieb

HUK-COBURG
Bahnhofsplatz
96444 Coburg

Telefon:

09561 96-44110

Telefax:

09561 96-44104

E-Mail:

renar.grunenb...@huk-coburg.de<mailto:renar.grunenb...@huk-coburg.de>

Internet:

Re: [gpfsug-discuss] Quota and hardlimit enforcement

2017-07-31 Thread Grunenberg, Renar 
Hallo Kevin,
thanks for your hint i will check these tomorrow, and yes as root, lol.
Regards Renar


Renar Grunenberg
Abteilung Informatik – Betrieb

HUK-COBURG
Bahnhofsplatz
96444 Coburg
Telefon:09561 96-44110
Telefax:09561 96-44104
E-Mail: renar.grunenb...@huk-coburg.de
Internet:   www.huk.de

HUK-COBURG Haftpflicht-Unterstützungs-Kasse kraftfahrender Beamter Deutschlands 
a. G. in Coburg
Reg.-Gericht Coburg HRB 100; St.-Nr. 9212/101/00021
Sitz der Gesellschaft: Bahnhofsplatz, 96444 Coburg
Vorsitzender des Aufsichtsrats: Prof. Dr. Heinrich R. Schradin.
Vorstand: Klaus-Jürgen Heitmann (Sprecher), Stefan Gronbach, Dr. Hans Olav 
Herøy, Dr. Jörg Rheinländer (stv.), Sarah Rössler, Daniel Thomas (stv.).

Diese Nachricht enthält vertrauliche und/oder rechtlich geschützte 
Informationen.
Wenn Sie nicht der richtige Adressat sind oder diese Nachricht irrtümlich 
erhalten haben,
informieren Sie bitte sofort den Absender und vernichten Sie diese Nachricht.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Nachricht ist 
nicht gestattet.

This information may contain confidential and/or privileged information.
If you are not the intended recipient (or have received this information in 
error) please notify the
sender immediately and destroy this information.
Any unauthorized copying, disclosure or distribution of the material in this 
information is strictly forbidden.

Von: gpfsug-discuss-boun...@spectrumscale.org 
[mailto:gpfsug-discuss-boun...@spectrumscale.org] Im Auftrag von Buterbaugh, 
Kevin L
Gesendet: Montag, 31. Juli 2017 21:22
An: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org>
Betreff: Re: [gpfsug-discuss] Quota and hardlimit enforcement

Hi Renar,

I’m sure this is the case, but I don’t see anywhere in this thread where this 
is explicitly stated … you’re not doing your tests as root, are you?  root, of 
course, is not bound by any quotas.

Kevin

On Jul 31, 2017, at 2:04 PM, Grunenberg, Renar 
<renar.grunenb...@huk-coburg.de<mailto:renar.grunenb...@huk-coburg.de>> wrote:


Hallo J. Eric, hallo Jaime,
Ok after we hit the softlimit we see that the graceperiod are go to 7 days. I 
think that’s the default. But was does it mean.
After we reach the ‘hard’-limit. we see additionaly the gbytes  in_doubt.
My interpretation now we can write many gb to the nospace-left event in the 
filesystem.
But our intention is to restricted some application to write only to the 
hardlimit in the fileset. Any hints to accomplish this?


Renar Grunenberg
Abteilung Informatik – Betrieb

HUK-COBURG
Bahnhofsplatz
96444 Coburg
Telefon:

09561 96-44110

Telefax:

09561 96-44104

E-Mail:

renar.grunenb...@huk-coburg.de<mailto:renar.grunenb...@huk-coburg.de>

Internet:

www.huk.de<http://www.huk.de/>


HUK-COBURG Haftpflicht-Unterstützungs-Kasse kraftfahrender Beamter Deutschlands 
a. G. in Coburg
Reg.-Gericht Coburg HRB 100; St.-Nr. 9212/101/00021
Sitz der Gesellschaft: Bahnhofsplatz, 96444 Coburg
Vorsitzender des Aufsichtsrats: Prof. Dr. Heinrich R. Schradin.
Vorstand: Klaus-Jürgen Heitmann (Sprecher), Stefan Gronbach, Dr. Hans Olav 
Herøy, Dr. Jörg Rheinländer (stv.), Sarah Rössler, Daniel Thomas (stv.).

Diese Nachricht enthält vertrauliche und/oder rechtlich geschützte 
Informationen.
Wenn Sie nicht der richtige Adressat sind oder diese Nachricht irrtümlich 
erhalten haben,
informieren Sie bitte sofort den Absender und vernichten Sie diese Nachricht.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Nachricht ist 
nicht gestattet.

This information may contain confidential and/or privileged information.
If you are not the intended recipient (or have received this information in 
error) please notify the
sender immediately and destroy this information.
Any unauthorized copying, disclosure or distribution of the material in this 
information is strictly forbidden.





Von: 
gpfsug-discuss-boun...@spectrumscale.org<mailto:gpfsug-discuss-boun...@spectrumscale.org>
 [mailto:gpfsug-discuss-boun...@spectrumscale.org] Im Auftrag von J. Eric 
Wonderley
Gesendet: Montag, 31. Juli 2017 19:55
An: gpfsug main discussion list 
<gpfsug-discuss@spectrumscale.org<mailto:gpfsug-discuss@spectrumscale.org>>
Betreff: Re: [gpfsug-discuss] Quota and hardlimit enforcement

Hi Renar:
What does 'mmlsquota -j fileset filesystem' report?
I did not think you would get a grace period of none unless the 
hardlimit=softlimit.

On Mon, Jul 31, 2017 at 1:44 PM, Grunenberg, Renar 
<renar.grunenb...@huk-coburg.de<mailto:renar.grunenb...@huk-coburg.de>> wrote:
Hallo All,
we are on Version 4.2.3.2 and see some missunderstandig in the enforcement of 
hardlimit definitions on a flieset quota. What we see is we put some 200 GB 
files on following quo

Re: [gpfsug-discuss] Quota and hardlimit enforcement

2017-07-31 Thread Jaime Pinto 

Renar

For as long as the usage is below the hard limit (space or inodes) and  
below the grace period you'll be able to write.


I don't think you can set the grace period to an specific value as a  
quota parameter, such as none. That is set at the filesystem creation  
time. BTW, grace period limit has been a mystery to me for many years.  
My impression is that GPFS keeps changing it internally depending on  
the position of the moon. I think ours is 2 hours, but at times I can  
see users writing for longer.


Jaime



Quoting "Grunenberg, Renar" :


Hallo All,
we are on Version 4.2.3.2 and see some missunderstandig in the   
enforcement of hardlimit definitions on a flieset quota. What we see  
 is we put some 200 GB files on following quota definitions:  quota   
150 GB Limit 250 GB Grace none.
After the creating of one 200 GB we hit the softquota limit, thats   
ok. But After the the second file was created!! we expect an io   
error but it don?t happen. We define all well know Parameters   
(-Q,..) on the filesystem . Is this a bug or a Feature? mmcheckquota  
 are already running at first.

Regards Renar.


Renar Grunenberg
Abteilung Informatik ? Betrieb

HUK-COBURG
Bahnhofsplatz
96444 Coburg
Telefon:09561 96-44110
Telefax:09561 96-44104
E-Mail: renar.grunenb...@huk-coburg.de
Internet:   www.huk.de

HUK-COBURG Haftpflicht-Unterstützungs-Kasse kraftfahrender Beamter  
Deutschlands a. G. in Coburg

Reg.-Gericht Coburg HRB 100; St.-Nr. 9212/101/00021
Sitz der Gesellschaft: Bahnhofsplatz, 96444 Coburg
Vorsitzender des Aufsichtsrats: Prof. Dr. Heinrich R. Schradin.
Vorstand: Klaus-Jürgen Heitmann (Sprecher), Stefan Gronbach, Dr.  
Hans Olav Herøy, Dr. Jörg Rheinländer (stv.), Sarah Rössler, Daniel  
Thomas (stv.).


Diese Nachricht enthält vertrauliche und/oder rechtlich geschützte  
Informationen.
Wenn Sie nicht der richtige Adressat sind oder diese Nachricht  
irrtümlich erhalten haben,

informieren Sie bitte sofort den Absender und vernichten Sie diese Nachricht.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser   
Nachricht ist nicht gestattet.


This information may contain confidential and/or privileged information.
If you are not the intended recipient (or have received this   
information in error) please notify the

sender immediately and destroy this information.
Any unauthorized copying, disclosure or distribution of the material  
 in this information is strictly forbidden.










 
  TELL US ABOUT YOUR SUCCESS STORIES
 http://www.scinethpc.ca/testimonials
 
---
Jaime Pinto
SciNet HPC Consortium - Compute/Calcul Canada
www.scinet.utoronto.ca - www.computecanada.ca
University of Toronto
661 University Ave. (MaRS), Suite 1140
Toronto, ON, M5G1M1
P: 416-978-2755
C: 416-505-1477


This message was sent using IMP at SciNet Consortium, University of Toronto.

___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] Quota and hardlimit enforcement

2017-07-31 Thread J. Eric Wonderley 
Hi Renar:

What does 'mmlsquota -j fileset filesystem' report?

I did not think you would get a grace period of none unless the
hardlimit=softlimit.

On Mon, Jul 31, 2017 at 1:44 PM, Grunenberg, Renar <
renar.grunenb...@huk-coburg.de> wrote:

> Hallo All,
> we are on Version 4.2.3.2 and see some missunderstandig in the enforcement
> of hardlimit definitions on a flieset quota. What we see is we put some 200
> GB files on following quota definitions:  quota 150 GB Limit 250 GB Grace
> none.
> After the creating of one 200 GB we hit the softquota limit, thats ok. But
> After the the second file was created!! we expect an io error but it don’t
> happen. We define all well know Parameters (-Q,..) on the filesystem . Is
> this a bug or a Feature? mmcheckquota are already running at first.
> Regards Renar.
>
>
>
> Renar Grunenberg
> Abteilung Informatik – Betrieb
>
> HUK-COBURG
> Bahnhofsplatz
> 96444 Coburg
> Telefon: 09561 96-44110
> Telefax: 09561 96-44104
> E-Mail: renar.grunenb...@huk-coburg.de
> Internet: www.huk.de
>
> --
> HUK-COBURG Haftpflicht-Unterstützungs-Kasse kraftfahrender Beamter
> Deutschlands a. G. in Coburg
> Reg.-Gericht Coburg HRB 100; St.-Nr. 9212/101/00021
> Sitz der Gesellschaft: Bahnhofsplatz, 96444 Coburg
> Vorsitzender des Aufsichtsrats: Prof. Dr. Heinrich R. Schradin.
> Vorstand: Klaus-Jürgen Heitmann (Sprecher), Stefan Gronbach, Dr. Hans Olav
> Herøy, Dr. Jörg Rheinländer (stv.), Sarah Rössler, Daniel Thomas (stv.).
> --
> Diese Nachricht enthält vertrauliche und/oder rechtlich geschützte
> Informationen.
> Wenn Sie nicht der richtige Adressat sind oder diese Nachricht irrtümlich
> erhalten haben,
> informieren Sie bitte sofort den Absender und vernichten Sie diese
> Nachricht.
> Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Nachricht
> ist nicht gestattet.
>
> This information may contain confidential and/or privileged information.
> If you are not the intended recipient (or have received this information
> in error) please notify the
> sender immediately and destroy this information.
> Any unauthorized copying, disclosure or distribution of the material in
> this information is strictly forbidden.
> --
>
>
> ___
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss