Re: [Beowulf] Storage Best Practices

[Richter, Brian J {BIS}]

I'd like to thank everyone for your responses!

Brian J. Richter
Global R&D Senior Analyst * Information Technology
617 W Main St, Barrington, IL 60010
Office: 847-304-2356 * Mobile: 847-305-6306
brian.j.rich...@pepsico.com


From: Richter, Brian J {BIS}
Sent: Monday, February 19, 2018 8:12 AM
To: 'Beowulf@beowulf.org' 
Subject: Storage Best Practices

Hey All,

I was hoping to get some recommendations for Storage. Last year we set up our 
first HPC and I'm looking for a good strategy moving forward for Storage. We 
set up a dedicated space on the cluster for Storage that has 5.5 TB of space. 
This space can be quickly chewed up depending on the project the business is 
working on. Do you guys typically do a retention policy on the cluster to 
ensure there is always enough space? I was thinking anything older than a month 
should be cleaned up.

Thanks!

Brian J. Richter
Global R&D Senior Analyst * Information Technology
617 W Main St, Barrington, IL 60010
Office: 847-304-2356 * Mobile: 847-305-6306
brian.j.rich...@pepsico.com


___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf

Re: [Beowulf] Storage Best Practices

[Adam DeConinck]

The best success I've seen is a mix of strategies: some amount storage
that's considered "permanent" with aggressive quota limits (per user
or per project), plus a larger no-quota space that's cleaned up based
on some retention policy. I've used this basic scheme in small
environments, and also seen it used successfully in large institutions
(where I was not directly responsible for storage).

That said, there's no way to make everyone happy with this -- people
will complain when a file is deleted unexpectedly, or when they hit
their quota, or both. Time-based retention policies that work by
automatically checking file modification time also invite users to
write "touch" scripts to try to fool the cleaner. It can turn into a
fun game. But I've also used these kinds of policies to justify new
storage purchases, and make users into allies in funding them.

On a lighter note, this is a good excuse for me to link my favorite
James Cuff post ;-) -
http://blog.jcuff.net/2011/12/cautionary-tale-about-storage-and.html

On Mon, Feb 19, 2018 at 6:12 AM, Richter, Brian J {BIS}
 wrote:
> Hey All,
>
>
>
> I was hoping to get some recommendations for Storage. Last year we set up
> our first HPC and I’m looking for a good strategy moving forward for
> Storage. We set up a dedicated space on the cluster for Storage that has 5.5
> TB of space. This space can be quickly chewed up depending on the project
> the business is working on. Do you guys typically do a retention policy on
> the cluster to ensure there is always enough space? I was thinking anything
> older than a month should be cleaned up.
>
>
>
> Thanks!
>
>
>
> Brian J. Richter
>
> Global R&D Senior Analyst • Information Technology
>
> 617 W Main St, Barrington, IL 60010
> Office: 847-304-2356 • Mobile: 847-305-6306
>
> brian.j.rich...@pepsico.com
>
>
>
>
>
>
> ___
> Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
>
___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf

Re: [Beowulf] Storage Best Practices

[Adam DeConinck]

The best success I've seen is a mix of strategies: some amount storage
that's considered "permanent" with aggressive quota limits (per user
or per project), plus a larger no-quota space that's cleaned up based
on some retention policy. I've used this basic scheme in small
environments, and also seen it used successfully in large institutions
(where I was not directly responsible for storage).

That said, there's no way to make everyone happy with this -- people
will complain when a file is deleted unexpectedly, or when they hit
their quota, or both. Time-based retention policies that work by
automatically checking file modification time also invite users to
write "touch" scripts to try to fool the cleaner. It can turn into a
fun game. But I've also used these kinds of policies to justify new
storage purchases, and make users into allies in funding them.

On a lighter note, this is a good excuse for me to link my favorite
James Cuff post ;-) -
http://blog.jcuff.net/2011/12/cautionary-tale-about-storage-and.html

Cheers,
Adam


On Mon, Feb 19, 2018 at 6:12 AM, Richter, Brian J {BIS}
 wrote:
> Hey All,
>
>
>
> I was hoping to get some recommendations for Storage. Last year we set up
> our first HPC and I’m looking for a good strategy moving forward for
> Storage. We set up a dedicated space on the cluster for Storage that has 5.5
> TB of space. This space can be quickly chewed up depending on the project
> the business is working on. Do you guys typically do a retention policy on
> the cluster to ensure there is always enough space? I was thinking anything
> older than a month should be cleaned up.
>
>
>
> Thanks!
>
>
>
> Brian J. Richter
>
> Global R&D Senior Analyst • Information Technology
>
> 617 W Main St, Barrington, IL 60010
> Office: 847-304-2356 • Mobile: 847-305-6306
>
> brian.j.rich...@pepsico.com
>
>
>
>
>
>
> ___
> Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
>
___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf

Re: [Beowulf] Storage Best Practices

[Tim Cutts]

No policy here.  People can keep stuff as long as they like.  I don’t agree 
with that lack of policy, but that’s where we are.

We did propose a 90 day limit, about 10 years ago.  It lasted about, er, 90 
days, before faculty started screaming.  ☺

Tim

On 19/02/2018, 15:31, "Beowulf on behalf of Skylar Thompson" 
mailto:beowulf-boun...@beowulf.org> on behalf of 
skylar.thomp...@gmail.com> wrote:

For our larger groups, we'll meet with them regularly to discuss their space 
usage (and other IT needs). Even that's unlikely to be frequent enough, so we 
direct usage alerts to their designated "data manager" if they're getting close 
to running out of space. Aside from regularly clearing out scratch space, 
though, we try to stay out of deletion decisions, to avoid problems stemming 
from miscommunication between the group and IT, or even within the group (left 
hand doesn't know what the right hand is doing).
Skylar

On Mon, Feb 19, 2018 at 6:12 AM, Richter, Brian J {BIS} 
mailto:brianj.rich...@pepsico.com>> wrote:
Hey All,

I was hoping to get some recommendations for Storage. Last year we set up our 
first HPC and I’m looking for a good strategy moving forward for Storage. We 
set up a dedicated space on the cluster for Storage that has 5.5 TB of space. 
This space can be quickly chewed up depending on the project the business is 
working on. Do you guys typically do a retention policy on the cluster to 
ensure there is always enough space? I was thinking anything older than a month 
should be cleaned up.

Thanks!

Brian J. Richter
Global R&D Senior Analyst • Information Technology
617 W Main St, Barrington, IL 60010
Office: 847-304-2356 • Mobile: 
847-305-6306
brian.j.rich...@pepsico.com



___
Beowulf mailing list, Beowulf@beowulf.org sponsored 
by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf




-- 
 The Wellcome Sanger Institute is operated by Genome Research 
 Limited, a charity registered in England with number 1021457 and a 
 company registered in England with number 2742969, whose registered 
 office is 215 Euston Road, London, NW1 2BE.___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf

Re: [Beowulf] Storage Best Practices

[Skylar Thompson]

For our larger groups, we'll meet with them regularly to discuss their
space usage (and other IT needs). Even that's unlikely to be frequent
enough, so we direct usage alerts to their designated "data manager" if
they're getting close to running out of space. Aside from regularly
clearing out scratch space, though, we try to stay out of deletion
decisions, to avoid problems stemming from miscommunication between the
group and IT, or even within the group (left hand doesn't know what the
right hand is doing).

Skylar

On Mon, Feb 19, 2018 at 6:12 AM, Richter, Brian J {BIS} <
brian.j.rich...@pepsico.com> wrote:

> Hey All,
>
>
>
> I was hoping to get some recommendations for Storage. Last year we set up
> our first HPC and I’m looking for a good strategy moving forward for
> Storage. We set up a dedicated space on the cluster for Storage that has
> 5.5 TB of space. This space can be quickly chewed up depending on the
> project the business is working on. Do you guys typically do a retention
> policy on the cluster to ensure there is always enough space? I was
> thinking anything older than a month should be cleaned up.
>
>
>
> Thanks!
>
>
>
> *Brian J. Richter*
>
> Global R&D Senior Analyst *•* Information Technology
>
> 617 W Main St, Barrington, IL 60010
> Office: 847-304-2356 <(847)%20304-2356> *• *Mobile: 847-305-6306
> <(847)%20305-6306>
>
> brian.j.rich...@pepsico.com
>
>
>
>
>
> ___
> Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
>
>
___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf