Re: [cfarm-users] Please cleanup your /home on farm machines

[Stefan Ring via cfarm-users]

On Sat, Jan 20, 2018 at 9:44 AM, Noah Misch via cfarm-users
 wrote:
>> We can determine the users consuming the largest amount of diskspace
>> and request that they delete it.
>
> I agree with all that.  (Automated /tmp and perhaps /var/tmp cleanup is fine
> with me, but not other files.)

And the call to action was clearly successful.

I don't mind a little cleanup work once a year.
___
cfarm-users mailing list
cfarm-users@lists.tetaneutral.net
https://lists.tetaneutral.net/listinfo/cfarm-users

Re: [cfarm-users] Please cleanup your /home on farm machines

[Noah Misch via cfarm-users]

On Fri, Jan 19, 2018 at 11:15:53AM -0500, David Edelsohn via cfarm-users wrote:
> On Fri, Jan 19, 2018 at 10:47 AM, Segher Boessenkool via cfarm-users
>  wrote:
> > On Fri, Jan 19, 2018 at 03:05:48PM +0100, Martin Guy via cfarm-users wrote:
> >> find /home \( \
> >> \( -name '*.o' -a \! -atime +7 \) \
> >
> > I veto this 400%.  People have valuable .o files, and this is not very
> > effective anyway (other files take up more space).
> 
> I agree with Segher.
> 
> Automatically deleting files is completely unacceptable and contrary
> to the purpose of the GNU Compile Farm.  The Compile Farm requires
> that users are considerate of each other and the shared resources that
> are provided gratis.
> 
> We can determine the users consuming the largest amount of diskspace
> and request that they delete it.

I agree with all that.  (Automated /tmp and perhaps /var/tmp cleanup is fine
with me, but not other files.)
___
cfarm-users mailing list
cfarm-users@lists.tetaneutral.net
https://lists.tetaneutral.net/listinfo/cfarm-users

Re: [cfarm-users] Please cleanup your /home on farm machines

[CM Graff via cfarm-users]

I agree with Segher on this. the only way for us to all share the
machines gracefully without constricting system limits and automatic
deletions is for each user to heuristically address their own usage
needs and delete what they don't need accordingly. For me, that
happens to be nearly everything, and this makes room for those that
need more permanent storage.

On 1/19/18, Segher Boessenkool via cfarm-users
 wrote:
> On Fri, Jan 19, 2018 at 09:29:23PM +0100, Martin Guy via cfarm-users wrote:
>> On 19/01/2018, David Edelsohn via cfarm-users
>>  wrote:
>> > On Fri, Jan 19, 2018 at 10:47 AM, Segher Boessenkool via cfarm-users
>> >  wrote:
>> >> On Fri, Jan 19, 2018 at 03:05:48PM +0100, Martin Guy via cfarm-users
>> >> wrote:
>> >>> find /home \( \
>> >>> \( -name '*.o' -a \! -atime +7 \) \
>>
>> >> I veto this 400%.  People have valuable .o files, and this is not very
>> >> effective anyway (other files take up more space).
>> >
>> > Automatically deleting files is completely unacceptable and contrary
>> > to the purpose of the GNU Compile Farm
>>
>> This is debatable as they should be regenerated automatically at the
>> next make,
>
> I have many .o files that are not generated by a build system (heck, some
> that are not object files!)
>
> No.
>
>
> Segher
> ___
> cfarm-users mailing list
> cfarm-users@lists.tetaneutral.net
> https://lists.tetaneutral.net/listinfo/cfarm-users
>
___
cfarm-users mailing list
cfarm-users@lists.tetaneutral.net
https://lists.tetaneutral.net/listinfo/cfarm-users

Re: [cfarm-users] Please cleanup your /home on farm machines

[Segher Boessenkool via cfarm-users]

On Fri, Jan 19, 2018 at 09:29:23PM +0100, Martin Guy via cfarm-users wrote:
> On 19/01/2018, David Edelsohn via cfarm-users
>  wrote:
> > On Fri, Jan 19, 2018 at 10:47 AM, Segher Boessenkool via cfarm-users
> >  wrote:
> >> On Fri, Jan 19, 2018 at 03:05:48PM +0100, Martin Guy via cfarm-users
> >> wrote:
> >>> find /home \( \
> >>> \( -name '*.o' -a \! -atime +7 \) \
> 
> >> I veto this 400%.  People have valuable .o files, and this is not very
> >> effective anyway (other files take up more space).
> >
> > Automatically deleting files is completely unacceptable and contrary
> > to the purpose of the GNU Compile Farm
> 
> This is debatable as they should be regenerated automatically at the
> next make,

I have many .o files that are not generated by a build system (heck, some
that are not object files!)

No.


Segher
___
cfarm-users mailing list
cfarm-users@lists.tetaneutral.net
https://lists.tetaneutral.net/listinfo/cfarm-users

Re: [cfarm-users] Please cleanup your /home on farm machines

[Martin Guy via cfarm-users]

For experimental modifications to the source tree, instead, patch files.
Not only does this reduce disk usag by 99.%, but it documents
the build process in a way that is testable (if it doesn't run, it
isn't right)

cheers

   M
___
cfarm-users mailing list
cfarm-users@lists.tetaneutral.net
https://lists.tetaneutral.net/listinfo/cfarm-users

Re: [cfarm-users] Please cleanup your /home on farm machines

[Martin Guy via cfarm-users]

Excuse the prolixity. The big stuff is the source. What i usually do
is, while downloading, unpacking, configuring and compiling stuff, is
to keep a window open in which I record the runes to do all this,
modifying it as I find solutions that work. In practice i usually have
to run such a script by hand, ilne by line cos it never works 100% but
sonetimes yes.

blessings

   M
___
cfarm-users mailing list
cfarm-users@lists.tetaneutral.net
https://lists.tetaneutral.net/listinfo/cfarm-users

Re: [cfarm-users] Please cleanup your /home on farm machines

[Martin Guy via cfarm-users]

However, automativally cleaning /tmp /var/tmp /usr/tmp is always good,
say after 24h of non-use. Someone said that there is bulk accumulating
in /tmp on some machines that would be cleaned at reboot but, of
course, the GCC Compile Farm machines run for years uninterruptedly...

   m
___
cfarm-users mailing list
cfarm-users@lists.tetaneutral.net
https://lists.tetaneutral.net/listinfo/cfarm-users

Re: [cfarm-users] Please cleanup your /home on farm machines

[Martin Guy via cfarm-users]

On 19/01/2018, David Edelsohn via cfarm-users
 wrote:
> On Fri, Jan 19, 2018 at 10:47 AM, Segher Boessenkool via cfarm-users
>  wrote:
>> On Fri, Jan 19, 2018 at 03:05:48PM +0100, Martin Guy via cfarm-users
>> wrote:
>>> find /home \( \
>>> \( -name '*.o' -a \! -atime +7 \) \

>> I veto this 400%.  People have valuable .o files, and this is not very
>> effective anyway (other files take up more space).
>
> Automatically deleting files is completely unacceptable and contrary
> to the purpose of the GNU Compile Farm

This is debatable as they should be regenerated automatically at the
next make, and though smaller than the source, they are still a bulk.
Idem core files. If you haven't worked on it for a week (-atime) I'd
say you can recompile them.

Still, it's up the the admins to decide as there are too widely
difffering opinions and needs here.

Yes, if everyone were more careful to clean when done the problem
wouldn't arise. That's another possible solution. i remember, still in
the late 80s, when I had to upgrade the OS in a way that needed
copying the user files over the local net, reformat, install and
replace user files. I put up a motd that explained this, and the disk
usage dropped from 90$ to 30% in 24 hours. I was glad to have such
excellent responsive users.

Another technique is to put a hall of shame in the motd so that
everyon sees who are the top 10 hogs. That seems to be effective too
(unless the hogs have a war to see who gets to be No1...)

How about we see how it goes, and if the problem arises again, we can
consider an automated solution, with all the work and discussion that
implementing that involves.

M
___
cfarm-users mailing list
cfarm-users@lists.tetaneutral.net
https://lists.tetaneutral.net/listinfo/cfarm-users

Re: [cfarm-users] Please cleanup your /home on farm machines

[Sebastian Huber via cfarm-users]

Maybe we could combine an automated cleaner with a user defined ignore file.
___
cfarm-users mailing list
cfarm-users@lists.tetaneutral.net
https://lists.tetaneutral.net/listinfo/cfarm-users

Re: [cfarm-users] Please cleanup your /home on farm machines

[Jeffrey Walton via cfarm-users]

On Fri, Jan 19, 2018 at 11:15 AM, David Edelsohn via cfarm-users
 wrote:
> On Fri, Jan 19, 2018 at 10:47 AM, Segher Boessenkool via cfarm-users
>  wrote:
>> On Fri, Jan 19, 2018 at 03:05:48PM +0100, Martin Guy via cfarm-users wrote:
>>> find /home \( \
>>> \( -name '*.o' -a \! -atime +7 \) \
>>
>> I veto this 400%.  People have valuable .o files, and this is not very
>> effective anyway (other files take up more space).
>
> I agree with Segher.
>
> Automatically deleting files is completely unacceptable and contrary
> to the purpose of the GNU Compile Farm.  The Compile Farm requires
> that users are considerate of each other and the shared resources that
> are provided gratis.

Is there a way to make the service "opt-in" through the control panel?
Is it a change the farm is willing to entertain?

Jeff
___
cfarm-users mailing list
cfarm-users@lists.tetaneutral.net
https://lists.tetaneutral.net/listinfo/cfarm-users

Re: [cfarm-users] Please cleanup your /home on farm machines

[David Edelsohn via cfarm-users]

On Fri, Jan 19, 2018 at 10:47 AM, Segher Boessenkool via cfarm-users
 wrote:
> On Fri, Jan 19, 2018 at 03:05:48PM +0100, Martin Guy via cfarm-users wrote:
>> find /home \( \
>> \( -name '*.o' -a \! -atime +7 \) \
>
> I veto this 400%.  People have valuable .o files, and this is not very
> effective anyway (other files take up more space).

I agree with Segher.

Automatically deleting files is completely unacceptable and contrary
to the purpose of the GNU Compile Farm.  The Compile Farm requires
that users are considerate of each other and the shared resources that
are provided gratis.

We can determine the users consuming the largest amount of diskspace
and request that they delete it.  The Compile Farm is a privilege, not
a right.  Users who consume an excessive amount of diskspace (or CPU)
and don't respond to requests to clean up politely should be told to
find other resources for their project.

Thanks, David
___
cfarm-users mailing list
cfarm-users@lists.tetaneutral.net
https://lists.tetaneutral.net/listinfo/cfarm-users

Re: [cfarm-users] Please cleanup your /home on farm machines

[Segher Boessenkool via cfarm-users]

On Fri, Jan 19, 2018 at 03:05:48PM +0100, Martin Guy via cfarm-users wrote:
> find /home \( \
> \( -name '*.o' -a \! -atime +7 \) \

I veto this 400%.  People have valuable .o files, and this is not very
effective anyway (other files take up more space).


Segher
___
cfarm-users mailing list
cfarm-users@lists.tetaneutral.net
https://lists.tetaneutral.net/listinfo/cfarm-users

Re: [cfarm-users] Please cleanup your /home on farm machines

[Jeffrey Walton via cfarm-users]

On Fri, Jan 19, 2018 at 9:02 AM, Martin Guy via cfarm-users
 wrote:
> On 18/01/2018, Jeffrey Walton via cfarm-users
>  wrote:
>> I can't speak for others, but I would not mind being subject to a
>> script that removes temporary object files that are over N days old.
>> In fact I would welcome it because I am forgetful at times.
>
> Traditionally, such a script is called "fileclean", essentialy a
> humungous "find" that deletes *.tmp ~*, *.o older than a week, core
> older than a day and so on.

Sounds perfect.

It makes me wonder why a problem that existed in 1987 still exists to day :)

Jeff
___
cfarm-users mailing list
cfarm-users@lists.tetaneutral.net
https://lists.tetaneutral.net/listinfo/cfarm-users

Re: [cfarm-users] Please cleanup your /home on farm machines

[Martin Guy via cfarm-users]

On 19/01/2018, Martin Guy  wrote:
> find /home \( \
> \( -name '*.o' -a \! -atime +7 \) \

Sorry, finger trouble.
___
cfarm-users mailing list
cfarm-users@lists.tetaneutral.net
https://lists.tetaneutral.net/listinfo/cfarm-users

Re: [cfarm-users] Please cleanup your /home on farm machines

[Martin Guy via cfarm-users]

find /home \( \
\( -name '*.o' -a \! -atime +7 \) \


On 19/01/2018, Martin Guy  wrote:
> On 18/01/2018, Jeffrey Walton via cfarm-users
>  wrote:
>> I can't speak for others, but I would not mind being subject to a
>> script that removes temporary object files that are over N days old.
>> In fact I would welcome it because I am forgetful at times.
>
> Traditionally, such a script is called "fileclean", essentialy a
> humungous "find" that deletes *.tmp ~*, *.o older than a week, core
> older than a day and so on.
>
>M
>
___
cfarm-users mailing list
cfarm-users@lists.tetaneutral.net
https://lists.tetaneutral.net/listinfo/cfarm-users

Re: [cfarm-users] Please cleanup your /home on farm machines

[Martin Guy via cfarm-users]

On 18/01/2018, Jeffrey Walton via cfarm-users
 wrote:
> I can't speak for others, but I would not mind being subject to a
> script that removes temporary object files that are over N days old.
> In fact I would welcome it because I am forgetful at times.

Traditionally, such a script is called "fileclean", essentialy a
humungous "find" that deletes *.tmp ~*, *.o older than a week, core
older than a day and so on.

   M
___
cfarm-users mailing list
cfarm-users@lists.tetaneutral.net
https://lists.tetaneutral.net/listinfo/cfarm-users

Re: [cfarm-users] Please cleanup your /home on farm machines

[Jeffrey Walton via cfarm-users]

On Thu, Jan 18, 2018 at 6:17 AM, Baptiste Jonglez via cfarm-users
 wrote:
> Hi everyone,
>
> Let's start the new year with a positive action for the compile farm: home 
> cleanup!
>
> The farm machines with the most critical disk usage are:
>
> - gcc110 (99% used out of 1.6 TB)
> - gcc20  (98% used out of 826 GB)
> - gcc13  (95% used out of 459 GB)
> - gcc112 (92% used out of 1.8 TB)
> - gcc16  (85% used out of 459 GB)
>
> Please check each of these machines: if you have unused data in your home
> directory, it should be deleted!

I can't speak for others, but I would not mind being subject to a
script that removes temporary object files that are over N days old.
In fact I would welcome it because I am forgetful at times.

I don't know how long a "log running" job is for some users. Maybe you
can exclude users who are logged in, and clean objects that meet
criteria for the rest.

Jeff
___
cfarm-users mailing list
cfarm-users@lists.tetaneutral.net
https://lists.tetaneutral.net/listinfo/cfarm-users

Re: [cfarm-users] Please cleanup your /home on farm machines

[Hans via cfarm-users]

Baptiste Jonglez via cfarm-users wrote on 20180118:
> Let's start the new year with a positive action for the compile farm: home 
> cleanup!

Good idea !

> The farm machines with the most critical disk usage are:

Done, all less than 36 kilobytes ;-)

Also cleaned gcc14 and gcc21 and gcc76 where I knew I had obsolete stellarium 
and open-phd-guiding things stored.

-- Hans
___
cfarm-users mailing list
cfarm-users@lists.tetaneutral.net
https://lists.tetaneutral.net/listinfo/cfarm-users

Re: [cfarm-users] Please cleanup your /home on farm machines

[CM Graff via cfarm-users]

For those of you like myself who don't have any important data stored
on the build machines, I used a fairly crude script to finally clean
out all of my work. Even on machines that I may have forgotten that I
used:

#!/bin/sh
if [ $# = 0 ]
thenCOUNT=20
elseCOUNT=$1
fi
USERNAME=cgraff1

while [ "$COUNT" -lt 130 ]
do  printf "%s\n" "attempting to clean
${USERNAME}@gcc${COUNT}.fsffrance.org"
ssh "${USERNAME}@gcc${COUNT}.fsffrance.org" -f 'rm -rf *'
COUNT=$((COUNT + 1))
done

On 1/18/18, Bart Van Assche via cfarm-users
 wrote:
> On 01/18/18 03:17, Baptiste Jonglez via cfarm-users wrote:
>> Please check each of these machines: if you have unused data in your home
>> directory, it should be deleted!
>
> Hello Baptiste,
>
> How about /tmp? There are several systems in the cluster with thousands
> of files in /tmp. How about installing a cron job that e.g. removes all
> files and directories from /tmp that are older than one week? This is
> the shell command I use on my own systems to remove files from /tmp that
> are older than one day:
>
> find "${1:-/tmp}" -maxdepth 1 ! -mtime 0 -ls -exec rm -rf {} \;
>
> Bart.
> ___
> cfarm-users mailing list
> cfarm-users@lists.tetaneutral.net
> https://lists.tetaneutral.net/listinfo/cfarm-users
>
___
cfarm-users mailing list
cfarm-users@lists.tetaneutral.net
https://lists.tetaneutral.net/listinfo/cfarm-users

Re: [cfarm-users] Please cleanup your /home on farm machines

[Martin Guy via cfarm-users]

Thanks. My /home is always clean because I work, get results, delete.
Maybe I keep a script to recreate whetever I was doing.

This is known as "The tragedy of the commons" : when something is free
for everyone, some people  use up the resources as if they were sea
water, resulting in there being nothing left, and whatver was free
dries up.

Mind u, it's nothing new. Here's a Christmas greeting from our Unix
sysadmin (Peter Collinson) from 1987

Disc is full, Disc is full,
People need to work,
Delete those junk files from the disc
whereso'ever they lurk,
HEY!
Disc is full, Disc is full,
The Eagle disc is packed,
Then we can go on holiday
leaving Eagle to be hacked.

(Sing loudly to a well known tune)

:)

   M
___
cfarm-users mailing list
cfarm-users@lists.tetaneutral.net
https://lists.tetaneutral.net/listinfo/cfarm-users

[cfarm-users] Please cleanup your /home on farm machines

[Baptiste Jonglez via cfarm-users]

Hi everyone,

Let's start the new year with a positive action for the compile farm: home 
cleanup!

The farm machines with the most critical disk usage are:

- gcc110 (99% used out of 1.6 TB)
- gcc20  (98% used out of 826 GB)
- gcc13  (95% used out of 459 GB)
- gcc112 (92% used out of 1.8 TB)
- gcc16  (85% used out of 459 GB)

Please check each of these machines: if you have unused data in your home
directory, it should be deleted!

The disk usage of farm machines in the last year can be seen here: 
https://cfarm.tetaneutral.net/munin/disk-year.html#Disk%20usage%20in%20percent

Thanks for your help in making the compile farm great again,
Baptiste


PS: for a quick & dirty overview of your files on those machines, you can
use the following one-liner:

for i in gcc{110,20,13,112,16}.fsffrance.org; do echo $i; ssh $i ls -lh; 
echo; done


signature.asc
Description: PGP signature
___
cfarm-users mailing list
cfarm-users@lists.tetaneutral.net
https://lists.tetaneutral.net/listinfo/cfarm-users