Re: [Haifux] SVN quota
On Tue, Jan 19, 2010, Shahar Dag wrote about Re: [Haifux] SVN quota: It is true that users can't delete data from the SVN. The only thing users can do is ask for additional quota. If they know that they are running out of quota in advanced, they can ask to enlarge the quota before they are out of disk space (which usually happens 01:00 ). Then I can check if the disk space usage is justify and respond on time. Shachar Shemesh made a very good point. There's very little point in a quota system if the person who is reaching the end of his quota can't do anything about it to clean up his act. If the only thing he can do is to ask you to increase his quota (and you never refuse), why have a quota in the first place - to make people feel bad about taking up space? To add beaurocracy? I have several years experience with using and administering SVN repositories, and my experience is that problems with repository sizes always stem from one issue: that somebody, usually mistakenly or due to lack of understanding of what SVN is supposed to do, did one huge commit. E.g., somebody added some huge test data or outputs to the repository, which they shouldn't have done in the first place. Once the user added a huge commit, Subversion doesn't give you any (convenient) facility to remove this commit, or the huge files in it, from the history. You can get mails that you are close to your quota until hell freezes over - and there's nothing you can do about it but say I'm sorry... If you must have SVN quotas, one thing I'd try first is to try to enforce first some sort of limit the size of a single commit. For example, if you give people a 100 MB quota for the repository, limit a single commit size to 30 MB; If somebody fills up more than 30% of his quota in a single commit - something is probably wrong. If you let him do it, he'll probably be sorry later because he'll not have enough space to continue using his repository. Of course, you can play with this 30% number. Perhaps enforce this single-commit-quota just after 50% of the total quota is finished - or come with whatever policy makes sense for you. Also, you'll need to figure out how to enforce this policy :-) (some sort of commmit hook might work, but I really didn't try to implement this idea). Nadav. -- Nadav Har'El| Thursday, Jan 21 2010, 6 Shevat 5770 n...@math.technion.ac.il |- Phone +972-523-790466, ICQ 13349191 |Why aren't fishmongers generous? Their http://nadav.harel.org.il |business makes them selfish. ___ Haifux mailing list Haifux@haifux.org http://hamakor.org.il/cgi-bin/mailman/listinfo/haifux
Re: [Haifux] SVN quota
If anything, make the quota soft, with no hard quota - and set up mails to be sent when the quota is reached - just to be aware that something is going on. On Thu, Jan 21, 2010 at 2:09 PM, Shahar Dag d...@cs.technion.ac.il wrote: Hello As I already said, setting quota on an SVN project is more a meter of awareness, then a real disk space limitation. The idea of using the pre-commit hock to limit the size of a commit is very interesting. Quick google came with several sites: http://www.davidgrant.ca/limit_size_of_subversion_commits_with_this_hook http://svn.haxx.se/users/archive-2009-09/0928.shtml I also found this, but I totally don't understand it http://lists.freebsd.org/pipermail/svn-src-svnadmin/2008-November/09.html I don't have time now to dig into the suggestions. I will update the list after I will have a working hock Shahar - Original Message - From: Nadav Har'El n...@math.technion.ac.il To: Shahar Dag d...@cs.technion.ac.il Cc: Haifaux haifux@haifux.org Sent: Thursday, January 21, 2010 10:43 AM Subject: Re: [Haifux] SVN quota On Tue, Jan 19, 2010, Shahar Dag wrote about Re: [Haifux] SVN quota: It is true that users can't delete data from the SVN. The only thing users can do is ask for additional quota. If they know that they are running out of quota in advanced, they can ask to enlarge the quota before they are out of disk space (which usually happens 01:00 ). Then I can check if the disk space usage is justify and respond on time. Shachar Shemesh made a very good point. There's very little point in a quota system if the person who is reaching the end of his quota can't do anything about it to clean up his act. If the only thing he can do is to ask you to increase his quota (and you never refuse), why have a quota in the first place - to make people feel bad about taking up space? To add beaurocracy? I have several years experience with using and administering SVN repositories, and my experience is that problems with repository sizes always stem from one issue: that somebody, usually mistakenly or due to lack of understanding of what SVN is supposed to do, did one huge commit. E.g., somebody added some huge test data or outputs to the repository, which they shouldn't have done in the first place. Once the user added a huge commit, Subversion doesn't give you any (convenient) facility to remove this commit, or the huge files in it, from the history. You can get mails that you are close to your quota until hell freezes over - and there's nothing you can do about it but say I'm sorry... If you must have SVN quotas, one thing I'd try first is to try to enforce first some sort of limit the size of a single commit. For example, if you give people a 100 MB quota for the repository, limit a single commit size to 30 MB; If somebody fills up more than 30% of his quota in a single commit - something is probably wrong. If you let him do it, he'll probably be sorry later because he'll not have enough space to continue using his repository. Of course, you can play with this 30% number. Perhaps enforce this single-commit-quota just after 50% of the total quota is finished - or come with whatever policy makes sense for you. Also, you'll need to figure out how to enforce this policy :-) (some sort of commmit hook might work, but I really didn't try to implement this idea). Nadav. -- Nadav Har'El| Thursday, Jan 21 2010, 6 Shevat 5770 n...@math.technion.ac.il |- Phone +972-523-790466, ICQ 13349191 |Why aren't fishmongers generous? Their http://nadav.harel.org.il |business makes them selfish. ___ Haifux mailing list Haifux@haifux.org http://hamakor.org.il/cgi-bin/mailman/listinfo/haifux -- Man is the only animal that laughs and weeps, for he is the only animal that is struck with the difference between what things are and what they ought to be. - William Hazlitt Ohad Lutzky ___ Haifux mailing list Haifux@haifux.org http://hamakor.org.il/cgi-bin/mailman/listinfo/haifux
Re: [Haifux] SVN quota
Hello The SVN is used for programming projects done by groups, so the quota is a group quota per project. My users are students at the Technion (computer science). I believe it is a good thing that users knows that they have limited resources, this way they use them wisely. If all groups are near the quota limits, I know it is a real need, if one group uses 500 the rest are below 100, I know something is odd here and with the instruction of the course I decide what to do. The problem of SVN is that there is no easy way to erase un-needed data. The only way I know of is to: - create a full backup of the repository while filtering unwanted data - erase the entire repository - import the data from the filtered backup. I must admit that I prefer to give more disk space then to do this work. Shahar - Original Message - From: Etzion Bar-Noy To: Shahar Dag Cc: Haifaux Sent: Wednesday, January 20, 2010 12:06 AM Subject: Re: [Haifux] SVN quota If SVN quota is group-wise, and if it's on a dedicated filesystem, and if you can't remove data from the SVN (due to its nature, anyhow, the users cant remove data), and if they ask for more space and you grant them that, may I ask what is the point in setting up quota for that specific storage? Ez On Tue, Jan 19, 2010 at 9:19 AM, Shahar Dag d...@cs.technion.ac.il wrote: Hello Max Home folder is on one file system and the SVN repository is on another file system. This is because home folders are limited by user quota while SVN projects are limited by group quota. If the 2 folders would be on the same file system then every write to the SVN repository will eat from the users quota (which I don't want). Shahar - Original Message - From: Maxim Kovgan kovg...@gmail.com To: Shachar Shemesh shac...@shemesh.biz Cc: Shahar Dag d...@cs.technion.ac.il; Haifaux haifux@haifux.org Sent: Tuesday, January 19, 2010 6:28 AM Subject: Re: [Haifux] SVN quota Shalom Shcharim! Possibly, the repositories are located on the same storage device as users' homes, so they might be able to delete other files to release some space, but you're completely right, the repository can only grow up :-) 2010/1/18 Shachar Shemesh shac...@shemesh.biz: Shahar Dag wrote: Hello I have a LINUX server (RHE 4) that among other things serves as SVN server (the SVN server runs over apache). Users can access their repository via https, but they can't login to the server (not even with ssh). To access the SVN, users must supply their NIS user name password. Every SVN repository has it's own quota limitation. The question is: do SVN users have any way to test their quota? (I don't want to write a cron job that warns the users when they are over 80% of the quota) Excuse me for taking this on a tangent, but what use is the quota warning going to do? It's not as if users can delete stuff in order save space. Shachar -- Shachar Shemesh Lingnu Open Source Consulting Ltd. http://www.lingnu.com ___ Haifux mailing list Haifux@haifux.org http://hamakor.org.il/cgi-bin/mailman/listinfo/haifux -- Maxim Kovgan ___ Haifux mailing list Haifux@haifux.org http://hamakor.org.il/cgi-bin/mailman/listinfo/haifux -- ___ Haifux mailing list Haifux@haifux.org http://hamakor.org.il/cgi-bin/mailman/listinfo/haifux ___ Haifux mailing list Haifux@haifux.org http://hamakor.org.il/cgi-bin/mailman/listinfo/haifux
Re: [Haifux] SVN quota
hi Shahar! A possible solution if your svn repository is stored on a filesystem could be a pre-commit hook script, that would basically: * check user's/project's repository quota against filesystem * act accordingly to the result, upon: * OK - do commit * Warning (quota threshold reached) - print the result, and commit * Quota exceeded - print the result and NOT commit Of course take care of the return values of the script properly. The script may even send an email to the user, if you have email in NIS. I do not know how different SVN clients would respond to the non-standard messages in the output of SVN client. This is what you will have to check for yourself. Regards, M. 2010/1/18 Shahar Dag d...@cs.technion.ac.il: Hello I have a LINUX server (RHE 4) that among other things serves as SVN server (the SVN server runs over apache). Users can access their repository via https, but they can't login to the server (not even with ssh). To access the SVN, users must supply their NIS user name password. Every SVN repository has it's own quota limitation. The question is: do SVN users have any way to test their quota? (I don't want to write a cron job that warns the users when they are over 80% of the quota) Thanks Shahar ___ Haifux mailing list Haifux@haifux.org http://hamakor.org.il/cgi-bin/mailman/listinfo/haifux -- Maxim Kovgan ___ Haifux mailing list Haifux@haifux.org http://hamakor.org.il/cgi-bin/mailman/listinfo/haifux
Re: [Haifux] SVN quota
Max Writing pre-commit script writing a cron-job that test the quota are almost equal effort. I am not sure which one is better. What I am looking for is a tool that will enable the users check the quota on the svn machine Shahar - Original Message - From: Maxim Kovgan kovg...@gmail.com To: Shahar Dag d...@cs.technion.ac.il Cc: Haifaux haifux@haifux.org Sent: Monday, January 18, 2010 9:56 AM Subject: Re: [Haifux] SVN quota hi Shahar! A possible solution if your svn repository is stored on a filesystem could be a pre-commit hook script, that would basically: * check user's/project's repository quota against filesystem * act accordingly to the result, upon: * OK - do commit * Warning (quota threshold reached) - print the result, and commit * Quota exceeded - print the result and NOT commit Of course take care of the return values of the script properly. The script may even send an email to the user, if you have email in NIS. I do not know how different SVN clients would respond to the non-standard messages in the output of SVN client. This is what you will have to check for yourself. Regards, M. 2010/1/18 Shahar Dag d...@cs.technion.ac.il: Hello I have a LINUX server (RHE 4) that among other things serves as SVN server (the SVN server runs over apache). Users can access their repository via https, but they can't login to the server (not even with ssh). To access the SVN, users must supply their NIS user name password. Every SVN repository has it's own quota limitation. The question is: do SVN users have any way to test their quota? (I don't want to write a cron job that warns the users when they are over 80% of the quota) Thanks Shahar ___ Haifux mailing list Haifux@haifux.org http://hamakor.org.il/cgi-bin/mailman/listinfo/haifux -- Maxim Kovgan ___ Haifux mailing list Haifux@haifux.org http://hamakor.org.il/cgi-bin/mailman/listinfo/haifux ___ Haifux mailing list Haifux@haifux.org http://hamakor.org.il/cgi-bin/mailman/listinfo/haifux
Re: [Haifux] SVN quota
well, I'm not aware of anything ready to use, but there might be one - a CGI/FastCGI or WSGI something. you're interested in just showing quota command output with proper args. If you choose Perl, you may use Quota; On Mon, Jan 18, 2010 at 12:03 PM, Shahar Dag d...@cs.technion.ac.il wrote: Max Writing pre-commit script writing a cron-job that test the quota are almost equal effort. I am not sure which one is better. What I am looking for is a tool that will enable the users check the quota on the svn machine Shahar - Original Message - From: Maxim Kovgan kovg...@gmail.com To: Shahar Dag d...@cs.technion.ac.il Cc: Haifaux haifux@haifux.org Sent: Monday, January 18, 2010 9:56 AM Subject: Re: [Haifux] SVN quota hi Shahar! A possible solution if your svn repository is stored on a filesystem could be a pre-commit hook script, that would basically: * check user's/project's repository quota against filesystem * act accordingly to the result, upon: * OK - do commit * Warning (quota threshold reached) - print the result, and commit * Quota exceeded - print the result and NOT commit Of course take care of the return values of the script properly. The script may even send an email to the user, if you have email in NIS. I do not know how different SVN clients would respond to the non-standard messages in the output of SVN client. This is what you will have to check for yourself. Regards, M. 2010/1/18 Shahar Dag d...@cs.technion.ac.il: Hello I have a LINUX server (RHE 4) that among other things serves as SVN server (the SVN server runs over apache). Users can access their repository via https, but they can't login to the server (not even with ssh). To access the SVN, users must supply their NIS user name password. Every SVN repository has it's own quota limitation. The question is: do SVN users have any way to test their quota? (I don't want to write a cron job that warns the users when they are over 80% of the quota) Thanks Shahar ___ Haifux mailing list Haifux@haifux.org http://hamakor.org.il/cgi-bin/mailman/listinfo/haifux -- Maxim Kovgan ___ Haifux mailing list Haifux@haifux.org http://hamakor.org.il/cgi-bin/mailman/listinfo/haifux ___ Haifux mailing list Haifux@haifux.org http://hamakor.org.il/cgi-bin/mailman/listinfo/haifux -- Maxim Kovgan ___ Haifux mailing list Haifux@haifux.org http://hamakor.org.il/cgi-bin/mailman/listinfo/haifux
Re: [Haifux] SVN quota
I think there is a problem to run the quota query from a web server, b/c the UID of the webserver process is not allowed to run quota stuff, and all the web technologies usually limit setuid() related stuff. And you don't want to run a CGI script as root, you really don't. So, the only relatively safe solution would be to build 2 scripts: 1. script to generate the quota database, and it should run once per 1-5 minutes 2. CGI/FastCGI/WSCGI script to query that database If users' repositories reside on the expense of their home directories, maybe a webmin/usermin web application allows to watch quota status with some nice module, SQLite looks like a good candidate, but even CSV is ok, as long as you make sure you use flock() like functions to lock the db file properly when you read/write, etc. Regards. On Mon, Jan 18, 2010 at 4:07 PM, Maxim Kovgan kovg...@gmail.com wrote: well, I'm not aware of anything ready to use, but there might be one - a CGI/FastCGI or WSGI something. you're interested in just showing quota command output with proper args. If you choose Perl, you may use Quota; On Mon, Jan 18, 2010 at 12:03 PM, Shahar Dag d...@cs.technion.ac.il wrote: Max Writing pre-commit script writing a cron-job that test the quota are almost equal effort. I am not sure which one is better. What I am looking for is a tool that will enable the users check the quota on the svn machine Shahar - Original Message - From: Maxim Kovgan kovg...@gmail.com To: Shahar Dag d...@cs.technion.ac.il Cc: Haifaux haifux@haifux.org Sent: Monday, January 18, 2010 9:56 AM Subject: Re: [Haifux] SVN quota hi Shahar! A possible solution if your svn repository is stored on a filesystem could be a pre-commit hook script, that would basically: * check user's/project's repository quota against filesystem * act accordingly to the result, upon: * OK - do commit * Warning (quota threshold reached) - print the result, and commit * Quota exceeded - print the result and NOT commit Of course take care of the return values of the script properly. The script may even send an email to the user, if you have email in NIS. I do not know how different SVN clients would respond to the non-standard messages in the output of SVN client. This is what you will have to check for yourself. Regards, M. 2010/1/18 Shahar Dag d...@cs.technion.ac.il: Hello I have a LINUX server (RHE 4) that among other things serves as SVN server (the SVN server runs over apache). Users can access their repository via https, but they can't login to the server (not even with ssh). To access the SVN, users must supply their NIS user name password. Every SVN repository has it's own quota limitation. The question is: do SVN users have any way to test their quota? (I don't want to write a cron job that warns the users when they are over 80% of the quota) Thanks Shahar ___ Haifux mailing list Haifux@haifux.org http://hamakor.org.il/cgi-bin/mailman/listinfo/haifux -- Maxim Kovgan ___ Haifux mailing list Haifux@haifux.org http://hamakor.org.il/cgi-bin/mailman/listinfo/haifux ___ Haifux mailing list Haifux@haifux.org http://hamakor.org.il/cgi-bin/mailman/listinfo/haifux -- Maxim Kovgan -- Maxim Kovgan ___ Haifux mailing list Haifux@haifux.org http://hamakor.org.il/cgi-bin/mailman/listinfo/haifux
Re: [Haifux] SVN quota
Maybe there is a way to have an ssh server which will always run quota and exit before starting the user's shell? That way SSHing into the server will never let a user do anything but knowing his quota. On Mon, Jan 18, 2010 at 8:27 PM, Maxim Kovgan kovg...@gmail.com wrote: I think there is a problem to run the quota query from a web server, b/c the UID of the webserver process is not allowed to run quota stuff, and all the web technologies usually limit setuid() related stuff. And you don't want to run a CGI script as root, you really don't. So, the only relatively safe solution would be to build 2 scripts: 1. script to generate the quota database, and it should run once per 1-5 minutes 2. CGI/FastCGI/WSCGI script to query that database If users' repositories reside on the expense of their home directories, maybe a webmin/usermin web application allows to watch quota status with some nice module, SQLite looks like a good candidate, but even CSV is ok, as long as you make sure you use flock() like functions to lock the db file properly when you read/write, etc. Regards. On Mon, Jan 18, 2010 at 4:07 PM, Maxim Kovgan kovg...@gmail.com wrote: well, I'm not aware of anything ready to use, but there might be one - a CGI/FastCGI or WSGI something. you're interested in just showing quota command output with proper args. If you choose Perl, you may use Quota; On Mon, Jan 18, 2010 at 12:03 PM, Shahar Dag d...@cs.technion.ac.il wrote: Max Writing pre-commit script writing a cron-job that test the quota are almost equal effort. I am not sure which one is better. What I am looking for is a tool that will enable the users check the quota on the svn machine Shahar - Original Message - From: Maxim Kovgan kovg...@gmail.com To: Shahar Dag d...@cs.technion.ac.il Cc: Haifaux haifux@haifux.org Sent: Monday, January 18, 2010 9:56 AM Subject: Re: [Haifux] SVN quota hi Shahar! A possible solution if your svn repository is stored on a filesystem could be a pre-commit hook script, that would basically: * check user's/project's repository quota against filesystem * act accordingly to the result, upon: * OK - do commit * Warning (quota threshold reached) - print the result, and commit * Quota exceeded - print the result and NOT commit Of course take care of the return values of the script properly. The script may even send an email to the user, if you have email in NIS. I do not know how different SVN clients would respond to the non-standard messages in the output of SVN client. This is what you will have to check for yourself. Regards, M. 2010/1/18 Shahar Dag d...@cs.technion.ac.il: Hello I have a LINUX server (RHE 4) that among other things serves as SVN server (the SVN server runs over apache). Users can access their repository via https, but they can't login to the server (not even with ssh). To access the SVN, users must supply their NIS user name password. Every SVN repository has it's own quota limitation. The question is: do SVN users have any way to test their quota? (I don't want to write a cron job that warns the users when they are over 80% of the quota) Thanks Shahar ___ Haifux mailing list Haifux@haifux.org http://hamakor.org.il/cgi-bin/mailman/listinfo/haifux -- Maxim Kovgan ___ Haifux mailing list Haifux@haifux.org http://hamakor.org.il/cgi-bin/mailman/listinfo/haifux ___ Haifux mailing list Haifux@haifux.org http://hamakor.org.il/cgi-bin/mailman/listinfo/haifux -- Maxim Kovgan -- Maxim Kovgan ___ Haifux mailing list Haifux@haifux.org http://hamakor.org.il/cgi-bin/mailman/listinfo/haifux ___ Haifux mailing list Haifux@haifux.org http://hamakor.org.il/cgi-bin/mailman/listinfo/haifux
Re: [Haifux] SVN quota
Shahar Dag wrote: Hello I have a LINUX server (RHE 4) that among other things serves as SVN server (the SVN server runs over apache). Users can access their repository via https, but they can't login to the server (not even with ssh). To access the SVN, users must supply their NIS user name password. Every SVN repository has it's own quota limitation. The question is: do SVN users have any way to test their quota? (I don't want to write a cron job that warns the users when they are over 80% of the quota) Excuse me for taking this on a tangent, but what use is the quota warning going to do? It's not as if users can delete stuff in order save space. Shachar -- Shachar Shemesh Lingnu Open Source Consulting Ltd. http://www.lingnu.com ___ Haifux mailing list Haifux@haifux.org http://hamakor.org.il/cgi-bin/mailman/listinfo/haifux
Re: [Haifux] SVN quota
Shalom Shcharim! Possibly, the repositories are located on the same storage device as users' homes, so they might be able to delete other files to release some space, but you're completely right, the repository can only grow up :-) 2010/1/18 Shachar Shemesh shac...@shemesh.biz: Shahar Dag wrote: Hello I have a LINUX server (RHE 4) that among other things serves as SVN server (the SVN server runs over apache). Users can access their repository via https, but they can't login to the server (not even with ssh). To access the SVN, users must supply their NIS user name password. Every SVN repository has it's own quota limitation. The question is: do SVN users have any way to test their quota? (I don't want to write a cron job that warns the users when they are over 80% of the quota) Excuse me for taking this on a tangent, but what use is the quota warning going to do? It's not as if users can delete stuff in order save space. Shachar -- Shachar Shemesh Lingnu Open Source Consulting Ltd. http://www.lingnu.com ___ Haifux mailing list Haifux@haifux.org http://hamakor.org.il/cgi-bin/mailman/listinfo/haifux -- Maxim Kovgan ___ Haifux mailing list Haifux@haifux.org http://hamakor.org.il/cgi-bin/mailman/listinfo/haifux
Re: [Haifux] SVN quota
Hello It is true that users can't delete data from the SVN. The only thing users can do is ask for additional quota. If they know that they are running out of quota in advanced, they can ask to enlarge the quota before they are out of disk space (which usually happens 01:00 ). Then I can check if the disk space usage is justify and respond on time. Shahar - Original Message - From: Shachar Shemesh To: Shahar Dag Cc: Haifaux Sent: Monday, January 18, 2010 10:01 PM Subject: Re: [Haifux] SVN quota Shahar Dag wrote: Hello I have a LINUX server (RHE 4) that among other things serves as SVN server (the SVN server runs over apache). Users can access their repository via https, but they can't login to the server (not even with ssh). To access the SVN, users must supply their NIS user name password. Every SVN repository has it's own quota limitation. The question is: do SVN users have any way to test their quota? (I don't want to write a cron job that warns the users when they are over 80% of the quota) Excuse me for taking this on a tangent, but what use is the quota warning going to do? It's not as if users can delete stuff in order save space. Shachar -- Shachar Shemesh Lingnu Open Source Consulting Ltd. http://www.lingnu.com -- ___ Haifux mailing list Haifux@haifux.org http://hamakor.org.il/cgi-bin/mailman/listinfo/haifux ___ Haifux mailing list Haifux@haifux.org http://hamakor.org.il/cgi-bin/mailman/listinfo/haifux
