Re: [vchkpw] vpopmail development
> Joshua Megerman wrote: >> One last note - the idea of maintaining a list of backfill "slots" in a >> text >> file is a pretty good one, but it still doesn't address the issue of not >> properly calculating the number of users in a directory... > > What are you referring to when you say it doesn't properly calculate the > number > of users? The current hashing structure keeps track of additions only. > Once users > are removed, it is no longer up to date with correct user counts. That's > what > we're addressing with this proposed patch. Hmm... you may be right (that's the problem with posting first thing in the morning). The issue probably is more of a factor for existing systems that have done lots of adds & deletes over time than for new installations. As long as you provide some sort of script to properly calculate the new backfill info file, I suppose the issue becomes moot. Of course, there's still the issue of overpopulated directories and moving them out, but that's a different problem... Josh Joshua Megerman SJGames MIB #5273 - OGRE AI Testing Division You can't win; You can't break even; You can't even quit the game. - Layman's translation of the Laws of Thermodynamics vpopm...@honorablemenschen.com !DSPAM:496b5efa32678902916388!
Re: [vchkpw] vpopmail development
Matt Brookings wrote: Remember that this feature does not yet exist, and that there are probably many systems with backfilling needs that go back years. Potentially this patch could hit a system with four levels of hashing simply because there's been a lot of additions and deletions. If the backfill patch doesn't take this into consideration, we may need to consider writing some sort of utility to analyze and "clean", a system that is "overhashed". My system would be one of those, here are the stats from just one domain after 4 years of use. I have been putting off hacking together a Perl script to move everything around and update the MySQL tables. Honestly, I cannot say there is any performance hit even with the dirs this messed up. [r...@newnfs:/usr/local/scripts/old-scripts]# ./dircheck.sh tls.net dir 0 ->35 dir 1 ->38 dir 2 ->36 dir 3 ->30 dir 4 ->32 dir 5 ->38 dir 6 ->32 dir 7 ->33 dir 8 ->38 dir 9 ->33 dir A ->31 dir B ->26 dir C ->45 dir D ->32 dir E ->32 dir F ->19 dir G ->36 dir H ->42 dir I ->39 dir J ->30 dir K ->34 dir L ->30 dir M ->33 dir N ->31 dir O ->33 dir P ->26 dir Q ->27 dir R ->24 dir S ->29 dir T ->31 dir U ->32 dir V ->25 dir W ->38 dir X ->45 dir Y ->30 dir Z ->30 dir a ->31 dir b ->11 dir c ->31 dir d ->36 dir e -> 3 dir f -> 2 dir g -> 5 dir h ->64 dir i ->14 dir j ->13 dir k ->13 dir l ->13 dir m ->26 dir n ->17 dir o ->36 dir p ->16 dir q ->17 dir r ->33 dir s ->23 dir t ->30 dir u -> 620 dir v -> 759 DAve -- The whole internet thing is sucking the life out of me, there ain't no pony in there. !DSPAM:496b560c32679005657564!
Re: [vchkpw] vpopmail development
Joshua Megerman wrote: > One last note - the idea of maintaining a list of backfill "slots" in a text > file is a pretty good one, but it still doesn't address the issue of not > properly calculating the number of users in a directory... What are you referring to when you say it doesn't properly calculate the number of users? The current hashing structure keeps track of additions only. Once users are removed, it is no longer up to date with correct user counts. That's what we're addressing with this proposed patch. -- /* Matt BrookingsGnuPG Key D9414F70 Software developer Systems technician Inter7 Internet Technologies, Inc. (815)776-9465 */
Re: [vchkpw] vpopmail development
ISP Lists wrote: > Can someone please provide a brief discussion as to when a vpopmail hashed > folder tree becomes "big enough" to warrant backfilling? Or, is "big" > just one concern amongst others such as: "rate of deletes and adds", > "filesystem choice"... > I'm not quite picking up why the backfill is important. You've got it backwards. Backfilling becomes important when adding users. vpopmail hashes directories at around 100 users per directory. It also does this with domain directories. The problem is that the hashing does not take user removal into account. If you add 1000 users, and delete the first 500, the hashing leaves empty hash directories and continues to add new ones, rather than re-using previously created hash directories that are no longer full. -- /* Matt BrookingsGnuPG Key D9414F70 Software developer Systems technician Inter7 Internet Technologies, Inc. (815)776-9465 */
Re: [vchkpw] vpopmail development
Manvendra Bhangui wrote: > Now let say I delete a user who has a directory > in /var/vpopmail/domains/1 > The backfill code will put the entry '1' in the first line in the file > dir_control_free. > > So after deleting 3 users, the file dir_control_free will have 3 lines > 1 > 2 > 2 > Each time the function backfill() is called it will deplete the file > dir_control_free by one line and will always return the first line as > the user_hash. When all lines get depleted, backfill() will return NULL > in which case the regular dir_control will again come into effect and > start from where it had left earlier. Okay. I can definitely see how this would work. It is a reasonable solution, and I'd be very interested to see a completed patch against the CVS head. The one comment I would make is that it's okay for user deletion to be an expensive call since it won't be being done nearly as much as queries for user information, but that depending upon the number of users a system has, for instance, where the hash levels have tripled up, the dir_control_free file would become very large and your solution requires a re-write of the file occasionally. It would be interesting to see a more efficient method where duplicates, as in your example, the hash directory 2, could be listed a single time. Remember that this feature does not yet exist, and that there are probably many systems with backfilling needs that go back years. Potentially this patch could hit a system with four levels of hashing simply because there's been a lot of additions and deletions. If the backfill patch doesn't take this into consideration, we may need to consider writing some sort of utility to analyze and "clean", a system that is "overhashed". > The advantage of this method is that you can use the find command to > generate the missing directories in dir_control_free to catch up with > the actual dir_control. > > Another way to explain this is that when backfill is in operation, > dir_control stops working and when backfill() gets depleted and stops > working, dir_control starts working Agreed. -- /* Matt BrookingsGnuPG Key D9414F70 Software developer Systems technician Inter7 Internet Technologies, Inc. (815)776-9465 */
Re: [vchkpw] vpopmail development
On Monday 12 January 2009 07:48:17 am ISP Lists wrote: > Can someone please provide a brief discussion as to when a vpopmail hashed > folder tree becomes "big enough" to warrant backfilling? Or, is "big" > just one concern amongst others such as: "rate of deletes and adds", > "filesystem choice"... > I'm not quite picking up why the backfill is important. > Well, I don't know what other people are considering "too big", but I actually wrote a backfill patch when I was working at a medium-sized college. We kept all 62 top-level hash directories on separate partitions, but didn't ever want to go to second level hashes - and with ~1200 adds (incoming freshmen) and deletes (outgoing seniors) every year, this became an issue pretty quick. The other issue with "backfill" is that the current implementation makes it so that you can easily exceed the 100 users per hash dir limit by deleting users from prior hash dirs and then adding new ones since the only check for a new hash dir is "total users/100". My patch and the reasons therefore can be found at http://sourceforge.net/tracker/index.php?func=detail&aid=1619600&group_id=85937&atid=577800. The main reason it's not currently slated for inclusion is that it's for the mysql backend only, and whatever process is used to provide backfill must be available for all backends. One last note - the idea of maintaining a list of backfill "slots" in a text file is a pretty good one, but it still doesn't address the issue of not properly calculating the number of users in a directory... Josh -- Joshua Megerman SJGames MIB #5273 - OGRE AI Testing Division You can't win; You can't break even; You can't even quit the game. - Layman's translation of the Laws of Thermodynamics j...@honorablemenschen.com !DSPAM:496b440e32671399810511!
Re: [vchkpw] vpopmail development
> On Fri, 2009-01-09 at 08:57 -0600, Matt Brookings wrote: >> This would not work because users can be deleted out of the hash tree >> anywhere. It appears your patch assumes a FILO ordering of user >> additions >> and deletions. > I have not been able to explain properly. It would be FIFO. > >> If the hashes, 'a' through 'd' existed, and the 'b' hash directory >> cleared >> out, your method would fail to backfill correctly. > Let's take an example > suppose > there are 100 users (with 100 directories) in /var/vpopmail/domains > there are 100 users (with 100 directories) in /var/vpopmail/domains/0 > there are 100 users (with 100 directories) in /var/vpopmail/domains/1 > there are 100 users (with 100 directories) in /var/vpopmail/domains/2 > there are 50 users (with 50 direcotires) in /var/vpopmail/domains/3 > > Now let say I delete a user who has a directory > in /var/vpopmail/domains/1 > The backfill code will put the entry '1' in the first line in the file > dir_control_free. > Also let us say that we delete two users in /var/vpopmail/domains/2 > The backfill code in vdeluser will put entry '2' twice in the file > dir_control_free > > So after deleting 3 users, the file dir_control_free will have 3 lines > 1 > 2 > 2 > > > So now we have 99 users in /var/vpopmail/domains/1 > andwe have 98 users in /var/vpopmail/domains/2 > > Now the modified vadduser will call a function called backfill() which > will open this file, lock the file and pickup the first line, delete the > line and return the value as user_hash > > #ifdef USERS_BIG_DIR > /* go into a user hash dir if required */ > if (!(user_hash = backfill(domain))) > { > open_big_dir(domain, uid, gid); > user_hash = next_big_dir(uid, gid); > close_big_dir(domain, uid, gid); > chdir(user_hash); > } > #endif > > Each time the function backfill() is called it will deplete the file > dir_control_free by one line and will always return the first line as > the user_hash. When all lines get depleted, backfill() will return NULL > in which case the regular dir_control will again come into effect and > start from where it had left earlier. > > The advantage of this method is that you can use the find command to > generate the missing directories in dir_control_free to catch up with > the actual dir_control. > > Another way to explain this is that when backfill is in operation, > dir_control stops working and when backfill() gets depleted and stops > working, dir_control starts working > > > > > Can someone please provide a brief discussion as to when a vpopmail hashed folder tree becomes "big enough" to warrant backfilling? Or, is "big" just one concern amongst others such as: "rate of deletes and adds", "filesystem choice"... I'm not quite picking up why the backfill is important. !DSPAM:496b3c1332671349816825!
Re: [vchkpw] vpopmail development
On Fri, 2009-01-09 at 08:57 -0600, Matt Brookings wrote: > This would not work because users can be deleted out of the hash tree > anywhere. It appears your patch assumes a FILO ordering of user additions > and deletions. I have not been able to explain properly. It would be FIFO. > If the hashes, 'a' through 'd' existed, and the 'b' hash directory cleared > out, your method would fail to backfill correctly. Let's take an example suppose there are 100 users (with 100 directories) in /var/vpopmail/domains there are 100 users (with 100 directories) in /var/vpopmail/domains/0 there are 100 users (with 100 directories) in /var/vpopmail/domains/1 there are 100 users (with 100 directories) in /var/vpopmail/domains/2 there are 50 users (with 50 direcotires) in /var/vpopmail/domains/3 Now let say I delete a user who has a directory in /var/vpopmail/domains/1 The backfill code will put the entry '1' in the first line in the file dir_control_free. Also let us say that we delete two users in /var/vpopmail/domains/2 The backfill code in vdeluser will put entry '2' twice in the file dir_control_free So after deleting 3 users, the file dir_control_free will have 3 lines 1 2 2 So now we have 99 users in /var/vpopmail/domains/1 andwe have 98 users in /var/vpopmail/domains/2 Now the modified vadduser will call a function called backfill() which will open this file, lock the file and pickup the first line, delete the line and return the value as user_hash #ifdef USERS_BIG_DIR /* go into a user hash dir if required */ if (!(user_hash = backfill(domain))) { open_big_dir(domain, uid, gid); user_hash = next_big_dir(uid, gid); close_big_dir(domain, uid, gid); chdir(user_hash); } #endif Each time the function backfill() is called it will deplete the file dir_control_free by one line and will always return the first line as the user_hash. When all lines get depleted, backfill() will return NULL in which case the regular dir_control will again come into effect and start from where it had left earlier. The advantage of this method is that you can use the find command to generate the missing directories in dir_control_free to catch up with the actual dir_control. Another way to explain this is that when backfill is in operation, dir_control stops working and when backfill() gets depleted and stops working, dir_control starts working !DSPAM:496b235e32678184414047!
Re: [vchkpw] vpopmail development
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Manvendra Bhangui wrote: > 1. vdeluser appends the directory being deleted in a MySQL table (for >file for cdb backend). The latest directory deleted will be at the >end. Let's say this filename is dir_control_free This would not work because users can be deleted out of the hash tree anywhere. It appears your patch assumes a FILO ordering of user additions and deletions. > 2. vaddduser locks this file (dir_control_free). Picks up the first >entry and assigns the new user the first prefix and after creating >the user and the Maildir, deletes the line. No changes will be made >to dir_control if a entry exists in this file. When all entries are >backfilled, dir_control will again get used. If the hashes, 'a' through 'd' existed, and the 'b' hash directory cleared out, your method would fail to backfill correctly. Furthermore, if I'm assuming other details correctly, it seems as if dir_control would never pick up again because you would never backfill 'all entries' because dir_control and dir_control_free would never match. If you have some further details, please let me know. I don't believe that the current hashing solution will support backfilling without major hacks. It may make sense to re-implement hashing, keeping backfilling in mind. Fortunately the dir_control method is quite simple, and in the event a new hashing method is devised, conversion to a newer method should be rather straightforward. - -- /* Matt BrookingsGnuPG Key D9414F70 Software developer Systems technician Inter7 Internet Technologies, Inc. (815)776-9465 */ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJZ2XF6QgvSNlBT3ARAi5XAJ9VFl6ajjM48TyTBnR/6s5/wleU3gCfRl2z WOnitBBSXhg9P8iI3FB8Xog= =bhWd -END PGP SIGNATURE-
Re: [vchkpw] vpopmail development
On Thu, 2009-01-08 at 17:10 -0500, DAve wrote: > My reward for fixing the dir_control to backfill still stands. > > http://pixelhammer.com/Dan/aargh.jpg > > Follow the thread... > http://www.mail-archive.com/vchkpw@inter7.com/msg25272.html > > I could improve on the prize some as well, how about two dozen award > winning homemade cookies and a snapped drive shaft u-joint from the > Seattle Monorail? Following the thread above I have the following idea on how to achieve this 1. vdeluser appends the directory being deleted in a MySQL table (for file for cdb backend). The latest directory deleted will be at the end. Let's say this filename is dir_control_free 2. vaddduser locks this file (dir_control_free). Picks up the first entry and assigns the new user the first prefix and after creating the user and the Maildir, deletes the line. No changes will be made to dir_control if a entry exists in this file. When all entries are backfilled, dir_control will again get used. I will work on this and have a patch ready for vpopmail and incorporate in Indimail - http://www.indimail.org !DSPAM:4966e0e232671449321458!
Re: [vchkpw] vpopmail development
Matt Brookings wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Remo Mattei wrote: I agree with you.. So I hope this stays. vpopmail and qmailadmin development are high on our list of items this year. We're finishing up leftover items from the final quarter of 2008, and then we're to schedule vpopmail and qmailadmin development to start the first quarter of 2009. Cheers! My reward for fixing the dir_control to backfill still stands. http://pixelhammer.com/Dan/aargh.jpg Follow the thread... http://www.mail-archive.com/vchkpw@inter7.com/msg25272.html I could improve on the prize some as well, how about two dozen award winning homemade cookies and a snapped drive shaft u-joint from the Seattle Monorail? DAve -- The whole internet thing is sucking the life out of me, there ain't no pony in there. !DSPAM:49667b6832671673621925!
Re: [vchkpw] vpopmail development
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Remo Mattei wrote: > I agree with you.. So I hope this stays. vpopmail and qmailadmin development are high on our list of items this year. We're finishing up leftover items from the final quarter of 2008, and then we're to schedule vpopmail and qmailadmin development to start the first quarter of 2009. Cheers! - -- /* Matt BrookingsGnuPG Key D9414F70 Software developer Systems technician Inter7 Internet Technologies, Inc. (815)776-9465 */ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJZmpK6QgvSNlBT3ARAiPnAJ0ek/y2dv3HpXoW5z0v/lCM7DVN7wCeNg1B Ko5POOjjvjvFYqC6tltzMR4= =dQYF -END PGP SIGNATURE-
Re: [vchkpw] vpopmail development
I agree with you.. So I hope this stays. Remo DAve On 1/8/09 1:56 PM, "DAve" wrote: > Remo Mattei wrote: >> Are people moving to a different options? Just wonder.. And if they are what >> are those options? > > I hope not, I've not seen anything that comes close to the abilities of > vpopmail. We are preparing to completely overhaul our backend server and > integrate vpopmaild into our intranet. > > DAve > >> >> Thanks, >> Remo >> >> >> Tom Collins On 1/8/09 12:55 PM, "Tom Collins" wrote: >> >>> On Jan 6, 2009, at 3:54 AM, aledr wrote: There's a lot of patches on SourceForge tracker to be accepted and I have my own patches and improvements for vpopmail that I would like to see in the package. There's no activity on the SF CVS for months, is It in active development yet? >>> >>> It's not in active development. All of the past "lead developers" on >>> the project (including myself) have been too busy to dedicate any >>> time to it. >>> >>> Some of the patches on SourceForge won't ever go into the mainstream >>> releases (as least as far as I'm concerned), but remain available for >>> those who want to add that functionality. >>> >>> Unfortunately, doing anything with vpopmail and qmailadmin has moved >>> very far down my to do list. >>> >>> -Tom >>> >>> >>> >>> >>> >>> >>> >> >> >> >> >> >> > !DSPAM:496668f732671796021211!
Re: [vchkpw] vpopmail development
Remo Mattei wrote: Are people moving to a different options? Just wonder.. And if they are what are those options? I hope not, I've not seen anything that comes close to the abilities of vpopmail. We are preparing to completely overhaul our backend server and integrate vpopmaild into our intranet. DAve Thanks, Remo Tom Collins On 1/8/09 12:55 PM, "Tom Collins" wrote: On Jan 6, 2009, at 3:54 AM, aledr wrote: There's a lot of patches on SourceForge tracker to be accepted and I have my own patches and improvements for vpopmail that I would like to see in the package. There's no activity on the SF CVS for months, is It in active development yet? It's not in active development. All of the past "lead developers" on the project (including myself) have been too busy to dedicate any time to it. Some of the patches on SourceForge won't ever go into the mainstream releases (as least as far as I'm concerned), but remain available for those who want to add that functionality. Unfortunately, doing anything with vpopmail and qmailadmin has moved very far down my to do list. -Tom -- The whole internet thing is sucking the life out of me, there ain't no pony in there. !DSPAM:4966689e32671466455160!
Re: [vchkpw] vpopmail development
Are people moving to a different options? Just wonder.. And if they are what are those options? Thanks, Remo Tom Collins On 1/8/09 12:55 PM, "Tom Collins" wrote: > On Jan 6, 2009, at 3:54 AM, aledr wrote: >> There's a lot of patches on SourceForge tracker to be accepted and I >> have my own patches and improvements for vpopmail that I would like to >> see in the package. >> There's no activity on the SF CVS for months, is It in active >> development yet? > > > It's not in active development. All of the past "lead developers" on > the project (including myself) have been too busy to dedicate any > time to it. > > Some of the patches on SourceForge won't ever go into the mainstream > releases (as least as far as I'm concerned), but remain available for > those who want to add that functionality. > > Unfortunately, doing anything with vpopmail and qmailadmin has moved > very far down my to do list. > > -Tom > > > > > > > !DSPAM:49665acf32671042813611!
Re: [vchkpw] vpopmail development
On Jan 6, 2009, at 3:54 AM, aledr wrote: There's a lot of patches on SourceForge tracker to be accepted and I have my own patches and improvements for vpopmail that I would like to see in the package. There's no activity on the SF CVS for months, is It in active development yet? It's not in active development. All of the past "lead developers" on the project (including myself) have been too busy to dedicate any time to it. Some of the patches on SourceForge won't ever go into the mainstream releases (as least as far as I'm concerned), but remain available for those who want to add that functionality. Unfortunately, doing anything with vpopmail and qmailadmin has moved very far down my to do list. -Tom !DSPAM:49665a2332672864611840!
[vchkpw] vpopmail development
Good day! There's a lot of patches on SourceForge tracker to be accepted and I have my own patches and improvements for vpopmail that I would like to see in the package. There's no activity on the SF CVS for months, is It in active development yet? Thanks. -- [ ]'s Aledr - Alexandre "OpenSource Solutions for SmallBusiness Problems" !DSPAM:4963469132677243920070!
Re: [vchkpw] vpopmail development
On Wednesday, October 23, 2002, at 12:51 PM, Matthias Trevarthan wrote: Granted, vpopmail does a great job as is... I'd just hate to see this project die for lack of renovation someday. What's the licensing like? Could I modify the source and create another project? (Not that I want to... trust me. I've got plenty of better things to do.) GPL. This is easily found on at on the vpopmail website, and in the source code. Bill
Re: [vchkpw] vpopmail development
Granted, vpopmail does a great job as is... I'd just hate to see this project die for lack of renovation someday. What's the licensing like? Could I modify the source and create another project? (Not that I want to... trust me. I've got plenty of better things to do.) Matthias On Wednesday 23 October 2002 13:41, Bill Shupp wrote: > On Wednesday, October 23, 2002, at 07:03 AM, Matthias Trevarthan wrote: > > Howdy, > > > > Just curious: Is vpopmail still being actively developed? > > Yes. > > > Is it a single developer, or is there a CVS and an open development > > community? > > > > I ask because I see people talking about patches a lot. If we had a > > CVS and an > > active development team, we wouldn't need patches, would we? > > > > Isn't that the strength of Open Source? The product evolves very > > quickly in > > response to it's user's needs. This gets foiled if there is only one > > developer... > > Ken Jones and Inter7 are the maintainers. I've been helping out with > putting together releases and adding patches, etc.., but this decision > is really up to Ken. Several people have offered to run CVS, and of > course there's sourceforge. I'm not opposed to this. > > Regards, > > Bill Shupp
Re: [vchkpw] vpopmail development
On Wednesday, October 23, 2002, at 07:03 AM, Matthias Trevarthan wrote: Howdy, Just curious: Is vpopmail still being actively developed? Yes. Is it a single developer, or is there a CVS and an open development community? I ask because I see people talking about patches a lot. If we had a CVS and an active development team, we wouldn't need patches, would we? Isn't that the strength of Open Source? The product evolves very quickly in response to it's user's needs. This gets foiled if there is only one developer... Ken Jones and Inter7 are the maintainers. I've been helping out with putting together releases and adding patches, etc.., but this decision is really up to Ken. Several people have offered to run CVS, and of course there's sourceforge. I'm not opposed to this. Regards, Bill Shupp
[vchkpw] vpopmail development
Howdy, Just curious: Is vpopmail still being actively developed? Is it a single developer, or is there a CVS and an open development community? I ask because I see people talking about patches a lot. If we had a CVS and an active development team, we wouldn't need patches, would we? Isn't that the strength of Open Source? The product evolves very quickly in response to it's user's needs. This gets foiled if there is only one developer... Matthias