Re: [vchkpw] Why is vadduser creating a hierarchy?
Bert JW Regeer wrote: Hey Ismail, I would like to see some stats on this. Do you have any facts or evidence to back this up? It's true - if you don't have some sort of directory-hashing (UFS_DIRHASH in FreeBSD-land) in place, which for FreeBSD has been default since sometime in the early 4.x days, IIRC. On current systems 7000 directories inside a directory should not be a problem. Most employ hashing of some sort to speed up this kind of thing. On my FreeBSD system there is currently a directory with 10,010 directories, and it is no slower than if that same directory had only 128 directories in it for example. Several of my users are on several mailing lists for open source projects, and some of their Maildir's have cur directories with over 30,000 emails in them. Biggest one is 150,000, with no slow downs. No extra load on my server. DJB gave qmail's queue split directories, See above. DJB was or is a (Free)-BSD user (when he started, Linux was a toy anyway), which back in these days had this problem. why I do not understand, and I might never, since clearly he did not create his Maildir's to have the same sort of split directories for speedy access by IMAP/POP3 or other mail protocols. I always disable vpopmail's big dir stuff, as writing scripts for it is harder, extra sub directories to traverse. Just use the output of vuserinfo -d BTW: Does the latest version of vpopmail include the patch someone posted that fills up earlier hash-directories, where domains have been deleted from, instead of creating new ones? cheers, Rainer
Re: [vchkpw] Why is vadduser creating a hierarchy?
On Oct 14, 2006, at 16:06:30 MST, Rainer Duffner wrote: Bert JW Regeer wrote: Hey Ismail, I would like to see some stats on this. Do you have any facts or evidence to back this up? It's true - if you don't have some sort of directory-hashing (UFS_DIRHASH in FreeBSD-land) in place, which for FreeBSD has been default since sometime in the early 4.x days, IIRC. We are talking about new systems that exist right now. There is no need to do dirhashing in an application anymore. On current systems 7000 directories inside a directory should not be a problem. Most employ hashing of some sort to speed up this kind of thing. On my FreeBSD system there is currently a directory with 10,010 directories, and it is no slower than if that same directory had only 128 directories in it for example. Several of my users are on several mailing lists for open source projects, and some of their Maildir's have cur directories with over 30,000 emails in them. Biggest one is 150,000, with no slow downs. No extra load on my server. DJB gave qmail's queue split directories, See above. DJB was or is a (Free)-BSD user (when he started, Linux was a toy anyway), which back in these days had this problem. Agreed, however his Maildir approach did not include hashing in any way shape or form, so how did file systems back then handle over 1000's of email messages in an Inbox? why I do not understand, and I might never, since clearly he did not create his Maildir's to have the same sort of split directories for speedy access by IMAP/POP3 or other mail protocols. I always disable vpopmail's big dir stuff, as writing scripts for it is harder, extra sub directories to traverse. Just use the output of vuserinfo -d Not always what I need. BTW: Does the latest version of vpopmail include the patch someone posted that fills up earlier hash-directories, where domains have been deleted from, instead of creating new ones? cheers, Rainer Greets, Bert JW Regeer smime.p7s Description: S/MIME cryptographic signature
Re: [vchkpw] Why is vadduser creating a hierarchy?
On Saturday 14 October 2006 19:06, Rainer Duffner wrote: BTW: Does the latest version of vpopmail include the patch someone posted that fills up earlier hash-directories, where domains have been deleted from, instead of creating new ones? Nope, it doesn't (I wrote the patch, and last I heard it hadn't even been considered... ) I posted the patch here a while back - while it may take a little hacking, it should work fine with the latest vpopmail. Josh -- Joshua Megerman SJGames MIB #5273 - OGRE AI Testing Division You can't win; You can't break even; You can't even quit the game. - Layman's translation of the Laws of Thermodynamics [EMAIL PROTECTED]
Re: [vchkpw] Why is vadduser creating a hierarchy?
On Oct 14, 2006, at 4:16 PM, Joshua Megerman wrote: Nope, it doesn't (I wrote the patch, and last I heard it hadn't even been considered... ) I posted the patch here a while back - while it may take a little hacking, it should work fine with the latest vpopmail. Please re-email it directly to me and I'll put it in my queue of patched for vpopmail. A slew of them were just posted to SourceForge, so I'll probably try to get a 5.4.18 release out before too long. -- Tom Collins - [EMAIL PROTECTED] Vpopmail - virtual domains for qmail: http://vpopmail.sf.net/ QmailAdmin - web interface for Vpopmail: http://qmailadmin.sf.net/
Re: [vchkpw] Why is vadduser creating a hierarchy?
On 10/14/06, Bert JW Regeer [EMAIL PROTECTED] wrote: On Oct 14, 2006, at 16:06:30 MST, Rainer Duffner wrote: See above. DJB was or is a (Free)-BSD user (when he started, Linux was a toy anyway), which back in these days had this problem. Agreed, however his Maildir approach did not include hashing in any way shape or form, so how did file systems back then handle over 1000's of email messages in an Inbox? Poorly. I had a server stall after a user stopped checking mail and accumulated 100,000 messages in a single folder. The filesystem was basically unusable. That was on FreeBSD 4.6, IIRC. Once I cleared that up, the mysterious slowness in general of the machine disappeared. -- Jon
Re: Re[2]: [vchkpw] Why is vadduser creating a hierarchy?
Hey Ismail, I would like to see some stats on this. Do you have any facts or evidence to back this up? On current systems 7000 directories inside a directory should not be a problem. Most employ hashing of some sort to speed up this kind of thing. On my FreeBSD system there is currently a directory with 10,010 directories, and it is no slower than if that same directory had only 128 directories in it for example. Several of my users are on several mailing lists for open source projects, and some of their Maildir's have cur directories with over 30,000 emails in them. Biggest one is 150,000, with no slow downs. No extra load on my server. DJB gave qmail's queue split directories, why I do not understand, and I might never, since clearly he did not create his Maildir's to have the same sort of split directories for speedy access by IMAP/POP3 or other mail protocols. I always disable vpopmail's big dir stuff, as writing scripts for it is harder, extra sub directories to traverse. Bert JW Regeer On Oct 7, 2006, at 19:53:36 MST, Ismail YENIGUL wrote: Dave, Please note that creating 7000 sub directories in a single directory will effect your performance negatively. Friday, October 6, 2006, 11:50:26 PM, you wrote: Rick Macdougall wrote: Dave Richardson wrote: I'm using a script to add thousands of user accounts as part of a migration for a single domain. It's a perl script making repeated calls to /home/vpopmail/bin/vadduser -e dsfgskjghaekjrgkr [EMAIL PROTECTED] The scripting is working fine, I see the accounts correctly in MySQL's vpopmail table. However, I'm seeing vadduser create a hierarchy of folders after about the first 80-100 users are added. Using subfolders A-z,0-9. I only have about 7,000 users to manage and would rather NOT subtree (whatever the term is) this user hierarchy. What logic controls when vadduser decides to subtree the folders for a particular domain? Or, should I just let my script run out all the migrations, create the user/Maildirs wherever, and then start moving them to the root of the domain folder? That leaves some nasty work in SQL to clean up the home folder field! Hi, Configure vpopmail with --disable-users-big-dir. --disable-users-big-dirDisable hashing of user directories. Regards, Rick Thanks Rick and Jon! -- Ismail YENIGUL Proje Yöneticisi / Project Manager [EMAIL PROTECTED] http://www.endersys.com smime.p7s Description: S/MIME cryptographic signature
Re[2]: [vchkpw] Why is vadduser creating a hierarchy?
Dave, vuserinfo -d [EMAIL PROTECTED] will give you the user's Maildir directory. # ~vpopmail/bin/vuserinfo -d [EMAIL PROTECTED] /home/vpopmail/domains/domain.com/postmaster You can add the following lines to the your script (I don't know what kind of script (perl, shell etc) do you use, so I will write a shell code. newdir=`vuserinfo -d [EMAIL PROTECTED]; oldir=/nfsmountdir/domainname/user; # don't delete the users under ~vpopmail/domains/ base dir if [ ! $newdir == $oldir ]; then rm -fr $newdir mv $oldir $newdir chown -R vpopmail:vchkpw $newdir fi I wrote this script to give you an idea. Please try it in your lab and Use at your own risk! I highly suggest you to take a backup of your existing data. Sunday, October 8, 2006, 11:56:47 PM, you wrote: Sure, I've tried to get vconvert to work on the source server, but it's a really old version of vpopmail and the files for the users on an NFS mount in a non-standard folder. There are other issues with the source configuration, but in short, they have all 7000 users in one primary domain folder. I have written a script (you know that now) that is parsing this source folder, then parsing the /etc/shadow file, and creating a series of commands to directly call /home/vpopmail/vadduser with the required values (per user) for every user. So the problem is that I've got to solve the sync of the user Maildirs once we're ready to migrate completely. My sources are in a sinpgle domain folder, now my targets are in a single domain folder. Once I rsync the folder data, I'm essentially ready to go. My thinking was that I could repair the big-dirs issue after the fact by slowly moving users' Maildirs from the domain folder to a domain subfolder. The alternative solution would be to find a much smarter way to rysnc the data based upon where the big-dirs enabled target migration puts the users' maildirs. I suppose I could go down that road too, scripting for it, but I suspect the rsync activity would take considerably longer than the few hours it does now. We're moving almost 25G of email, albeit within a local LAN. Appreciate your consideration of options, opinions welcome! Ismail YENIGUL wrote: Dave, What is the problem with big-dirs while migrating the users? If you can tell us the reason of the disabling big-dirs, We can try to find out a solution without disabling big-dirs. I guess it is related with the script? Sunday, October 8, 2006, 11:29:56 PM, you wrote: OK, fair point. So let me ask this please. If I migrate with big-dirs DISABLED, then I recompile to enable big-dirs AND I move some accounts into subfolders a, b, c, etc while making appropriate adjustments in vpopmail table, is that an appropriate way to control this risk? THANKS! Ismail YENIGUL wrote: Dave, Please note that creating 7000 sub directories in a single directory will effect your performance negatively. Friday, October 6, 2006, 11:50:26 PM, you wrote: Rick Macdougall wrote: Dave Richardson wrote: I'm using a script to add thousands of user accounts as part of a migration for a single domain. It's a perl script making repeated calls to /home/vpopmail/bin/vadduser -e dsfgskjghaekjrgkr [EMAIL PROTECTED] The scripting is working fine, I see the accounts correctly in MySQL's vpopmail table. However, I'm seeing vadduser create a hierarchy of folders after about the first 80-100 users are added. Using subfolders A-z,0-9. I only have about 7,000 users to manage and would rather NOT subtree (whatever the term is) this user hierarchy. What logic controls when vadduser decides to subtree the folders for a particular domain? Or, should I just let my script run out all the migrations, create the user/Maildirs wherever, and then start moving them to the root of the domain folder? That leaves some nasty work in SQL to clean up the home folder field! Hi, Configure vpopmail with --disable-users-big-dir. --disable-users-big-dirDisable hashing of user directories. Regards, Rick Thanks Rick and Jon! -- Ismail YENIGUL [EMAIL PROTECTED] http://www.endersys.com
Re: [vchkpw] Why is vadduser creating a hierarchy?
OK, fair point. So let me ask this please. If I migrate with big-dirs DISABLED, then I recompile to enable big-dirs AND I move some accounts into subfolders a, b, c, etc while making appropriate adjustments in vpopmail table, is that an appropriate way to control this risk? THANKS! Ismail YENIGUL wrote: Dave, Please note that creating 7000 sub directories in a single directory will effect your performance negatively. Friday, October 6, 2006, 11:50:26 PM, you wrote: Rick Macdougall wrote: Dave Richardson wrote: I'm using a script to add thousands of user accounts as part of a migration for a single domain. It's a perl script making repeated calls to /home/vpopmail/bin/vadduser -e dsfgskjghaekjrgkr [EMAIL PROTECTED] The scripting is working fine, I see the accounts correctly in MySQL's vpopmail table. However, I'm seeing vadduser create a hierarchy of folders after about the first 80-100 users are added. Using subfolders A-z,0-9. I only have about 7,000 users to manage and would rather NOT subtree (whatever the term is) this user hierarchy. What logic controls when vadduser decides to subtree the folders for a particular domain? Or, should I just let my script run out all the migrations, create the user/Maildirs wherever, and then start moving them to the root of the domain folder? That leaves some nasty work in SQL to clean up the home folder field! Hi, Configure vpopmail with --disable-users-big-dir. --disable-users-big-dirDisable hashing of user directories. Regards, Rick Thanks Rick and Jon!
Re[2]: [vchkpw] Why is vadduser creating a hierarchy?
Dave, What is the problem with big-dirs while migrating the users? If you can tell us the reason of the disabling big-dirs, We can try to find out a solution without disabling big-dirs. I guess it is related with the script? Sunday, October 8, 2006, 11:29:56 PM, you wrote: OK, fair point. So let me ask this please. If I migrate with big-dirs DISABLED, then I recompile to enable big-dirs AND I move some accounts into subfolders a, b, c, etc while making appropriate adjustments in vpopmail table, is that an appropriate way to control this risk? THANKS! Ismail YENIGUL wrote: Dave, Please note that creating 7000 sub directories in a single directory will effect your performance negatively. Friday, October 6, 2006, 11:50:26 PM, you wrote: Rick Macdougall wrote: Dave Richardson wrote: I'm using a script to add thousands of user accounts as part of a migration for a single domain. It's a perl script making repeated calls to /home/vpopmail/bin/vadduser -e dsfgskjghaekjrgkr [EMAIL PROTECTED] The scripting is working fine, I see the accounts correctly in MySQL's vpopmail table. However, I'm seeing vadduser create a hierarchy of folders after about the first 80-100 users are added. Using subfolders A-z,0-9. I only have about 7,000 users to manage and would rather NOT subtree (whatever the term is) this user hierarchy. What logic controls when vadduser decides to subtree the folders for a particular domain? Or, should I just let my script run out all the migrations, create the user/Maildirs wherever, and then start moving them to the root of the domain folder? That leaves some nasty work in SQL to clean up the home folder field! Hi, Configure vpopmail with --disable-users-big-dir. --disable-users-big-dirDisable hashing of user directories. Regards, Rick Thanks Rick and Jon! -- Ismail YENIGUL Proje Yöneticisi / Project Manager [EMAIL PROTECTED] http://www.endersys.com
Re: [vchkpw] Why is vadduser creating a hierarchy?
Sure, I've tried to get vconvert to work on the source server, but it's a really old version of vpopmail and the files for the users on an NFS mount in a non-standard folder. There are other issues with the source configuration, but in short, they have all 7000 users in one primary domain folder. I have written a script (you know that now) that is parsing this source folder, then parsing the /etc/shadow file, and creating a series of commands to directly call /home/vpopmail/vadduser with the required values (per user) for every user. So the problem is that I've got to solve the sync of the user Maildirs once we're ready to migrate completely. My sources are in a single domain folder, now my targets are in a single domain folder. Once I rsync the folder data, I'm essentially ready to go. My thinking was that I could repair the big-dirs issue after the fact by slowly moving users' Maildirs from the domain folder to a domain subfolder. The alternative solution would be to find a much smarter way to rysnc the data based upon where the big-dirs enabled target migration puts the users' maildirs. I suppose I could go down that road too, scripting for it, but I suspect the rsync activity would take considerably longer than the few hours it does now. We're moving almost 25G of email, albeit within a local LAN. Appreciate your consideration of options, opinions welcome! Ismail YENIGUL wrote: Dave, What is the problem with big-dirs while migrating the users? If you can tell us the reason of the disabling big-dirs, We can try to find out a solution without disabling big-dirs. I guess it is related with the script? Sunday, October 8, 2006, 11:29:56 PM, you wrote: OK, fair point. So let me ask this please. If I migrate with big-dirs DISABLED, then I recompile to enable big-dirs AND I move some accounts into subfolders a, b, c, etc while making appropriate adjustments in vpopmail table, is that an appropriate way to control this risk? THANKS! Ismail YENIGUL wrote: Dave, Please note that creating 7000 sub directories in a single directory will effect your performance negatively. Friday, October 6, 2006, 11:50:26 PM, you wrote: Rick Macdougall wrote: Dave Richardson wrote: I'm using a script to add thousands of user accounts as part of a migration for a single domain. It's a perl script making repeated calls to /home/vpopmail/bin/vadduser -e dsfgskjghaekjrgkr [EMAIL PROTECTED] The scripting is working fine, I see the accounts correctly in MySQL's vpopmail table. However, I'm seeing vadduser create a hierarchy of folders after about the first 80-100 users are added. Using subfolders A-z,0-9. I only have about 7,000 users to manage and would rather NOT subtree (whatever the term is) this user hierarchy. What logic controls when vadduser decides to subtree the folders for a particular domain? Or, should I just let my script run out all the migrations, create the user/Maildirs wherever, and then start moving them to the root of the domain folder? That leaves some nasty work in SQL to clean up the home folder field! Hi, Configure vpopmail with --disable-users-big-dir. --disable-users-big-dirDisable hashing of user directories. Regards, Rick Thanks Rick and Jon!
Re[2]: [vchkpw] Why is vadduser creating a hierarchy?
Dave, Please note that creating 7000 sub directories in a single directory will effect your performance negatively. Friday, October 6, 2006, 11:50:26 PM, you wrote: Rick Macdougall wrote: Dave Richardson wrote: I'm using a script to add thousands of user accounts as part of a migration for a single domain. It's a perl script making repeated calls to /home/vpopmail/bin/vadduser -e dsfgskjghaekjrgkr [EMAIL PROTECTED] The scripting is working fine, I see the accounts correctly in MySQL's vpopmail table. However, I'm seeing vadduser create a hierarchy of folders after about the first 80-100 users are added. Using subfolders A-z,0-9. I only have about 7,000 users to manage and would rather NOT subtree (whatever the term is) this user hierarchy. What logic controls when vadduser decides to subtree the folders for a particular domain? Or, should I just let my script run out all the migrations, create the user/Maildirs wherever, and then start moving them to the root of the domain folder? That leaves some nasty work in SQL to clean up the home folder field! Hi, Configure vpopmail with --disable-users-big-dir. --disable-users-big-dirDisable hashing of user directories. Regards, Rick Thanks Rick and Jon! -- Ismail YENIGUL Proje Yöneticisi / Project Manager [EMAIL PROTECTED] http://www.endersys.com
[vchkpw] Why is vadduser creating a hierarchy?
I'm using a script to add thousands of user accounts as part of a migration for a single domain. It's a perl script making repeated calls to /home/vpopmail/bin/vadduser -e dsfgskjghaekjrgkr [EMAIL PROTECTED] The scripting is working fine, I see the accounts correctly in MySQL's vpopmail table. However, I'm seeing vadduser create a hierarchy of folders after about the first 80-100 users are added. Using subfolders A-z,0-9. I only have about 7,000 users to manage and would rather NOT subtree (whatever the term is) this user hierarchy. What logic controls when vadduser decides to subtree the folders for a particular domain? Or, should I just let my script run out all the migrations, create the user/Maildirs wherever, and then start moving them to the root of the domain folder? That leaves some nasty work in SQL to clean up the home folder field!
Re: [vchkpw] Why is vadduser creating a hierarchy?
OK, I RTFM'd and found this from Ken... but CAN I TURN IT OFF? Is anyone out there? Virtual domain user directory structure Vpopmail uses an adaptive directory structure based on a state file .dir-control which is automatically managed by the core vpopmail api functions vadduser and vdeluser. For sites with 100 users or less, all user directories are stored in the virtual domain directory. For sites that go above 100 users the adaptive directory structure goes into effect. The basic idea is to break up the user Maildir directories across multple directories and sub directories so that there are never more than 100 user directories in a single directory. The default directory setup allows for 62 directories in 3 levels and 100 user directories per directory. The total number of user directories is equal to 100 + (62 * 100) + (62 * 62 * 100) + (62 * 62 * 62 * 100) = over 24 million directories. This should be more than sufficent for any site and probably goes beyond the technology of directory structures. If you are going to be storing large numbers of user directories, make sure you set your file system to have a higher than normal percentage of inodes. Vpopmail will automatically create these directories and sub directories as needed and populate each directory with up to 100 user accounts. As soon as a directory reaches 100 users it will create the next directory or sub directory and store the new users directory there. Look in the source code release directory contrib/ for a contributed directory reorganization program. Dave Richardson wrote: I'm using a script to add thousands of user accounts as part of a migration for a single domain. It's a perl script making repeated calls to /home/vpopmail/bin/vadduser -e dsfgskjghaekjrgkr [EMAIL PROTECTED] The scripting is working fine, I see the accounts correctly in MySQL's vpopmail table. However, I'm seeing vadduser create a hierarchy of folders after about the first 80-100 users are added. Using subfolders A-z,0-9. I only have about 7,000 users to manage and would rather NOT subtree (whatever the term is) this user hierarchy. What logic controls when vadduser decides to subtree the folders for a particular domain? Or, should I just let my script run out all the migrations, create the user/Maildirs wherever, and then start moving them to the root of the domain folder? That leaves some nasty work in SQL to clean up the home folder field!
Re: [vchkpw] Why is vadduser creating a hierarchy?
Dave Richardson wrote: I'm using a script to add thousands of user accounts as part of a migration for a single domain. It's a perl script making repeated calls to /home/vpopmail/bin/vadduser -e dsfgskjghaekjrgkr [EMAIL PROTECTED] The scripting is working fine, I see the accounts correctly in MySQL's vpopmail table. However, I'm seeing vadduser create a hierarchy of folders after about the first 80-100 users are added. Using subfolders A-z,0-9. I only have about 7,000 users to manage and would rather NOT subtree (whatever the term is) this user hierarchy. What logic controls when vadduser decides to subtree the folders for a particular domain? Or, should I just let my script run out all the migrations, create the user/Maildirs wherever, and then start moving them to the root of the domain folder? That leaves some nasty work in SQL to clean up the home folder field! Hi, Configure vpopmail with --disable-users-big-dir. --disable-users-big-dirDisable hashing of user directories. Regards, Rick
Re: [vchkpw] Why is vadduser creating a hierarchy?
On 10/6/06, Dave Richardson [EMAIL PROTECTED] wrote: However, I'm seeing vadduser create a hierarchy of folders after about the first 80-100 users are added. Using subfolders A-z,0-9. I only have about 7,000 users to manage and would rather NOT subtree (whatever the term is) this user hierarchy. Be sure your filesystem can handle this. Certain filesystems (ffs for example) can take a major performance hit with a large number of files in a single directory. What logic controls when vadduser decides to subtree the folders for a particular domain? When running configure: --disable-users-big-dirDisable hashing of user directories. -- Jon
Re: [vchkpw] Why is vadduser creating a hierarchy?
Rick Macdougall wrote: Dave Richardson wrote: I'm using a script to add thousands of user accounts as part of a migration for a single domain. It's a perl script making repeated calls to /home/vpopmail/bin/vadduser -e dsfgskjghaekjrgkr [EMAIL PROTECTED] The scripting is working fine, I see the accounts correctly in MySQL's vpopmail table. However, I'm seeing vadduser create a hierarchy of folders after about the first 80-100 users are added. Using subfolders A-z,0-9. I only have about 7,000 users to manage and would rather NOT subtree (whatever the term is) this user hierarchy. What logic controls when vadduser decides to subtree the folders for a particular domain? Or, should I just let my script run out all the migrations, create the user/Maildirs wherever, and then start moving them to the root of the domain folder? That leaves some nasty work in SQL to clean up the home folder field! Hi, Configure vpopmail with --disable-users-big-dir. --disable-users-big-dirDisable hashing of user directories. Regards, Rick Thanks Rick and Jon!
