Re: [vchkpw] Shared libvpopmail thoughts

[DAve]

Rick Widmer wrote:



DAve wrote:
Traction! Finally! I'll up the ante to guaranteed 24 hour turn around 
on testing any fixes, I'll setup a VMWare server just so I can test 
any patches.


But that's not all... I will also throw in a pristine copy of a Mac 
rescue CD with all OSes from 7.1 to 8.6 including all patches, AND 
bottle of Jose Gold. Shipped of course, at no cost to the programmer 
who fixes dir-control.


DAve



Looks like I need to start the vpopmail 6.0.0 branch...


Rick




All of this, and more, could be yours ;^)

http://pixelhammer.com/Dan/aargh.jpg

DAve

--
Three years now I've asked Google why they don't have a
logo change for Memorial Day. Why do they choose to do logos
for other non-international holidays, but nothing for
Veterans?

Maybe they forgot who made that choice possible.

Re: [vchkpw] Shared libvpopmail thoughts

[Joshua Megerman]

> Tren Blackburn wrote:
>>> -Original Message-
>>> From: DAve [mailto:[EMAIL PROTECTED]

>>> Trying to figure out dir-control gave me a
>>> headache and I never did get anyone to own up to knowing how it
>> worked.
>>> If someone would fix it, I would provide a 6 pack of Jones (any
>>> flavor),
>>> a carton of Marlborough reds, some tokens, a Magic Eight Ball, and a
>>> big
>>> bag of twizzlers to sweeten the deal.
>>>
>>> Dave
>>
>> Oooh! That's a sweet deal...I'll throw in a 6 pack of Beer from Canada
>> from the brewery of your choice!
>>
>
> Traction! Finally! I'll up the ante to guaranteed 24 hour turn around on
> testing any fixes, I'll setup a VMWare server just so I can test any
> patches.
>
> But that's not all... I will also throw in a pristine copy of a Mac
> rescue CD with all OSes from 7.1 to 8.6 including all patches, AND
> bottle of Jose Gold. Shipped of course, at no cost to the programmer who
> fixes dir-control.
>
I don't have any suggestions on exactly _how_ to fix dir-control at this
time, but I can try to explain how it works, since I did a lot of digging
into it to figure out how to do backfills for my patch.  This info is
based on the mysql dir_control table, but it should be (mostly) correct
for all backends.

The first entry in the domain column, dom_# is the placeholder for the
domain directory, and uses the vpopmail uid as the value of #.  All other
entries use the domain name for the real vpopmail domain (no aliases).

The second entry, cur_users, is the current number of entries in the
domain - this is important, because when this number is divisible cleanly
by 100, it increments the directory counter.  This is why I wrote my
patch, as the number doesn't accurately reflect the number of users in any
given directory, just the total for the domain.  Also, the 100 is a
compile-time value, although I have a separate patch that allows an extra
column in this table to change it on a per-domain basis.

The next 2 columns, level_cur and level_max, indicate how deep the tree 
is and how deep it can go.  0 means you're in the root directory, with no
hash dirs.  I think vpopmail supports going 3 levels deep
(///), but I've only ever seen it go 2 (and
onle because of lots of deletions causing the above problem).  I'm not
sure why this is in the databse, but I guess it's there so you can change
it to 1 or 2 to prevent hashing or limit it to 1 deep - I haven't tried
it.

After that, there are 3 each level_start and level_end columns.  This
indicates what characters in the hash list are available for use.  The
list consists of "[0-9][A-Z][a-z]" - I'm guessing these fields are there
so you can limit the hash characters to less than that, but again, I
haven't tried it.  Also, whil you can change the hash list, it's
hard-coded at compile time and I've never seen reason to mess with it -
see bigdir.c for details.

Next, there are 3 level_mod fields.  These are actually kinda pointless,
since they tell you the offset from the end of the root domain name where
the hash directory is located, but since the hash dirs are always single
characters (unless you set it all up manually - which I guess you could
do), these values are static and cannot change, so why they're in a
database I have no idea.  Level 0 is 0 because the hash dir is at
/X, level 1 is 2 because it's /X/Y, and level 2 is 4 because
it's /X/Y/Z.

Next we have the level indexes, which is just the index in the hash list
for that level of hashing.  It starts at 0 for each level when the DB is
created, but really gets initialized to level_start when it's first used. 
When a deeper level maxes out at level_end, the higer level increments and
the maxxed out level goes back to level_start.  If the topmost level maxes
out, it creates a new level - no idea what it does if it maxes out at
level 3, as that would imply (although by no means guarantee, since
there's the dir increment issue mentioned above) 99 + (100*62) +
(100*(62^2)) + (100*(62^3)) users are already on the system (that's
potentially over 24M users, and even with deleting and re-adding users,
would require over 200K instances where the user count was exactly
divisible by 100).

Finally there's the_dir, which is the actual string of the current hash
directory - an empty string for no hash, "X" for level0, "X/Y" for level1,
and "X/Y/Z" for level2.  This is used for creating the new user directory
when adding users, and gets changed whenever the hash changes.

I hope this helps - I know just writing it has given me some ideas about
things that can be changed/simplified with it, but I have to get back to
me real job for the moment :)

Josh

Joshua Megerman
SJGames MIB #5273 - OGRE AI Testing Division
You can't win; You can't break even; You can't even quit the game.
  - Layman's translation of the Laws of Thermodynamics
[EMAIL PROTECTED]

Re: [vchkpw] Shared libvpopmail thoughts

[Charlie Garrison]

Good afternoon,

On 25/9/07 at 12:41 PM -0400, John Simpson <[EMAIL PROTECTED]> wrote:

>i'm not sure exactly how far "underlying" you don't understand, but here's a 
>fairly simple overview 
>of the seedy underside of program linking and the difference between static 
>(i.e. compile-time) and 
>dynamic (i.e. run-time) linking.

That was a GREAT overview; thanks for the info.


Charlie

-- 
   Charlie Garrison  <[EMAIL PROTECTED]>
   PO Box 141, Windsor, NSW 2756, Australia

O< ascii ribbon campaign - stop html mail - www.asciiribbon.org
http://www.ietf.org/rfc/rfc1855.txt

Re: [vchkpw] Shared libvpopmail thoughts

[Joshua Megerman]

On Tuesday 25 September 2007 08:00:16 pm John Simpson wrote:
> On 2007-09-25, at 1331, Joshua Megerman wrote:
> >> i vote for "a" and "c" during a transition period, then "c" as the
> >> only option after that.
> >>
> >> in either case, i think "d" might be taking the idea too far.
> >
> > Yeah, I realized that after the fact.  I'm great ad coming up with
> > lots of
> > ideas of what can be dine, and then I need to rein them in as to what
> > _should_ be done :)
>
> oh i see, so you're an "idea rat" (pardon the dilbert reference)...
> surprising, most idea rats end up in marketing because they can't
> program their way out of a wet paper bag, but from what i've seen
> your code seems pretty solid. maybe you're a better patcher than a
> writer? i suffer from that myself- i find i do a better job of
> programming if i'm patching existing code, or writing against a
> previous design... problem is that when i start something from
> scratch, i tend to not do that separate design step ahead of time and
> end up slowing myself down.
>
> the other thing i seem to be good at is the "sanity checking", poking
> holes in ideas (my own included.) at past jobs, windows developers
> always hated to have me sit in on their meetings, because i would
> always bring up security issues before they had even started writing
> code... they eventually got over it, after their web apps got hacked
> and they had to very quickly add the changes i had originally
> suggested, while i rebuilt the server and restored from a backup
> taken just before they installed their stuff... i just don't like it
> when other peoples' stupidity causes me to have extra work.
>
Yeah, that sounds about right - I'm great with patching (I learned C (I had 
experience with Basic & Pascal previously) by pulling in code from about half 
a dozen different MUDs and combining them into one), and I can take existing 
concepts and make programs for them moderately well, but I've rarely done 
full-scale development from the ground up.  I've got a couple of things that 
I made from ground-zero that I'll get around to releasing someday, but I'm 
less inclined to spend my day doing all of the necessary things for proper 
software development than tinkering :)

And I'm right with you as to poking holes in ideas - I have a tendency 
to "think outside the box", occasionally to the annoyance of the people I 
work with...

> >>> 3) In all cases, even if the vpopmail binaries are linked against
> >>> the
> >>> shared library, the static library libvpopmail.a should be built
> >>> since
> >>> some programs expect it.
> >>
> >> maybe for interim versions, to give other programs' developers time
> >> to deal with the change... but i think that a "vpopmail version 6"
> >> should be "shared only".
> >
> > I don't see why there shouldn't be a static library that has
> > exactly the
> > same ABI as the shared one, in case someone wants (or needs) a static
> > binary.  But they should be interchangable at compile time.
>
> maybe we can always build the shared version, and have a configure
> option "--with-static-library" (which would not be active by default)
> which builds a static "libvpopmail.a" for people who may need it,
> even though the programs in the vpopmail package won't need it?
>
I can see it either way, but it's one of those things that doesn't really 
drive the code development, so it can be decided later :)

> either way, i think we need to move the thread over to the devel
> list. anybody who's interested should subscribe and look for it there.
>
Sounds good to me - I'm on the list, so I'll see it there (and am cc'ing it 
here).

Josh
-- 
Joshua Megerman
SJGames MIB #5273 - OGRE AI Testing Division
You can't win; You can't break even; You can't even quit the game.
  - Layman's translation of the Laws of Thermodynamics
[EMAIL PROTECTED]

Re: [vchkpw] Shared libvpopmail thoughts

[John Simpson]

On 2007-09-25, at 1836, Rick Widmer wrote:


The difference between shared and static libraries isn't in the  
code. It is all compiler options and when the linking actually  
happens.  The problem as I see it is ./configure options like -- 
enable-clear-password.  This option changes the table definition in  
sql back ends, and the structure of the vpasswd and vpasswd.cdb  
files in the cdb back end. It may even change the struct that  
contains the password data.


not an issue... make the SQL tables HAVE that field, as a varchar()  
which allows NULL, and leave the item in the struct, so that the only  
difference between "with" and "without" clear passwords is whether or  
not those fields in the SQL table or vpasswd file are used or not.  
basically if clear passwords are NOT supported, then we would always  
store NULL or an empty string in those fields.


The solution as I see it is to compile with everything on, disable  
undesired options at runtime based on the configuration file, and  
always provide variables and fields, even for disabled items, when  
library functions are called.


err... yeah, what he said.

my suggestion would be to use 5.5 as the "testing ground" as we  
migrate closer to a single API and a shared library, with 6.0  
being the "release" with a shared library only.


OK, I won't start anything yet...  I am partial to 6.0 being dev  
and 6.1 being stable.  If we unify the library interface and change  
the database structure it is a major change.


usually the "stable" versions have even sub-version numbers (i.e. for  
the linux kernel, 2.3 and 2.5 are experimental, while 2.4 and 2.6 are  
stable.) maybe start a 5.9 as the test-bed for what will eventually  
become 6.0?


i also have an idea which has been suggested, in various forms, by  
several users... a field in the vpasswd file and/or whatever SQL  
table serves its function, for a timestamp when a particular password  
was set, and one which, if non-zero, will "expire" a password. this  
allows for "aging" schemes to be built- some companies have policies  
which require users to change passwords every 90 days or whatever,  
and right now vpopmail has no way to support this.


i also wonder if it wouldn't make sense to change the format of the  
vpasswd file itself- instead of fields separated by ":" with each  
field having its meaning based on its ordinal number within the line,  
how hard would it be to support the "userdb" format- fields separated  
by "|", and each field being of the format "name=value"... i would  
imagine the code to parse this format already exists somewhere, it  
would just be a case of moving it into vcdb.c. this would allow extra  
fields to be added in the future without any major code changes.


i think if we're going to start designing a major new version, now is  
the time to make these kinds of major changes.



| John M. Simpson---   KG4ZOW   ---Programmer At Large |
| http://www.jms1.net/ <[EMAIL PROTECTED]> |

| http://video.google.com/videoplay?docid=-1656880303867390173 |





PGP.sig
Description: This is a digitally signed message part

Re: [vchkpw] Shared libvpopmail thoughts

[John Simpson]

On 2007-09-25, at 1348, Tom Collins wrote:


I'm all for keeping it, but someone should fix it.  On my server,  
with a cdb backend, I have the following structure:


main directory: 65 domains
0: 25 domains
1: 2 domains
2: 2 domains
3: 0 domains
4: 3 domains
5: 44 domains


to me this looks like a bunch of domains were deleted at some  
point... either that, or different versions of vpopmail have had or  
not had the "store domains in numbered directories" option at  
different times.


I'd love to see vadddomain do a better job of back-filling  
domains.  Maybe vadddomain and vdeldomain could work together to  
keep directories at a balanced level.


i'm not sure that vdeldomain has, or should have, anything to do with  
it.


Keep track of the next directory to fill in a file (which needs to  
be protected by a file lock).  The .dir-control file is supposed to  
work that way.


On vdeldomain, if the domain came out of a directory "less than"  
the next_directory, update next_directory.


On vadddomain, if next_directory has 100 domains after the  
addition, scan forward until you find a directory with <100 domains  
and update next_directory.


i think trying to track it in a file is overkill... unless you're  
adding several domains per minute, or you have multiple people adding  
domains, you should just be able to add the bucket-selection code to  
vadddomain().


It should be possible to make the code generalized enough to work  
for the domains directory and the individual domain directories  
(for managing users via vuseradd and vuserdel).


not too difficult... i'll throw something up on the dev list in a few  
minutes. i would have figured that logic was built into vadddomain()  
and vadduser() already though?


again... on the dev list.


| John M. Simpson---   KG4ZOW   ---Programmer At Large |
| http://www.jms1.net/ <[EMAIL PROTECTED]> |

| http://video.google.com/videoplay?docid=-1656880303867390173 |





PGP.sig
Description: This is a digitally signed message part

Re: [vchkpw] Shared libvpopmail thoughts

[John Simpson]

On 2007-09-25, at 1331, Joshua Megerman wrote:

i vote for "a" and "c" during a transition period, then "c" as the
only option after that.

in either case, i think "d" might be taking the idea too far.


Yeah, I realized that after the fact.  I'm great ad coming up with  
lots of

ideas of what can be dine, and then I need to rein them in as to what
_should_ be done :)


oh i see, so you're an "idea rat" (pardon the dilbert reference)...  
surprising, most idea rats end up in marketing because they can't  
program their way out of a wet paper bag, but from what i've seen  
your code seems pretty solid. maybe you're a better patcher than a  
writer? i suffer from that myself- i find i do a better job of  
programming if i'm patching existing code, or writing against a  
previous design... problem is that when i start something from  
scratch, i tend to not do that separate design step ahead of time and  
end up slowing myself down.


the other thing i seem to be good at is the "sanity checking", poking  
holes in ideas (my own included.) at past jobs, windows developers  
always hated to have me sit in on their meetings, because i would  
always bring up security issues before they had even started writing  
code... they eventually got over it, after their web apps got hacked  
and they had to very quickly add the changes i had originally  
suggested, while i rebuilt the server and restored from a backup  
taken just before they installed their stuff... i just don't like it  
when other peoples' stupidity causes me to have extra work.


3) In all cases, even if the vpopmail binaries are linked against  
the
shared library, the static library libvpopmail.a should be built  
since

some programs expect it.


maybe for interim versions, to give other programs' developers time
to deal with the change... but i think that a "vpopmail version 6"
should be "shared only".


I don't see why there shouldn't be a static library that has  
exactly the

same ABI as the shared one, in case someone wants (or needs) a static
binary.  But they should be interchangable at compile time.


maybe we can always build the shared version, and have a configure  
option "--with-static-library" (which would not be active by default)  
which builds a static "libvpopmail.a" for people who may need it,  
even though the programs in the vpopmail package won't need it?


either way, i think we need to move the thread over to the devel  
list. anybody who's interested should subscribe and look for it there.



| John M. Simpson---   KG4ZOW   ---Programmer At Large |
| http://www.jms1.net/ <[EMAIL PROTECTED]> |

| http://video.google.com/videoplay?docid=-1656880303867390173 |





PGP.sig
Description: This is a digitally signed message part

Re: [vchkpw] Shared libvpopmail thoughts

[Rick Widmer]

Tom Collins wrote:

On Sep 25, 2007, at 6:31 AM, Joshua Megerman wrote:

Perhaps the first step is to document the API as it currently stands, and
give people the option to build a shared library with the caviat that if
you reconfigure vpopmail, you need to rebuild those things that link
against it.  That would be a 5.5 branch, since it doesn't change the
current functionality (much).  Then we can in parallel start developing
the truly stable API and other changes that will become 6.0, and when we
do we can increment libvpopmail.so to indicate the ABI difference.


I would love to see this happen, but it is going to take a considerable 
amount of work.  I'm willing to provide lots of input on the API, but 
really don't have time to contribute actual code.


I would start with the things that qmailadmin calls, then add any 
additional things that vpopmaild calls. (They should be very close in 
their use of the vpopmail api)  That is the api, and everything else is 
considered internal, and subject to change.  At that point give the 
general users a chance to ask for anything else they need in the api.


As far as that goes, vpopmaild commands exploit pretty much everything I 
consider the vpopmail api.  (If something is found missing from 
vpopmaild it should be added.)  Re-state the vpopmaild documentaton in C 
terms and you are close.



Tom Collins wrote:
If we use two different names, could we retain backward compatibility by 
building a libvpopmail the way we do now (statically linked, apps may 
use vpopmail's config.h, etc.) in addition to the new-style, shared 
library with a well-defined API?


The difference between shared and static libraries isn't in the code. 
It is all compiler options and when the linking actually happens.  The 
problem as I see it is ./configure options like --enable-clear-password. 
 This option changes the table definition in sql back ends, and the 
structure of the vpasswd and vpasswd.cdb files in the cdb back end. It 
may even change the struct that contains the password data.


The solution as I see it is to compile with everything on, disable 
undesired options at runtime based on the configuration file, and always 
provide variables and fields, even for disabled items, when library 
functions are called.



John Simpson wrote:
>>On 2007-09-24, at 1120, Joshua Megerman wrote:
>>

2) There has been some question regarding performance of the vpopmail
programs when compiled against shared vs. static libraries.  I suggest the
following options be added for shared libraries at compile-time:
  a) --disable-shared - don't build libvpopmail.so, which is what vpopmail
does now.
  b) --enable-shared - build libvpopmail.so, but don't link the vpopmail
binaries against it - this gives other programs the ability to use the
shared library, but keeps the vpopmail binaries statically linked.
  c) --enable-shared-binaries - build libvpopmail.so and link the vpopmail
binaries against it.  Implies --enable-shared.
  d) possibly, if it's not to difficult, have a --enable-shared-binaries=
and/or --enable-static-binaries= option, which takes a list of binaries
to link against the stated library, and links the rest against the
other.  So you could have static vdelivermail and vchkpw, but not
vadduser, for example.  Not sure if that really is necessary, but static
linking does save space...


i vote for "a" and "c" during a transition period, then "c" as the only option 
after that.


I agree with a and c, but think the option should stay forever.



in either case, i think "d" might be taking the idea too far.


I agree.

I disagree that static links save space.  Each static executable 
contains all of the library code, so there are many copies.  On the 
other hand, I use static binding for key programs so I don't have to 
worry about updating the central library.  Replace the programs and the 
correct library is right there in the files.






If so, perhaps the speed solution for a dynamic
(e.g.) vdelivermail would be to run something that was dynamically linked
all the time, so libvpopmail stayed in memory...


if you're on a system which is busy enough that these few milliseconds are

> a significant issue, you will already have tens or hundreds of other
> processes with libvpopmail.so mapped into their memory space anyway- so
> again, it won't be an issue.

Like vpopmaild.  Once you fire that up libvpopmail should stay in  memory.



Once upon a time vpopmail was designed to be quick and tiny.  All options

>> were compiled in.  Since then at least 3 of the back ends have adopted a
>> configuration file.  Maybe it is time to look at moving most of the
>> ./configure options to a configuration file


add a vpopmail_init() function which reads that file and sets a bunch

> of "use_*" variables, which the other functions would then check. make
> it a requirement that all client programs must call this function first,
> or any functions whose operation depends on these variables being
> p

Re: [vchkpw] Shared libvpopmail thoughts

[Rick Widmer]

DAve wrote:
Traction! Finally! I'll up the ante to guaranteed 24 hour turn around on 
testing any fixes, I'll setup a VMWare server just so I can test any 
patches.


But that's not all... I will also throw in a pristine copy of a Mac 
rescue CD with all OSes from 7.1 to 8.6 including all patches, AND 
bottle of Jose Gold. Shipped of course, at no cost to the programmer who 
fixes dir-control.


DAve



Looks like I need to start the vpopmail 6.0.0 branch...


Rick

Re: [vchkpw] Shared libvpopmail thoughts

[DAve]

Tren Blackburn wrote:

-Original Message-
From: DAve [mailto:[EMAIL PROTECTED]
Sent: Tuesday, September 25, 2007 12:13 PM
To: vchkpw@inter7.com
Subject: Re: [vchkpw] Shared libvpopmail thoughts

Tom Collins wrote:

On Sep 25, 2007, at 10:06 AM, John Simpson wrote:

the idea of splitting the domains and mailboxes into different
directories has been around forever, there's no excuse for somebody

to

not have made the adjustment by now.

I'm all for keeping it, but someone should fix it.  On my server,

with a

cdb backend, I have the following structure:

main directory: 65 domains
0: 25 domains
1: 2 domains
2: 2 domains
3: 0 domains
4: 3 domains
5: 44 domains

Just for reference, here's the .dir-control file for that server:

116
0
3
0 0 0
61 61 61
0 2 2
5 0 0
5

I'd love to see vadddomain do a better job of back-filling domains.
Maybe vadddomain and vdeldomain could work together to keep

directories

at a balanced level.

Keep track of the next directory to fill in a file (which needs to

be

protected by a file lock).  The .dir-control file is supposed to

work

that way.

On vdeldomain, if the domain came out of a directory "less than" the
next_directory, update next_directory.

On vadddomain, if next_directory has 100 domains after the addition,
scan forward until you find a directory with <100 domains and update
next_directory.

It should be possible to make the code generalized enough to work

for

the domains directory and the individual domain directories (for
managing users via vuseradd and vuserdel).

dir-control is a train wreck. If we ever left vpopmail (unlikely) it
would be because of dir-control. I'm not a C programmer but I have

lots

of coding experience with IBM Basic, JCL, COBOL, Perl, PHP, Ruby,
JavaScript, Bash, etc. Trying to figure out dir-control gave me a
headache and I never did get anyone to own up to knowing how it

worked.

If someone would fix it, I would provide a 6 pack of Jones (any
flavor),
a carton of Marlborough reds, some tokens, a Magic Eight Ball, and a
big
bag of twizzlers to sweeten the deal.

Dave


Oooh! That's a sweet deal...I'll throw in a 6 pack of Beer from Canada
from the brewery of your choice!



Traction! Finally! I'll up the ante to guaranteed 24 hour turn around on 
testing any fixes, I'll setup a VMWare server just so I can test any 
patches.


But that's not all... I will also throw in a pristine copy of a Mac 
rescue CD with all OSes from 7.1 to 8.6 including all patches, AND 
bottle of Jose Gold. Shipped of course, at no cost to the programmer who 
fixes dir-control.


DAve


--
Three years now I've asked Google why they don't have a
logo change for Memorial Day. Why do they choose to do logos
for other non-international holidays, but nothing for
Veterans?

Maybe they forgot who made that choice possible.

RE: [vchkpw] Shared libvpopmail thoughts

[Tren Blackburn]

> -Original Message-
> From: DAve [mailto:[EMAIL PROTECTED]
> Sent: Tuesday, September 25, 2007 12:13 PM
> To: vchkpw@inter7.com
> Subject: Re: [vchkpw] Shared libvpopmail thoughts
> 
> Tom Collins wrote:
> > On Sep 25, 2007, at 10:06 AM, John Simpson wrote:
> >> the idea of splitting the domains and mailboxes into different
> >> directories has been around forever, there's no excuse for somebody
> to
> >> not have made the adjustment by now.
> >
> > I'm all for keeping it, but someone should fix it.  On my server,
> with a
> > cdb backend, I have the following structure:
> >
> > main directory: 65 domains
> > 0: 25 domains
> > 1: 2 domains
> > 2: 2 domains
> > 3: 0 domains
> > 4: 3 domains
> > 5: 44 domains
> >
> > Just for reference, here's the .dir-control file for that server:
> >
> > 116
> > 0
> > 3
> > 0 0 0
> > 61 61 61
> > 0 2 2
> > 5 0 0
> > 5
> >
> > I'd love to see vadddomain do a better job of back-filling domains.
> > Maybe vadddomain and vdeldomain could work together to keep
> directories
> > at a balanced level.
> >
> > Keep track of the next directory to fill in a file (which needs to
be
> > protected by a file lock).  The .dir-control file is supposed to
work
> > that way.
> >
> > On vdeldomain, if the domain came out of a directory "less than" the
> > next_directory, update next_directory.
> >
> > On vadddomain, if next_directory has 100 domains after the addition,
> > scan forward until you find a directory with <100 domains and update
> > next_directory.
> >
> > It should be possible to make the code generalized enough to work
for
> > the domains directory and the individual domain directories (for
> > managing users via vuseradd and vuserdel).
> 
> dir-control is a train wreck. If we ever left vpopmail (unlikely) it
> would be because of dir-control. I'm not a C programmer but I have
lots
> of coding experience with IBM Basic, JCL, COBOL, Perl, PHP, Ruby,
> JavaScript, Bash, etc. Trying to figure out dir-control gave me a
> headache and I never did get anyone to own up to knowing how it
worked.
> 
> If someone would fix it, I would provide a 6 pack of Jones (any
> flavor),
> a carton of Marlborough reds, some tokens, a Magic Eight Ball, and a
> big
> bag of twizzlers to sweeten the deal.
> 
> Dave

Oooh! That's a sweet deal...I'll throw in a 6 pack of Beer from Canada
from the brewery of your choice!

t.

> 
> --
> Three years now I've asked Google why they don't have a
> logo change for Memorial Day. Why do they choose to do logos
> for other non-international holidays, but nothing for
> Veterans?
> 
> Maybe they forgot who made that choice possible.

Re: [vchkpw] Shared libvpopmail thoughts

[DAve]

Tom Collins wrote:

On Sep 25, 2007, at 10:06 AM, John Simpson wrote:
the idea of splitting the domains and mailboxes into different 
directories has been around forever, there's no excuse for somebody to 
not have made the adjustment by now.


I'm all for keeping it, but someone should fix it.  On my server, with a 
cdb backend, I have the following structure:


main directory: 65 domains
0: 25 domains
1: 2 domains
2: 2 domains
3: 0 domains
4: 3 domains
5: 44 domains

Just for reference, here's the .dir-control file for that server:

116
0
3
0 0 0
61 61 61
0 2 2
5 0 0
5

I'd love to see vadddomain do a better job of back-filling domains.  
Maybe vadddomain and vdeldomain could work together to keep directories 
at a balanced level.


Keep track of the next directory to fill in a file (which needs to be 
protected by a file lock).  The .dir-control file is supposed to work 
that way.


On vdeldomain, if the domain came out of a directory "less than" the 
next_directory, update next_directory.


On vadddomain, if next_directory has 100 domains after the addition, 
scan forward until you find a directory with <100 domains and update 
next_directory.


It should be possible to make the code generalized enough to work for 
the domains directory and the individual domain directories (for 
managing users via vuseradd and vuserdel).


dir-control is a train wreck. If we ever left vpopmail (unlikely) it 
would be because of dir-control. I'm not a C programmer but I have lots 
of coding experience with IBM Basic, JCL, COBOL, Perl, PHP, Ruby, 
JavaScript, Bash, etc. Trying to figure out dir-control gave me a 
headache and I never did get anyone to own up to knowing how it worked.


If someone would fix it, I would provide a 6 pack of Jones (any flavor), 
a carton of Marlborough reds, some tokens, a Magic Eight Ball, and a big 
bag of twizzlers to sweeten the deal.


DAve

--
Three years now I've asked Google why they don't have a
logo change for Memorial Day. Why do they choose to do logos
for other non-international holidays, but nothing for
Veterans?

Maybe they forgot who made that choice possible.

Re: [vchkpw] Shared libvpopmail thoughts

[Joshua Megerman]

> On Sep 25, 2007, at 10:06 AM, John Simpson wrote:
>> the idea of splitting the domains and mailboxes into different
>> directories has been around forever, there's no excuse for somebody
>> to not have made the adjustment by now.
>
> I'm all for keeping it, but someone should fix it.  On my server,
> with a cdb backend, I have the following structure:
>
> main directory: 65 domains
> 0: 25 domains
> 1: 2 domains
> 2: 2 domains
> 3: 0 domains
> 4: 3 domains
> 5: 44 domains
>
> Just for reference, here's the .dir-control file for that server:
>
> 116
> 0
> 3
> 0 0 0
> 61 61 61
> 0 2 2
> 5 0 0
> 5
>
> I'd love to see vadddomain do a better job of back-filling domains.
> Maybe vadddomain and vdeldomain could work together to keep
> directories at a balanced level.



Just for reference, I have a patch that does a lot of this for the MySQL
backend.  It's up on SF in patches, #1619600, and could probably server as
a basis for this type of code everywhere.  There's also a separate patch
that allows per-domain control of the # of users per directory, also just
for MySQL, but again it could serve as a starting point.

Josh

Joshua Megerman
SJGames MIB #5273 - OGRE AI Testing Division
You can't win; You can't break even; You can't even quit the game.
  - Layman's translation of the Laws of Thermodynamics
[EMAIL PROTECTED]

Re: [vchkpw] Shared libvpopmail thoughts

[tonix (Antonio Nati)]

At 19.48 25/09/2007, you wrote:

On Sep 25, 2007, at 10:06 AM, John Simpson wrote:

the idea of splitting the domains and mailboxes into different
directories has been around forever, there's no excuse for somebody
to not have made the adjustment by now.


I'm all for keeping it, but someone should fix it.  On my server,
with a cdb backend, I have the following structure:

main directory: 65 domains
0: 25 domains
1: 2 domains
2: 2 domains
3: 0 domains
4: 3 domains
5: 44 domains

Just for reference, here's the .dir-control file for that server:

116
0
3
0 0 0
61 61 61
0 2 2
5 0 0
5

I'd love to see vadddomain do a better job of back-filling domains.
Maybe vadddomain and vdeldomain could work together to keep
directories at a balanced level.

Keep track of the next directory to fill in a file (which needs to be
protected by a file lock).  The .dir-control file is supposed to work
that way.

On vdeldomain, if the domain came out of a directory "less than" the
next_directory, update next_directory.

On vadddomain, if next_directory has 100 domains after the addition,
scan forward until you find a directory with <100 domains and update
next_directory.

It should be possible to make the code generalized enough to work for
the domains directory and the individual domain directories (for
managing users via vuseradd and vuserdel).


Would not it be more easy to add a hash logic and create paths 
following that hash logic?


Example: domain interazioni.it.

md5('interazioni.it') => 931d4b6acf6f23d224b7f57c76334286

Path --> /home/vpopmail/domains/in/te/93/1d/interazioni.it

Hash could be more or less long depending on site configuration.

For sure this hashed path is a lot more balanced than actual.

Tonino


--
Tom Collins  -  [EMAIL PROTECTED]
Vpopmail - virtual domains for qmail: http://vpopmail.sf.net/
QmailAdmin - web interface for Vpopmail: http://qmailadmin.sf.net/

Re: [vchkpw] Shared libvpopmail thoughts

[Tom Collins]

On Sep 25, 2007, at 10:06 AM, John Simpson wrote:
the idea of splitting the domains and mailboxes into different  
directories has been around forever, there's no excuse for somebody  
to not have made the adjustment by now.


I'm all for keeping it, but someone should fix it.  On my server,  
with a cdb backend, I have the following structure:


main directory: 65 domains
0: 25 domains
1: 2 domains
2: 2 domains
3: 0 domains
4: 3 domains
5: 44 domains

Just for reference, here's the .dir-control file for that server:

116
0
3
0 0 0
61 61 61
0 2 2
5 0 0
5

I'd love to see vadddomain do a better job of back-filling domains.   
Maybe vadddomain and vdeldomain could work together to keep  
directories at a balanced level.


Keep track of the next directory to fill in a file (which needs to be  
protected by a file lock).  The .dir-control file is supposed to work  
that way.


On vdeldomain, if the domain came out of a directory "less than" the  
next_directory, update next_directory.


On vadddomain, if next_directory has 100 domains after the addition,  
scan forward until you find a directory with <100 domains and update  
next_directory.


It should be possible to make the code generalized enough to work for  
the domains directory and the individual domain directories (for  
managing users via vuseradd and vuserdel).


--
Tom Collins  -  [EMAIL PROTECTED]
Vpopmail - virtual domains for qmail: http://vpopmail.sf.net/
QmailAdmin - web interface for Vpopmail: http://qmailadmin.sf.net/

Re: [vchkpw] Shared libvpopmail thoughts

[Joshua Megerman]

> On 2007-09-24, at 1120, Joshua Megerman wrote:
>>
>> First off, let me prefice this by saying that while I understand the
>> concept of shared libraries, I don't understand the underlying
>> mechanics
>> of how the OS handles them,
>
> i'm not sure exactly how far "underlying" you don't understand, but
> here's a fairly simple overview of the seedy underside of program
> linking and the difference between static (i.e. compile-time) and
> dynamic (i.e. run-time) linking.
>
Conceptually I understand the difference between static and dynamic
linking, I just don't understand the low-level mechanics of how dynamic
linking works at runtime.  But thank you for the comprehensable
explanation anyway - I'm sure there are people out there who also
appreciate it.

> you can see the various imports and exports in a .o or .a file using
> the program "nm". for example, in vpopmail 5.4.22, the file "md5.o"
> contains the following symbols:
>
> $ nm md5.o
> 07e0 T MD5Final
> 0038 T MD5Init
> 006c T MD5Transform
> 06ec T MD5Update
>  T byteReverse
>   U memcpy
>   U memset
>
> the symbols with "T" are exports, the functions in the module. these
> function names are available to be matched against other modules
> which may need them. the symbols with "U" are imports, names which
> need to be matched against other modules in order to build a final
> working program. in this case, the "memcpy" and "memset" functions
> are defined in the "memcpy.o" module within "libc.a" or "libc.so".
>
This part I didn't know - thanks!

> then, when the program is actually executed, the first thing it does
> is call a "run-time linker", usually called "ld.so". the run-time
> linker loads the necessary .so files into your program's memory
> space, performs the "fixups" (i.e. stores the final in-memory address
> of the library functions into the correct memory locations in your
> code), and then jumps to the starting point of your program.
>
> because modern CPUs support the concept of making a particular
> segment of memory "read only", and because most memory management
> hardware makes it possible to map a particular physical segment of
> memory to appear in any logical address within the address space, it
> is possible for shared libraries to physically exist in memory only
> one time, while visible to multiple processes as different addresses.
> this is why, if you look at a process with "ps" or "top", you'll see
> two memory-usage numbers- the "virtual size", which is how much total
> memory space is used if this process were the only one on the
> machine, and "resident set", which is how much memory is dedicated to
> just that one process. the difference in these two numbers is the
> amount used for shared memory, usually shared libraries like libc.so.
>
OK - I knew and/or guessed about 3/4 of this, but thanks for clarifying.

>>  and thus am not sure exactly how can be affected performance-wise.
>
> the vpopmail programs are already dynamically loaded- it's just the
> "libvpopmail.a" functions which are not loaded dynamically. the
> performance hit would be minimal- it already has to load libc.so at
> run-time, one more library won't take long enough to make any real
> difference.
>
That's what I was wondering about...

>> 1) A shared library with a stable API would make recompiling outside
>> programs (e.g., QmailAdmin) unnecessary, which would be a Good Thing
>> (tm).
>
> as long as it's the same API for all of the authentication modules.
>
> i can also see having "libvpopmail.so" for the client-facing
> programs, then modules like "libvpopmailauth_cdb.so",
> "libvpopmailauth_mysql.so", and so forth, for the back-end code to
> handle the mechanics for that particular authentication back-end,
> similar to how courier-authlib is structured.
>
That's not a bad idea - IIRC you can dynamically link shared libraries to
other shared libraries, so the actual function calls could be handled via
config data and function pointer arrays...

>> 2) There has been some question regarding performance of the vpopmail
>> programs when compiled against shared vs. static libraries.  I
>> suggest the
>> following options be added for shared libraries at compile-time:
>>   a) --disable-shared - don't build libvpopmail.so, which is what
>> vpopmail
>> does now.
>>   b) --enable-shared - build libvpopmail.so, but don't link the
>> vpopmail
>> binaries against it - this gives other programs the ability to use the
>> shared library, but keeps the vpopmail binaries statically linked.
>>   c) --enable-shared-binaries - build libvpopmail.so and link the
>> vpopmail
>> binaries against it.  Implies --enable-shared.
>>   d) possibly, if it's not to difficult, have a --enable-shared-
>> binaries=
>> and/or --enable-static-binaries= option, which takes a list of
>> binaries
>> to link against the stated library, and links the rest against the
>> other.  So you could have static vdelivermail and vchkpw, but not
>> vadduser, fo

Re: [vchkpw] Shared libvpopmail thoughts

[Joshua Megerman]

> On Sep 25, 2007, at 6:31 AM, Joshua Megerman wrote:
>> Perhaps the first step is to document the API as it currently
>> stands, and
>> give people the option to build a shared library with the caviat
>> that if
>> you reconfigure vpopmail, you need to rebuild those things that link
>> against it.  That would be a 5.5 branch, since it doesn't change the
>> current functionality (much).  Then we can in parallel start
>> developing
>> the truly stable API and other changes that will become 6.0, and
>> when we
>> do we can increment libvpopmail.so to indicate the ABI difference.
>
> I would love to see this happen, but it is going to take a
> considerable amount of work.  I'm willing to provide lots of input on
> the API, but really don't have time to contribute actual code.
>
I know the feeling...

> I can help out with documenting some of the ways that QmailAdmin
> interfaces with vpopmail.  Getting a new version of QmailAdmin to
> compile to a shared vpopmail lib with a single vpopmail.h to describe
> the API would be great.  As it is now, QmailAdmin actually uses
> vpopmail's config.h file at build time.  That will definitely have to
> go.
>
Yes, that would be a good step.  One of the things that I think needs to
be identified is all of the packages that we know of that interface with
libvpopmail, and how they do so.  That will help us develop the new API so
that we can provide all of the necessary program interfaces.

> If we use two different names, could we retain backward compatibility
> by building a libvpopmail the way we do now (statically linked, apps
> may use vpopmail's config.h, etc.) in addition to the new-style,
> shared library with a well-defined API?
>
I don't know enough about how shared vs. static libraries work at a
system/compiler level, but I would think this could work.  However, I
think long-term we want to eliminate the old-style API through the static
library just for ease of maintainability.  But I don't see a problem with
generating both shared and static libraries, and letting the person
compiling the package that uses them decide which one to use.

Josh

Joshua Megerman
SJGames MIB #5273 - OGRE AI Testing Division
You can't win; You can't break even; You can't even quit the game.
  - Layman's translation of the Laws of Thermodynamics
[EMAIL PROTECTED]

Re: [vchkpw] Shared libvpopmail thoughts

[John Simpson]

On 2007-09-25, at 0513, Rick Widmer wrote:

Joshua Megerman wrote:


1) A shared library with a stable API would make recompiling outside
programs (e.g., QmailAdmin) unnecessary, which would be a Good  
Thing(tm).


It is that 'stable API' that is the killer.  I know some ./ 
configure options change the interface to libvpopmail.  I don't  
know which ones they are.  :(  I do know if you change some of the  
options you can get some spectacular failures if you forget to re- 
compile everything that uses vpopmail.


maybe we need to change the API, standardize it... if there are  
functions which either "do" or "do not" need an extra parameter, then  
i would say to add that parameter anyway, and if the final program  
doesn't use it, it can pass a 0 or NULL or whatever, and the function  
would ignore it.


Once upon a time vpopmail was designed to be quick and tiny.  All  
options were compiled in.  Since then at least 3 of the back ends  
have adopted a configuration file.  Maybe it is time to look at  
moving most of the ./configure options to a configuration file


add a vpopmail_init() function which reads that file and sets a bunch  
of "use_*" variables, which the other functions would then check.  
make it a requirement that all client programs must call this  
function first, or any functions whose operation depends on these  
variables being properly set, could call this function instead... of  
course it would have a flag variable so that calling it multiple  
times doesn't result in the config files being read multiple times,  
something like this...


int flag_init = 0 ;
int use_auth_mysql = 0 ;
int use_auth_pgsql = 0 ;
int use_auth_ldap = 0 ;
int use_auth_sybase = 0 ;
int use_onchange = 0 ;
void vpopmail_init(void)
{
if ( flag_init ) return ;

/* read the config file, set all use_* vars
   if we encounter errors, exit(1) */
}

we also need a "reinit()" function, so that processes like vpopmaild  
can force the config to be re-read without having to be stopped and  
restarted.


void vpopmail_reinit(void)
{
flag_init = 0 ;
vpopmail_init() ;
}

and of course, the relevant functions need to check these variables  
in order to work properly. for example, i know the call_onchange()  
function is already written to early-exit if the ~vpopmail/etc/ 
onchange script doesn't exist or isn't executable, it would be  
trivial to early-exit if the "use_onchange" variable contains a zero  
instead of a one.


i think i'm probably being a bit too detailed for this list, this  
should probably be moved to the developer list.


and have only one vpopmail library interface for the entire life of  
a major (minor?) version.


Are we starting 5.5 or 6.0 if we change the library interface, and  
table layouts?


my suggestion would be to use 5.5 as the "testing ground" as we  
migrate closer to a single API and a shared library, with 6.0 being  
the "release" with a shared library only.


the other suggestion i have is this- there are options which make  
sense for larger systems, and don't hurt anything for smaller  
systems, the ones involving splitting the domains and mailboxes into  
numbered sub-directories in order to prevent having a single  
directory with 15,000 entries in it. these options should just plain  
be turned on for everybody, and the "options" should be removed.


the only argument i've ever heard for keeping them as options is that  
some people have written scripts which make assumptions about the  
directory structure. these scripts should run "vdominfo -d" and  
"vuserinfo -d" to get the directories, rather than assuming they will  
be in any particular location. i'm just not a fan of hanging on to  
options which serve no purpose other than to accomodate improperly  
written scripts- the idea of splitting the domains and mailboxes into  
different directories has been around forever, there's no excuse for  
somebody to not have made the adjustment by now.



| John M. Simpson---   KG4ZOW   ---Programmer At Large |
| http://www.jms1.net/ <[EMAIL PROTECTED]> |

| http://video.google.com/videoplay?docid=-1656880303867390173 |





PGP.sig
Description: This is a digitally signed message part

Re: [vchkpw] Shared libvpopmail thoughts

[John Simpson]

On 2007-09-24, at 1120, Joshua Megerman wrote:


First off, let me prefice this by saying that while I understand the
concept of shared libraries, I don't understand the underlying  
mechanics

of how the OS handles them,


i'm not sure exactly how far "underlying" you don't understand, but  
here's a fairly simple overview of the seedy underside of program  
linking and the difference between static (i.e. compile-time) and  
dynamic (i.e. run-time) linking.


the compiler generates ".o" files, containing the following:

- one or more "text" segments, which contain byte sequences of  
executable code


- a list of "exports", symbols which are available in the module,  
usually functions which may be called from, or global variables (such  
as "errno") which may be accessed by other modules.


- a list of "imports", symbols which this module needs in order to  
execute correctly, and the "fixup" locations where the final memory  
address of each symbol should be stored.


there are other types of files called ".a" files, which are basically  
a collection of .o files joined together for easier management- a  
"library", in other words. libvpopmail.a is one of these.


you can see the various imports and exports in a .o or .a file using  
the program "nm". for example, in vpopmail 5.4.22, the file "md5.o"  
contains the following symbols:


$ nm md5.o
07e0 T MD5Final
0038 T MD5Init
006c T MD5Transform
06ec T MD5Update
 T byteReverse
 U memcpy
 U memset

the symbols with "T" are exports, the functions in the module. these  
function names are available to be matched against other modules  
which may need them. the symbols with "U" are imports, names which  
need to be matched against other modules in order to build a final  
working program. in this case, the "memcpy" and "memset" functions  
are defined in the "memcpy.o" module within "libc.a" or "libc.so".


the compile-time linker gathers a bunch of these .o and .a files,  
matches up the "imports" with the "exports" from the various modules,  
and produces a final executable with any interior links resolved. for  
a statically linked program, ALL links must be resolved in order to  
have a working program- so if your "main()" called any or all of the  
MD5 functions listed above, your ".o" would have "MD5Init" and  
friends as imports, and the linker would match those against the  
md5.o module and add "memcpy" and "memset" to the list of imports, so  
the linker would then bring in the "memcpy.o" and "memset.o" modules  
from "libc.a", as part of your program's final executable.


there are two problems with this scenario:

- some functions, like printf(), have a LOT of dependencies. a three- 
line program which might normally generate a 4K executable, can grow  
to over 800K because of these dependencies.


- if the underlying library changes, you have to re-compile this  
program to gain the benefits (security fixes, new features, etc.) of  
the new library.


if a program is being compiled to support dynamic linking, then  
instead of looking at "libc.a", it looks at "libc.so". and instead of  
copying the code from the .so into the final executable, it builds a  
list of "run-time fixups", which is stored in the final executable.


then, when the program is actually executed, the first thing it does  
is call a "run-time linker", usually called "ld.so". the run-time  
linker loads the necessary .so files into your program's memory  
space, performs the "fixups" (i.e. stores the final in-memory address  
of the library functions into the correct memory locations in your  
code), and then jumps to the starting point of your program.


because modern CPUs support the concept of making a particular  
segment of memory "read only", and because most memory management  
hardware makes it possible to map a particular physical segment of  
memory to appear in any logical address within the address space, it  
is possible for shared libraries to physically exist in memory only  
one time, while visible to multiple processes as different addresses.  
this is why, if you look at a process with "ps" or "top", you'll see  
two memory-usage numbers- the "virtual size", which is how much total  
memory space is used if this process were the only one on the  
machine, and "resident set", which is how much memory is dedicated to  
just that one process. the difference in these two numbers is the  
amount used for shared memory, usually shared libraries like libc.so.



 and thus am not sure exactly how can be affected performance-wise.


the vpopmail programs are already dynamically loaded- it's just the  
"libvpopmail.a" functions which are not loaded dynamically. the  
performance hit would be minimal- it already has to load libc.so at  
run-time, one more library won't take long enough to make any real  
difference.



1) A shared library with a stable API would make recompiling outside
programs (e.g., QmailAdmin) unnecessar

Re: [vchkpw] Shared libvpopmail thoughts

[Tom Collins]

On Sep 25, 2007, at 6:31 AM, Joshua Megerman wrote:
Perhaps the first step is to document the API as it currently  
stands, and
give people the option to build a shared library with the caviat  
that if

you reconfigure vpopmail, you need to rebuild those things that link
against it.  That would be a 5.5 branch, since it doesn't change the
current functionality (much).  Then we can in parallel start  
developing
the truly stable API and other changes that will become 6.0, and  
when we

do we can increment libvpopmail.so to indicate the ABI difference.


I would love to see this happen, but it is going to take a  
considerable amount of work.  I'm willing to provide lots of input on  
the API, but really don't have time to contribute actual code.


I can help out with documenting some of the ways that QmailAdmin  
interfaces with vpopmail.  Getting a new version of QmailAdmin to  
compile to a shared vpopmail lib with a single vpopmail.h to describe  
the API would be great.  As it is now, QmailAdmin actually uses  
vpopmail's config.h file at build time.  That will definitely have to  
go.


If we use two different names, could we retain backward compatibility  
by building a libvpopmail the way we do now (statically linked, apps  
may use vpopmail's config.h, etc.) in addition to the new-style,  
shared library with a well-defined API?


--
Tom Collins  -  [EMAIL PROTECTED]
Vpopmail - virtual domains for qmail: http://vpopmail.sf.net/
QmailAdmin - web interface for Vpopmail: http://qmailadmin.sf.net/

Re: [vchkpw] Shared libvpopmail thoughts

[Joshua Megerman]

>> 1) A shared library with a stable API would make recompiling outside
>> programs (e.g., QmailAdmin) unnecessary, which would be a Good
>> Thing(tm).
>
> It is that 'stable API' that is the killer.  I know some ./configure
> options change the interface to libvpopmail.  I don't know which ones
> they are.  :(  I do know if you change some of the options you can get
> some spectacular failures if you forget to re-compile everything that
> uses vpopmail.
>
Hmm...  that's a good point, although AFAIK the API is "stable" with
respect to what the functions do for any given config.h.  But you're
correct that until things are documented and "blessed" (not to mention
creating some sort of "official" API change process for the future, even
if it's assigning 1 person as the only authority on changes).

> Once upon a time vpopmail was designed to be quick and tiny.  All
> options were compiled in.  Since then at least 3 of the back ends have
> adopted a configuration file.  Maybe it is time to look at moving most
> of the ./configure options to a configuration file and have only one
> vpopmail library interface for the entire life of a major (minor?)
> version.
>
> Are we starting 5.5 or 6.0 if we change the library interface, and table
> layouts?
>
Hmm...  if we're talking about doing major changes to existing
functionality, I'd call that a major version change, as you're going to
break things when you upgrade if you don't follow through with the upgrade
process properly.  As far as the table layout changes, while I understand
that it could improve performance some, I'm not sure if it's really that
big a deal.  I worked on a system with ~6.5K users that had no DB lag time
whatsoever, but we also used master/slave mysql over 6-8 frontends at any
given time.  But again, I'd definitely call that a major revision change.

Perhaps the first step is to document the API as it currently stands, and
give people the option to build a shared library with the caviat that if
you reconfigure vpopmail, you need to rebuild those things that link
against it.  That would be a 5.5 branch, since it doesn't change the
current functionality (much).  Then we can in parallel start developing
the truly stable API and other changes that will become 6.0, and when we
do we can increment libvpopmail.so to indicate the ABI difference.

I'm not sure how much I can do to help with this other than ideas, testing
and the occasional spurt of inspired programming, but I'm willing to do
what I can.

Josh

Joshua Megerman
SJGames MIB #5273 - OGRE AI Testing Division
You can't win; You can't break even; You can't even quit the game.
  - Layman's translation of the Laws of Thermodynamics
[EMAIL PROTECTED]

Re: [vchkpw] Shared libvpopmail thoughts

[Rick Widmer]

Joshua Megerman wrote:


1) A shared library with a stable API would make recompiling outside
programs (e.g., QmailAdmin) unnecessary, which would be a Good Thing(tm).


It is that 'stable API' that is the killer.  I know some ./configure 
options change the interface to libvpopmail.  I don't know which ones 
they are.  :(  I do know if you change some of the options you can get 
some spectacular failures if you forget to re-compile everything that 
uses vpopmail.


Once upon a time vpopmail was designed to be quick and tiny.  All 
options were compiled in.  Since then at least 3 of the back ends have 
adopted a configuration file.  Maybe it is time to look at moving most 
of the ./configure options to a configuration file and have only one 
vpopmail library interface for the entire life of a major (minor?) 
version.


Are we starting 5.5 or 6.0 if we change the library interface, and table 
layouts?

Re: [vchkpw] Shared libvpopmail thoughts

[Christopher Chan]

Also, just a supposition on my part, but if you're running (e.g.)
courier-authdaemon linked against libvpopmail.so all the time, wouldn't
that (theoretically) mean that other dynamically linked vpopmail programs
would run faster than the static version since the library would already
be loaded in memory?  If so, perhaps the speed solution for a dynamic
(e.g.) vdelivermail would be to run something that was dynamically linked
all the time, so libvpopmail stayed in memory...



Yes and no. If a third party program is statically linked the vpopmail 
calls are native to its memory space. So if the program is a long 
running process, it is actually faster than calling out to a shared 
vpopmail library. If the program is not a long running program, then it 
might take a bit longer due to the larger size of the binary.


The static library is a megabyte in size so it would be nice to have a 
shared version in case one has a program that is run on the order of 
hundreds concurrently.

Re: [vchkpw] Shared libvpopmail thoughts

[Len Burns]

Joshua,

I think this is a worthwhile endeavour for many of the reasons you
discuss below, as well as one more.  It makes linking applications
such as courier-authlib much easier.  I recently ran into trouble with
this under NetBSD-i386.  I did not know of a historical patch, I have
not been with vpopmail for long.  I ended up tweaking the build of
vpopmail-5.4.20 to create the shared library.  I also notice on this
OS that separate from this, the binaries are building as dynamically
linked.  I would definitely provide the option to build static; I
don't see a real need to do so selectively for different binaries.  As
regards performance, I am not qualified to speak but I doubt we would
see much difference.  Most modern operating systems are built on
shared libs.  Thanks to all for the hard work you do on vpopmail, it
is an impressive package.

-Len
On Mon, 24 Sep 2007, Joshua Megerman wrote:


Following up on my recent post to [vchkpw], I did some digging back into
the history of the shared libvpopmail patch and the discussions
surrounding it, and have the following thoughts.

First off, let me prefice this by saying that while I understand the
concept of shared libraries, I don't understand the underlying mechanics
of how the OS handles them, and thus am not sure exactly how can be
affected performance-wise.  That being said, here's my thoughts:

1) A shared library with a stable API would make recompiling outside
programs (e.g., QmailAdmin) unnecessary, which would be a Good Thing(tm).
2) There has been some question regarding performance of the vpopmail
programs when compiled against shared vs. static libraries.  I suggest the
following options be added for shared libraries at compile-time:
 a) --disable-shared - don't build libvpopmail.so, which is what vpopmail
does now.
 b) --enable-shared - build libvpopmail.so, but don't link the vpopmail
binaries against it - this gives other programs the ability to use the
shared library, but keeps the vpopmail binaries statically linked.
 c) --enable-shared-binaries - build libvpopmail.so and link the vpopmail
binaries against it.  Implies --enable-shared.
 d) possibly, if it's not to difficult, have a --enable-shared-binaries=
and/or --enable-static-binaries= option, which takes a list of binaries
to link against the stated library, and links the rest against the
other.  So you could have static vdelivermail and vchkpw, but not
vadduser, for example.  Not sure if that really is necessary, but static
linking does save space...
3) In all cases, even if the vpopmail binaries are linked against the
shared library, the static library libvpopmail.a should be built since
some programs expect it.

Also, just a supposition on my part, but if you're running (e.g.)
courier-authdaemon linked against libvpopmail.so all the time, wouldn't
that (theoretically) mean that other dynamically linked vpopmail programs
would run faster than the static version since the library would already
be loaded in memory?  If so, perhaps the speed solution for a dynamic
(e.g.) vdelivermail would be to run something that was dynamically linked
all the time, so libvpopmail stayed in memory...

Anyway, that's it for now - I haven't even tried the patch against the
latest vpopmail, though I'm guessing it should be fairly easy (albeing
possibly tedious) to integrate since it's not much in the way of actual
code changes...

Josh

Joshua Megerman
SJGames MIB #5273 - OGRE AI Testing Division
You can't win; You can't break even; You can't even quit the game.
 - Layman's translation of the Laws of Thermodynamics
[EMAIL PROTECTED]




--

"Do not go where the path may lead, go instead where there is no path
and leave a trail."
~ Ralph Waldo Emerson ~

[vchkpw] Shared libvpopmail thoughts

[Joshua Megerman]

Following up on my recent post to [vchkpw], I did some digging back into
the history of the shared libvpopmail patch and the discussions
surrounding it, and have the following thoughts.

First off, let me prefice this by saying that while I understand the
concept of shared libraries, I don't understand the underlying mechanics
of how the OS handles them, and thus am not sure exactly how can be
affected performance-wise.  That being said, here's my thoughts:

1) A shared library with a stable API would make recompiling outside
programs (e.g., QmailAdmin) unnecessary, which would be a Good Thing(tm).
2) There has been some question regarding performance of the vpopmail
programs when compiled against shared vs. static libraries.  I suggest the
following options be added for shared libraries at compile-time:
  a) --disable-shared - don't build libvpopmail.so, which is what vpopmail
does now.
  b) --enable-shared - build libvpopmail.so, but don't link the vpopmail
binaries against it - this gives other programs the ability to use the
shared library, but keeps the vpopmail binaries statically linked.
  c) --enable-shared-binaries - build libvpopmail.so and link the vpopmail
binaries against it.  Implies --enable-shared.
  d) possibly, if it's not to difficult, have a --enable-shared-binaries=
and/or --enable-static-binaries= option, which takes a list of binaries
to link against the stated library, and links the rest against the
other.  So you could have static vdelivermail and vchkpw, but not
vadduser, for example.  Not sure if that really is necessary, but static
linking does save space...
3) In all cases, even if the vpopmail binaries are linked against the
shared library, the static library libvpopmail.a should be built since
some programs expect it.

Also, just a supposition on my part, but if you're running (e.g.)
courier-authdaemon linked against libvpopmail.so all the time, wouldn't
that (theoretically) mean that other dynamically linked vpopmail programs
would run faster than the static version since the library would already
be loaded in memory?  If so, perhaps the speed solution for a dynamic
(e.g.) vdelivermail would be to run something that was dynamically linked
all the time, so libvpopmail stayed in memory...

Anyway, that's it for now - I haven't even tried the patch against the
latest vpopmail, though I'm guessing it should be fairly easy (albeing
possibly tedious) to integrate since it's not much in the way of actual
code changes...

Josh

Joshua Megerman
SJGames MIB #5273 - OGRE AI Testing Division
You can't win; You can't break even; You can't even quit the game.
  - Layman's translation of the Laws of Thermodynamics
[EMAIL PROTECTED]