On 2007-01-08, at 1313, Rick Widmer wrote:
John Simpson wrote:

so you're going to design this new thing which is LIKE an alias, but it supports sequencing... and it can use the same code and probably the same database tables... but it's NOT an alias, simply because you give it some other name.

Yes, but it is not a _new_ thing. I want to freeze the alias interface
and provide a new interface for scripts.  Both can end up in either a
.qmail/.vpopmail file or a database.

here's a thought: the current function names become macros which call functions with new names and new functionality. the old API is preserved, and the new functionality is available with reasonable defaults to programs whose source code calls the old function names.

only two functions really need to be changed:

        valias_insert(alias,domain,line)
        valias_remove(alias,domain,line)

the functions would be renamed and gain a new parameter...

        valias_insertseq(alias,domain,seq,line)
        valias_removeset(alias,domain,seq,line)

and the existing function names become macros which call the new functions:

        #define valias_insert(a,d,l) valias_insertseq((a),(d),INT_MAX,(l))
        #define valias_remove(a,d,l) valias_removeseq((a),(d),-1,(l))

valias_insertseq() would be written to treat a "seq" parameter beyond the end of the alias as "insert after the end". the SQL versions would include logic to find the current highest sequence within the alias and replace the huge number with "max plus one", so that the lines within a given alias are always maintained as 0,1,2,3,etc. with no gaps and no duplications.

valias_removeseq() would treat a "seq" parameter lower than zero as "ignore the sequence and search for the text instead".


By having separate functions we can better optimize the update process
for its storage method.  I think most scripts will be the result of
template expansion or copied out of a text area and there will be more
than one line that needs to be added.  For example when writing to a
file the write-script function can copy an entire script with one open /
close / rename.  If I have to use alias calls each line added will
require the file to be copied to a temp file then renamed.

i see the value of the template processing stuff, it gives an administrator a way to pre-define templates for users who want spam filtered into a spam folder and things like that. it IS a good idea.

however, i see this implemented as a new function call:

int valias_replace ( const char *alias , const char *domain , const char **lines )

which would start by wiping the current contents of the alias (if any) and then just writing the new lines into the alias, with sequences 0,1,2,3,etc... all within a single BEGIN TRANSACTION ... COMMIT block (or simply by writing the new lines to a ".new.qmail" file, chmod()ing it as needed, then rename()ing it over any existing ".qmail" file.

Aliases are an unordered collection of delivery commands that you add or
remove a single entry at a time.

NO. aliases are, and have always been, a layer of abstraction above .qmail files, and are therefore ORDERED collections of delivery targets.

They stay what they are in 5.4.18 for a MySQL or pgsql user.

if current users are not expecting a specific sequence, and are used to not having any specific sequence, then how does it hurt things if suddenly they ARE sequenced? if nothing else, it's still random... it's just that it's the same random every time.

besides, if every line in an alias is an email address, then the sequence (or lack thereof) doesn't matter.

The key to an alias is that you work with it one address (line) at a time. An email coming in to an alias when you have only written two of three destination addresses isn't as much of a disaster as a script that is not completely written.

you're thinking of the race condition where a delivery might happen while a .qmail file is halfway through being written. there are two solutions to this:

- djb's solution, which is to enable the sticky bit for the "home directory", which suspends all deliveries to that directory (for vpopmail, the "home directory" is the domain directory.) this is less than optimal, because it suspends all deliveries for the entire domain, rather than just the one alias you're updating. it also doesn't work with vdelivermail, because vdelivermail doesn't look at the sticky bit (it should.)

- update the files or entries in a manner which is guaranteed to be atomic, either by using BEGIN TRANSACTION ... COMMIT, or by writing your new collection of lines to some other filename, setting its permissions, and then using rename() to give it the ".qmail" name which makes it "live". this is better.

A mail script is an ordered collection of delivery commands that is
created or updated as a group.  The key to a script is that you update
the whole thing at once.  I know there are multiple lines to most
scripts so its handler should accept all lines before making the script
available to receive mail.  If you use the existing add alias call and
the script receives an email when only two of three lines have been
written you may have a disaster.

hence the new valias_replace() function, which writes an entire alias at once, in an atomic manner (again, using BEGIN TRANSACTION ... COMMIT, or writing to a temp file, chmod()ing it, and rename()ing it.)

I'm thinking a pointer to an array of string pointers is a good way to
pass scripts around at the library level. The daemon will pass a block
of lines terminated with a line containing just a period.  The command
line program will work like crontab -e.

however you want to do it... although i think if you're going to use an array of string pointers, the "end of array" marker should be a NULL pointer, like how argv and envp are done. the code is easier to write that way.

and for the maintenance programs, it's one extra parameter passed to (1) the command, (2) the vpopmail.c stub function, and (3) the various back-end functions... and the extra code added to each back- end function.

One extra parameter to the valias functions means everyone that is
currently using them has to modify existing code when they upgrade.

not if you re-define the existing names as macros which call the new functions and provide a reasonable default for the new parameter.


the hard parts are going to be coordinating the changes to all of the back-end functions at once, and then testing all of it. if somebody has the time, it won't be difficult for one person (myself, if needed) to write the changes for all of the back-end libraries and add them all to the CVS repository. finding testers will be interesting though... i'm sure there are no shortage of cdb and mysql users, but what about the others? not just finding users, but finding people who are able to TEST the new code, without directly impacting their existing services? i could probably set up a test box with postgresql on it, but the others (oracle, sybase, and LDAP) i wouldn't be able to touch.

MySQL and pgsql are the only back ends that have valias support code at
all.  The rest have very limited functionality compared to the three
front runners.  Even pgsql probably won't get more that a test to make
sure the code compiles until someone who wants to use it downloads a
copy.  I think it was over a year between when the the last pgsql bug
was added to the code and someone actually tried to use it.  It was
fixed within a few days.

i think, since only two functions need to be changed, it won't be such a big deal.

My policy is that cdb and mysql have to have matching apis, and get
tested before release. Changes to mysql are ported to pgsql, and I make sure it compiles. The rest only get big changes like vauth_open which I
considered very important for all back ends to support.  I think that
meets or exceeds what has happened in the past.

until/unless somebody using one of the other back-ends complains about it, i guess that's reasonable.

Until someone needs a new function in one of the other back ends it
probably won't happen.  At the very least they would have to provide
access to a system where one of the developers could work. More likely
they will do the work using one of the leading backends as an example
and submit patches.  The only way the other back ends will get more
support is if one of its users steps up. (I think we recently dumped a
back end that hasn't been updated in years, and there may be more that
can go.)

maybe we should do a survey- how many people are actually using vpopmail with oracle, sybase, or LDAP?


and that's certainly an option. however, i'm sure there are users now who rely on being able to edit the contents of an alias by modifying an SQL table, even if they can't currently control the order in which the instructions within the alias will be processed by the delivery program. i don't think it's right to take away their current partial functionality and force them to find an entirely new solution (a way to edit text files, or to start using vpopmaild.)

Currently there is no partial functionality for ordered alias entries in
a database.

i consider it partial- they have alias entries, but they're not ordered.

and once we add the sequence field and add the macros so that the old source code still compiles, they will start seeing it ordered whether they expect it or not.

if they're used to the addresses coming out in a random order, they won't have any expectations about ordering- which means if they suddenly start coming out in the same order every time, they still won't care.

I want to add a new api so we don't require existing code
to change. Everything that works now continues to work in exactly the
same way.  If you need order, use the new set of functions to edit.

and if you don't need order, you use the macros- which still call the new functions.

As far as where to implement the template system, its a choice between
implementing it in both qmailadmin and vpopmaild, or just vpopmail, I'd rather put it in the library. It would be a layer above the script api.

again.... valias_replace().

The only problem with putting it there is the need to provide programs
to manage the templates distributed with vpopmail.  (Black hole,
autoresponder, vacation, ?)  Oh well... I guess if I add script
templates, I need to add the programs to manage them.

you'll also need some kind of substitution engine... a regex library would work nicely, unless you really want to write your own (which i have done, in perl.)



and if both processes are doing this "stat, sleep, stat" routine at roughly the same time, what's to prevent the same race condition from happening during the second round of stat() calls?

Vdelivermail should fail if the second attempt to open both files fails.
Maybe it should just fail (temporary) if the first attempt fails.  By
the time qmail schedules another delivery attempt the writer process
should be done.

this kind of thinking is where race conditions come from... "by the time the other process does so-and-so, such-and-such SHOULD BE done." if you cannot guarantee that it will be done, either by writing the code to ensure that it's done, or by falling back to explicit statements in the API, then you cannot safely assume that it will be done.

my OS design instructor called this racism, because it leads to race conditions. he drilled into our heads, how to think about concurrently running processes and explicitly arbitrating access to shared resources which should only be used by one process at a time... after three months it became like second nature to me.



I don't think it checks the sticky bit of the directory, but that is
probably the directory level lock we should use since it is used by
qmail-local.

yup.

except that if two processes SET the sticky bit at the "same time" (i.e. close enough that neither one realizes that the other one has done it) and then one CLEARS it before the other is ready to have it cleared... bingo, there's another race condition.

this is why none of djb's code ever manually manipulates the sticky bit- it reads it, and honours whatever setting it dictates, but it doesn't change it. this way the only process which ever changes the sticky bit should be the between the administrator's ears.



i hate to say it, but the more i think about it, the more i think it's not such a good idea to try and rename files "on the fly" like this. (keep reading, the next bit explains the other reason why.)

If it does not rename on the fly, then the rename needs to be part of
make install.  I don't want to have to depend on the user to do it.

i can understand that... but there's only so much you can do to protect people. the administrator (what you and i think of as "the user", the person out there who's administering a vpopmail system) needs to know, and needs to control, what's going on with his own server. i really think the renaming process needs to be an explicit step that they do, on their own, while qmail-send is not running... just so that they KNOW it's been done.

obviously you warn people, maybe so far as to including something like requiring a new undocumented ./configure option "--i-have- renamed-the-dot-qmail-files-to-dot-vpopmail" before allowing the new version to compile without a documented "--stick-with-the-old-dot- qmail-names" option.



i respect charles and his opinion about .qmail and .vpopmail files... but making charles happy is not a primary goal of vpopmail, and it's certainly not an excuse to unsafely force this change on all of the existing vpopmail systems out there. i think it makes more sense to explain the situation to the administrators, provide them with a tool to manually rename the files en masse (and identify any problem cases where both .qmail and .vpopmail files already exist) and TELL them that it should be done- but the final decision about whether and when to do the change should be left in the hands of the administrator of each system.

OK, but the default needs to be on.  Experts can decide to stay with
.qmail files, newbies get .vpopmail. Its the newbies who don't read the documentation who need .vpopmail files the most. If it can't default on
it should not be done at all.

that makes sense to me.



i can SORTA see having a list of domains within the server, a list of users within a domain, or a list of aliases within a domain, and keeping those lists in a pre-sorted format... and while i think it's a waste of time and CPU cycles, it doesn't really hurt anything- because these entities (domains, users, and aliases) are all things which exist independently of each other. there is no NEED for them to be maintained in a certain order within their parent sets, therefore it's harmless if you want to sort them for cosmetic reasons.

Once upon a time I was testing vpopmaild using a test setup with 1500
domains and one domain with over 3000 users. List operations took over
...

o vpopmail insert operation - The winner.  For file based storage an
insertion sort is used to maintain order in the file.  The database
needs to use ORDER BY.  If you are worried about wasting cpu cycles,
this is by far the winner.  The sort only happens when the data is
stored, not each time its read.

I was amazed how easy the sort was.  The program already copied the
source file to a temporary file then wrote the new entry before closing
both files.  Changing from this 'append at end' model to an insertion
sort was simple.  I also added a second if that looks for out of order
entries in the file and causes the file to be sorted if it finds any.

hrmmm... actually the way you've explained it makes sense. and the fact that (1) you tested it and can point to concrete results, and (2) it's already written and working... i'll give you that one.



however, the set of lines within an alias is a different beast entirely. there IS a need for them to be stored in a specific sequence, because when they are being used to control a delivery, they need to be processed in a certain order.

So use the new functions instead of the existing valias functions which
don't have any way to control sequence.

yes and no... yes to "use the new functions", no to "instead of".

if somebody uses the old function name to add a line without specifying the sequence, it goes at the end. they end up getting the same "sorta" sequence mechanism that the .qmail files give them right now.


I'm OK if you want to try to extend the functionality of Vpopmail to do more stuff, just please don't break the current way of doing things.

I don't think I'm breaking any existing functionality.  If you see
something that my plans would break, please let me know.  Backwards
compatibility is very important.

unfortunately, the current design of vpopmail was broken to begin with. the original design of the alias-related commands should have included support for sequencing from the start. it never did. whatever scripts or web interface you're using now will have to be modified, unless you're happy with only being able to add new lines to the end of an existing alias.

I wish we could redesign aliases from scratch today, but they have been
in use for years now.

but how many people are actually using the vpopmail API to edit aliases? other than qmailadmin and courier-authlib, i'm not aware of any... and even if there are, by providing the macros we make it possible for older source code to still compile with the new stuff, and instead of "randomly sorted" they start doing "sorted by order of insertion" or something like that... and if they were used to "random" before, then they're not expecting any particular order- so they probably won't care if the entries are suddenly in the same order every time... just as i hadn't really noticed when suddeny my rcpthosts, virtualdomains, and users/assign files were suddenly in alphabetical order.

using the macros will ensure that existing source code won't have to be changed, unless they specifically want to control sequences. otherwise, existing code will work, the only difference is that instead of being random, any entries added without an explicit sequence will end up being in the order they were added- which matches the current behavior of the .qmail file mechanism anyway.



and it could be done. the only reason i can see for not wanting to do it is because it does involve writing and testing more code. however, i think we as developers need to remember that the point of writing a program is to make our users' lives easier, not to make our own programming jobs easier.

I think making the users life easier means not changing two + year old
valias code, and providing a separate way to edit ordered data.  Any
changes we make to the alias functions will require existing users to
modify their code.

all it takes is two macros for add sequencing support to aliases, and one new API function to support atomically creating or editing multi- line aliases (or "scripts", to use your term.)

the API changes, the old API still works, and we add native support for multi-line aliases- which provides the perfect tool for your templates framework.

        #define valias_insert(a,d,l) valias_insertseq((a),(d),INT_MAX,(l))
        #define valias_remove(a,d,l) valias_removeseq((a),(d),-1,(l))

int valias_replace( const char *alias , const char *domain , const char **lines ) ;

the only things which WILL need to change are:

- any external applications which modify the valias table directly. vpopmail will distribute a "contrib" script to go through the valias table and add sequence numbers to all of the entries- single-line aliases all get sequence 0, and multi-line aliases get 0,1,2,3... in random order.

i'm assuming that anybody who has written such a beast- any kind of interface or plug-in for some other system which uses the valias table- is reading this list... i think it's safe to say at this point that there will be a field called "seq" added to the valias table, it will be an integer (32-bit, unsigned if the database engine supports it) and the values within a given alias will be 0,1,2,3... so if you want to start planning ahead, now is the time to do it.

- any shell scripts which call the "valias" command, and want to start explicitly controlling sequences.

- vpopmaild... ideas:

        add_alias [EMAIL PROTECTED] target (current)
                adds "target" as a new line at the end of the alias

        add_alias [EMAIL PROTECTED] @2 target (new)
                inserts "target" before line 2 of the alias

        remove_alias [EMAIL PROTECTED] target
                removes any line from the alias which matches the target

        remove_alias [EMAIL PROTECTED] @2
                removes line 2 from the alias

        replace_alias [EMAIL PROTECTED]
        target
        target
        target
        .
                creates or re-creates a multi-line alias (script)

the "@2" idea will work, i think, because "@" is not a valid character for the first column of an alias line. (my original idea was to use "#2", but somebody might want to put comments into their aliases...)

----------------------------------------------------------------
| John M. Simpson    ---   KG4ZOW   ---    Programmer At Large |
| http://www.jms1.net/                         <[EMAIL PROTECTED]> |
----------------------------------------------------------------
| http://video.google.com/videoplay?docid=-4312730277175242198 |
----------------------------------------------------------------


Attachment: PGP.sig
Description: This is a digitally signed message part

Reply via email to