First here is the into to the documentation for what I have in mind for
vpopmail script handling, then I'll answer points from John's email.



vpopmail mail script handling.

Vpopmail provides a way to manage mail scripts that allows you to choose
at configure time between storing scripts in the database or in .qmail
or .vpopmail files.  The mail script api is available for the cdb back
end, but the configure options to use a table for storage are not available.

Actually, for historic reasons, there are two api's available.  The
first, aliases stores a list of addresses to forward messages to.  You
are allowed to deliver to a program, but can not depend on the order of
entries in an alias.  Aliases are controlled by --enable-valias which
causes vpopmail to store aliases in the databases.

By the time lack of support for order in aliases became a problem there
were a number of people using valias in programs they have written.
Rather than changing the existing api ordered mail scripts have been
provided by adding a new api, mail scripts.  Mail scripts are controlled
by --enable-scripts which causes vpopmail to store scripts in the database.

Depending on the settings of --enable-valias and --enable-scripts [1]
both may be stored in the same place.  If so, it is safe to edit an
alias as a script, but editing a script as an alias may not retain the
order of lines within the script, causing it to fail.

Mail scripts allow the calling program to store or retrieve the script
as a .qmail or .vpopmail file, even when configure options specify it
should be in the database. [2] If a file and database entry both exist,
the file is executed and the database entry is ignored. [3]  Files in
the domain directory are executed by qmail-local, so they don't have the
overhead of executing vdelivermail.  Files in user level directories are
executed by vdelivermail so there isn't as much advantage to using real
files there.


[1] Both of the configure options probably need to be renamed to reflect
that they cause their associated data to be stored in a database table.

[2] This can be used if the user chooses to use files instead of the
database, and for programs like vconvert and dotqmail2valias which need
to access both files and the database.

[3] Sometimes you may have to delete a script twice, once to delete the
file, then to delete the database entry exposed when the file disappeared.


Note that this provides two switches that allow the system administrator
to control storage of aliases and scripts separately.  It also allows
the programmer to chose to use a file to store a script even when the
system administrator has selected database storage.


John Simpson wrote:

so you're going to design this new thing which is LIKE an alias, but it supports sequencing... and it can use the same code and probably the same database tables... but it's NOT an alias, simply because you give it some other name.

Yes, but it is not a _new_ thing.  I want to freeze the alias interface
and provide a new interface for scripts.  Both can end up in either a
.qmail/.vpopmail file or a database.


explain to me how your "scripts" are any different from "aliases with a sequence field", or "a .qmail file". either way, the functional result is the same: a set of delivery targets, guaranteed to be processed in a certain order.

The calling conventions, and the idea that having two sets of calls
allows you to optimize how things are handled in the back end.  For
example most scripts I can think of have two or more lines, and you
don't want to attempt a delivery to a partially written script.

By having separate functions we can better optimize the update process
for its storage method.  I think most scripts will be the result of
template expansion or copied out of a text area and there will be more
than one line that needs to be added.  For example when writing to a
file the write-script function can copy an entire script with one open /
close / rename.  If I have to use alias calls each line added will
require the file to be copied to a temp file then renamed.

Aliases are an unordered collection of delivery commands that you add or
remove a single entry at a time.  They stay what they are in 5.4.18 for
a MySQL or pgsql user.  The key to an alias is that you work with it one
address (line) at a time.  An email coming in to an alias when you have
only written two of three destination addresses isn't as much of a
disaster as a script that is not completely written.

A mail script is an ordered collection of delivery commands that is
created or updated as a group.  The key to a script is that you update
the whole thing at once.  I know there are multiple lines to most
scripts so its handler should accept all lines before making the script
available to receive mail.  If you use the existing add alias call and
the script receives an email when only two of three lines have been
written you may have a disaster.

I'm thinking a pointer to an array of string pointers is a good way to
pass scripts around at the library level.  The daemon will pass a block
of lines terminated with a line containing just a period.  The command
line program will work like crontab -e.


and for the maintenance programs, it's one extra parameter passed to (1) the command, (2) the vpopmail.c stub function, and (3) the various back-end functions... and the extra code added to each back- end function.

One extra parameter to the valias functions means everyone that is
currently using them has to modify existing code when they upgrade.


the hard parts are going to be coordinating the changes to all of the back-end functions at once, and then testing all of it. if somebody has the time, it won't be difficult for one person (myself, if needed) to write the changes for all of the back-end libraries and add them all to the CVS repository. finding testers will be interesting though... i'm sure there are no shortage of cdb and mysql users, but what about the others? not just finding users, but finding people who are able to TEST the new code, without directly impacting their existing services? i could probably set up a test box with postgresql on it, but the others (oracle, sybase, and LDAP) i wouldn't be able to touch.


MySQL and pgsql are the only back ends that have valias support code at
all.  The rest have very limited functionality compared to the three
front runners.  Even pgsql probably won't get more that a test to make
sure the code compiles until someone who wants to use it downloads a
copy.  I think it was over a year between when the the last pgsql bug
was added to the code and someone actually tried to use it.  It was
fixed within a few days.

My policy is that cdb and mysql have to have matching apis, and get
tested before release.  Changes to mysql are ported to pgsql, and I make
sure it compiles.  The rest only get big changes like vauth_open which I
considered very important for all back ends to support.  I think that
meets or exceeds what has happened in the past.

Until someone needs a new function in one of the other back ends it
probably won't happen.  At the very least they would have to provide
access to a system where one of the developers could work.  More likely
they will do the work using one of the leading backends as an example
and submit patches.  The only way the other back ends will get more
support is if one of its users steps up.  (I think we recently dumped a
back end that hasn't been updated in years, and there may be more that
can go.)


There are good arguments that scripts are best implemented  as .qmail or
.vpopmail files but vdelivermail already supports all the delivery
methods from a database, and it will continue to do so.

actually, this i agree with. as long as vdelivermail can be taught to add an ORDER BY clause to the SELECT query it uses to fetch the individual "lines" from the alias "file" in the database, there should be no difference whether the instructions come from a text file or a database.

I agree.  I also believe we are pretty close.  Vdelivermail doesn't need
to know know if it is processing an alias or a script.  The difference
is only in the library calls you use to _edit_ the data.


and that's certainly an option. however, i'm sure there are users now who rely on being able to edit the contents of an alias by modifying an SQL table, even if they can't currently control the order in which the instructions within the alias will be processed by the delivery program. i don't think it's right to take away their current partial functionality and force them to find an entirely new solution (a way to edit text files, or to start using vpopmaild.)

Currently there is no partial functionality for ordered alias entries in
a database.  I want to add a new api so we don't require existing code
to change.   Everything that works now continues to work in exactly the
same way.  If you need order, use the new set of functions to edit.


Once the script functions are added I plan to add a template system  that
allows the system admin (or anyone with admin and expert abilities) to
manage the templates.

that's fine (actually it's a great idea) but i think it's external to the core of vpopmail. what it all comes down to is that an "alias" is nothing more than an ordered set of instructions, stored somewhere- either in a .qmail file, a .vpopmail file, or a database table.

The problem is, a valias has been defined as an unordered set of
instructions for over two years now.  I don't want to change anything
that already exists.  (Other than maybe sorting the alias addresses for
display since you can't depend on their order now anyway.)

As far as where to implement the template system, its a choice between
implementing it in both qmailadmin and vpopmaild, or just vpopmail, I'd
rather put it in the library.  It would be a layer above the script api.
 The only problem with putting it there is the need to provide programs
to manage the templates distributed with vpopmail.  (Black hole,
autoresponder, vacation, ?)  Oh well... I guess if I add script
templates, I need to add the programs to manage them.


the easiest "safe" way around this would be to assert an flock() on the directory (or on a specific filename within the directory) which contains the .qmail and/or .vpopmail files, before searching for either filename... and only release the lock after reading the contents of whichever file you're going to use (if any) into memory... and also ensuring that every piece of code in the vpopmail suite (including any external management scripts which may have been written by others) which touches these files, participates in (and honours) the flock() scheme.

The way I plan to handle things is to copy the existing code that
manages the files in /var/qmail/control.  It already has locking,
creating a temporary file then using rename() to change from one file
version to another.


The open needs to handle the case where the file suddenly disappears,

which you would detect HOW? by doing another stat()?

By the fact that the fopen in read only mode fails even though the stat
on the line before it says there should be a file to read.


and if both processes are doing this "stat, sleep, stat" routine at roughly the same time, what's to prevent the same race condition from happening during the second round of stat() calls?

Vdelivermail should fail if the second attempt to open both files fails.
Maybe it should just fail (temporary) if the first attempt fails.  By
the time qmail schedules another delivery attempt the writer process
should be done.  I don't think it checks the sticky bit of the
directory, but that is probably the directory level lock we should use
since it is used by qmail-local.

The writer process should create the .vpopmail file before it deletes
the .qmail file.  Maybe there should be short sleep to make sure there
is never a time where neither file exists for other processes.  Even
without the sleep I don't think there will be a time where the reader
won't find one file or the other.


i hate to say it, but the more i think about it, the more i think it's not such a good idea to try and rename files "on the fly" like this. (keep reading, the next bit explains the other reason why.)

If it does not rename on the fly, then the rename needs to be part of
make install.  I don't want to have to depend on the user to do it.


i respect charles and his opinion about .qmail and .vpopmail files... but making charles happy is not a primary goal of vpopmail, and it's certainly not an excuse to unsafely force this change on all of the existing vpopmail systems out there. i think it makes more sense to explain the situation to the administrators, provide them with a tool to manually rename the files en masse (and identify any problem cases where both .qmail and .vpopmail files already exist) and TELL them that it should be done- but the final decision about whether and when to do the change should be left in the hands of the administrator of each system.

OK, but the default needs to be on.  Experts can decide to stay with
.qmail files, newbies get .vpopmail.  Its the newbies who don't read the
documentation who need .vpopmail files the most.  If it can't default on
it should not be done at all.


of course i would also remove the "either/or" filename logic from the NEXT version of vpopmail, so that if they haven't renamed the files, they become broken and it's their own fault.

<Evil grin!>  Sounds good to me.  I'm not sure its realistic though.


i can SORTA see having a list of domains within the server, a list of users within a domain, or a list of aliases within a domain, and keeping those lists in a pre-sorted format... and while i think it's a waste of time and CPU cycles, it doesn't really hurt anything- because these entities (domains, users, and aliases) are all things which exist independently of each other. there is no NEED for them to be maintained in a certain order within their parent sets, therefore it's harmless if you want to sort them for cosmetic reasons.

Once upon a time I was testing vpopmaild using a test setup with 1500
domains and one domain with over 3000 users.  List operations took over
a minute to paint a web form.  Each time I wanted to paint a page of
entries I had to download the entire list, sort it then display the page
of interest.  How to fix it?  Move the sort.  Where to move it?

o Client - BAD!  Must extract, transfer, sort and select all the data
for each page sent.

o vpopmaild - Data is extracted, sorted and selected on the server, only
the desired records are transferred to the client.  Much faster because
the selection is done on the server, but the sorted data is only
available to daemon clients.

o vpopmail list operation - Similar to vpopmaild, except sorted data is
available to qmailadmin and other programs built on the library.

o vpopmail insert operation - The winner.  For file based storage an
insertion sort is used to maintain order in the file.  The database
needs to use ORDER BY.  If you are worried about wasting cpu cycles,
this is by far the winner.  The sort only happens when the data is
stored, not each time its read.

I was amazed how easy the sort was.  The program already copied the
source file to a temporary file then wrote the new entry before closing
both files.  Changing from this 'append at end' model to an insertion
sort was simple.  I also added a second if that looks for out of order
entries in the file and causes the file to be sorted if it finds any.


however, the set of lines within an alias is a different beast entirely. there IS a need for them to be stored in a specific sequence, because when they are being used to control a delivery, they need to be processed in a certain order.

So use the new functions instead of the existing valias functions which
don't have any way to control sequence.


to me it looks like this all started as code patches to make vpopmaild able to handle "list_" commands which may return multiple items- basically, being able to serve up "page 4 of 19" from some list.

Exactly.  It was the only way I could get acceptable performance with
more than a handful of entries in a web based environment where every
operation has to start from ground zero.


honestly, i think those kinds of issues belong with the client program which calls vpopmaild, rather than with vpopmaild itself. sure, it makes it easier to write a web interface if the server back- end can sort and paginate the list for you... but the sorting and pagination are functions of the program which is doing the presenting- in this case, the web interface itself.

Having to transfer an entire large list to the client for sorting and
selection is very inefficient.  Small lists seemed to work well, but
about 100 entries or so the delay is annoying, 1500 entries was
unusable.  I know there are larger sites out there.


how much easier would vpopmaild be to write and maintain if it didn't have to deal with these kinds of issues- if it could just deal with the raw data, in the order it finds it?

Sure it is easier to write, but its unusable for large sites.


look at how many changes have been made to other parts of vpopmail over the past few months, just to keep things like rcpthosts and virtualdomains and users/assign in some cosmetically sorted order... and for what? to save some PHP programmer from having to call a sort() function?

No, to save the poor user vast quantities of wasted time managing a
server with a large number of entries.

If you look closely, I think you will find the changes very simple, and
there is very little additional overhead most of the time.  Unless you
muck with the files, they will need to be sorted the first time they are
accessed, and the insertion sort will keep then in proper order from
then on.

The fact that rcpthosts and virtualdomains got sorted is a side effect
of sorting users/assign.  It turns out vpopmail uses the same function
to write all of those files so I was faced with the question of sorting
them all, or having to have multiple functions to write the files.  I
sorted them all.

there is a line between "data" and "presentation"... and i think it's been drawn in the wrong place.

Have you timed the daemon when it is returning 1500 domain entries so
the client can sort them?  I have.  It sucks.  You won't wait that long
for a web page!


3) Most importantly, to me the idea of storing aliases in the DB is just
that - it's for aliases.  If you have a program that you want to pipe
the message through, or send a copy of your mail offsite, I'd much
rather have that be via .qmail file and not as a database line.


i don't recognize this paragraph, although it's written in a style similar to my own so i may have written it... regardless, it brings another point to mind- one which is probably the heart of this whole issue.

Sorry, I probably forgot to attribute this block to Joahua Mergeman.


what exactly do we mean when we say "alias"?

I consider an alias to be the subset of mail scripts where the message
is delivered to one or more recipients, including possibly programs, but
the order of entries is not defined.  Aliases are edited one line at at
time.  Using this definition allows the existing valias code to continue
working unchanged.


if it corresponds to a .qmail file, then we DO need a way to store sequences in the databases, and we DO need ways to maintain those sequence values using the command line tools, and using vpopmaild, and using qmailadmin. and 80% of this discussion goes away- we can stop trying to figure out whether or not to make certain changes, and we can figure out how to do them in such a way as to provide the functions people need while changing as little of the API, command line tools, and vpopmaild, as possible.

Yes, but we don't have to use the existing valias calls to do it with.
If order is important, that implies that there is more than one line,
and you probably want to make sure the entire script is written before
it becomes available to receive mail.  Writing multiple lines before
making an alias available certainly wasn't a consideration when the
existing alias code was written.


and if somebody's going to make the executive decision that "alias" is just a bag of email addresses, and that sequence is not important... then storing my own aliases in any database back-end becomes impossible (no great loss for me personally, i'm happy with the cdb files) and people who do use database back-ends remain limited in the kinds of aliases they can set up.

It does not make it impossible, it just requires you to use a new,
separate interface to edit the entries using the library, or follow some
special rules on how you manage the sequence field.  (All sequence
values in an alias are 0, and all sequence values in a script must be
unique and non-zero.  If you have to co-exist with qmailadmin or
pmailadmin and the valias and vscript commands make the first line a
comment that identifies what kind of script it is.)  You should also be
sure to put a transaction around any update of a script to make sure it
is an all or nothing update.


(1) add a sequence field to the database, and write the code necessary to support it. this will result in the users of the database back-ends GAINING the same functionality that users of the cdb back-end already have- namely, the ability to sort the lines within a given alias, in a certain order.

I think we agree on what to do to the database, and how vdelivermail
will handle it.  The only disagreement is you want to mess with the
existing alias calls, and I want to provide a new set of calls to edit
ordered scripts.


I'm OK if you want to try to extend the functionality of Vpopmail to do more stuff, just please don't break the current way of doing things.


I don't think I'm breaking any existing functionality.  If you see
something that my plans would break, please let me know.  Backwards
compatibility is very important.


unfortunately, the current design of vpopmail was broken to begin with. the original design of the alias-related commands should have included support for sequencing from the start. it never did. whatever scripts or web interface you're using now will have to be modified, unless you're happy with only being able to add new lines to the end of an existing alias.

I wish we could redesign aliases from scratch today, but they have been
in use for years now.


and it could be done. the only reason i can see for not wanting to do it is because it does involve writing and testing more code. however, i think we as developers need to remember that the point of writing a program is to make our users' lives easier, not to make our own programming jobs easier.

I think making the users life easier means not changing two + year old
valias code, and providing a separate way to edit ordered data.  Any
changes we make to the alias functions will require existing users to
modify their code.





Reply via email to