On 2006-12-15, at 1144, Rick Widmer wrote:
John Simpson wrote:
On 2006-12-13, at 0211, Rick Widmer wrote:

Extra credit if the addresses are sorted like the /var/qmail/ congrol files so domains sort together.

not unless every single line in the .qmail file is an address.

The way I see it without a sequence field and order by clause you can't depend on the order of entries in a database. Maybe some are, but all it takes is a few deletes and adds and they will find out why they should not be depending on the order of entries...

Since you can't depend on the order now, I see no reason not to go ahead and sort the entries in a way that makes it easy to find the one you are looking for in a list.

actually, i use vpasswd.cdb and .qmail-* files, no database involved... which means that i CAN, and (for this one alias) i DO, depend on the order right now. i don't use the vpopmail API to modify this particular .qmail-* file, i edit it by hand.

i had one incident where the domain's admin didn't realize that this one alias was "special"- he accidentally added a new employee after the text searching program instead of before it. i ended up having to fix it by hand (after he realized that he couldn't fix it using qmailadmin and called me about it) and at the same time i made it owned by root and "chmod 644", so it works but qmailadmin can't change it by accident any more.

granted, it's the only such alias i have, and i know the people involved and their job functions well enough to be able to re-create the .qmail-help file from memory if needed, but not everybody knows (or wants to know) their clients' businesses that well.

If you need the entries to stay in the order you specify you need to be using the new calls that we are designing.

... or continue to use the current calls, and just not use a database back-end (which i've been doing for years.)

for this one alias, i would NEVER want this automatic sorting to happen, regardless of which back-end i happen to be using. the fact is that right now the file back-end is the only one that i *can* use, because it's the only one where any kind of sequencing is available, even if it's not explicitly controlled by vpopmail's API.

besides, i'm not sure i understand the point of sorting the entries, either within a .(q|vpop)mail file, or within the qmail control files. there are certainly no performance reasons, and if you're trying to encourage people to use the API instead of manually editing the files (and potentially introducing errors) then i would think as long as the files work the way they are, why worry about sorting them at all?

there are cases where program deliveries need to be processed in a specific order- i have one alias, for example, which always delivers to two mailboxes, then runs a program which does a text search, and if certain words are found in the body of the message, delivers to three other mailboxes as well.

I would say that now you are depending on blind luck that these entries stay in order, unless you already have some kind of sequence field. This needs to be done using the new calls, once they exist.

no, i'm depending on a .qmail-* file, and the fact that the lines are processed by qmail-local in the order in which they exist in the file.

The fun part is making a .qmail/.vpopmail file look like a table. I think it will be easier to efficiently make the file look like a database rather than treating the database like a file.

i'm not sure what you mean here. the format of a .qmail file is defined by what qmail-local understands, because if a .qmail file exists in a domain's directory, qmail-local processes it directly and vpopmail's executables never have a chance to modify how it works (unless the .qmail file contains an explicit call to "vdelivermail".)

Yes, that is one complication, we can't redefine the structure of the .qmail file. On the other hand I absolutely DO NOT want qmailadmin, vpopmaild or any other program built on top of this to have to know what back end is being used. I want to be able to swap out back ends without re-coding higher level programs.


the file-based back-end already has a working sequence for deliveries- it's implied by the order in which the lines appear in the .qmail file. the part which is lacking is the vpopmail API to do anything with this sequence.

adding it to the API is not difficult at all- just pick a new function prototype and write a function within each back-end module that does that job. the hard part is going to be the fact that everybody who is currently using a SQL-based back-end is going to have to convert their valias table from the existing "no sequence" format to the new "with sequence" format, and for any aliases containing multiple lines with at least one program delivery, manually verifying that the new sequences are correct in order to meet the needs of the owner of the mailbox.

i've always thought the best way to do it was to just add a sequence field to the SQL table containing the aliases, and have "vdelivermail" just duplicate what qmail-local would do if presented with a .qmail file containing the same lines in the specified sequence.

I see two approaches:

1.  Delete all entries, then re-add them in the desired order.

2. Provide a new way to manage ordered sets of data by adding an order field to the functions that add entries.

The first is drop dead easy for cdb, and not all that hard to code for a database, but really sucks for efficiency if you are using a database back end. Its not all that great for the user either.

The second is very easy to code for database, but will be much harder to code the cdb part. I think users would prefer it, I know I do. The extra work to make cdb look like a database for alias updates is "once work." You do it once and forget it.

actually, #2 is not really that hard at all. the trick isn't to make it look like a database table- just treat either physical representation as an ordered list. the sequence is there, but the sequence values are not as explicitly controllable as you might like. if you can live with the sequence values always being 0,1,2,3,etc. then it's just a case of writing the appropriate code and/or SQL queries.

I think maybe it should look something like:

//  load the cdb file, or BEGIN TRANSACTION on the database
update_alias( alias_name );

//  add an entry at a specific location
add_ordered_alias( alias_name, alias_line, sequence );
//  repeat as needed...

//  write the cdb file, or COMMIT TRANSACTION on the database

Once you 'open' an alias you can do multiple add and delete operations on its lines, with explicit control of the sequence. This provides hooks so you can read the cdb file, manipulate it in memory, and write it out.

your description sounds like you want to "open an alias" (which would return a handle of some kind), then do various operations (add line, delete line, reorder, etc.) on that alias/handle, then either commit () the alias (which would write the changes to the back-end and invalidate the handle) or rollback() the alias (which would invalidate the handle without saving any changes.)

i see where you're going, but i think it's a lot more general-purpose than we really need. other than allowing the command line "valias" command to insert or delete multiple lines, how often is one program invocation going to actually perform more than one operation on a given alias? not very often, if at all... which makes me think that the whole framework needed in order to support these kinds of "handles", and working on an in-memory image of an alias, are just overkill.

how about this instead?

alias_read ( alias_name ) ;
        // returns all lines, in sequence, separated by '\n'
        for file back-end:
                read the file
                return all text
        for database back-end:
                SELECT valias_line FROM valias WHERE ... ORDER BY seq

alias_line_read ( alias_name , sequence ) ;
        // returns one specific line from an alias
        for file back-end
                open file
                skip "seq" lines
                read one line
                close the file
                return that line
        for database back-end
                SELECT valias_line FROM valias WHERE ... AND seq = ___

alias_line_add ( alias_name , seq , newline ) ;
        // adds a new line, after existing line with sequence "blah"
// entries are re-numbered in the process, so that the sequence is always 0,1,2,3 etc.
        file back-end:
                chmod +t directory
                open existing and temp files
                copy "seq" lines from existing file to temp file
                write new line to temp file
                copy remainder of lines from existing file to temp file
                close existing and temp files
                chown() chgrp() chmod() temp file to match existing file
                rename() temp file over existing file
                chmod -t directory
        database back-end:
                BEGIN TRANSACTION
                UPDATE valias SET seq = seq + 1 WHERE ... AND seq >= ___
INSERT INTO valias ( ... , seq , valias_line ) VALUES ( ... , ___ , ________ )

alias_line_delete ( alias_name , seq ) ;
        // deletes line number "seq" from the alias
// entries are re-numbered in the process, so that the sequence is always 0,1,2,3 etc.
        file back-end:
                chmod +t directory
                open existing and temp files
                copy "seq"-1 lines from existing file to temp file
                read 1 line from existing file (but don't write it anywhere)
                copy remainder of lines from existing file to temp file
                close existing and temp files
                chown() chgrp() chmod() temp file to match existing file
                rename() temp file over existing file
                chmod -t directory
        database back-end:
                BEGIN TRANSACTION
                DELETE FROM valias WHERE ... AND seq=___
                UPDATE valias SET seq = seq - 1 WHERE ... AND seq > ___

alias_line_reorder ( alias_name , oldseq , newseq ) ;
        // moves an entry from one sequence to another
        all back-end modules:
                if ( oldseq == newseq ) return ;
                work = alias_line_read ( alias_name , oldseq ) ;
                alias_line_add ( alias_name , newseq , work ) ;
                alias_line_del ( alias_del , oldseq + 1 ) ;

I also think the commit phase should re-number the sequence fields in a real database so they are always nubered by 10's or 100's so there is room between entries. The cdb files will always be 'renumbered' because there is no place to store the sequence within the file.

at first i agreed with you about "leaving room" between sequence values (my first idea was to use a floating point number for the sequence field) but then i thought about it, and realized that by using transactions, adding an "UPDATE ... SET seq=seq+1 WHERE seq >= ___" before each add, and a -1 after each deletion, the API can very easily keep the sequence values as 0,1,2,3,etc. at all times.

| John M. Simpson - KG4ZOW - Programmer At Large |
| http://www.jms1.net/           <[EMAIL PROTECTED]> |
| Mac OS X proves that it's easier to make UNIX  |
| pretty than it is to make Windows secure.      |

Attachment: PGP.sig
Description: This is a digitally signed message part

Reply via email to