Hi list,
I am in the works of building a Xmail-RPC (Remote Procedure Call) system for
handling external filter/mailproc procedures via a rpc deamon and I would like to
hear any comments, suggestions or ideas or whatever concerning how to implement
this thing correctly right from the start. If you are a scriptwise knowledged guy
(PERL / PHP / Python / WSH) or a C/C++ dude you are most welcome with suggestions,
but ofcourse, any XMail user, please do read on and let me hear your point of view.
Preamble.
------------------------------------------------------------------------------------------
Anyone having used scriptbased external filters/mailproc stuff in a XMail context
(or in any interpreted language context for that matter) knows that this causes a lot
of extra
burden on the CPU since a script interpreter has to be re-launched for each call. In
the case
of PERL on a Win32 box a standard "external" 2KB script causes an additional 3.5+ MB
loading/unloading of the PERL interpreter + autoloaded modules just in order to
execute it.
Using WSH this is about 1MB extra.
Well, if you are running a 1000 mail/day box, this shouldn't pose much of a problem,
but if running a 10000+/day it might bring your CPU and HDD to its knees fairly
severely
from time to time. Now, my setup is not target for such amounts of mails a day,
but my XMail is running on a multi-service box (HTTP/NNTP/SMTP/FTP) that
all uses a fair amount of load. And since my box isn't a Sun HP 10000 with RAIDed
SCSI multi-arrays, if I can reduce load in any way I am willing to look into it.
Performance has always been an issue with using script based languages, this is why
high-load CGI isn't very common nowadays. Launching another X MB to execute a
script with Y KB size isn't very efficient in the long run. The solution is interpreter
persistance and bytecode caching. Having the script interpreter in RAM already,
makes the script execute with a minimum of overhead. Apache's mod_perl, mod_python,
mod_rexx, mod_whitebeam, ASP and PHP and any other modern HTTP based script
language for instance behaves exactly in this manner. The difference here is that
mentioned
solutions all are linked in, statically or dynamically with the main application at
compile-time
or at startup and therefore the communication between the main application and the
particular interpreter are carried out "in-process", while my Xmail-RPC idea depends on
an external communication layer namely UDP via XMail and script interpreter carrying
out the "external" filters/mprocs operations with pre-compiled bytecode.
(hey, isn't that what that big $+ PHP's ZEND Accelerator is all about :)
After doing some thinking, I came up with the idea of knock up a RPC system for
handling CPU expensive scriptbased filters/mprocs under XMail to a level close
to a compiled language, still with the maintained versatility of a scripted language.
The RPC system concists of two parts, a client and a server, utilizing a connectionless
UDP datagram protocol as transport-layer, since it is faster than common TCP
for repetative exchange of data.
The Client. (xmrpcc)
---------------------------------------------------------------------------------------
Being fairly new to C, with 20% cut n' paste and 80% trial and error, I now have
a to me stunningly super-tiny rpc-client working, compiled size 4KB / 16KB under
Win32 / Linux. (didn't thought it was possible to achieve a 4KB app with Win32 VC)
Default server address 127.0.0.1:7887. Executed from a filter.tab or a mailproc.tab,
cmdalies, custdomains it simply passes on the call for procedure and any defined
arguments to the RPC-server AND the turns the response into its OWN
exitcode. (0, 97-100).
The client could be utilized in the following situations:
* Filters
* Mailprocs
* CmdAliases
* Custdomains
* ...hey, davide. what about those outbound filters you spoke about some months ago.
*** C/C++ guys, any help appreciated in improving the code... ***
The Server (aka. daemon) (xmprcd)
----------------------------------------------------------------------------------------------
Written in PERL (... could by any scriptbased language really)
Sits in the background listening on an UDP port for orders from a the client carrying
them out as per defined procedure id. A procedure is actually only another name for
a pre-compiled bytecode-chunk representation of a PERL module stored in-cache.
Besides the big win of the in-cache bytecode the second main idea about the server
is that anyone fairly experienced with PERL could knock up and add a new
procedure for his special needs and share it with the community if applicable.
I have a XMail::RPC::Procedure API on the way that will make it close to trivial
to create a new module with only few lines of code.
(read section "Server modules" below)
Preferrably this daemon should run on the same box as XMail itself,
but theoretically it could be located in Bagdad, Antarctica or in a Russian
Sub roaming longside California coastline as long as it sits on a box connected
to the Internet.
The latter scenarios however would require some vastly obnoxious long-distance
shares/nfs/samba/smbfs mounting if we wanted the server to muck with the actual
message structure locally. But if we had the RPC-server on a local LAN on the
other hand, we would call it distributing the workload, which is a scenario
pretty common nowadays.
Upon a procedure call from the client, the server executes the mapped code
assigned to that procedure. It could be a (or a series of procedures)
simple content scanner, archiver, a SpamAssasin wrapper, RBL checking or
whatever. The important thing is that any otherwise vastly expensive
script-operations would be carried out instantly since the procedure/s to execute
is compiled into bytecode and stored in RAM already.
Server modules (procedures)
------------------------------
These are pretty much plug and play and can be copied to any system as
any other PERL module. Some may have configuration options needed
for the local Xmail environment. Modules are re-loaded by the server
upon change, so you can comfortable modify them without restarting the
server.
I found the easiest way to incorporate new procedures is to package them
as vanilla PERL modules having a common "handler" method available.
Supplied parameters passed with the procedure call are passed in with it.
Global stuff like logging methods etc, are available globally.
I still have to knock up some modules for this beast, but theoretically
anything that can be done with PERL is go. Ideas so far:
* Virus Scanning.
* Mail archiver
* Header mucking.
* Common mailproc stuff "lredirect|redirect|smtp" via spool/local.
* SpamAssassin wrapper.
* Razor wrapper (or maybe it's part of the SpAss suit nowadays ?)
* RPL map checks
* Auto Responder.
* Anti-multiparting. (attachment removal)
* xm2nntp gatewaying (my cool little piece)
* Mailinglist manager.
* XMail::CtrlClnt wrapper. (as sysop adding/removing users etc, just by sending a
mail)
* XMail Queue Manager (shell version) wrapper
* Sysop commands. (would require some sort of in-message authentication,
preferrably ;) )
...with Davide implementing outbound filters this would render in another load of
cool modules:
* Automatic PGP message armouring.
* Message footers
* PGP key /VCARD attachments ( compiled from usertabs ?!?)
* Outbound Virus Scanning.
* ...
Realworld "mailproc.tab" scenario.
-----------------------------------
XMail
|
----> "external" "/path/to/xmrpcc" "-cmd /some.procedure" "-from" "@@FROM" "-file"
"@@FILE"
|
-----> xmrpcc (20-200 bytes, depending on arguments passed)
/* tells the server to execute "/some.procedure" passing on -from
and -file as args */
|
* xmrpcd carries out the procedure (logs the action eventually)
|
<----- xmrpcd replies with an integer, 1-3 bytes (**)
|
<----- xmrpcc takes the response and uses it as its own exitcode, telling XMail how to
continue.
**) Exitcodes doesn't matter for mprocs, while it's crucial for filters. (0, 97-100)
-Phew, this message turned out way longer than I thought it would...
Anyways, guys. What do you think about this ?
I have the client a basic server up an going on my devbox right now. But there is
still work
to do with the server API. All stuff is available upon request, as I was hoping for
some ideas
and co-op from the community to pull this thing thru. Everything is going into the GNU
GPL
stable upon release, so don't worry about me profiting on your time.
--
Cheers and Evel Knievel for President,
Thomas Loo
___________________________________________________
[EMAIL PROTECTED] ������������������������������������
http://www.saltstorm.net
________________________________S_a_l_t_s_t_o_r_m____
-
To unsubscribe from this list: send the line "unsubscribe xmail" in
the body of a message to [EMAIL PROTECTED]
For general help: send the line "help" in the body of a message to
[EMAIL PROTECTED]
