On Fri, 16 Aug 2002, Newsmirror wrote:
> > Hi list, > > > I am in the works of building a Xmail-RPC (Remote Procedure Call) system for > handling external filter/mailproc procedures via a rpc deamon and I would like to > hear any comments, suggestions or ideas or whatever concerning how to implement > this thing correctly right from the start. If you are a scriptwise knowledged guy > (PERL / PHP / Python / WSH) or a C/C++ dude you are most welcome with suggestions, > but ofcourse, any XMail user, please do read on and let me hear your point of view. > > > > > Preamble. > >------------------------------------------------------------------------------------------ > Anyone having used scriptbased external filters/mailproc stuff in a XMail context > (or in any interpreted language context for that matter) knows that this causes a >lot of extra > burden on the CPU since a script interpreter has to be re-launched for each call. In >the case > of PERL on a Win32 box a standard "external" 2KB script causes an additional 3.5+ MB > loading/unloading of the PERL interpreter + autoloaded modules just in order to >execute it. > Using WSH this is about 1MB extra. > > Well, if you are running a 1000 mail/day box, this shouldn't pose much of a problem, > but if running a 10000+/day it might bring your CPU and HDD to its knees fairly >severely > from time to time. Now, my setup is not target for such amounts of mails a day, > but my XMail is running on a multi-service box (HTTP/NNTP/SMTP/FTP) that > all uses a fair amount of load. And since my box isn't a Sun HP 10000 with RAIDed > SCSI multi-arrays, if I can reduce load in any way I am willing to look into it. > > Performance has always been an issue with using script based languages, this is why > high-load CGI isn't very common nowadays. Launching another X MB to execute a > script with Y KB size isn't very efficient in the long run. The solution is >interpreter > persistance and bytecode caching. Having the script interpreter in RAM already, > makes the script execute with a minimum of overhead. Apache's mod_perl, mod_python, > mod_rexx, mod_whitebeam, ASP and PHP and any other modern HTTP based script > language for instance behaves exactly in this manner. The difference here is that >mentioned > solutions all are linked in, statically or dynamically with the main application at >compile-time > or at startup and therefore the communication between the main application and the > particular interpreter are carried out "in-process", while my Xmail-RPC idea depends >on > an external communication layer namely UDP via XMail and script interpreter carrying > out the "external" filters/mprocs operations with pre-compiled bytecode. > (hey, isn't that what that big $+ PHP's ZEND Accelerator is all about :) > > After doing some thinking, I came up with the idea of knock up a RPC system for > handling CPU expensive scriptbased filters/mprocs under XMail to a level close > to a compiled language, still with the maintained versatility of a scripted language. > The RPC system concists of two parts, a client and a server, utilizing a >connectionless > UDP datagram protocol as transport-layer, since it is faster than common TCP > for repetative exchange of data. > > > > > The Client. (xmrpcc) > >--------------------------------------------------------------------------------------- > Being fairly new to C, with 20% cut n' paste and 80% trial and error, I now have > a to me stunningly super-tiny rpc-client working, compiled size 4KB / 16KB under > Win32 / Linux. (didn't thought it was possible to achieve a 4KB app with Win32 VC) > Default server address 127.0.0.1:7887. Executed from a filter.tab or a mailproc.tab, > cmdalies, custdomains it simply passes on the call for procedure and any defined > arguments to the RPC-server AND the turns the response into its OWN > exitcode. (0, 97-100). > > The client could be utilized in the following situations: > * Filters > * Mailprocs > * CmdAliases > * Custdomains > * ...hey, davide. what about those outbound filters you spoke about some months >ago. > > *** C/C++ guys, any help appreciated in improving the code... *** > > > > > The Server (aka. daemon) (xmprcd) > >---------------------------------------------------------------------------------------------- > Written in PERL (... could by any scriptbased language really) > Sits in the background listening on an UDP port for orders from a the client carrying > them out as per defined procedure id. A procedure is actually only another name for > a pre-compiled bytecode-chunk representation of a PERL module stored in-cache. > > Besides the big win of the in-cache bytecode the second main idea about the server > is that anyone fairly experienced with PERL could knock up and add a new > procedure for his special needs and share it with the community if applicable. > I have a XMail::RPC::Procedure API on the way that will make it close to trivial > to create a new module with only few lines of code. > (read section "Server modules" below) > > Preferrably this daemon should run on the same box as XMail itself, > but theoretically it could be located in Bagdad, Antarctica or in a Russian > Sub roaming longside California coastline as long as it sits on a box connected > to the Internet. > > The latter scenarios however would require some vastly obnoxious long-distance > shares/nfs/samba/smbfs mounting if we wanted the server to muck with the actual > message structure locally. But if we had the RPC-server on a local LAN on the > other hand, we would call it distributing the workload, which is a scenario > pretty common nowadays. > > Upon a procedure call from the client, the server executes the mapped code > assigned to that procedure. It could be a (or a series of procedures) > simple content scanner, archiver, a SpamAssasin wrapper, RBL checking or > whatever. The important thing is that any otherwise vastly expensive > script-operations would be carried out instantly since the procedure/s to execute > is compiled into bytecode and stored in RAM already. > > > Server modules (procedures) > ------------------------------ > These are pretty much plug and play and can be copied to any system as > any other PERL module. Some may have configuration options needed > for the local Xmail environment. Modules are re-loaded by the server > upon change, so you can comfortable modify them without restarting the > server. > > I found the easiest way to incorporate new procedures is to package them > as vanilla PERL modules having a common "handler" method available. > Supplied parameters passed with the procedure call are passed in with it. > Global stuff like logging methods etc, are available globally. > > I still have to knock up some modules for this beast, but theoretically > anything that can be done with PERL is go. Ideas so far: > > * Virus Scanning. > * Mail archiver > * Header mucking. > * Common mailproc stuff "lredirect|redirect|smtp" via spool/local. > * SpamAssassin wrapper. > * Razor wrapper (or maybe it's part of the SpAss suit nowadays ?) > * RPL map checks > * Auto Responder. > * Anti-multiparting. (attachment removal) > * xm2nntp gatewaying (my cool little piece) > * Mailinglist manager. > * XMail::CtrlClnt wrapper. (as sysop adding/removing users etc, just by sending >a mail) > * XMail Queue Manager (shell version) wrapper > * Sysop commands. (would require some sort of in-message authentication, >preferrably ;) ) > > ...with Davide implementing outbound filters this would render in another load of >cool modules: > * Automatic PGP message armouring. > * Message footers > * PGP key /VCARD attachments ( compiled from usertabs ?!?) > * Outbound Virus Scanning. > * ... > > > > Realworld "mailproc.tab" scenario. > ----------------------------------- > XMail > | > ----> "external" "/path/to/xmrpcc" "-cmd /some.procedure" "-from" "@@FROM" "-file" >"@@FILE" > | > -----> xmrpcc (20-200 bytes, depending on arguments passed) > /* tells the server to execute "/some.procedure" passing on >-from and -file as args */ > | > * xmrpcd carries out the procedure (logs the action eventually) > | > <----- xmrpcd replies with an integer, 1-3 bytes (**) > | > <----- xmrpcc takes the response and uses it as its own exitcode, telling XMail how >to continue. > > > **) Exitcodes doesn't matter for mprocs, while it's crucial for filters. (0, 97-100) > > > > -Phew, this message turned out way longer than I thought it would... > Anyways, guys. What do you think about this ? > I have the client a basic server up an going on my devbox right now. But there is >still work > to do with the server API. All stuff is available upon request, as I was hoping for >some ideas > and co-op from the community to pull this thing thru. Everything is going into the >GNU GPL > stable upon release, so don't worry about me profiting on your time. extremely interesting. a few notes : rpc is a standard and implementing it would require some overhead in both client and server, maybe it's better a simpler parameter passing method. udp is faster but unreliable, so having remote servers forces you to put extra code to ensure reliability. - Davide - To unsubscribe from this list: send the line "unsubscribe xmail" in the body of a message to [EMAIL PROTECTED] For general help: send the line "help" in the body of a message to [EMAIL PROTECTED]
