Hello, I wonder if it would be possible to add client support for compressed HTTP/POST requests to scheduler and file_upload_handler. Basically any modern Apache server compiled/configured with mod_deflate (and probably mod_gzip also) is capable of decompressing the input on-the-fly as long as there's a header that tells it to do so. I've already tried this on a small scale on my own server and it works just fine without any changes in the server environment. The only thing that needs to be changed is the apache configuration file:
<Location /enigma_cgi> SetInputFilter DEFLATE </Location> That's it for the server side. The client side is a bit more complicated; I have only the basic knowledge about c/c++ programming, so I wasn't able to alter the BOINC's http code as it requires way higher skills than I have. Instead, I wrote a small proxy server which reads the sched request, parses HTTP POST data, takes all the POST content and runs it through gzip, updates the content-length and adds Content-Encoding: gzip header. Not only the client behind this proxy had no problems at all communicating with the project's server, but the amount of data send to the server was reduced by a lot. On average, the compression ratio is well above 80%, in some cases (client running debug app which outputs a lot of info to stderr) it peaks at about 95%. Here on my own project this would save more than 50% of average monthly traffic, as most of the traffic are sched requests. Before I tried the proxy, I ran manual steps to check if the server side is in fact able to read compressed POSTs. Here's what I did: - I took sched_request_www.enigmaathome.net.xml and renamed it sched.xml, then I created uncompressed POST request just to test if my method works (note that the content-length is sched.xml actual size): POST /enigma_cgi/cgi HTTP/1.0 User-Agent: BOINC client (windows_intelx86 6.11.7) Host: enigmaathome.net Accept: */* Accept-Encoding: deflate, gzip Content-Type: application/x-www-form-urlencoded Content-Length: 22163 then I attached the sched.xml file here by executing cat sched.xml >>uncompressed and used 'cat uncompressed | nc enigmaathome.net 80 >uncompressed_reply' to check communication; I read the reply file to check if/what the server replied; it worked fine and I got sched's reply, so I proceeded to another step: I ran the sched.xml through gzip: 'cat sched.xml | gzip -9 >sched.gz', took the new content-length (sched.gz size) and altered the post headers: POST /enigma_cgi/cgi HTTP/1.0 User-Agent: BOINC client (windows_intelx86 6.11.7) Host: enigmaathome.net Accept: */* Accept-Encoding: deflate, gzip Content-Type: application/x-www-form-urlencoded Content-Encoding: gzip Content-Length: 4907 Then again I used cat to attach the POST data: 'cat sched.gz >>compressed' and I sent it to the project's scheduler: 'cat compressed | nc enigmaathome.net 80 >compressed_reply'. Once again scheduler replied without any problems, that proves the server can read compressed data. One problem I see right now is that the Apache itself does not tell the client if it can read compressed POST data. I thought that the server will announce it in it's headers, for example by adding "Accept-Encoding: gzip", but it doesn't, so that leaves enabling compression somewhere in the project's config. That however should be failsafe, so it needs a way to detect situations when the config file says server accepts compression, but it doesn't. So I also tried what happens if I do compressed POST request to the server with deflate filter disabled; I thought that Apache will reply with "unacceptable" or something, but it just passes the compressed data to sched and the scheduler replies with "no start tag" - I believe thit could be used to signal the client that something is wrong. /TJM http://www.enigmaathome.net _______________________________________________ boinc_dev mailing list [email protected] http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev To unsubscribe, visit the above URL and (near bottom of page) enter your email address.
