El Lunes 22 Jun 2009 17:43:15 Nicolás Alvarez escribió: > El Lunes 22 Jun 2009 17:13:52 David Anderson escribió: > > It's not clear to me how to solve this problem. > > Synopsis: > > - projects can configure their web server to send files in > > a compressed form to clients that can handle it; see > > http://boinc.berkeley.edu/trac/wiki/FileCompression > > In modern (>= 5.4) BOINC clients, > > the client tells libCurl to handle such compression. > > - The client keeps track of the bytes transferred N, > > and if a transfer fails and is retried, > > it sends a Range: header telling the web server to skip the first N > > bytes. NOTE: N is a count of uncompressed bytes; as far as I can tell, > > libCurl tells us only how many uncompressed bytes it's read. > > But when send Range:N to the server, it skips N compressed bytes, > > which causes the error Kevin points out below. > > > > I can think of 3 solutions, 2 of which I don't know how to implement: > > > > 1) If a file is compressed, always start the transfer from the beginning. > > Problem: how does the client know if a file is compressed? > > This is negotiated between libCurl and the server. > > 2) Keep track of the number M of compressed bytes transferred, > > and tell the server to skip M bytes on retries. > > Problems: > > a) libCurl doesn't tell us M, as far as I can tell. > > b) I'm not sure that Apache supports this. > > 3) Allow a <no_partial_transfer> flag in <file_info> elements, > > telling the client to always start transfers from the beginning. > > > > Does anyone know how to implement 1) or 2), or have other ideas? > > If not, should I go ahead and implement 3)? > > I'm discussing on IRC (for the n'th time) in what order HTTP expects range > and content-encoding to happen. Are we supposed to get a piece of the > compressed file, or a compressed piece of the original file? > > In the former case, we *can't* let libcurl handle compression for us, since > when we resume a transfer, we have to concatenate the new data to the > partial *compressed* file. If libcurl handles content-encoding compression > transparently, then the partial compressed data is never saved anywhere.
Okay, apparently it's not explicit in the HTTP spec, but Range is supposed to happen after Content-Encoding. See sections 3.2 and 4 http://tinyurl.com/maqdsn So we shouldn't let libcurl handle Content-Encoding compression for us. Download the compressed file to disk, resume it if needed, and once the transfer is done, decompress it all. But are hashes and digital signatures applied before or after compression? Ugh, this needs some more thinking. _______________________________________________ boinc_dev mailing list [email protected] http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev To unsubscribe, visit the above URL and (near bottom of page) enter your email address.
