On Sat, May 14, 2011 at 5:12 PM, Thomas S Hatch <[email protected]> wrote:
> Great! This is what I was hoping to hear! If the entire file is getting read > into ram anyway, how would I go about making a file server that would send > the files in chunks and not read the whole thing into ram? I think that you > can see me intention here, but I gues I am still not entirely sure how to do > it yet. So here's one design. You read/write files in chunks of say 1MB, each a 0MQ message. Your protocol to send a file is: * OPEN filename - requests to send a new file, reply gives a staging handle and the size of the file already received, if GT 0, * STAGE message - sends a chunk to a staging handle. * DELIVER handle - closes the file staging, publishes the file I'd use a router/dealer socket pair so that I can do the open and deliver commands synchronously, and the stage command asynchronously. This gives you a good level of performance and lets you send massive files that don't fit in memory, with recovery if there's a network issue or node crash. If you're sending 10K config files, of course, this is overkill, just slurp and send the file in one message. -Pieter _______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
