On Sat, May 14, 2011 at 1:59 PM, Thomas S Hatch <[email protected]> wrote:
> Excellent, this is the info I need, now I should be able to figure it out. > Thanks! > > I will probably have more questions, I will post it once I get it figured > out. My plan is to make a simple api to embed a simple zmq file server into > a python app. Then use it in salt. > On May 14, 2011 1:38 PM, "Pieter Hintjens" <[email protected]> wrote: > > On Sat, May 14, 2011 at 5:12 PM, Thomas S Hatch <[email protected]> > wrote: > > > >> Great! This is what I was hoping to hear! If the entire file is getting > read > >> into ram anyway, how would I go about making a file server that would > send > >> the files in chunks and not read the whole thing into ram? I think that > you > >> can see me intention here, but I gues I am still not entirely sure how > to do > >> it yet. > > > > So here's one design. You read/write files in chunks of say 1MB, each > > a 0MQ message. Your protocol to send a file is: > > > > * OPEN filename - requests to send a new file, reply gives a staging > > handle and the size of the file already received, if GT 0, > > * STAGE message - sends a chunk to a staging handle. > > * DELIVER handle - closes the file staging, publishes the file > > > > I'd use a router/dealer socket pair so that I can do the open and > > deliver commands synchronously, and the stage command asynchronously. > > > > This gives you a good level of performance and lets you send massive > > files that don't fit in memory, with recovery if there's a network > > issue or node crash. > > > > If you're sending 10K config files, of course, this is overkill, just > > slurp and send the file in one message. > > > > -Pieter > > _______________________________________________ > > zeromq-dev mailing list > > [email protected] > > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > Ok, I think that this is on the right track, can you all let me know/ give suggestions? https://github.com/thatch45/sandbox/blob/master/examples/zmq/file_get2.py https://github.com/thatch45/sandbox/blob/master/examples/zmq/file_serve2.py obviously there is a ways to go, since this is still the same level of simplicity as the last attempt, but this should at least give the chunks seperately
_______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
