Dan, push or pull only matters for who initiates the connection. The reading too fast and buffering issue is 100% local. It's between the code reading from the local disk and the code writing to the local kernel buffer. But by monitoring when the local kernel buffer is full or not, you can make sure to not send data faster than it's able to leave.
On Thu, Feb 23, 2012 at 3:50 PM, Dan North <[email protected]> wrote: > ** > The "best practice" here is to prefer pull over push. The remote daemon > will always knbow best r > ------------------------------ > *From: * Tim Caswell <[email protected]> > *Sender: * [email protected] > *Date: *Thu, 23 Feb 2012 09:00:30 -0600 > *To: *<[email protected]> > *ReplyTo: * [email protected] > *Subject: *Re: [nodejs] Best practice for pushing lots of data over TCP > to a distant server > > Tcp itself handles a lot of the connection issues, so it's fairly > reliable. You can still get disconnects at this level and you'll need to > write reconnect logic if you want to be that robust. As far as bandwidth > throttling, use Stream.prototype.pipe which implements "backpressure". > That is, it stops reading from the source when the destination is full and > not writable. When the dest is ready for more data, it resumes the stream. > This way you won't end up buffering all the data in memory in the likely > case that the source can read data faster than the remote dest can receive > it. > > On Wed, Feb 22, 2012 at 9:30 PM, Jules <[email protected]> wrote: > >> Daemon A needs to push lots of data to Daemon B, which is located on >> another continent. The bandwidth available is usually quite high, but >> it's highly variable and might drop out entirely at times. >> >> Is there an accepted "best strategy" for implementing this in Node? >> >> Does A just write as fast as it can to the socket and hope for the >> best? Should it detect errors and re-connect, resuming from the last >> offset that B successfully received? >> >> Does a socket ever become non-writeable? i.e. Does daemon A need to >> catch "socket not writeable" exceptions and throttle itself to suit >> the available bandwidth? >> >> I imagine some articles have been written on this topic, but I haven't >> found them yet. >> >> -- >> Job Board: http://jobs.nodejs.org/ >> Posting guidelines: >> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >> You received this message because you are subscribed to the Google >> Groups "nodejs" group. >> To post to this group, send email to [email protected] >> To unsubscribe from this group, send email to >> [email protected] >> For more options, visit this group at >> http://groups.google.com/group/nodejs?hl=en?hl=en >> > > -- > Job Board: http://jobs.nodejs.org/ > Posting guidelines: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > You received this message because you are subscribed to the Google > Groups "nodejs" group. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/nodejs?hl=en?hl=en > > -- > Job Board: http://jobs.nodejs.org/ > Posting guidelines: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > You received this message because you are subscribed to the Google > Groups "nodejs" group. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/nodejs?hl=en?hl=en > -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en
