Yes. I am deliberately abusing the stream with the goal of understanding behavior.
I would like to understand why it drains so slowly to disk, on the order of a few KB/sec. I would imagine that a stream created specifically with a file as its drain would be implemented to dump the biggest chunks it could on every write to the filesystem. Hence I wonder whether the low rate is really a node stream behavior or a virtual filesystem (buffering) behavior. I guess I also need to ask how nodes own streams are backed. I really want to understand just enough to predict bounds on scaling. On Mar 20, 2012 12:30 PM, "Matt" <[email protected]> wrote: > You're not "streaming" data into the stream. You're pumping it full as > quickly as possible and letting it drain in its own time. Obviously this is > sub-optimal. > > On Tue, Mar 20, 2012 at 3:20 PM, C. Mundi <[email protected]> wrote: > >> >> Hi. I am trying to learn how to use streams properly in node. >> >> The attached script stakes a single argv parameter N and writes 2^N bytes >> to a file via a stream. >> >> I expect time behavior O(2^N). But what I am seeing is scaling faster. >> >> I collected data like this in bash on Linux 3.0.0.-16 x64: >> >> for i in {0..20}; do time node simple.js $i; done >> >> and plotted the results in the attached spreadhseet (OpenOffice format). >> >> Watching it execute, I see that almost all of the time for N>10 is spent >> waiting for the stream to close. It seems to be draining very slowly! The >> maximum memory use by node at any time is 180 MB, less than 10% of physical >> memory. For N greater than about 10, node takes 100% CPU while closing the >> stream. The system is otherwise not under load. The system is not >> swapping much if at all. >> >> The straight lines in the plot are a guide to the eyes -- not a fit! You >> can see that the time behavior is superlinear on the log-log plot. >> >> If I were using a lot of memory, I would expect the GC to kick in and >> then all bets are off. But my biggest run (N=20) is only a megabyte. >> >> Could the pushback be coming from the Linux VFS? The scaling of the >> system time makes me wonder. >> >> Please help me understand why I should not do what I've done and where to >> learn about how to use streams efficiently in node. >> >> Note: The code is a demonstration. I would not typically create files >> this way. >> >> Thanks for cluing the noob (again). >> >> >> -- >> Job Board: http://jobs.nodejs.org/ >> Posting guidelines: >> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >> You received this message because you are subscribed to the Google >> Groups "nodejs" group. >> To post to this group, send email to [email protected] >> To unsubscribe from this group, send email to >> [email protected] >> For more options, visit this group at >> http://groups.google.com/group/nodejs?hl=en?hl=en >> > > -- > Job Board: http://jobs.nodejs.org/ > Posting guidelines: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > You received this message because you are subscribed to the Google > Groups "nodejs" group. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/nodejs?hl=en?hl=en > -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en
