I changed kPoolSize to 128 * 1024, and compiled node again. Now, Node tries to read larger chunks in each read. Although it never reaches 131072 bytes, but its usually in the range of 128000 to 130000. It reads each 128*1024 chunk in two reads, first one about 128000 bytes, and the second one the remaining number of bytes. Although not totally efficient, two reads is better than the 4 reads we were getting earlier. Also, this is very consisted with Java's stream.read function, which also behaves in the same way.
- Gill On Friday, 3 August 2012 03:36:48 UTC-7, Gill wrote: > > In this particular application, I do not have to worry too much about > memory usage. The focus is to read as much data from the file system as > possible, in one read operation. > > I will try increasing the kPoolSize and recompile Node. I will update this > thread with results. > > Thanks guys. > > - Gill > > > > On Friday, 3 August 2012 02:48:49 UTC-7, Dominic wrote: >> >> Gill, the kPool is a large buffer that ReadStream cuts buffers out of, >> I think, to minimize the allocation of memory... >> >> I was reading through this the other day, and noticed that the default >> buffer size is larger than the default kPoolSize. that seems wrong. >> >> you'd need to recompile node, of course, but making kPoolSize larger >> should work. >> >> On Fri, Aug 3, 2012 at 8:50 AM, Gill <[email protected]> wrote: >> > I think this is related: https://github.com/joyent/node/issues/2098 >> > >> > - Gill >> > >> > >> > On Thursday, 2 August 2012 23:32:39 UTC-7, Gill wrote: >> >> >> >> Ben, thanks for the reply. I have a doubt that its just a hint, >> because >> >> how come it is exactly 40960 bytes every time. The underlying >> filesystem is >> >> a custom coded one, which WILL return the exact number of bytes that >> were >> >> asked for. Line 38 for /lib/fs.js says: >> >> >> >> var kPoolSize = 40 * 1024; >> >> >> >> Do you think changing it to 128 * 1024 will change anything? >> >> >> >> - Gill >> >> >> >> On Thursday, 2 August 2012 16:29:43 UTC-7, Ben Noordhuis wrote: >> >>> >> >>> On Thu, Aug 2, 2012 at 9:21 PM, Gill <[email protected]> wrote: >> >>> > I have a code where the NodeJS server reads a file and streams it >> to >> >>> > response, it looks like: >> >>> > >> >>> > var fStream = fs.createReadStream(filePath, {'bufferSize': 128 * >> >>> > 1024}); >> >>> > fStream.pipe(response); >> >>> > >> >>> > The issue is, Node reads the file exactly 40960 bytes a time. >> However, >> >>> > my >> >>> > app would be much more efficient (due to reasons not applicable to >> this >> >>> > question), if it reads 131072 (128 * 1024) bytes at a time. >> >>> > >> >>> > Is there a way to force Node to read 128 * 1024 bytes at a time >> from >> >>> > the >> >>> > stream? >> >>> > >> >>> > Thanks in advance! >> >>> >> >>> No. bufferSize is a hint, not an imperative. It's up to the operating >> >>> system to honor it. >> > >> > -- >> > Job Board: http://jobs.nodejs.org/ >> > Posting guidelines: >> > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >> > You received this message because you are subscribed to the Google >> > Groups "nodejs" group. >> > To post to this group, send email to [email protected] >> > To unsubscribe from this group, send email to >> > [email protected] >> > For more options, visit this group at >> > http://groups.google.com/group/nodejs?hl=en?hl=en >> > -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en
