Re: [nodejs] Streaming fs.readdir?

Pipes vave limites sizes in Linux: https://www.google.com.br/search?q=linux+pipe+size+limit&oq=lonux+pipe+size+limit&aqs=chrome.1.69i57j0l3.14217j0j4&client=chrome-mobile&sourceid=chrome-mobile&espv=1&ie=UTF-8 Em 09/10/2014 11:30, "Matt" escreveu: > The buffer size of a pipe between your process

Re: [nodejs] Streaming fs.readdir?

The buffer size of a pipe between your process and ls should fill up eventually and ls's calls to write to stdout should eventually block not allowing ls to continue until you've consumed the buffer. At least that's my theory - maybe you aren't applying backpressure right? On Wed, Oct 8, 2014 at

Re: [nodejs] Streaming fs.readdir?

Thanks Forrest for the confirmation! I'll subscribe. And thanks all for the streaming `ls [-f]` suggestion. I should have mentioned I had tried that, but it doesn't really solve this problem. The reason is because unless you are processing each file instantly, the backpressure is going to build

Re: [nodejs] Streaming fs.readdir?

On Oct 8, 2014, at 11:34 AM, Matt wrote: > Please note that the suggestions to shell out to "ls" are dangerous as "ls" > has a bunch of default options (some set by the environment) and will by > default try and sort (reading the whole directory into memory). > > At the minimum you want the "

Re: [nodejs] Streaming fs.readdir?

You could stream from a child process written in another language. For example you could do 'opendir(DIR,$ARGV[1]); print "$_\n" while $_=readdir(DIR)' as a perl script (with the directory as the first command line argument). On Tue, Oct 7, 2014 at 10:48 PM, Forrest Norvell wrote: > I know there

Re: [nodejs] Streaming fs.readdir?

I know there’s been some talk about adding streaming readdir() to libuv recently, but that’s of little use to you now. There are also some modules (like Thorsten Lorenz’s readdirp) that have streaming interfaces, but that only works if you want a recursi

[nodejs] Streaming fs.readdir?

Hi there, I have a directory with a very large number of files in them (over 1M). I need to process them, so I'm using fs.readdir() on the directory. The problem is, fs.readdir() returns everything at once, causing my script to suddenly consume >1 GB of RAM. AFAICT, there's no way to stream this