Here is a simple solution with ez-streams and child_process:
var ez = require('ez-streams');
var cp = require('child_process');
var reader = ez.devices.child_process.reader(cp.spawn('ls'), {
encoding: "utf8"
}).transform(ez.transforms.lines.parser())
// just check that it works
reader.pipe(_, ez.devices.console.log);
It will handle the backpressure on the child process standard output.
Bruno
On Thursday, October 9, 2014 9:27:05 PM UTC+2, Bruno Jouhier wrote:
>
> Hi Aseem,
>
> You could use ez-streams 's ez.devices.file.list(path) function. It will
> give you a stream of file names over a directory tree. Unfortunately it
> won't really stream over the content of a single dir because there is no
> native API to support this (it will just do virtual streaming over an array
> of names in memory in this case).
>
> If you are generating the files you could arrange to dispatch them in
> subdirectories. But if they are generated by something you don't control
> you are probably toasted.
>
> You may also run into perf problem with the file system itself. How well
> does it cope with 1M+ entries per directory?
>
> Bruno
>
> On Wednesday, October 8, 2014 3:49:52 AM UTC+2, Aseem Kishore wrote:
>>
>> Hi there,
>>
>> I have a directory with a very large number of files in them (over 1M). I
>> need to process them, so I'm using fs.readdir() on the directory.
>>
>> The problem is, fs.readdir() returns everything at once, causing my
>> script to suddenly consume >1 GB of RAM.
>>
>> AFAICT, there's no way to stream this list instead of returning it all at
>> once. Is there anything equivalent that I can do?
>>
>> Thanks!
>>
>> Aseem
>>
>>
--
Job board: http://jobs.nodejs.org/
New group rules:
https://gist.github.com/othiym23/9886289#file-moderation-policy-md
Old group rules:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
---
You received this message because you are subscribed to the Google Groups
"nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/nodejs/92f2680f-dde8-42cf-8370-d0cd49971c77%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
