Hi Aseem, You could use ez-streams 's ez.devices.file.list(path) function. It will give you a stream of file names over a directory tree. Unfortunately it won't really stream over the content of a single dir because there is no native API to support this (it will just do virtual streaming over an array of names in memory in this case).
If you are generating the files you could arrange to dispatch them in subdirectories. But if they are generated by something you don't control you are probably toasted. You may also run into perf problem with the file system itself. How well does it cope with 1M+ entries per directory? Bruno On Wednesday, October 8, 2014 3:49:52 AM UTC+2, Aseem Kishore wrote: > > Hi there, > > I have a directory with a very large number of files in them (over 1M). I > need to process them, so I'm using fs.readdir() on the directory. > > The problem is, fs.readdir() returns everything at once, causing my script > to suddenly consume >1 GB of RAM. > > AFAICT, there's no way to stream this list instead of returning it all at > once. Is there anything equivalent that I can do? > > Thanks! > > Aseem > > -- Job board: http://jobs.nodejs.org/ New group rules: https://gist.github.com/othiym23/9886289#file-moderation-policy-md Old group rules: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines --- You received this message because you are subscribed to the Google Groups "nodejs" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/nodejs/19af3e59-6b8a-4485-8088-ebef520c6f49%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
