You could stream from a child process written in another language. For example you could do 'opendir(DIR,$ARGV[1]); print "$_\n" while $_=readdir(DIR)' as a perl script (with the directory as the first command line argument).
On Tue, Oct 7, 2014 at 10:48 PM, Forrest Norvell <[email protected]> wrote: > I know there’s been some talk <https://github.com/joyent/libuv/issues/931> > about adding streaming readdir() to libuv recently, but that’s of little > use to you now. There are also some modules (like Thorsten Lorenz’s > readdirp) that have streaming interfaces, but that only works if you want > a recursive readdir(); it’s no help if you’ve got one enormous directory > (like you do). Unless you want to write your own native module that follows > the approach described in that libuv thread, I think you’re out of luck. > Sorry! :/ > > F > > > On Tue, Oct 7, 2014 at 6:04 PM, Aseem Kishore <[email protected]> > wrote: > >> Hi there, >> >> I have a directory with a very large number of files in them (over 1M). I >> need to process them, so I'm using fs.readdir() on the directory. >> >> The problem is, fs.readdir() returns everything at once, causing my >> script to suddenly consume >1 GB of RAM. >> >> AFAICT, there's no way to stream this list instead of returning it all at >> once. Is there anything equivalent that I can do? >> >> Thanks! >> >> Aseem >> >> -- >> Job board: http://jobs.nodejs.org/ >> New group rules: >> https://gist.github.com/othiym23/9886289#file-moderation-policy-md >> Old group rules: >> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >> --- >> You received this message because you are subscribed to the Google Groups >> "nodejs" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/nodejs/CAETHeihHkaQAbj9Ug5MMeNR6MHPR%3DfV21F%3DaMYMHQ-cNsQakPw%40mail.gmail.com >> <https://groups.google.com/d/msgid/nodejs/CAETHeihHkaQAbj9Ug5MMeNR6MHPR%3DfV21F%3DaMYMHQ-cNsQakPw%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > > -- > Job board: http://jobs.nodejs.org/ > New group rules: > https://gist.github.com/othiym23/9886289#file-moderation-policy-md > Old group rules: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > --- > You received this message because you are subscribed to the Google Groups > "nodejs" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/nodejs/CAHG3qKoE484fVCmP5Tzqwv-G1k82Z%2B0OMypt7vc%3D_UA%2B2gBtig%40mail.gmail.com > <https://groups.google.com/d/msgid/nodejs/CAHG3qKoE484fVCmP5Tzqwv-G1k82Z%2B0OMypt7vc%3D_UA%2B2gBtig%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- Job board: http://jobs.nodejs.org/ New group rules: https://gist.github.com/othiym23/9886289#file-moderation-policy-md Old group rules: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines --- You received this message because you are subscribed to the Google Groups "nodejs" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/nodejs/CAPJ5V2Z1p5mcjSFG8%3D7%2BRuwZzCqzkbMO6KZmXrX3iBJ%2BU6e7jw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
