I'm using https://github.com/substack/node-seq as my default async helper,
and it supports limiting the concurrency without changing to an explicit
queue construct. Just call parEach, parMap etc. with the desired number as
its first argument:
seq(arrayWithManyFileNames)
.parEach(20, function (fileName) {
var cb = this;
// ... Read file, do pattern matching, then call cb() or cb(err)
when done.
})
.seq(function () {
// All done!
})
.catch(function (err) {
// Whoops, it didn't work out
});
Best regards,
Andreas Lind Petersen (papandreou)
On Monday, July 2, 2012 1:56:12 AM UTC+2, Adeel Qureshi wrote:
>
> I am getting the EMFILE error when I am trying to read a bunch of files to
> search for user provided strings. I understand that its because node is
> running out of file descriptors with too many files open but i cant seem to
> find any good solution for this. Is there any design pattern to limit the
> number of open files. The only thing I have come across is to increase the
> number of open descriptors but I am hoping for a programmatic solution.
>
> Also since all i am doing here is reading the files to make a match for a
> given string how different that is from doing an actual grep child process,
> i didnt go that route becoz I wasnt able to figure out how to get grep
> working .. I think i tried something like
>
> var grep = spawn('grep', [matchStr, dirPath, filePrefix]);
>
> and it didnt seemed to work.
>
> Any comments would be appreciated.
>
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
