Blake, You mentioned large amount of directory recursion. I hit a similar wall at roughly 2000 files where the process would start to slow down drastically. My process was very similar to what you described.
After wasting considerable time I noticed that native 'fs' module was complete rewritten in iojs. After switching from node to iojs the problem went away completely. What version and OS are you running this on? > On Jun 3, 2015, at 2:31 PM, Blake McBride <[email protected]> wrote: > > Greetings, > > I have a large, very recursive process that (at best) takes 6 minutes to run. > This process does a lot of IO, and a lot of directory recursion. In order > to play nicely in a single threading event oriented system, I: > > 1. use non-blocking function whenever available > > 2. rather than perform recursion directly, I call the next step with > nextTick or setImmediate and exit > > This has worked well in my smaller databases, and the first time through the > large one (6 minutes). But when trying to run the large one a second time, > it gets slower and slower until it kind of never ends. I don't know what is > causing the slowdown. > > One thing I thought of is that perhaps I am filling the event queue in such a > way so that the IO never gets a chance and it has to go through many, many > events before any real IO can occur. Don't know. > > Sure appreciate any ideas. > > Thanks. > > Blake McBride > > -- > Job board: http://jobs.nodejs.org/ > New group rules: > https://gist.github.com/othiym23/9886289#file-moderation-policy-md > Old group rules: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > --- > You received this message because you are subscribed to the Google Groups > "nodejs" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/nodejs/e80bfd1f-fcd2-4ca2-847a-4c0425c6db5d%40googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- Job board: http://jobs.nodejs.org/ New group rules: https://gist.github.com/othiym23/9886289#file-moderation-policy-md Old group rules: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines --- You received this message because you are subscribed to the Google Groups "nodejs" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/nodejs/E12CA090-84DC-4F2D-8FCD-4B2FA1CD9DCB%40gmail.com. For more options, visit https://groups.google.com/d/optout.
