I am running node 0.12.4 on current 64 bit Linux. Also, I added a counter to increment each time I called setImmediate, and decremented each time the callback got called - to get the queue size. It remained at one.
Thanks! On Thursday, June 4, 2015 at 10:00:14 AM UTC-5, nwhite wrote: > > Blake, > > You mentioned large amount of directory recursion. I hit a similar wall at > roughly 2000 files where the process would start to slow down drastically. > My process was very similar to what you described. > > After wasting considerable time I noticed that native 'fs' module was > complete rewritten in iojs. After switching from node to iojs the problem > went away completely. > > What version and OS are you running this on? > > > On Jun 3, 2015, at 2:31 PM, Blake McBride <[email protected] <javascript:>> > wrote: > > Greetings, > > I have a large, very recursive process that (at best) takes 6 minutes to > run. This process does a lot of IO, and a lot of directory recursion. In > order to play nicely in a single threading event oriented system, I: > > 1. use non-blocking function whenever available > > 2. rather than perform recursion directly, I call the next step with > nextTick or setImmediate and exit > > This has worked well in my smaller databases, and the first time through > the large one (6 minutes). But when trying to run the large one a second > time, it gets slower and slower until it kind of never ends. I don't know > what is causing the slowdown. > > One thing I thought of is that perhaps I am filling the event queue in > such a way so that the IO never gets a chance and it has to go through > many, many events before any real IO can occur. Don't know. > > Sure appreciate any ideas. > > Thanks. > > Blake McBride > > -- > Job board: http://jobs.nodejs.org/ > New group rules: > https://gist.github.com/othiym23/9886289#file-moderation-policy-md > Old group rules: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > --- > You received this message because you are subscribed to the Google Groups > "nodejs" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] <javascript:>. > To post to this group, send email to [email protected] <javascript:> > . > To view this discussion on the web visit > https://groups.google.com/d/msgid/nodejs/e80bfd1f-fcd2-4ca2-847a-4c0425c6db5d%40googlegroups.com > > <https://groups.google.com/d/msgid/nodejs/e80bfd1f-fcd2-4ca2-847a-4c0425c6db5d%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > > -- Job board: http://jobs.nodejs.org/ New group rules: https://gist.github.com/othiym23/9886289#file-moderation-policy-md Old group rules: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines --- You received this message because you are subscribed to the Google Groups "nodejs" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/nodejs/1577729e-9091-4b01-bd3f-4c1e510da294%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
