I only had a few minutes to look for obvious things, so I don't have a solution, but I do have some observations.
1. You've done a pretty good job at organizing the code from what I've see, so +1 to that. 2. You seem to be under the impression that child_process.fork() spawns threads, when it is actually spawning entire new node processes. In light of #2, you could be hitting OS enforced limits for things like the number of processes per user. Given that each process creates a new IPC channel, you could also be hitting a file descriptor limit. It's not clear which concurrency parameter you are referring to in your post, but I would suggest keeping your "thread" parameter tied to the number of CPU cores (0.5x-2x depending on the type of code being run). ~Ryan On Sun, Aug 7, 2016, 11:14 PM <[email protected]> wrote: > > Is there such a thing as a maximum number of functions you can put in the > event loop? > > I have a highly asynchronous app, which results in a ton of async > callbacks getting pushed into the event loop. One "operation" could result > in several tens of async functions. Everything runs fine at low concurrency > (like 5000 concurrent operations or so), but when I crank it up (20,000), > it runs fine for a little while (less than a second), and then gets stuck. > It queues up a bunch of callbacks, and they never get resolved. The app it > just at a standstill. Is there any technical reason this would be > happening? > > The app in question is basically a task runner: > https://github.com/catdad/grandma/ > > At first, I suspected that IPC was the issue (because multiple threads), > so I wrote a whole custom IPC implementation using named sockets. Still the > same issue. The IPC bits work just fine, more process.nextTick instances > get called, but the function never executes once the concurrency is high > enough. > > I have tried setting --max_old_space_size to a high number (thinking it > might be a memory issue) and that did not help. I have also > set UV_THREADPOOL_SIZE to 128, 256, and 512, thinking that maybe the libuv > thread pool is getting saturated, and that doesn't seem to be helping > either. > > I have found that when using longer async tasks (process.nextTick vs. > setTimeout(cb, 10) vs. an http request) that longer tasks get stuck at a > much lower concurrency. Assuming there is no bug in the code (I know, > everyone hates doing that), is there any technical reason for such an issue? > > -- > Job board: http://jobs.nodejs.org/ > New group rules: > https://gist.github.com/othiym23/9886289#file-moderation-policy-md > Old group rules: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > --- > You received this message because you are subscribed to the Google Groups > "nodejs" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/nodejs/267a1ccf-27cc-4a82-8009-d3794c56fcf8%40googlegroups.com > <https://groups.google.com/d/msgid/nodejs/267a1ccf-27cc-4a82-8009-d3794c56fcf8%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- ~Ryan -- Job board: http://jobs.nodejs.org/ New group rules: https://gist.github.com/othiym23/9886289#file-moderation-policy-md Old group rules: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines --- You received this message because you are subscribed to the Google Groups "nodejs" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/nodejs/CAGjmZGwFx7piSkr3n24Go3B7_u4f_BHz0RLM_TQiV%3D6P_N9NKg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
