If you are looking to spawn 10k worker nodes all on the same machine, you are probably doing it wrong. Try scaling out, instead of up. Maybe node isn't the right choice for what you are trying to do.
As for spawning up several hundred instances of a node.js application, we've put a lot of work into our node application server haibu ( https://github.com/nodejitsu/haibu ) On Tue, Jun 12, 2012 at 5:06 PM, Andrew Finnell <[email protected]>wrote: > Isolates was my last saving grace while doing NodeJS research. I am > attempting to do a proof of concept for a product I am working on. > > Scenario: > > 10,000 distinct'ish workers on a single machine. As it stands I'd need to > spawn 10,000 separate nodejs instances which amounts to about 300 gigs of > memory for a very simple task. I was hoping that Isolates or something > similar that could share common memory in some fashion would of alleviated > this problem. I could maybe have 200 nodejs clusters that each run 50 > internal instances. I do not need to share variables etc across threads or > processes. I am very appreciate and understanding of the mailbox/message > model for communication between processes. I just need to reduce the > handle/memory requirements to something more manageable. > > Of all the workers, the idea is that a single one can go down and I still > have all the other ones working. In a typical Java implementation one might > create a cluster of a few VM nodes 3-5. Even in this scenario 1 node going > down would bring down a substantial number of workers and the memory > overhead of running a JVM is too high. > > In a little more detail, lets say I wanted to create a management > framework that allowed developers to write their own nodejs applications > that could run in the management framework. I might need to spawn 400 > instances of App A, 100 of App B, 90 or App C, etc. The separate instances > of the Apps do not share state nor need to. The key issue I am facing is > that these app's are written independently of each other, thus it makes it > difficult to run them all in the same event loop. Too many bad things can > happen with differently written apps. > > > Any suggestions? I have been looking into Vert.x as an alternative. > > - > Andrew > > On Friday, February 3, 2012 4:28:54 PM UTC-5, Isaac Schlueter wrote: >> >> The Isolates feature was intended to make it possible to run >> child_process.fork() in a thread, rather than a full process. The >> justification was to make it cheaper to spin up new child node >> instances, as well as allowing for fast message-passing using shared >> memory in binary addons, while retaining the semantics of node's >> child_process implementation by keeping them in completely isolated v8 >> instances. >> >> It was a very informative experiment, but has ultimately turned out to >> cause too much instability in node's internal functionality to justify >> continuing with it at this time. It requires a lot of complexity to >> be added to libuv and node, and isn't likely to yield enough gains to >> be worth the investment. >> >> We're going to roll back the changes in master that were added to >> support Isolates, and instead focus on Domains and other things that >> promise to increase stability and make debugging easier. This change >> will land in 0.7.3. It's entirely possible that we may decide to use >> v8 isolates in some future version of node, but they will not be in >> 0.8. >> >> If you were eagerly looking forward to using this feature, and find >> yourself shocked or frustrated by this decision, please contact me >> directly. It's not our intention to leave anyone stuck, and I do >> understand that this feature was promised for some time. If this >> causes hardship for you, let's figure out another way to get your use >> cases handled. >> >> It's never easy to back-pedal, but doing experimental things means >> sometimes finding out that you were headed in the wrong direction. >> The sooner we make this change, the easier it will be. >> >> Thanks. >> >> -- > Job Board: http://jobs.nodejs.org/ > Posting guidelines: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > You received this message because you are subscribed to the Google > Groups "nodejs" group. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/nodejs?hl=en?hl=en > -- -- Marak Squires Co-founder and Chief Evangelist Nodejitsu, Inc. [email protected] -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en
