Thanks, yes, this is what I'm leaning toward. I may even simply create a single "digest" file that each process loads at startup rather than passing a massive message. The file system caching should insure that each processes load of the digest will go very quickly.
Steve On Friday, November 30, 2012 3:08:55 PM UTC-5, ajlopez wrote: > > And, what about: > > - Master process loads the files, and build the result you need > - Then, it sends the result to every child process via a message > > Unless the desired result is child-dependent > > On Fri, Nov 30, 2012 at 5:04 PM, spqr <[email protected] <javascript:>>wrote: > >> The child process paradigm is fine for what I'm doing and there should be >> no problems with the message passing. >> My main concern is "system startup" not message passing. Each process >> has to have loaded (potentially) thousands of >> JavaScript files and built some summary data off the loaded files. When >> it does process the messages it receives >> all of the JavaScript objects are part of the processes state and it can >> execute the ones it needs to based on the messages it >> receives. I don't really know if loading those thousands of small >> JavaScripts will be slow or fast, maybe there is no problem. >> It just seems wasteful to have each process go through the exact same >> loading process and it seems like there should be >> some way to do most of that work just once. I was hoping there might be >> some clever and tricky way to do it ;-) >> >> >> On Friday, November 30, 2012 2:35:21 PM UTC-5, ajlopez wrote: >> >>> Umm... maybe I misunderstood the problem, or the below proposal: use >>> cluster (or child process >>> http://nodejs.org/api/**child_process.html<http://nodejs.org/api/child_process.html>) >>> >>> and send a message from master to workers. I don't know the limitations of >>> this kind of communitacion >>> >>> http://nodejs.org/api/cluster.**html<http://nodejs.org/api/cluster.html> >>> >>> example will echo back all messages from the master: >>> >>> if (cluster.isMaster) { >>> var worker = cluster.fork(); >>> worker.send('hi there'); >>> } else if (cluster.isWorker) { >>> process.on('message', function(msg) { >>> process.send(msg); >>> });} >>> >>> >>> On Fri, Nov 30, 2012 at 4:21 PM, spqr <[email protected]> wrote: >>> >>>> Thanks for connecting the dots. I was missing the connection to the OS >>>> exec environment. >>>> Trying to use the "env" definitely won't be a good solution to my >>>> particular problem. The thousands >>>> of scripts I will be loading are variable sized and may vary quite a >>>> bit depending on how much data >>>> they contain. A lot of data structures have to be populated as the >>>> scripts are loaded. It looks to me >>>> like my only choice is to have each process perform the load. One >>>> other thought I have is to maybe have >>>> the master communication process load all the files and write out some >>>> sort of compilation file that >>>> all the child processes would load instead or re-executing the entire >>>> process. >>>> >>>> On Friday, November 30, 2012 11:35:13 AM UTC-5, Ben Noordhuis wrote: >>>> >>>>> On Fri, Nov 30, 2012 at 5:16 PM, spqr <[email protected]> wrote: >>>>> > So, I guess this is a fairly stupid question because I can pass >>>>> everything >>>>> > to the process as part of the "env" >>>>> > Maybe a new question is that since I'm talking about "quite a lot of >>>>> junk," >>>>> > is there any negative consequence >>>>> > of passing a really large env to the child process. >>>>> >>>>> Not intrinsically but most (all?) operating systems put some arbitrary >>>>> upper limit on the environment size. On modern Linux, for example, >>>>> the maximum length of arguments + environment is usually (but not >>>>> always) 2 MB. >>>>> >>>>> In case of doubt, consult the limitations section of `man execve`. >>>>> >>>> -- >>>> Job Board: http://jobs.nodejs.org/ >>>> Posting guidelines: https://github.com/joyent/**node/wiki/Mailing-List- >>>> **Posting-Guidelines<https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines> >>>> You received this message because you are subscribed to the Google >>>> Groups "nodejs" group. >>>> To post to this group, send email to [email protected] >>>> >>>> To unsubscribe from this group, send email to >>>> nodejs+un...@**googlegroups.com >>>> >>>> For more options, visit this group at >>>> http://groups.google.com/**group/nodejs?hl=en?hl=en<http://groups.google.com/group/nodejs?hl=en?hl=en> >>>> >>> >>> -- >> Job Board: http://jobs.nodejs.org/ >> Posting guidelines: >> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >> You received this message because you are subscribed to the Google >> Groups "nodejs" group. >> To post to this group, send email to [email protected]<javascript:> >> To unsubscribe from this group, send email to >> [email protected] <javascript:> >> For more options, visit this group at >> http://groups.google.com/group/nodejs?hl=en?hl=en >> > > -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en
