I should clarify that those 10.000 files are not loaded on startup, it is really just around 300 files. I could defer loading to a later time but since require is always sync this would just mean to block the V8 thread at a later time which is not what I want to do.
Moving the files off the UNC to a different place is unfortunately not an option in my scenario. On Monday, July 29, 2013 4:18:26 PM UTC+2, azer wrote: > > How about bundling it in parts ? > > On Mon, Jul 29, 2013 at 6:58 AM, Angel Java Lopez > <[email protected]<javascript:>> > wrote: > > No experience.... but I would measure the file system time in serving > your > > application. > > > > You could store the files in a SSD disk (a "flash" disk, sorry, I'm not > a > > hardware guy). Comparing times will give you more context about the > issue. > > > > > > On Mon, Jul 29, 2013 at 10:48 AM, Matt <[email protected] <javascript:>> > wrote: > >> > >> Did you think about a possibility of 4) redesign your application so > it's > >> not loading 10k modules? That's an INSANE amount of dependencies. Even > >> making them parallel in some way is still going to require thousands of > >> syscalls. In general I don't think it's going to be a priority for node > to > >> optimise for startup time, but I could be wrong - I'm not a core team > >> member. > >> > >> > >> On Mon, Jul 29, 2013 at 9:28 AM, Benjamin Pasero > >> <[email protected] <javascript:>> wrote: > >>> > >>> Hi, > >>> > >>> in my environment it is very important to startup a node.js server > >>> instance in a very short time. Now, my server has many dependencies > (around > >>> 10.000 files in node_modules). My measurements show that the pure fact > that > >>> node.js resolves all require() calls from the entry point is making > the > >>> startup very slow. In this case, around 500 files are being loaded > through > >>> require(). The problem is even worse for me when I host node.js from a > UNC > >>> share where loading files one by one is a real bottleneck. > >>> > >>> I would like to ask the community if there is any idea for a solution > >>> that would speed up require() in node.js. So far I had two ideas: > >>> > >>> parallelize fs operations in require() > >>> allow to resolve require() calls from a zip file > >>> implement true async module loading through requireJS or similar > >>> > >>> Option 1 is the obvious: Instead of fs.readFileSync each required file > >>> one by one, do them all in one batch and collect the results. This > should > >>> increase the performance by an order of magnitude, especially if the > disk is > >>> slow (as in my UNC case). The only drawback from this solution I could > see > >>> is that it would no longer be possible to overwrite the behavior of > the > >>> require() function from within a dependency because file loading would > >>> happen in parallel. But this seems like not a typical use case. > >>> > >>> Option 2 would allow to specify an archive file to resolve require() > >>> against. This seems like actually supporting commonjs packages in zip > >>> archive form (http://wiki.commonjs.org/wiki/Packages/1.0). The idea > is that > >>> my entire node_modules folder could be one zip file that gets loaded > into > >>> memory on startup (at least the file/folder structure) and all > require() > >>> calls resolve against the archive. > >>> > >>> Option 3 is something I was hoping to get already using the requireJS > >>> node module. However they make clear that module loading is sync to > the > >>> execution context. So this is not an option currently, but maybe there > is a > >>> plugin that does this? > >>> > >>> Any other ideas are greatly appreciated. Maybe there is even a > solution > >>> out there I could benefit from ? > >>> Thanks, > >>> Ben > >>> > >>> -- > >>> -- > >>> Job Board: http://jobs.nodejs.org/ > >>> Posting guidelines: > >>> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > >>> You received this message because you are subscribed to the Google > >>> Groups "nodejs" group. > >>> To post to this group, send email to [email protected]<javascript:> > >>> To unsubscribe from this group, send email to > >>> [email protected] <javascript:> > >>> For more options, visit this group at > >>> http://groups.google.com/group/nodejs?hl=en?hl=en > >>> > >>> --- > >>> You received this message because you are subscribed to the Google > Groups > >>> "nodejs" group. > >>> To unsubscribe from this group and stop receiving emails from it, send > an > >>> email to [email protected] <javascript:>. > >>> For more options, visit https://groups.google.com/groups/opt_out. > >>> > >>> > >> > >> > >> -- > >> -- > >> Job Board: http://jobs.nodejs.org/ > >> Posting guidelines: > >> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > >> You received this message because you are subscribed to the Google > >> Groups "nodejs" group. > >> To post to this group, send email to [email protected]<javascript:> > >> To unsubscribe from this group, send email to > >> [email protected] <javascript:> > >> For more options, visit this group at > >> http://groups.google.com/group/nodejs?hl=en?hl=en > >> > >> --- > >> You received this message because you are subscribed to the Google > Groups > >> "nodejs" group. > >> To unsubscribe from this group and stop receiving emails from it, send > an > >> email to [email protected] <javascript:>. > >> For more options, visit https://groups.google.com/groups/opt_out. > >> > >> > > > > > > -- > > -- > > Job Board: http://jobs.nodejs.org/ > > Posting guidelines: > > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > > You received this message because you are subscribed to the Google > > Groups "nodejs" group. > > To post to this group, send email to [email protected]<javascript:> > > To unsubscribe from this group, send email to > > [email protected] <javascript:> > > For more options, visit this group at > > http://groups.google.com/group/nodejs?hl=en?hl=en > > > > --- > > You received this message because you are subscribed to the Google > Groups > > "nodejs" group. > > To unsubscribe from this group and stop receiving emails from it, send > an > > email to [email protected] <javascript:>. > > For more options, visit https://groups.google.com/groups/opt_out. > > > > > -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en --- You received this message because you are subscribed to the Google Groups "nodejs" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
