Brice, This looks great and it's great to see some real(ish) numbers that indicate real performance gains. I'm curious as to how they would look on mongrel.
+100 Rein Henrichs http://puppetlabs.com On Sat, Apr 10, 2010 at 8:32 AM, Brice Figureau < brice-pup...@daysofwonder.com> wrote: > On 04/04/10 22:55, Luke Kanies wrote: > > On Apr 3, 2010, at 9:22 AM, Brice Figureau wrote: > >> I'm proud to post here two patches forming the Puppet Streaming patch. > >> It's rather different and more leightweight than my previous attempt at > >> solving this issue. But it doesn't cover streaming the catalog, it only > >> involves streaming file content, either server side or client side. > >> > >> [snipped myself] > > > > Very nice - I'm really excited you got this done. I'd love to see some > > test results for memory and performance differences. > > I performed a couple of benchmarks. > My test is to run a master and a puppetd on the same machine (with > plenty of RAM). The puppetd requests a manifests that consists of > sourcing a 100MiB file, while both process are sampled every 1s to > capture VSZ and RES. > > The master is runnin > > This bench ran with ruby 1.8.7 on osx 10.5. > > VSZ and RES are in KiB. > > Here are the results: > > without the patch: > time: 27.47s > > VSZmin VSZmax VSZdiff RESmin RESmax RESdiff > master 632096 1052040 419944 31992 349676 317684 > puppetd 612860 879732 266872 13012 227012 214000 > > > with the patch: > time: 28.80s > > VSZmin VSZmax VSZdiff RESmin RESmax RESdiff > master 632096 642496 10400 31976 42432 10456 > puppetd 612860 630060 17200 13020 28580 15560 > > Conclusion: > The patch did great in terms of memory consumption increase (17MiB > compared to 266MiB on the client, and 10Mib compared to 420MiB on the > server). > > Unfortunately it also has an impact on catalog application time (an > increase of 1.4s, 5%). > > I did a couple of research and found that the culprit was Webrick buffer > handling while sending file. On ruby 1.8.7 it uses a buffer of 4 KiB. > Increasing this buffer to 1MiB drastically reduced the catalog > application time to less than 16s. > > This buffer is used only when streaming content in webrick. When sending > whole content as a string, this one is directly sent to the OS which > uses much larger buffers able to sustain the bandwitdh of localhost. > > So, I think we can keep the streaming patch, it adds more advantages > than drawbacks :-) > -- > Brice Figureau > My Blog: http://www.masterzen.fr/ > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Developers" group. > To post to this group, send email to puppet-...@googlegroups.com. > To unsubscribe from this group, send email to > puppet-dev+unsubscr...@googlegroups.com<puppet-dev%2bunsubscr...@googlegroups.com> > . > For more options, visit this group at > http://groups.google.com/group/puppet-dev?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To post to this group, send email to puppet-...@googlegroups.com. To unsubscribe from this group, send email to puppet-dev+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-dev?hl=en.
