On Sat, Apr 10, 2010 at 1:42 PM, Michael DeHaan <mich...@puppetlabs.com>wrote:
> On Sat, Apr 10, 2010 at 11:32 AM, Brice Figureau > <brice-pup...@daysofwonder.com> wrote: > > On 04/04/10 22:55, Luke Kanies wrote: > >> On Apr 3, 2010, at 9:22 AM, Brice Figureau wrote: > >>> I'm proud to post here two patches forming the Puppet Streaming patch. > >>> It's rather different and more leightweight than my previous attempt at > >>> solving this issue. But it doesn't cover streaming the catalog, it only > >>> involves streaming file content, either server side or client side. > >>> > >>> [snipped myself] > >> > >> Very nice - I'm really excited you got this done. I'd love to see some > >> test results for memory and performance differences. > > > > I performed a couple of benchmarks. > > My test is to run a master and a puppetd on the same machine (with > > plenty of RAM). The puppetd requests a manifests that consists of > > sourcing a 100MiB file, while both process are sampled every 1s to > > capture VSZ and RES. > > > > The master is runnin > > > > This bench ran with ruby 1.8.7 on osx 10.5. > > > > VSZ and RES are in KiB. > > > > Here are the results: > > > > without the patch: > > time: 27.47s > > > > VSZmin VSZmax VSZdiff RESmin RESmax RESdiff > > master 632096 1052040 419944 31992 349676 317684 > > puppetd 612860 879732 266872 13012 227012 214000 > > > > > > with the patch: > > time: 28.80s > > > > VSZmin VSZmax VSZdiff RESmin RESmax RESdiff > > master 632096 642496 10400 31976 42432 10456 > > puppetd 612860 630060 17200 13020 28580 15560 > > > > Conclusion: > > The patch did great in terms of memory consumption increase (17MiB > > compared to 266MiB on the client, and 10Mib compared to 420MiB on the > > server). > > > > Unfortunately it also has an impact on catalog application time (an > > increase of 1.4s, 5%). > > > > I did a couple of research and found that the culprit was Webrick buffer > > handling while sending file. On ruby 1.8.7 it uses a buffer of 4 KiB. > > Increasing this buffer to 1MiB drastically reduced the catalog > > application time to less than 16s. > > > > This buffer is used only when streaming content in webrick. When sending > > whole content as a string, this one is directly sent to the OS which > > uses much larger buffers able to sustain the bandwitdh of localhost. > > > > So, I think we can keep the streaming patch, it adds more advantages > > than drawbacks :-) > > -- > > Brice Figureau > > My Blog: http://www.masterzen.fr/ > > I like the idea of trading 5% runtime performance for such a large > memory savings. > > This was done with puppettest, yes? I'm thinking of spending some > time adding things to it. If you're using something else or have > additional performance testing ideas, let me know. > puppettest is working on 0.25.x again? > > It's hard to gauge the "how many of X can we run against a > puppetmaster", but I'd love to have more data on relatively comparing > releases, OSes, and implementations. Both time and resources. > ++ What I really want to be able to do... is simply take the cached yaml data from a puppet server for facts etc, and 'replay' those client requests against the server with specified splay, concurrent clients, etc. I have some proof of concept stuff I've worked on to do this, but I think I'm faking out too much of the client for it to be a truly accurate indicator. Capacity planning with puppet is difficult. > > --Michael > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Developers" group. > To post to this group, send email to puppet-...@googlegroups.com. > To unsubscribe from this group, send email to > puppet-dev+unsubscr...@googlegroups.com<puppet-dev%2bunsubscr...@googlegroups.com> > . > For more options, visit this group at > http://groups.google.com/group/puppet-dev?hl=en. > > -- nigel -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To post to this group, send email to puppet-...@googlegroups.com. To unsubscribe from this group, send email to puppet-dev+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-dev?hl=en.
