On Sat, Apr 10, 2010 at 11:32 AM, Brice Figureau <brice-pup...@daysofwonder.com> wrote: > On 04/04/10 22:55, Luke Kanies wrote: >> On Apr 3, 2010, at 9:22 AM, Brice Figureau wrote: >>> I'm proud to post here two patches forming the Puppet Streaming patch. >>> It's rather different and more leightweight than my previous attempt at >>> solving this issue. But it doesn't cover streaming the catalog, it only >>> involves streaming file content, either server side or client side. >>> >>> [snipped myself] >> >> Very nice - I'm really excited you got this done. I'd love to see some >> test results for memory and performance differences. > > I performed a couple of benchmarks. > My test is to run a master and a puppetd on the same machine (with > plenty of RAM). The puppetd requests a manifests that consists of > sourcing a 100MiB file, while both process are sampled every 1s to > capture VSZ and RES. > > The master is runnin > > This bench ran with ruby 1.8.7 on osx 10.5. > > VSZ and RES are in KiB. > > Here are the results: > > without the patch: > time: 27.47s > > VSZmin VSZmax VSZdiff RESmin RESmax RESdiff > master 632096 1052040 419944 31992 349676 317684 > puppetd 612860 879732 266872 13012 227012 214000 > > > with the patch: > time: 28.80s > > VSZmin VSZmax VSZdiff RESmin RESmax RESdiff > master 632096 642496 10400 31976 42432 10456 > puppetd 612860 630060 17200 13020 28580 15560 > > Conclusion: > The patch did great in terms of memory consumption increase (17MiB > compared to 266MiB on the client, and 10Mib compared to 420MiB on the > server). > > Unfortunately it also has an impact on catalog application time (an > increase of 1.4s, 5%). > > I did a couple of research and found that the culprit was Webrick buffer > handling while sending file. On ruby 1.8.7 it uses a buffer of 4 KiB. > Increasing this buffer to 1MiB drastically reduced the catalog > application time to less than 16s. > > This buffer is used only when streaming content in webrick. When sending > whole content as a string, this one is directly sent to the OS which > uses much larger buffers able to sustain the bandwitdh of localhost. > > So, I think we can keep the streaming patch, it adds more advantages > than drawbacks :-) > -- > Brice Figureau > My Blog: http://www.masterzen.fr/
I like the idea of trading 5% runtime performance for such a large memory savings. This was done with puppettest, yes? I'm thinking of spending some time adding things to it. If you're using something else or have additional performance testing ideas, let me know. It's hard to gauge the "how many of X can we run against a puppetmaster", but I'd love to have more data on relatively comparing releases, OSes, and implementations. Both time and resources. --Michael -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To post to this group, send email to puppet-...@googlegroups.com. To unsubscribe from this group, send email to puppet-dev+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-dev?hl=en.
