Jeff Schroeder wrote: > On Tue, Mar 24, 2009 at 4:11 PM, Michael DeHaan <[email protected]> wrote: > >> Michael DeHaan wrote: >> >>> Ok, recently Scott Henson, John Eckersberg, and I were trying to speed >>> up cobbler ... a lot. We got a lot of good things done, including >>> cutting the sync time in half. One thing we also achieved is we found >>> out, that using the python profiler, even though the new yaml >>> implementation is more robust (good, that's the most important thing!), >>> it's also slow... even with libyaml installed. That all being said, >>> the old implementation was also slow. We want /fast/. >>> >>> Some quick benchmarks on my end show that saving objects using >>> simplejson instead of PyYAML result in 10-12 second times to edit 200 >>> objects versus 28 seconds. >>> >>> As a result of this, I'm going to go ahead and add "serializer_catalog2" >>> to the distribution, though I should warn you ... there's no upgrade path. >>> >>> The way to replicate from catalog to catalog2 is to edit modules.conf on >>> a /new/ cobbler machine and cobbler replicate the data to it (since the >>> wire protocol is exactl y the same). >>> >>> Simplejson offers two output modes -- one is a "pretty printed" mode, >>> this mode is maybe 30% slower. It depends how cleanly readable we need >>> the output to be, but the answer is "not really", since we can always >>> get a pretty printed version of a data structure someone pastebins using >>> ipython if absolutely needed. I am in favor of not using the pretty >>> printed version. >>> >>> Just for kicks, I think I'll benchmark cPickle again -- shelve (which >>> uses things like berkley DB) tends to have issues. >>> >>> We may make the JSON implementation the new default in 1.8, we'll see. >>> >>> --Michael >>> _______________________________________________ >>> cobbler mailing list >>> [email protected] >>> https://fedorahosted.org/mailman/listinfo/cobbler >>> >>> >> Further data... it appears, and I don't really have the data to back >> this up just yet, that with, say 3000 systems in cobbler's >> configuration, "cobbler list" and "cobbler report" on these systems are >> basically instantaneous. >> >> sync took 30 seconds with 3000 systems (using hardlinks), where before >> it could take 15 seconds with 300. Roughly, this seems like it's >> infinitely faster for loading. Almost seems too good to be true. >> >> Writing is maybe 2x fast. >> >> It would be trivial to write a seperate (not shipped) python script for >> migrating folks, if this proves as fast as I think it is in further tests. >> >> The last time we tried this it was fed very large files (catalog style) >> -- now with the smaller ones it seems very very happy. >> > > And this is why folks like us using cobbler in moderate to large > environments love how fast you release. Great work Michael and crew. > > FYI -- I've decided to just teach catalog to update folks automatically, saving new files with an extension showing the type.
I can do this seamlessly, so why not :) --Michael _______________________________________________ cobbler mailing list [email protected] https://fedorahosted.org/mailman/listinfo/cobbler
