On Jul 6, 2009, at 6:17 PM, Sam Rowe wrote: > > On Fri, Jul 3, 2009 at 11:29 AM, Brice > Figureau<[email protected]> wrote: >> That's our eldorado :-) >> But it isn't as easy as it sounds first, if you look to how puppet >> uses >> the database for collection, and storeconfigs. >> Then there's the issue of finding the proper storage engine. > > I'll admit that I know next to nothing about puppet's use of the > database from the code side, but from the database side it sure looks > like the data is already being stored as key/value pairs which is TC's > whole raison d''être.
Really, Storeconfigs builds hash-like structures (resources are a hash with arrays for values, effectively) into the db, which basically sucks: Every resource is at least two associations: has_many :param_names and has_many :param_values. This is slow and sucky, but unavoidable with a standard db. > >>> It seems like using an RDBMS for this data not only doesn't benefit >>> from RDBMS features, but also makes it harder to store and retrieve >>> it. Would something like Tokyo Cabinet be a better fit? >> >> I didn't know Tokyo Cabinet, but the main issue with all key=value >> storage subsystem will be to adapt the collection query (currently >> rails >> only as pointed Luke) to something more versatile. >> I mean, when you write: >> File <<| tag== myfile |>> >> You need to be able to find the correct resource, so you have to >> create >> some special indices to map resources tags to resources, and it has >> to >> be done by hand. >> You need clever tricks to say that the key tag_myfile is pointing >> to a >> resource keyed by it's Type[title] string. Then perform a second >> look-up >> to look-up the resource itself. >> Basically what is cumbersome is mapping the complex storeconfigs >> namespace to a 1D system... > > Right, I think that there are a couple of things at play here. One, > it's a complex task to use an RDBMS as a key/value store. That's data > going in and coming out. Right now that complexity lives in the RoR > code so it seems "free." Changing to something like TC will move the > complex code out of RoR and into puppet, but there's actually a chance > that it will reduce complexity because it wouldn't be trying to do k/v > in RDBMS. And I guarantee (money back if not delighted) that it'll > increase performance. Heh. We tried couchdb, and with the large catalog sizes one customer was using, it blew up no matter how we modeled it. And it wasn't any faster. Sure, that was a naïve implementation, but it wasn't a drastic difference. That being said, I think the long term is some kind of solution like this. I haven't looked at Tokyo Cabinet, ftr. I think the main point that Brice was making is that the primary barrier to this is we would need to convert our existing query language into an abstract query, which could then transparently support any kind of db backend. Yay, rewriting sql, I guess. > >>> Of course you might say "It's not hard for me, RoR does it for me!" >>> but we've had other uses for our storedconfigs database that weren't >>> exactly RoR-friendly. >> >> Can you share with us your other usage patterns? > > We've had a variety of uses, but the first that comes to mind is using > a mediawiki bot to populate certain facts into our wiki. > > > -- In our civilization, and under our republican form of government, intelligence is so highly honored that it is rewarded by exemption from the cares of office. --Ambrose Bierce --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/puppet-dev?hl=en -~----------~----~----~----~------~----~------~--~---
