On Mon, May 21, 2012 at 2:04 PM, Marc Zampetti <[email protected]>wrote:
> > > On Mon May 21 15:33:44 2012, Deepak Giridharagopal wrote: > >> On Mon, May 21, 2012 at 11:02 AM, Marc Zampetti >> <[email protected] >> <mailto:marc.zampetti@gmail.**com<[email protected]>>> >> wrote: >> >> Is Puppet Labs saying they are ending support of MySQL and instead >> will only support PostgreSQL? That is going to be a big problems >> for shops that do not support PostgresSQL, or are only allowed to >> run DB systems on an approved list. >> >> >> What is on your approved list? Oracle? SQL Server? DB2? That kind of >> information helps us plan things for the future. >> > > MySQL, Sybase, Oracle are all supported platforms, but since only MySQL is > the one without a license cost, that is really my only choice. Introducing > yet another SQL store into the environment is just not an option from a > management and support perspective. We are also supporting new solutions, > like MongoDB, so that would be a better choice for us. Oracle is actually a reasonable target for us, though I sympathize with your concerns about licensing costs. In terms of supportability of PuppetDB, I can point out a few things that may mitigate some concerns: 1) The data stored in PuppetDB is entirely driven by puppetmasters compiling catalogs for agents. If your entire database exploded and lost all data, everything will be 100% repopulated within around $runinterval minutes. 2) If the PuppetDB daemon is up, but it loses connectivity to the database for some reason, outstanding requests simply block until connectivity is re-established. Incoming catalogs and facts will queue up, and will be processed once connectivity is re-established. This allows for DB maintenance without impacting the overall availability of the Puppet infrastructure. 3) No other Puppet tools other than PuppetDB itself require direct access to the database; as such, the security and authentication requirements are pretty simple. I know that doesn't directly address your immediate concern about MySQL support, but hopefully that helps add some context around the operational requirements of PuppetDB's data store. It's much less of a "single db that all puppet tools dump data into" and much more of a "data store for just this one daemon and nothing else". > > >> Why wouldn't a DB-agnostic model be used? >> >> >> The short answer is performance. To effectively implement things we've >> got on our roadmap, we need things that (current) MySQL doesn't >> support: array types are critical for efficiently supporting things >> like parameter values, recursive query support is critical for fast >> graph traversal operations, things like INTERSECT are handy for query >> generation, and we rely on fast joins (MySQL's nested loop joins don't >> always cut it). It's much easier for us to support databases with >> these features than those that don't. For fairly divergent database >> targets, it becomes really hard to get the performance we want while >> simultaneously keeping our codebase manageable. >> > > I understand the need to not support everything. Having designed a number > of systems that require some of the features you say you need, I can say > with confidence that most of those issues can be handled without having an > RDBMS that has all those advanced features. So I will respectfully disagree > that you need features you listed. Yes, you may not be able to use > something like ActiveRecord or Hibernate, and have to hand-code your SQL > more often, but there are a number of techniques that can be used to at > least achieve similar performance characteristics. I think it is a bit > dangerous to assume that your user base can easily and quickly switch out > their RDBMS systems as easy as this announcement seems to suggest. I'm > happy to be wrong if the overall community thinks that is true, but for > something that is as core to one's infrastructure as Puppet, making such a > big change seems concerning. > We aren't using ActiveRecord or Hibernate, and we are using hand-coded SQL where necessary to wring maximum speed out of the underlying data store. I'm happy to go into much greater detail about why the features I listed are important, but I think that's better suited to puppet-dev than puppet-users. We certainly didn't make this decision cavalierly; it was made after around a month of benchmarking various solutions ranging from traditional databases like PostgreSQL to document stores like MongoDB to KV stores such as Riak to graph databases like Neo4J. For Puppet's particular type of workload, with Puppet's volume of data, with Puppet's required durability and safety requirements...I maintain this was the best choice. While I don't doubt that given a large enough amount of time and enough engineers we could get PuppetDB working fast enough on arbitrary backing stores (MySQL included), we have limited time and resources. From a pragmatic standpoint, we felt that supporting a database that was available on all platforms Puppet supports, that costs nothing, that has plenty of modules on the Puppet Forge to help set it up, that has a great reliability record, that meets our performance needs, and that in the worst case has free/cheap hosted offerings (such as Heroku) was a reasonable compromise. The catalog, resource and fact wire formats that the PuppetDB terminus uses are published at docs.puppetlabs.com; we've designed those interchange formats such that you could write your own backing store that persists things and answers queries. While PuppetDB does include a specific backing store for those formats, you can use the PuppetDB terminus code on the frontend and your own custom stuff on the backend if you like. I very much encourage such endeavors, and I'm happy to help in any way I can! We certainly don't expect users to "quickly switch out their RDBMS systems" with ease, and I sincerely apologize if I gave that impression. A few points of clarification: 1) This is the first PuppetDB release 2) It's a beta release; things will be fixed as we move to RC and then to 1.0 3) This feature is entirely optional; while I think everyone should use storeconfigs, you can certainly use puppet without it 4) Current storeconfigs users aren't abandoned; the existing, ActiveRecord-based storeconfigs code is fully supported in Puppet 3.0 Again, apologies if the original announcement wasn't clear on these points! > >> >> Right now, I can say that due to these types of issues, I cannot >> even evaluate PuppetDB, and will not be able to for the >> foreseeable future. >> >> >> How many hosts do you have? Would the built-in, embedded database work >> for you as an interim solution? >> > > No, I'm already managing several hundred hosts. > > > >> >> Also, does this mean that the existing inventory service and store >> configs functionality goes away? >> >> >> The existing inventory service API is still supported, and in fact >> PuppetDB works as a backing store for that API. So tools and code that >> use that API currently will continue to work. Puppet 3.0 still >> includes the old ActiveRecord-based storeconfigs backend, which still >> works. >> >> But is there a commitment from Puppet Labs that storeconfigs on top of > MySQL will be supported for some time? It doesn't really do me any good to > build my infrastructure on store config (primarily for external resources), > and then find out 6 - 12 months from now that it is going away simply > because the extra work to support MySQL is too hard. > The current, ActiveRecord-based storeconfigs support will likely be marked 'deprecated' soon, but it shouldn't go away until Puppet 4.0, which is slated for over a year from now (semantic versioning and all that). Cheers, deepak -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
