** Final Release ** PuppetDB 2.0.0 final - May 6, 2014.
PuppetDB 2.0.0 Downloads ------------------------ Available in native package format in the release repositories at: http://yum.puppetlabs.com and http://apt.puppetlabs.com For information on how to enable the Puppet Labs repos, see: http://docs.puppetlabs.com/guides/puppetlabs_package_repositories.html#open-source-repositories Binary tarball: http://downloads.puppetlabs.com/puppetdb/ Source: http://github.com/puppetlabs/puppetdb Please report feedback via the Puppet Labs tickets site, using an affected PuppetDB version of 2.0.0: https://tickets.puppetlabs.com/browse/PDB Documentation: http://docs.puppetlabs.com/puppetdb/2.0 Puppet module: http://forge.puppetlabs.com/puppetlabs/puppetdb PuppetDB 2.0.0 Release Notes -------------------------------- PuppetDB 2.0.0 is a major release focusing on environments support. Note that *this is a major version bump* and there are several breaking changes, including dropping support for versions of PostgreSQL prior to version 8.4 and Java 1.6. See the "Deprecations and potentially breaking changes" section below for more information. Things to take note of before upgrading: * There is a known issue in Puppet 3.4.0, 3.4.1 and 3.4.2 with duplicate failed events being tracked here: https://tickets.puppetlabs.com/browse/PUP-1524. This is being fixed in an upcoming release of Puppet. Since 1.6.0 this error will become more obvious as we are now raising full stack traces, so you may notice it more readily after the upgrade. * If you receive the error “Could not open /etc/puppet/log4j.properties”, this is because we have changed the packaged config.ini to point at a new logging configuration file: logback.xml. However during package installation some package managers will cowardly refuse to just update config.ini, this in particular affects RPM. After upgrading you should ensure any .rpmnew files are reviewed and that changes to our vendored version are now merged with your version of config.ini on disk. See this ticket for more information: https://tickets.puppetlabs.com/browse/PDB-656 * Make sure all your PuppetDB instances are shut down and only upgrade one at a time. * As usual, don’t forget to upgrade your puppetdb-terminus package also (on the host where your Puppet Master lives), and restart your master service. New features: * (PDB-452,453,454,456,457,526,557) Adding support for storing, querying and importing/exporting environments This change forced a new revision of the `replace facts`, `replace catalog` and `store report` commands. The PuppetDB terminus also needed to be updated to support new environment information being sent to PuppetDB. USERS MUST ALSO UPDATE THE PUPPETDB TERMINUS. Previous versions of those commands (and wire formats) are now deprecated. See the PuppetDB API docs for more information. Environments support has been added to the new v4 (currently experimental) query API. The following query endpoints now include environment in the response: - facts - resources - nodes - catalogs - reports - events - event counts - aggregate event counts The below query endpoints now allow query/filtering by environment: - facts - resources - nodes - catalogs - reports - events This release also includes a new environments query endpoint to list all known environments and allow an easy filtering based on environment for things like events, facts reports and resources. See the query API docs for more information. PuppetDB import/export/anonymization and benchmark tools also now have environment support. Storeconfigs export does not include environments as that information is not being stored in the old storeconfigs module. Environments that are no longer associated with a fact set, report or catalog will be "garbage collected" by being removed from the database. * (PDB-581) Add subqueries to events query endpoint The events endpoint now supports select-resources and select-facts * (PDB-234) Add v4 query API, deprecate v2 query API This patch adds the new code relevant for doing any future v4 work. It has been raised as an experimental end-point only so there are no commitments to its interface yet. Once stable we will need another patch to declare it as so. This patch also deprecates the v2 end-point in documentation and by adding the same headers we used to use for the v1 end-point. * (PDB-470) Provide new db setting 'statements-cache-size' with a default of 1000 This setting adjusts how many SQL prepared statements get cached via BoneCP. By using this setting we've seen an almost 40% decrease in wall clock time it takes to store a new catalog. This patch adds the new configuration item as a user configurable one, with a default set to 1000 for now. Documentation has also been added for this setting. * (PDB-221) Add facts to import/export This commit imports/exports facts similar to how we currently import/export catalogs and reports. Anonymize doesn't currently work for facts, which is going to be added separately. * (PDB-469) - Support Anonymizing Facts This commit adds support for the anonymization of facts. The levels of anonymization supported are: - none - no anonymization - low - only values with a fact name of secret, password etc - moderate - recognized "safe" facts are untouched, recognized facts with sensitive information (i.e. ipaddress) have their values anonymized - full - all fact names and values are anonymized Deprecations and potentially breaking changes: * (PDB-88, PDB-271) JDK 1.6 no longer supported * (PDB-308) Drop 2.7.x support This patch removes support for Puppet 2.7.x in several ways: * New check for every entry point in terminus will return an error if the version of Puppet is not supported. This is done in a 'soft' manner to avoid Puppet from not working. * Documentation now only references Puppet 3 * Documentation now states only latest version of Puppet is supported * Packaging now has hard dependencies on the latest version of Puppet * Contrib gemspec has been updated * Gemfile for tests have been updated * (PDB-552) Pin support for Puppet to 3.5.1 and above Starting PuppetDB with a Puppet earlier than 3.5.1 will now fail on startup. * (PDB-605) Pin facter requirement to 1.7.0 Using prior versions of Facter will cause PuppetDB to fail on startup * (PDB-592) Removing support for Ubuntu Raring Raring went EOL in Jan 2014, so we are no longer building packages for it. * (PDB-79) Drop support for Postgres < 8.4 PuppetDB will now log an error and exit if it connects to an instance of Postgres older than 8.4. Users of older versions will need to upgrade (especially EL 5 users as it defaults to 8.1). The acceptance tests for EL 5 have been updated to be explicit about using Postgres 8.4 packages instead. * (PDB-204) Ensure all commands no longer need a serialized payload Some previous commands required the payload of the command to be JSON serialized strings as opposed to the relevant JSON type directly in the payload. All commands no longer require the payload to be serialized to a string first. * (PDB-238) - Remove v1 API This commit removes the v1 API and builds on the HTTP api refactor. This commit contains: - Remove all v1 namespaces and the namespaces calling them - Remove api tests excercising the v1 routes - Remove v1 references in the docs * (PDB-354) Deprecate old versions of commands This patch drops a warning whenever an old version of the commands API is used and updates the documentation to warn the user these old commands are deprecated. * (PDB-570) Remove planetarium endpoint Old endpoint that had significant overlap with the current catalogs endpoint * (PDB-113) Remove swank As swank is now a deprecated project. This patch removes swank support completely from the code base. Notable improvements and fixes: * (PDB-473) Support POST using application/json data in the body This patch adds to the commands end-point the ability to simply POST using application/json with the JSON content in the body. It also switches the terminus to use this mechanism. We found that the url encode/decode required to support x-www-form-urlencoded was actually quite an overhead in a number of ways: * The urlencode on the terminus added overhead * The urldecode in the server added overhead * The interim strings created during this encode/decode process can get quite large increasing the amount of garbage collection required This feature has been implemented by providing a new middleware that will move the body into the parameter :body-string of the request when the content-type is not set to application/x-www-form-urlencoded. This provides a convenient backwards compatible layer so that the old form url encoding can still be supported for older versions of the API. * (PDB-567,191) Use hash not config_version for report export files This fixes a bug related to config_versions containing characts not safe to be use in file names (such as '/'). * (PDB-518) Fix bug storeconfig export of arrays For exported Resources with parameters which value is a Array the storeconfig export fails to collect them. Instead of collecting all the parameter values into a array it simply override the value with each value in turn. * (PDB-228) Use JSON in terminus instead of PSON The PuppetDB API specifies that it is JSON, so we should parse it as that and not as PSON. Some Puppet classes (Puppet::Node and Puppet::Node::Facts) don't support JSON serialization, so continue to use PSON serialization for them. In Puppet 3.4.0+ they have methods to do seralization in other formats than PSON though, so once support for older versions of Puppet is dropped they can be seralized in JSON as well. * (PDB-476) Decorate the terminus code with Puppet profiling blocks This patch adds some select profiling blocks to the PuppetDB terminus code. The profiler is provided by puppet core from Puppet::Util::Puppetdb#profile, which has recently become public for our use. We provide here in our own utils library our own wrapper implementation that can be mixed in. Key areas of our terminus functionality have now been profiled with this patch: * Entry points are profiled and identified by their entry methods (save, find, search etc.) * Remote calls, HTTP gets/posts * Code that does any form of encoding/decoding that might be potentially slow at capacity. The style of messages I've used follow along with the existing Puppet profiling examples already in place so as to be readable together. We have prefixed our profile message with "PuppetDB" for easy searchability also. I have provided a small FAQ entry that explains in brief the process of debugging, although we lack something to link to in Puppet for a more detailed explanation. This will probably need to be fixed if better documentation comes available. * (PDB-472) - Annotate MQ messages without parsing payload Received time and a UUID are currently added to incoming (via HTTP) messages before placing them on the queue. This commit adds those annotations to the MQ message header no longer requires parsing the incoming message payload before placing it on the queue. * (PDB-87) Port PuppetDB to TrapperKeeper TrapperKeeper is a new container that PuppetDB will be deployed in. This is mainly a refactoring of existing code and error handling to use the centralized TrapperKeeper service. More information on TrapperKeeper can be found here: http://puppetlabs.com/blog/new-era-application-services-puppet-labs. * (PDB-401) Upgrade to TrapperKeeper 0.3.4 This commit updates PuppetDB to use the new trapperkeeper 0.3.4 API. This includes: * Slightly modified syntax for defining services and service lifecycle behavior * Switch from log4j to logback, update documentation and packaging accordingly * Switch from jetty7 to jetty9 * Add example of how to use "reloaded" interactive development pattern in REPL * Upgrade to kitchensink 0.5.3, with bouncycastle fix for improved HTTPS performance * (PDB-529) Added latest-report? example to the events docs * (PDB-512) Upgrade to Clojure 1.6.0 * (PDB-521) Switch to using /dev/urandom (using java.security.egd) * (PDB-564) Added OpenBSD-specific variables to puppetdb.env Adding OpenBSD specific variables allows the OpenBSD package maintained downstream in the OpenBSD ports tree to be greatly simplified. * (PDB-177) Replace ssl-host default with 0.0.0.0 By trying to use a hostname, the amount of issues people suffer with during setup times related to hostname resolution is quite high. This patch replaces the hostname with 0.0.0.0 which by default listens on all interfaces. * (PDB-402) Remove ahead-of-time compilation This patch removes AOT compilation from our leiningen project and updates all relevant shell scripts to use the non-AOT methodology for invoking clojure projects. * (PDB-576) Update beaker tests to use host.hostname instead of host.name * (PDB-481) Added Arch Linux build/install support * (PDB-572) New community packages of puppet install in vendorlibdir * (PDB-575) Updated install from source docs * (PDB-254) Change benchmark to mutate catalog resources and edges send a defined number of messages Previously benchmark.clj when it would mutate a catalog would only add a single resource. This change will add a new resource or mutate a random existing resource. It will also add a new or change an existing edge. One of these four mutations is picked at random. This commit also adds a new parameter to benchmark.clj to allow the syncronous sending of a specified number of commands (per host) via the -N argument * (PDB-591) Allow gem source to come from env vars * (PDB-595) Added docs for the load testing and benchmarking tool Mostly a developer tool, but documented how to use it in case it's useful to others. See "Load Testing" in the Usage / Admin section of the docs site * (PDB-602) Updated acceptance tests to use a proper release of leiningen * (DOCUMENT-6) Update config page for PuppetDB module's improved settings behavior PuppetDB 2.0.0 Contributors --------------------------- Alice Nodelman, Aric Gardner, Chris Price, Daniele Sluijters, Deepak Giridharagopal, Erik Dalén, Jasper Lievisse Adriaanse, Ken Barber, Marius Rieder, Matthaus Owens, Melissa Stone, Nate Wolfe, Nick Fagerlund, Niels Abspoel, Riley Shott, Ryan Senior PuppetDB 2.0.0 Changlog ----------------------- Alice Nodelman (1): 974a19e (maint) have beaker tests use the host.hostname instead of host.name Aric Gardner (3): d4a3773 added example for latest-report? e549822 flubbed the initial operator. 6325e42 removed a space Chris Price (15): 97755c5 Port PuppetDB to run on trapperkeeper 703699b Introduce `error-shutdown!` function for handling fatal errors 324d67a Add environment variable to allow SNAPSHOT dependencies cac398c Update to released versions of deps, use new cli exceptions e92f9ab Add clarifying comments to `bootstrap.cfg` a62b6dd Remove SNAPSHOT dependency from test profile 6d50bbf Change trapperkeeper docs link to point to head of 0.1.x branch f2bfd00 Remove unused Jetty configuration 088da45 Remove unused dependency on ring-jetty-adapter 198129d Fix tk version string and test deps 1fadc84 Fix acceptance test race condition related to stop_puppetdb 61c9b2a Port to trapperkeeper 0.3.4 bf6b089 Add back in the placeholder log4j.properties.erb 3fe1191 Add `modify_config_setting` helper method for acceptance c15ba49 Update CLI tools to use new form of specs for args Daniele Sluijters (1): b38ea55 PDB-521: Switch to using /dev/urandom Deepak Giridharagopal (4): 59fa50a Fixes for Clojure 1.6 compatibility 228d854 (PDB-512) Update to Clojure 1.6.0-RC1 50309b6 Add jdk8 tests to travis-ci 29d0007 Allow gem source to come from env vars Erik Dalén (3): 660522f (#23422) Use JSON in terminus instead of PSON e3f3fbe (PDB-581) Add subqueries to events query endpoint abb8f2f Fix command API Content-Type Jasper Lievisse Adriaanse (1): 20efae8 [PDB-564] Add OpenBSD-specific variables to puppetdb.env Ken Barber (51): 9da1078 PDB-177 Replace ssl-host default with 0.0.0.0 ef96659 PDB-177 Remove superflous fqdn lookups b4b85ee Fix merge up mistake f96946d PDB-234 Add v4, deprecate v2 query API 40c42e3 PDB-308 Drop 2.7.x support 424d104 PDB-308 Fix missing files from packaging 4acafb0 Fix puppetdb-env.erb so we don't get stripped carriage returns 9e6beab PDB-354 Deprecate old versions of commands 1b42c1d PDB-113 Remove swank 08d849a PDB-402 Remove ahead-of-time compilation bfc45a0 PDB-476 Decorate the terminus code with Puppet profiling blocks 295edd7 PDB-470 Provide new db setting 'statements-cache-size' with a default of 1000 15d68d8 PDB-473 Support POST using application/json data in the body c789867 PDB-473 Add a content-type verification middleware 0b604cf PDB-473 Additional tests for verify-content-type and other cleanups ef2ea91 PDB-473 Convert submit-command-via-http! to use application/json POST 1185307 PDB-473 Use (seq foo) on its own 9dd010b PDB-473 Adapt for new queue message format 4aaf0a2 Use Clojure 1.6.0-rc3 e03b4f0 Clojure 1.6.0-RC4 e5c3f72 Clojure 1.6.0 final 0685e8b PDB-453 Add environments to all terminii 619600e PDB-453 Normalize request.environment with to_s to avoid 'stack level too deep' fa79af6 PDB-554 Switch to using new experimental ec2 beaker driver 63360a0 PDB-554 Remove temp refspec entry 856edea (PDB-526) Garbage collect unreferenced environments f578757 (PDB-581) Add tests and minor cleanups for event subqueries 89ba090 (PDB-552) Pin support for Puppet to 3.5.1 and above 5c0d6f6 Use ActiveRecord 3.2.x instead of 2.3.x 6c9ad0d Use rubygem-sqlite3 for fedora 19/20 e34a661 Run storeconfig tests regardless 5be8845 Fix Fedora case statement and install sqlite3 from gem 4b98fea (PDB-594) Ensure libsqlite3-dev is installed 52df5c8 (PDB-594) Install builder to help with Ubuntu 10.04 rubygems failure to follow dependencies 5ea8b27 (PDB-594) Manually install i18n for the sake of Ubuntu 10.04 old rubygems 1619cde (PDB-594) Confine legacy import/export scripts to avoid Ubuntu 10.04 d4cf33d (PDB-567 & PDB-191) Use hash not config_version for report export files 4821eeb (PDB-465) Remove Ubuntu 14.04 trust support for master c334ef4 (PDB-555) Environments for /nodes & hyphenation consistency fixes 42be0b4 (PDB-204) Ensure all commands no longer need a serialized payload 05da2c4 (PDB-204) Fix terminus unit tests to avoid seralization for facts/deactivate 0abbd53 (PDB-204) Fix submit command for terminus so it returns the commands response 2daf865 (PDB-555) Rename *-last-environment to -environment 4c794b4 (PDB-204) Simplify variable destructuring and other minor cleanups c1743e8 (PDB-605) Prepare all documentation headers for 2.0 release e55b34b (PDB-605) Revert some 1.6->2.0 changes 66d97b0 (PDB-605) Pin facter requirement to 1.7.0 ef8a2ca Fix environment end-point links to be compatible with our docs site a9a29b0 Wrap API examples in back-ticks to make the hyperlinks work ded3e1e Fix example query to be JSON, not clojure 5ea3901 Correct hyperlink to anchor syntax in FAQ Marius Rieder (1): 6952ac0 fix storeconfig export for array values Matthaus Owens (1): 5d3e90b (PDB-592) Raring went EOL in Jan 2014, so we should no longer build packages for it. Melissa Stone (1): 3f34e3c (PDB-465) Add ubuntu-14.04, trusty tahr Nate Wolfe (3): 0ea1949 Properly handle cli! errors 30bca1d Properly handle cli! --help 8720ff7 Hit actual endpoint to see if PuppetDB started Nick Fagerlund (1): ea700fd (DOCUMENT-6) Update config page for PuppetDB module's improved settings behavior Niels Abspoel (2): 35fcfed Aded Archlinux build/install support 5d9a2dc new community packages of puppet install in vendorlibdir Riley Shott (2): e16e308 Updated install from source commands 88ec143 Fixed typo. Ryan Senior (45): 7818a59 (PDB-254) Change benchmark to mutate catalog resources and edges send a defined number of messages 504431d Fixed report submission part of benchmark to change the start/end time and config version 455b4fa (PDB-79) Drop support for Postgres < 8.4 3236907 Added Postgres 8.4 declaration to the acceptance test PuppetDB package installation code 48e8fa1 Split metrics/version/command out of api.clj and into their own namespaces c01a774 (PDB-238) - Remove v1 API a6f42af (PDB-271) Update docs to reflect removal of JDK 1.6 support d88e89b (PDB-88) Fail on JDK version 1.6 173b9b2 Remove the option of a JDK 1.6 RPM dependency a850666 Bump the redhat JDK dep for acceptance tests to 1.7.0 4e7ad25 (PDB-221) Add facts to import/export 4ed5aca Added support for exporting facts/reports when no catalog has been persisted for the given node 6c9e4d3 PDB-469 - Support Anonymizing Facts e1961d0 Tighten the fact-value anonymization for the moderate profile f567003 (PDB-472) - Annotate MQ messages without parsing payload fa8453f Fixed a date related issue causing tests to fail (only at certain times of the day) 2585409 (PDB-452) Adding support for storing environments fd6ed60 Minor doc changes 60aea48 Removed unused command attribute "puppetdb-version" 432dd02 (PDB-454) - Updating the catalogs endpoint to return environments b571324 Fix for the terminus to stringify version per spec, added defensive conversion code in the back end. b2cdeb0 Changed to covert catalog version to a string for wire formats v1-v3. Also a minor doc update for the terminus. 1939ee3 (PDB-454) Added support for querying/returning environments for resources and facts dd75f69 Changed v2/v3 resource/fact API calls to remove environment from the response bf26afe Fixed issue with doseq over deftests in the events tests 20959ad (PDB-454) - Added environment support for querying reports, events, event-counts and aggregate-event-counts 40e1d2d (PDB-574) 404 should be returned when querying for a missing catalog 6a30c4c (PDB-457) - Update import/export/anonymization for v4 API f9124ac Added tests around exporting storeconfigs with resources containing arrays for param values. bab2401 Only pull in activerecord and sqlite3 on new versions of Ruby (1.9.3). This will cause the storeconfigs spec tests to only run on newer Rubies. 2943316 Changed the storeconfig acceptance tests to verify import by querying 3ea8a68 (PDB-457) More storeconfig acceptance test fixes 789802c (PDB-457) Storconfigs acceptance tests broken on Fedora e9cd65a (PDB-570) Remove planetarium endpoint 93ef9d1 (PDB-595) Added docs for the load testing and benchmarking tool 8ceed05 (PDB-589) Add indexes to environment foreign keys 91257df Bump catalog/report wire format versions for the benchmark tool ea4e576 (PDB-456) Added environments endpoint and related query functions 3b320d4 Added a ring middleware for creating a db transaction and a fixture for diagnosing routes that don't create a connection dd3a64c Fixed inconsistencies in the deprecation warning messages of commands 9c58a5b DRY'd up the environments docs a bit f121f26 Updated acceptance tests to use a proper release of leiningen 3ca240f (PDB-557) Add support for standard query operators and subqueries aa95bca Updated the release notes for the 2.0.0 release ff807b1 (PDB-626) Added some additional fixes/features to the 2.0.0 release notes -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CAE4bNTnHE7DuefmPSJjJp1to5%2Bz_e-p71V9X_EtRQyU0AzSFjQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
