Giuseppe Lavagetto has uploaded a new change for review. https://gerrit.wikimedia.org/r/172552
Change subject: hiera: allow regex-based searches ...................................................................... hiera: allow regex-based searches This will allow us to put most host-specific definitions in one single, simple file. Change-Id: I74ce2e5d942eb1d46d043128d25b7ad45acb3908 Signed-off-by: Giuseppe Lavagetto <[email protected]> --- M modules/puppetmaster/files/production.hiera.yaml M modules/wmflib/lib/hiera/backend/nuyaml_backend.rb 2 files changed, 59 insertions(+), 8 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/52/172552/1 diff --git a/modules/puppetmaster/files/production.hiera.yaml b/modules/puppetmaster/files/production.hiera.yaml index 1600ea5..f2dc321 100644 --- a/modules/puppetmaster/files/production.hiera.yaml +++ b/modules/puppetmaster/files/production.hiera.yaml @@ -11,6 +11,7 @@ :datadir: /etc/puppet/private/hieradata :hierarchy: - "hosts/%{::hostname}" + - "regex/%{::hostname}" - "%{::realm}" - "%{::site}" - "private/%{::site}" diff --git a/modules/wmflib/lib/hiera/backend/nuyaml_backend.rb b/modules/wmflib/lib/hiera/backend/nuyaml_backend.rb index 2f4e376..f3a9613 100644 --- a/modules/wmflib/lib/hiera/backend/nuyaml_backend.rb +++ b/modules/wmflib/lib/hiera/backend/nuyaml_backend.rb @@ -21,7 +21,6 @@ # :expand_path be expanded when looking the file up on disk. This # allows both to have a more granular set of files, but also to avoid # unnecessary cache evictions for cached data. -# # === Example # # Say your hiera.yaml has defined @@ -40,6 +39,30 @@ # inside the file #{datadir}/module_data/passwords/mysql.yaml # # Unless very small, all files should be split up like this. +# +# == Regexp matching +# +# As multiple hosts may correspond to the same rules/configs in a +# large cluster, we allow to define a self-contained "regex.yaml" file +# in your datadir, where each different class of servers may be +# represented by a label and a corresponding regexp. +# +# === Example +# Say you have a lookup for "cluster", and you have +#"regex/%{hostname}" in your hierarchy; also, let's say that your +# scope contains hostname = "web1001.local". So if your regex.yaml +# file contains: +# +# databases: +# __regex: !ruby/regex '/db.*\.local/' +# cluster: db +# +# webservices: +# __regex: !ruby/regex '/^web.*\.local$/' +# cluster: www +# +# This will make it so that "cluster" will assume the value "www" +# given the regex matches the "webservices" stanza # # == Dynamic lookup # @@ -82,6 +105,11 @@ def get_path(key, scope, source) config_section = :nuyaml + # Special case: regex + if m = /^regex\//.match(source) + return key, Backend.datafile(config_section, scope, 'regex', "yaml") + end + # Special case: 'private' repository. # We use a different datadir in this case. # Example: private/common will search in the common source @@ -110,6 +138,23 @@ return key, Backend.datafile(config_section, scope, source, "yaml") end + def plain_lookup(key, data, scope) + return nil unless data.include?(key) + return Backend.parse_answer(data[key], scope) + end + + def regex_lookup(key, matchon, data, scope) + data.each do |label, datahash| + next unless datahash[:__regex].match(matchon) + next unless datahash.include?(key) + return Backend.parse_answer(data[key]) + end + return nil + rescue => detail + Hiera.debug(detail) + return nil + end + def lookup(key, scope, order_override, resolution_type) answer = nil @@ -127,31 +172,37 @@ dynsource ||= 'default' source += "/#{dynsource}" end - Hiera.debug("Loading info from #{source} for #{key}") lookup_key, yamlfile = get_path(key, scope, source) Hiera.debug("Searching for #{lookup_key} in #{yamlfile}") - next if yamlfile.nil? + return nil if yamlfile.nil? Hiera.debug("Loading file #{yamlfile}") - next unless File.exist?(yamlfile) + return nil unless File.exist?(yamlfile) data = @cache.read(yamlfile, Hash) do |content| YAML.load(content) end - next if data.empty? - next unless data.include?(lookup_key) + return nil if data.empty? + + if m = /regex\/(.*)/.match(source) + matchto = m[1] + new_answer = regex_lookup(lookup_key, matchto, data, scope) + else + new_answer = plain_lookup(key, data, scope) + end + next if new_aswer.nil? # Extra logging that we found the key. This can be outputted # multiple times if the resolution type is array or hash but that # should be expected as the logging will then tell the user ALL the # places where the key is found. - Hiera.debug("Found #{lookup_key} in #{source}") + Hiera.debug("Found #{key} in #{source}") # for array resolution we just append to the array whatever # we find, we then goes onto the next file and keep adding to @@ -160,7 +211,6 @@ # for priority searches we break after the first found data # item - new_answer = Backend.parse_answer(data[lookup_key], scope) case resolution_type when :array raise Exception, "Hiera type mismatch: expected Array and got #{new_answer.class}" unless new_answer.kind_of? Array or new_answer.kind_of? String -- To view, visit https://gerrit.wikimedia.org/r/172552 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I74ce2e5d942eb1d46d043128d25b7ad45acb3908 Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Giuseppe Lavagetto <[email protected]> _______________________________________________ MediaWiki-commits mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
